[Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Philippe Mathieu-Daudé]

Hi,

With this series we can now write tests using Python rather than C.
For complex tests this can reduce the test development time, we can focus on
the test purposes instead of his implementation details.

- 1,2: we already have Python classes to run Block tests, move all the
  non Block-related methods to qtest.py,
- 3: default TEST_DIR to TMPDIR,
- 4: add a method to restrict tests to a list of supported machines,
- 5: since the Block tests are output sensitive, do not mess with their
  current tuned iotests::main(unittest), add a more generic one in qtest.py,
- 6: finally modify the tests Makefile to run C/Python tests with the same
  rule.

to have a better idea, here is a snippet from the next series:

    class TestSdCardSpecV2(qtest.QMPTestCase):
        [...]
        def test_power_on_spec_v2(self):
            self.bus.do_cmd(GO_IDLE_STATE)
            [...]
            # verify Card ID
            data = self.bus.do_cmd(ALL_SEND_CID)
            oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
            self.assertEqual(oid, "XY") # QEMU default
            self.assertEqual(pnm, "QEMU!") # QEMU default
            self.assertEqual(psn, 0xdeadbeef) # QEMU default

The test suite is ran the same way:

$ make check-qtest-aarch64 
  GTESTER check-qtest-aarch64
  PYTEST  check-qtest-aarch64

Adding the verbose flag we see the SD bus commands:

$ make check-qtest-aarch64 V=1
TEST: tests/test-hmp... (pid=16074)
...
  /aarch64/hmp/xlnx-zcu102:                                            OK
  /aarch64/hmp/lm3s811evb:                                             OK
  /aarch64/hmp/none+2MB:                                               OK
PASS: tests/test-hmp
PYTHONPATH=/source/qemu/scripts QEMU_PROG=aarch64-softmmu/qemu-system-aarch64 TEST_DIR=/tmp python -B /source/qemu/tests/sdcard_tests.py -v
INFO:root:CMD#00 -> (none)
INFO:root:CMD#08 -> 00000100
INFO:root:CMD#08 -> 00000200
INFO:root:CMD#08 -> 00000400
INFO:root:CMD#08 -> 00000800
INFO:root:CMD#08 -> 00001000
INFO:root:CMD#08 -> 00002000
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 00ffff00
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 80ffff00
INFO:root:CMD#02 -> aa585951454d552101deadbeef006219
INFO:root:CMD#03 -> 45670500
INFO:root:CMD#03 -> 8ace0700
.
----------------------------------------------------------------------
Ran 1 test in 0.070s

OK

Regards,

Phil.

Based-on: 20171213204436.5379-12-f4bug@amsat.org
          (QOM'ify SDBus, housekeeping)

Philippe Mathieu-Daudé (6):
  iotests.py: split BlockQMPTestCase class of QMPTestCase
  iotests.py: move the generic QMPTestCase to qtest.py
  qtest.py: use TMPDIR/TEMP if the TEST_DIR env var is missing
  qtest.py: add verify_machine(supported_machines)
  qtest.py: add a simple main() which calls unittest.main()
  tests: add a Makefile rule to run Python qtests

 scripts/qtest.py              | 152 ++++++++++++++++++++++++++++++++++++++++++
 tests/Makefile.include        |  11 ++-
 tests/qemu-iotests/030        |  14 ++--
 tests/qemu-iotests/040        |   2 +-
 tests/qemu-iotests/041        |  20 +++---
 tests/qemu-iotests/044        |   2 +-
 tests/qemu-iotests/045        |   4 +-
 tests/qemu-iotests/055        |   8 +--
 tests/qemu-iotests/056        |   4 +-
 tests/qemu-iotests/057        |   2 +-
 tests/qemu-iotests/065        |   2 +-
 tests/qemu-iotests/093        |   6 +-
 tests/qemu-iotests/096        |   2 +-
 tests/qemu-iotests/118        |   2 +-
 tests/qemu-iotests/124        |   2 +-
 tests/qemu-iotests/129        |   2 +-
 tests/qemu-iotests/132        |   2 +-
 tests/qemu-iotests/136        |   2 +-
 tests/qemu-iotests/139        |   2 +-
 tests/qemu-iotests/147        |   2 +-
 tests/qemu-iotests/148        |   2 +-
 tests/qemu-iotests/152        |   2 +-
 tests/qemu-iotests/155        |   2 +-
 tests/qemu-iotests/163        |   2 +-
 tests/qemu-iotests/165        |   2 +-
 tests/qemu-iotests/196        |   2 +-
 tests/qemu-iotests/iotests.py |  84 ++---------------------
 27 files changed, 212 insertions(+), 127 deletions(-)

-- 
2.15.1

[Qemu-devel] [PATCH 1/6] iotests.py: split BlockQMPTestCase class of QMPTestCase

[Philippe Mathieu-Daudé]

Add a new class BlockQMPTestCase based on QMPTestCase, which only contains the
block methods. This will allow to reuse the generic methods in the next patch.

Patch created mostly mechanically.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/qemu-iotests/iotests.py |  4 ++++
 tests/qemu-iotests/030        | 14 +++++++-------
 tests/qemu-iotests/040        |  2 +-
 tests/qemu-iotests/041        | 20 ++++++++++----------
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/045        |  4 ++--
 tests/qemu-iotests/055        |  8 ++++----
 tests/qemu-iotests/056        |  4 ++--
 tests/qemu-iotests/057        |  2 +-
 tests/qemu-iotests/065        |  2 +-
 tests/qemu-iotests/093        |  6 +++---
 tests/qemu-iotests/096        |  2 +-
 tests/qemu-iotests/118        |  2 +-
 tests/qemu-iotests/124        |  2 +-
 tests/qemu-iotests/129        |  2 +-
 tests/qemu-iotests/132        |  2 +-
 tests/qemu-iotests/136        |  2 +-
 tests/qemu-iotests/139        |  2 +-
 tests/qemu-iotests/147        |  2 +-
 tests/qemu-iotests/148        |  2 +-
 tests/qemu-iotests/152        |  2 +-
 tests/qemu-iotests/155        |  2 +-
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/165        |  2 +-
 tests/qemu-iotests/196        |  2 +-
 25 files changed, 50 insertions(+), 46 deletions(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 457984b8e9..2bb96d5ea9 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -27,7 +27,7 @@ backing_img = os.path.join(iotests.test_dir, 'backing.img')
 mid_img = os.path.join(iotests.test_dir, 'mid.img')
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
-class TestSingleDrive(iotests.QMPTestCase):
+class TestSingleDrive(iotests.BlockQMPTestCase):
     image_len = 1 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -153,7 +153,7 @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_qmp(result, 'error/class', 'GenericError')
 
 
-class TestParallelOps(iotests.QMPTestCase):
+class TestParallelOps(iotests.BlockQMPTestCase):
     num_ops = 4 # Number of parallel block-stream operations
     num_imgs = num_ops * 2 + 1
     image_len = num_ops * 1024 * 1024
@@ -385,7 +385,7 @@ class TestParallelOps(iotests.QMPTestCase):
                          qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[3]),
                          'image file map matches backing file after streaming')
 
-class TestQuorum(iotests.QMPTestCase):
+class TestQuorum(iotests.BlockQMPTestCase):
     num_children = 3
     children = []
     backing = []
@@ -441,7 +441,7 @@ class TestQuorum(iotests.QMPTestCase):
                          qemu_io('-f', iotests.imgfmt, '-c', 'map', self.backing[0]),
                          'image file map does not match backing file after streaming')
 
-class TestSmallerBackingFile(iotests.QMPTestCase):
+class TestSmallerBackingFile(iotests.BlockQMPTestCase):
     backing_len = 1 * 1024 * 1024 # MB
     image_len = 2 * backing_len
 
@@ -464,7 +464,7 @@ class TestSmallerBackingFile(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-class TestErrors(iotests.QMPTestCase):
+class TestErrors(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     # this should match STREAM_BUFFER_SIZE/512 in block/stream.c
@@ -694,7 +694,7 @@ class TestENOSPC(TestErrors):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-class TestStreamStop(iotests.QMPTestCase):
+class TestStreamStop(iotests.BlockQMPTestCase):
     image_len = 8 * 1024 * 1024 * 1024 # GB
 
     def setUp(self):
@@ -723,7 +723,7 @@ class TestStreamStop(iotests.QMPTestCase):
 
         self.cancel_and_wait(resume=True)
 
-class TestSetSpeed(iotests.QMPTestCase):
+class TestSetSpeed(iotests.BlockQMPTestCase):
     image_len = 80 * 1024 * 1024 # MB
 
     def setUp(self):
diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 90b5b4f2ad..1da0e7afec 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -32,7 +32,7 @@ backing_img = os.path.join(iotests.test_dir, 'backing.img')
 mid_img = os.path.join(iotests.test_dir, 'mid.img')
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
-class ImageCommitTestCase(iotests.QMPTestCase):
+class ImageCommitTestCase(iotests.BlockQMPTestCase):
     '''Abstract base class for image commit test cases'''
 
     def wait_for_complete(self, need_ready=False):
diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index a860a31e9a..e7ff6d4ea1 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -34,7 +34,7 @@ quorum_img3 = os.path.join(iotests.test_dir, 'quorum3.img')
 quorum_repair_img = os.path.join(iotests.test_dir, 'quorum_repair.img')
 quorum_snapshot_file = os.path.join(iotests.test_dir, 'quorum_snapshot.img')
 
-class TestSingleDrive(iotests.QMPTestCase):
+class TestSingleDrive(iotests.BlockQMPTestCase):
     image_len = 1 * 1024 * 1024 # MB
     qmp_cmd = 'drive-mirror'
     qmp_target = target_img
@@ -254,7 +254,7 @@ class TestSingleDriveUnalignedLength(TestSingleDrive):
 class TestSingleBlockdevUnalignedLength(TestSingleBlockdev):
     image_len = 1025 * 1024
 
-class TestMirrorNoBacking(iotests.QMPTestCase):
+class TestMirrorNoBacking(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -323,7 +323,7 @@ class TestMirrorNoBacking(iotests.QMPTestCase):
         self.assertTrue(iotests.compare_images(test_img, target_img),
                         'target image does not match source after mirroring')
 
-class TestMirrorResized(iotests.QMPTestCase):
+class TestMirrorResized(iotests.BlockQMPTestCase):
     backing_len = 1 * 1024 * 1024 # MB
     image_len = 2 * 1024 * 1024 # MB
 
@@ -371,7 +371,7 @@ class TestMirrorResized(iotests.QMPTestCase):
         self.assertTrue(iotests.compare_images(test_img, target_img),
                         'target image does not match source after mirroring')
 
-class TestReadErrors(iotests.QMPTestCase):
+class TestReadErrors(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     # this should be a multiple of twice the default granularity
@@ -526,7 +526,7 @@ new_state = "1"
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-class TestWriteErrors(iotests.QMPTestCase):
+class TestWriteErrors(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     # this should be a multiple of twice the default granularity
@@ -653,7 +653,7 @@ new_state = "1"
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-class TestSetSpeed(iotests.QMPTestCase):
+class TestSetSpeed(iotests.BlockQMPTestCase):
     image_len = 80 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -719,7 +719,7 @@ class TestSetSpeed(iotests.QMPTestCase):
 
         self.wait_ready_and_cancel()
 
-class TestUnbackedSource(iotests.QMPTestCase):
+class TestUnbackedSource(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -760,7 +760,7 @@ class TestUnbackedSource(iotests.QMPTestCase):
         self.complete_and_wait()
         self.assert_no_active_block_jobs()
 
-class TestGranularity(iotests.QMPTestCase):
+class TestGranularity(iotests.BlockQMPTestCase):
     image_len = 10 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -790,7 +790,7 @@ class TestGranularity(iotests.QMPTestCase):
         self.complete_and_wait(drive='drive0', wait_ready=False)
         self.assert_no_active_block_jobs()
 
-class TestRepairQuorum(iotests.QMPTestCase):
+class TestRepairQuorum(iotests.BlockQMPTestCase):
     """ This class test quorum file repair using drive-mirror.
         It's mostly a fork of TestSingleDrive """
     image_len = 1 * 1024 * 1024 # MB
@@ -1002,7 +1002,7 @@ class TestRepairQuorum(iotests.QMPTestCase):
 
 # Test mirroring with a source that does not have any parents (not even a
 # BlockBackend)
-class TestOrphanedSource(iotests.QMPTestCase):
+class TestOrphanedSource(iotests.BlockQMPTestCase):
     def setUp(self):
         blk0 = { 'node-name': 'src',
                  'driver': 'null-co' }
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 11ea0f4d35..4237d5187a 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -29,7 +29,7 @@ import subprocess
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
-class TestRefcountTableGrowth(iotests.QMPTestCase):
+class TestRefcountTableGrowth(iotests.BlockQMPTestCase):
     '''Abstract base class for image mirroring test cases'''
 
     def preallocate(self, name):
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 6be8fc4912..c1f813b2cc 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -28,7 +28,7 @@ image2 = os.path.join(iotests.test_dir, 'image2')
 image3 = os.path.join(iotests.test_dir, 'image3')
 image4 = os.path.join(iotests.test_dir, 'image4')
 
-class TestFdSets(iotests.QMPTestCase):
+class TestFdSets(iotests.BlockQMPTestCase):
 
     def setUp(self):
         self.vm = iotests.VM()
@@ -126,7 +126,7 @@ class TestFdSets(iotests.QMPTestCase):
         self.vm.shutdown()
 
 # Add fd at runtime, there are two ways: monitor related or fdset related
-class TestSCMFd(iotests.QMPTestCase):
+class TestSCMFd(iotests.BlockQMPTestCase):
     def setUp(self):
         self.vm = iotests.VM()
         qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055
index 8a5d9fd269..935c1e60a3 100755
--- a/tests/qemu-iotests/055
+++ b/tests/qemu-iotests/055
@@ -44,7 +44,7 @@ def tearDownModule():
     os.remove(test_img)
 
 
-class TestSingleDrive(iotests.QMPTestCase):
+class TestSingleDrive(iotests.BlockQMPTestCase):
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
 
@@ -163,7 +163,7 @@ class TestSingleDrive(iotests.QMPTestCase):
                              target='drive0', sync='full')
         self.assert_qmp(result, 'error/class', 'GenericError')
 
-class TestSetSpeed(iotests.QMPTestCase):
+class TestSetSpeed(iotests.BlockQMPTestCase):
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
 
@@ -249,7 +249,7 @@ class TestSetSpeed(iotests.QMPTestCase):
 
 # Note: We cannot use pause_drive() here, or the transaction command
 #       would stall.  Instead, we limit the block job speed here.
-class TestSingleTransaction(iotests.QMPTestCase):
+class TestSingleTransaction(iotests.BlockQMPTestCase):
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
 
@@ -453,7 +453,7 @@ class TestSingleTransaction(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
 
 
-class TestDriveCompression(iotests.QMPTestCase):
+class TestDriveCompression(iotests.BlockQMPTestCase):
     image_len = 64 * 1024 * 1024 # MB
     fmt_supports_compression = [{'type': 'qcow2', 'args': ()},
                                 {'type': 'vmdk', 'args': ('-o', 'subformat=streamOptimized')}]
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 04f2c3c841..dedebf1a66 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -29,7 +29,7 @@ backing_img = os.path.join(iotests.test_dir, 'backing.img')
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
 
-class TestSyncModesNoneAndTop(iotests.QMPTestCase):
+class TestSyncModesNoneAndTop(iotests.BlockQMPTestCase):
     image_len = 64 * 1024 * 1024 # MB
 
     def setUp(self):
@@ -82,7 +82,7 @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
         time.sleep(1)
         self.assertEqual(-1, qemu_io('-c', 'read -P0x41 0 512', target_img).find("verification failed"))
 
-class TestBeforeWriteNotifier(iotests.QMPTestCase):
+class TestBeforeWriteNotifier(iotests.BlockQMPTestCase):
     def setUp(self):
         self.vm = iotests.VM().add_drive_raw("file=blkdebug::null-co://,id=drive0,align=65536,driver=blkdebug")
         self.vm.launch()
diff --git a/tests/qemu-iotests/057 b/tests/qemu-iotests/057
index 9f0a5a3057..092fff8ed2 100755
--- a/tests/qemu-iotests/057
+++ b/tests/qemu-iotests/057
@@ -27,7 +27,7 @@ from iotests import qemu_img, qemu_io
 
 test_drv_base_name = 'drive'
 
-class ImageSnapshotTestCase(iotests.QMPTestCase):
+class ImageSnapshotTestCase(iotests.BlockQMPTestCase):
     image_len = 120 * 1024 * 1024 # MB
 
     def __init__(self, *args):
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 72aa9707c7..a30f117a2e 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -28,7 +28,7 @@ import unittest
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
-class TestImageInfoSpecific(iotests.QMPTestCase):
+class TestImageInfoSpecific(iotests.BlockQMPTestCase):
     '''Abstract base class for ImageInfoSpecific tests'''
 
     def setUp(self):
diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index 5c36a5fb4d..c8bca994ba 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -23,7 +23,7 @@ import iotests
 
 nsec_per_sec = 1000000000
 
-class ThrottleTestCase(iotests.QMPTestCase):
+class ThrottleTestCase(iotests.BlockQMPTestCase):
     test_img = "null-aio://"
     max_drives = 3
 
@@ -211,7 +211,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
 class ThrottleTestCoroutine(ThrottleTestCase):
     test_img = "null-co://"
 
-class ThrottleTestGroupNames(iotests.QMPTestCase):
+class ThrottleTestGroupNames(iotests.BlockQMPTestCase):
     test_img = "null-aio://"
     max_drives = 3
 
@@ -308,7 +308,7 @@ class ThrottleTestGroupNames(iotests.QMPTestCase):
             groupname = "group%d" % i
             self.verify_name(devname, groupname)
 
-class ThrottleTestRemovableMedia(iotests.QMPTestCase):
+class ThrottleTestRemovableMedia(iotests.BlockQMPTestCase):
     def setUp(self):
         self.vm = iotests.VM()
         if iotests.qemu_default_machine == 's390-ccw-virtio':
diff --git a/tests/qemu-iotests/096 b/tests/qemu-iotests/096
index aeeb3753cf..21368455e5 100644
--- a/tests/qemu-iotests/096
+++ b/tests/qemu-iotests/096
@@ -22,7 +22,7 @@
 import iotests
 import os
 
-class TestLiveSnapshot(iotests.QMPTestCase):
+class TestLiveSnapshot(iotests.BlockQMPTestCase):
     base_img = os.path.join(iotests.test_dir, 'base.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     group = 'mygroup'
diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 8a9e838c90..5fccfd6bff 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -28,7 +28,7 @@ from iotests import qemu_img
 old_img = os.path.join(iotests.test_dir, 'test0.img')
 new_img = os.path.join(iotests.test_dir, 'test1.img')
 
-class ChangeBaseClass(iotests.QMPTestCase):
+class ChangeBaseClass(iotests.BlockQMPTestCase):
     has_opened = False
     has_closed = False
 
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 8e76e62f93..94a04caa51 100644
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -91,7 +91,7 @@ class Bitmap:
                 try_remove(image)
 
 
-class TestIncrementalBackupBase(iotests.QMPTestCase):
+class TestIncrementalBackupBase(iotests.BlockQMPTestCase):
     def __init__(self, *args):
         super(TestIncrementalBackupBase, self).__init__(*args)
         self.bitmaps = list()
diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 9e87e1c8d9..4201094bc1 100644
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -22,7 +22,7 @@ import os
 import iotests
 import time
 
-class TestStopWithBlockJob(iotests.QMPTestCase):
+class TestStopWithBlockJob(iotests.BlockQMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     base_img = os.path.join(iotests.test_dir, 'base.img')
diff --git a/tests/qemu-iotests/132 b/tests/qemu-iotests/132
index f53ef6e391..ec18274424 100644
--- a/tests/qemu-iotests/132
+++ b/tests/qemu-iotests/132
@@ -26,7 +26,7 @@ from iotests import qemu_img, qemu_io
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
 
-class TestSingleDrive(iotests.QMPTestCase):
+class TestSingleDrive(iotests.BlockQMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     def setUp(self):
diff --git a/tests/qemu-iotests/136 b/tests/qemu-iotests/136
index 88b97ea7c6..76ce358ea6 100644
--- a/tests/qemu-iotests/136
+++ b/tests/qemu-iotests/136
@@ -29,7 +29,7 @@ bad_sector = 8192
 bad_offset = bad_sector * 512
 blkdebug_file = os.path.join(iotests.test_dir, 'blkdebug.conf')
 
-class BlockDeviceStatsTestCase(iotests.QMPTestCase):
+class BlockDeviceStatsTestCase(iotests.BlockQMPTestCase):
     test_img = "null-aio://"
     total_rd_bytes = 0
     total_rd_ops = 0
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index f8f02808a9..57457a8eab 100644
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -30,7 +30,7 @@ if iotests.qemu_default_machine == 's390-ccw-virtio':
 else:
     default_virtio_blk = 'virtio-blk-pci'
 
-class TestBlockdevDel(iotests.QMPTestCase):
+class TestBlockdevDel(iotests.BlockQMPTestCase):
 
     def setUp(self):
         iotests.qemu_img('create', '-f', iotests.imgfmt, base_img, '1M')
diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index 90f40ed245..03603ca0ea 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -37,7 +37,7 @@ def flatten_sock_addr(crumpled_address):
     return result
 
 
-class NBDBlockdevAddBase(iotests.QMPTestCase):
+class NBDBlockdevAddBase(iotests.BlockQMPTestCase):
     def blockdev_add_options(self, address, export=None):
         options = { 'node-name': 'nbd-blockdev',
                     'driver': 'raw',
diff --git a/tests/qemu-iotests/148 b/tests/qemu-iotests/148
index e01b061fe7..1b1153e4d8 100644
--- a/tests/qemu-iotests/148
+++ b/tests/qemu-iotests/148
@@ -34,7 +34,7 @@ event_rate = 1000000000
 sector_size = 512
 offset = 10
 
-class TestQuorumEvents(iotests.QMPTestCase):
+class TestQuorumEvents(iotests.BlockQMPTestCase):
     read_pattern = 'quorum'
 
     def create_blkdebug_file(self, blkdebug_file, bad_sector):
diff --git a/tests/qemu-iotests/152 b/tests/qemu-iotests/152
index fec546d033..f949a9a5cf 100644
--- a/tests/qemu-iotests/152
+++ b/tests/qemu-iotests/152
@@ -25,7 +25,7 @@ from iotests import qemu_img
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
 
-class TestUnaligned(iotests.QMPTestCase):
+class TestUnaligned(iotests.BlockQMPTestCase):
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, test_img, '512')
         self.vm = iotests.VM().add_drive(test_img)
diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155
index 0b86ea4e5c..5314205b72 100755
--- a/tests/qemu-iotests/155
+++ b/tests/qemu-iotests/155
@@ -46,7 +46,7 @@ target_img = os.path.join(iotests.test_dir, 'target.' + iotests.imgfmt)
 #                      target_blockdev_backing is not None
 #                      (None: same as target_backing)
 
-class BaseClass(iotests.QMPTestCase):
+class BaseClass(iotests.BlockQMPTestCase):
     target_blockdev_backing = None
     target_real_backing = None
 
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 403842354e..b2ca3fa7d8 100644
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -28,7 +28,7 @@ def size_to_int(str):
     suff = ['B', 'K', 'M', 'G', 'T']
     return int(str[:-1]) * 1024**suff.index(str[-1:])
 
-class ShrinkBaseClass(iotests.QMPTestCase):
+class ShrinkBaseClass(iotests.BlockQMPTestCase):
     image_len = '128M'
     shrink_size = '10M'
     chunk_size = '16M'
diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index a3932db3de..31b4e3e562 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -33,7 +33,7 @@ regions1 = ((0x0fff00, 0x10000),
 regions2 = ((0x10000000, 0x20000),
             (0x3fff0000, 0x10000))
 
-class TestPersistentDirtyBitmap(iotests.QMPTestCase):
+class TestPersistentDirtyBitmap(iotests.BlockQMPTestCase):
 
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size))
diff --git a/tests/qemu-iotests/196 b/tests/qemu-iotests/196
index 4116ebc92b..9c06095ac4 100755
--- a/tests/qemu-iotests/196
+++ b/tests/qemu-iotests/196
@@ -26,7 +26,7 @@ from iotests import qemu_img
 disk = os.path.join(iotests.test_dir, 'disk')
 migfile = os.path.join(iotests.test_dir, 'migfile')
 
-class TestInvalidateAutoclear(iotests.QMPTestCase):
+class TestInvalidateAutoclear(iotests.BlockQMPTestCase):
 
     def tearDown(self):
         self.vm_a.shutdown()
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 6f057904a9..e1f892c46f 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -315,6 +315,10 @@ class QMPTestCase(unittest.TestCase):
         result = self.dictpath(d, path)
         self.assertEqual(result, value, 'values not equal "%s" and "%s"' % (str(result), str(value)))
 
+
+class BlockQMPTestCase(QMPTestCase):
+    '''Abstract base class for Block QMP test cases'''
+
     def assert_no_active_block_jobs(self):
         result = self.vm.qmp('query-block-jobs')
         self.assert_qmp(result, 'return', [])
-- 
2.15.1

[Qemu-devel] [PATCH 2/6] iotests.py: move the generic QMPTestCase to qtest.py

[Philippe Mathieu-Daudé]

All those methods are generic and can be reused.
Add a few 'from qtest import ...' to keep the patch short.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qtest.py              | 102 ++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/iotests.py |  86 +++--------------------------------
 2 files changed, 107 insertions(+), 81 deletions(-)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index df0daf26ca..78db8d02c0 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -12,10 +12,33 @@
 #
 
 import socket
+import sys
 import os
 import qemu
+import re
+import unittest
+import logging
 
 
+qemu_prog = os.environ.get('QEMU_PROG', 'qemu')
+qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ')
+
+test_dir = os.environ.get('TEST_DIR')
+output_dir = os.environ.get('OUTPUT_DIR', '.')
+qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
+
+socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
+debug = False
+
+
+def filter_qmp_event(event):
+    '''Filter a QMP event dict'''
+    event = dict(event)
+    if 'timestamp' in event:
+        event['timestamp']['seconds'] = 'SECS'
+        event['timestamp']['microseconds'] = 'USECS'
+    return event
+
 class QEMUQtestProtocol(object):
     def __init__(self, address, server=False):
         """
@@ -107,3 +130,82 @@ class QEMUQtestMachine(qemu.QEMUMachine):
     def qtest(self, cmd):
         '''Send a qtest command to guest'''
         return self._qtest.cmd(cmd)
+
+
+def createQtestMachine(path_suffix=''):
+    name = "qemu%s-%d" % (path_suffix, os.getpid())
+    return QEMUQtestMachine(qemu_prog, qemu_opts, name=name,
+                             test_dir=test_dir,
+                             socket_scm_helper=socket_scm_helper)
+
+class QMPTestCase(unittest.TestCase):
+    '''Abstract base class for QMP test cases'''
+
+    index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
+
+    def dictpath(self, d, path):
+        '''Traverse a path in a nested dict'''
+        for component in path.split('/'):
+            m = QMPTestCase.index_re.match(component)
+            if m:
+                component, idx = m.groups()
+                idx = int(idx)
+
+            if not isinstance(d, dict) or component not in d:
+                self.fail('failed path traversal for "%s" in "%s"' % (path,
+                                                                      str(d)))
+            d = d[component]
+
+            if m:
+                if not isinstance(d, list):
+                    self.fail(('path component "%s" in "%s" is not a list ' +
+                              'in "%s"') % (component, path, str(d)))
+                try:
+                    d = d[idx]
+                except IndexError:
+                    self.fail(('invalid index "%s" in path "%s" ' +
+                              'in "%s"') % (idx, path, str(d)))
+        return d
+
+    def flatten_qmp_object(self, obj, output=None, basestr=''):
+        if output is None:
+            output = dict()
+        if isinstance(obj, list):
+            for i in range(len(obj)):
+                self.flatten_qmp_object(obj[i], output, basestr + str(i) + '.')
+        elif isinstance(obj, dict):
+            for key in obj:
+                self.flatten_qmp_object(obj[key], output, basestr + key + '.')
+        else:
+            output[basestr[:-1]] = obj # Strip trailing '.'
+        return output
+
+    def qmp_to_opts(self, obj):
+        obj = self.flatten_qmp_object(obj)
+        output_list = list()
+        for key in obj:
+            output_list += [key + '=' + obj[key]]
+        return ','.join(output_list)
+
+    def assert_qmp_absent(self, d, path):
+        try:
+            result = self.dictpath(d, path)
+        except AssertionError:
+            return
+        self.fail('path "%s" has value "%s"' % (path, str(result)))
+
+    def assert_qmp(self, d, path, value):
+        '''Assert that the value for a specific path in a QMP dict matches'''
+        result = self.dictpath(d, path)
+        self.assertEqual(result, value, ('values not equal "%s" ' +
+                         'and "%s"') % (str(result), str(value)))
+
+
+def notrun(reason):
+    '''Skip this test suite'''
+    print '%s not run: %s' % (seq, reason)
+    sys.exit(0)
+
+def verify_platform(supported_oses=['linux']):
+    if True not in [sys.platform.startswith(x) for x in supported_oses]:
+        notrun('not suitable for this OS: %s' % sys.platform)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index e1f892c46f..cb3d22855b 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -20,7 +20,6 @@ import errno
 import os
 import re
 import subprocess
-import string
 import unittest
 import sys
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts'))
@@ -30,6 +29,10 @@ import json
 import signal
 import logging
 
+# use the following imports from qtest instead of iotests.
+from qtest import qemu_prog, qemu_opts, test_dir, output_dir
+from qtest import qemu_default_machine, socket_scm_helper
+from qtest import filter_qmp_event, verify_platform
 
 # This will not work if arguments contain spaces but is necessary if we
 # want to support the override options that ./check supports.
@@ -45,18 +48,10 @@ qemu_nbd_args = [os.environ.get('QEMU_NBD_PROG', 'qemu-nbd')]
 if os.environ.get('QEMU_NBD_OPTIONS'):
     qemu_nbd_args += os.environ['QEMU_NBD_OPTIONS'].strip().split(' ')
 
-qemu_prog = os.environ.get('QEMU_PROG', 'qemu')
-qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ')
-
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
-test_dir = os.environ.get('TEST_DIR')
-output_dir = os.environ.get('OUTPUT_DIR', '.')
 cachemode = os.environ.get('CACHEMODE')
-qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
 
-socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
-debug = False
 
 def qemu_img(*args):
     '''Run qemu-img and return the exit code'''
@@ -134,14 +129,6 @@ chown_re = re.compile(r"chown [0-9]+:[0-9]+")
 def filter_chown(msg):
     return chown_re.sub("chown UID:GID", msg)
 
-def filter_qmp_event(event):
-    '''Filter a QMP event dict'''
-    event = dict(event)
-    if 'timestamp' in event:
-        event['timestamp']['seconds'] = 'SECS'
-        event['timestamp']['microseconds'] = 'USECS'
-    return event
-
 def log(msg, filters=[]):
     for flt in filters:
         msg = flt(msg)
@@ -257,66 +244,7 @@ class VM(qtest.QEMUQtestMachine):
                         command_line='qemu-io %s "%s"' % (drive, cmd))
 
 
-index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
-
-class QMPTestCase(unittest.TestCase):
-    '''Abstract base class for QMP test cases'''
-
-    def dictpath(self, d, path):
-        '''Traverse a path in a nested dict'''
-        for component in path.split('/'):
-            m = index_re.match(component)
-            if m:
-                component, idx = m.groups()
-                idx = int(idx)
-
-            if not isinstance(d, dict) or component not in d:
-                self.fail('failed path traversal for "%s" in "%s"' % (path, str(d)))
-            d = d[component]
-
-            if m:
-                if not isinstance(d, list):
-                    self.fail('path component "%s" in "%s" is not a list in "%s"' % (component, path, str(d)))
-                try:
-                    d = d[idx]
-                except IndexError:
-                    self.fail('invalid index "%s" in path "%s" in "%s"' % (idx, path, str(d)))
-        return d
-
-    def flatten_qmp_object(self, obj, output=None, basestr=''):
-        if output is None:
-            output = dict()
-        if isinstance(obj, list):
-            for i in range(len(obj)):
-                self.flatten_qmp_object(obj[i], output, basestr + str(i) + '.')
-        elif isinstance(obj, dict):
-            for key in obj:
-                self.flatten_qmp_object(obj[key], output, basestr + key + '.')
-        else:
-            output[basestr[:-1]] = obj # Strip trailing '.'
-        return output
-
-    def qmp_to_opts(self, obj):
-        obj = self.flatten_qmp_object(obj)
-        output_list = list()
-        for key in obj:
-            output_list += [key + '=' + obj[key]]
-        return ','.join(output_list)
-
-    def assert_qmp_absent(self, d, path):
-        try:
-            result = self.dictpath(d, path)
-        except AssertionError:
-            return
-        self.fail('path "%s" has value "%s"' % (path, str(result)))
-
-    def assert_qmp(self, d, path, value):
-        '''Assert that the value for a specific path in a QMP dict matches'''
-        result = self.dictpath(d, path)
-        self.assertEqual(result, value, 'values not equal "%s" and "%s"' % (str(result), str(value)))
-
-
-class BlockQMPTestCase(QMPTestCase):
+class BlockQMPTestCase(qtest.QMPTestCase):
     '''Abstract base class for Block QMP test cases'''
 
     def assert_no_active_block_jobs(self):
@@ -430,10 +358,6 @@ def verify_image_format(supported_fmts=[], unsupported_fmts=[]):
     if unsupported_fmts and (imgfmt in unsupported_fmts):
         notrun('not suitable for this image format: %s' % imgfmt)
 
-def verify_platform(supported_oses=['linux']):
-    if True not in [sys.platform.startswith(x) for x in supported_oses]:
-        notrun('not suitable for this OS: %s' % sys.platform)
-
 def supports_quorum():
     return 'quorum' in qemu_img_pipe('--help')
 
-- 
2.15.1

[Qemu-devel] [PATCH 3/6] qtest.py: use TMPDIR/TEMP if the TEST_DIR env var is missing

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qtest.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index 78db8d02c0..733686a6fe 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -18,12 +18,13 @@ import qemu
 import re
 import unittest
 import logging
+import tempfile
 
 
 qemu_prog = os.environ.get('QEMU_PROG', 'qemu')
 qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ')
 
-test_dir = os.environ.get('TEST_DIR')
+test_dir = os.environ.get('TEST_DIR', tempfile.gettempdir())
 output_dir = os.environ.get('OUTPUT_DIR', '.')
 qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
 
-- 
2.15.1

[Qemu-devel] [PATCH 4/6] qtest.py: add verify_machine(supported_machines)

[Philippe Mathieu-Daudé]

We can now restrict tests to specific machines.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qtest.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index 733686a6fe..429eb67c2a 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -19,6 +19,7 @@ import re
 import unittest
 import logging
 import tempfile
+import subprocess
 
 
 qemu_prog = os.environ.get('QEMU_PROG', 'qemu')
@@ -210,3 +211,18 @@ def notrun(reason):
 def verify_platform(supported_oses=['linux']):
     if True not in [sys.platform.startswith(x) for x in supported_oses]:
         notrun('not suitable for this OS: %s' % sys.platform)
+
+def verify_machine(supported_machines):
+    if 'any' in supported_machines:
+        return
+    args = list((qemu_prog, "-machine", "help"))
+    subp = subprocess.Popen(args, stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT)
+    exitcode = subp.wait()
+    if exitcode < 0:
+        sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode,
+                                                            ' '.join(args)))
+    machines = re.split("\n([\w\.-]+)\s.*", subp.communicate()[0])
+
+    if True not in [x in machines for x in supported_machines]:
+        notrun('not machine suitable for this test')
-- 
2.15.1

[Qemu-devel] [PATCH 5/6] qtest.py: add a simple main() which calls unittest.main()

[Philippe Mathieu-Daudé]

We can now write complex qtests with a high-level language (Python).
(partly copied from iotests::main)

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/qtest.py | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index 429eb67c2a..581451836c 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -226,3 +226,36 @@ def verify_machine(supported_machines):
 
     if True not in [x in machines for x in supported_machines]:
         notrun('not machine suitable for this test')
+
+def main(supported_oses=['linux'], supported_machines=['any']):
+    '''Run tests'''
+
+    global debug
+
+    verify_platform(supported_oses)
+    verify_machine(supported_machines)
+
+    verbosity = 0
+    level = logging.WARN
+    if '-d' in sys.argv:
+        verbosity = 2
+        level = logging.DEBUG
+    if '-v' in sys.argv:
+        verbosity = 1
+        level = logging.INFO
+    if '-q' in sys.argv:
+        level = logging.NOTSET
+
+    import StringIO
+    output = sys.stdout if verbosity else StringIO.StringIO()
+
+    logging.basicConfig(level=level)
+
+    class MyTestRunner(unittest.TextTestRunner):
+        def __init__(self, stream=output, descriptions=True,
+                     verbosity=verbosity):
+            unittest.TextTestRunner.__init__(self, stream, descriptions,
+                                             verbosity)
+
+    # unittest.main() will use sys.exit() so expect a SystemExit exception
+    unittest.main(testRunner=MyTestRunner)
-- 
2.15.1

[Qemu-devel] [PATCH 6/6] tests: add a Makefile rule to run Python qtests

[Philippe Mathieu-Daudé]

We can now add Python qtests to the common $(check-qtest-y) /
$(check-qtest-$(TARGET)-y) variable.

Using the $(filter...) function, the check-qtest-% rule splits between C/Python
qtests and can run both types.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/Makefile.include | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index c002352134..13673f6d1d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -828,7 +828,7 @@ endif
 TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGET_DIRS)))
 ifeq ($(CONFIG_POSIX),y)
 QTEST_TARGETS = $(TARGETS)
-check-qtest-y=$(foreach TARGET,$(TARGETS), $(check-qtest-$(TARGET)-y))
+check-qtest-y=$(foreach TARGET,$(TARGETS), $(filter-out %.py, $(check-qtest-$(TARGET)-y)))
 check-qtest-y += $(check-qtest-generic-y)
 else
 QTEST_TARGETS =
@@ -843,6 +843,7 @@ tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y)
 SPEED = quick
 GTESTER_OPTIONS = -k $(if $(V),--verbose,-q)
 GCOV_OPTIONS = -n $(if $(V),-f,)
+QPYTHONS_OPTIONS = $(if $(V),-v,)
 
 # gtester tests, possibly with verbose output
 
@@ -852,11 +853,15 @@ $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-softmmu
 	$(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
 		QTEST_QEMU_IMG=qemu-img$(EXESUF) \
 		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
-		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@")
+		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(filter-out %.py,$(check-qtest-$*-y) $(check-qtest-generic-y)),"GTESTER","$@")
 	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$*-y) $(gcov-files-generic-y); do \
 	  echo Gcov report for $$f:;\
 	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
 	done,)
+	$(if $(filter %.py,$(check-qtest-$*-y)), \
+		$(call quiet-command,PYTHONPATH=$(SRC_PATH)/scripts \
+			QEMU_PROG=$*-softmmu/qemu-system-$* \
+			$(PYTHON) $(SRC_PATH)/$(filter %.py,$(check-qtest-$*-y)) $(QPYTHONS_OPTIONS),"PYTEST","$@"),)
 
 .PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y))
 $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: %
@@ -874,7 +879,7 @@ $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: %
 $(patsubst %, check-report-qtest-%.xml, $(QTEST_TARGETS)): check-report-qtest-%.xml: $(check-qtest-y)
 	$(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
 		QTEST_QEMU_IMG=qemu-img$(EXESUF) \
-	  gtester -q $(GTESTER_OPTIONS) -o $@ -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@")
+	  gtester -q $(GTESTER_OPTIONS) -o $@ -m=$(SPEED) $(filter-out %.py,$(check-qtest-$*-y) $(check-qtest-generic-y)),"GTESTER","$@")
 
 check-report-unit.xml: $(check-unit-y)
 	$(call quiet-command,gtester -q $(GTESTER_OPTIONS) -o $@ -m=$(SPEED) $^,"GTESTER","$@")
-- 
2.15.1

Re: [Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> With this series we can now write tests using Python rather than C.
> For complex tests this can reduce the test development time, we can focus on
> the test purposes instead of his implementation details.
> 
> - 1,2: we already have Python classes to run Block tests, move all the
>   non Block-related methods to qtest.py,
> - 3: default TEST_DIR to TMPDIR,
> - 4: add a method to restrict tests to a list of supported machines,
> - 5: since the Block tests are output sensitive, do not mess with their
>   current tuned iotests::main(unittest), add a more generic one in qtest.py,
> - 6: finally modify the tests Makefile to run C/Python tests with the same
>   rule.

Python tests are great for functional tests of qemu-system-* that
interact via the command-line and QMP monitor.  From this perspective I
think the series is good.

> to have a better idea, here is a snippet from the next series:
> 
>     class TestSdCardSpecV2(qtest.QMPTestCase):
>         [...]
>         def test_power_on_spec_v2(self):
>             self.bus.do_cmd(GO_IDLE_STATE)
>             [...]
>             # verify Card ID
>             data = self.bus.do_cmd(ALL_SEND_CID)
>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>             self.assertEqual(oid, "XY") # QEMU default
>             self.assertEqual(pnm, "QEMU!") # QEMU default
>             self.assertEqual(psn, 0xdeadbeef) # QEMU default

Device qtests are better done in C than Python.  Python is not good at
binary I/O and porting this to Python 3 will be extra work later (Python
2 is set for End-of-Life in 2020, see https://pythonclock.org/).

More importantly, we already have libqos in C with a guest memory
allocator, PCI, and virtio support.  Fragmenting the small amount effort
that goes into device testing will delay libqos reaching critical mass.
Critical mass is where libqos provides all the infrastructure you need
to set up a device and focus on your actual test instead of machine,
bus, or device initialization.  Starting a Python device testing effort
will just lead to duplication and 2 underdeveloped device testing
frameworks.

Is there a specific reason why adding SD Card support to libqos is not
possible in C?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]

On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>>             # verify Card ID
>>             data = self.bus.do_cmd(ALL_SEND_CID)
>>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>>             self.assertEqual(oid, "XY") # QEMU default
>>             self.assertEqual(pnm, "QEMU!") # QEMU default
>>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> 
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.

I agree that fragmentation is bad.  However, libqos is small (about 4k
lines of code, maybe 3k in Python).

I also agree that any qtest written in Python should be written in
Python 3 from the beginning (in fact we should consider dropping Python
2.x support in 2.12).  Doing so should not make binary I/O much
different than C.

From my point of view, the main advantage of Python is the reflection
mechanisms.  Those would make it possible to automatically discover the
set of runnable tests; for example, based on:

- two machine descriptions (including malloc) for ARM -M virt and x86 -M q35

- a driver for the generic PCIe host bridge

- a driver for virtio-pci

- a driver for virtio-scsi

- a set of SCSI tests

... it should be possible to run the same tests as either ARM or x86
qtests.  This would be a very different framework compared to the C libqos.

Thanks,

Paolo

> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

Hi Stefan,

On 12/14/2017 06:39 AM, Stefan Hajnoczi wrote:
[...]
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> 
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.
> 
> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?

Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)

Now thinking about the specific reasons...

What I intend to do is add qtests for the SDHCI implementation.
There are different revisions of the standard specs.

Currently QEMU implements a mixed subset of v1.10 and v2.00, while the
real SDHCI hardware implemented is v2.00 or v3.01. I also saw 2 series
adding SDHCI devices supporting the v4.00 (not yet in /master).

With the current codebase we can plug SDSC/SDHC cards (up to 32 GB) but
we can not plug a SDXC card, so:
- plugging a 64GB card into the ZynqMP machine fails,
- plugging a 64GB card into the SABRE Lite machine fails,
- booting a Win10 guest on a exFAT formatted SD card also fails.

A later test create different cards with [2GB, 16GB, 64GB] and test if
the SDHCI correctly access them.
Machines using the Exynos4210 SoC which only support the v2.00 spec are
expected to fail with 64GB SDXC cards.

The Zynq machine has 2 SD buses, another later test unplug/replug the
same card between the buses. This test is expected to be useful to test
the custom SDHOST controller from the BCM2835 SoC (Raspi2 machine).

It is surely possible to write all those tests in C, but it resulted way
faster to me to write them in Python...

I could think of a bunch of I2C tests too, more generally I found using
Python for bus backend testing is much easier, as with the Block backends.

Regards,

Phil.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Nir Soffer]

On Thu, Dec 14, 2017 at 11:39 AM Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> > Hi,
> >
> > With this series we can now write tests using Python rather than C.
> > For complex tests this can reduce the test development time, we can
> focus on
> > the test purposes instead of his implementation details.
> >
> > - 1,2: we already have Python classes to run Block tests, move all the
> >   non Block-related methods to qtest.py,
> > - 3: default TEST_DIR to TMPDIR,
> > - 4: add a method to restrict tests to a list of supported machines,
> > - 5: since the Block tests are output sensitive, do not mess with their
> >   current tuned iotests::main(unittest), add a more generic one in
> qtest.py,
> > - 6: finally modify the tests Makefile to run C/Python tests with the
> same
> >   rule.
>
> Python tests are great for functional tests of qemu-system-* that
> interact via the command-line and QMP monitor.  From this perspective I
> think the series is good.
>
> > to have a better idea, here is a snippet from the next series:
> >
> >     class TestSdCardSpecV2(qtest.QMPTestCase):
> >         [...]
> >         def test_power_on_spec_v2(self):
> >             self.bus.do_cmd(GO_IDLE_STATE)
> >             [...]
> >             # verify Card ID
> >             data = self.bus.do_cmd(ALL_SEND_CID)
> >             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> >             self.assertEqual(oid, "XY") # QEMU default
> >             self.assertEqual(pnm, "QEMU!") # QEMU default
> >             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O


Why do you think that? what is missing in python 2 for binary io?


> and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>

You can write today code that work with both python 2 and 3. For binary
io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
and CStringIO.

Nir


>
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.
>
> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?
>

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Nir Soffer]

On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
> >>             # verify Card ID
> >>             data = self.bus.do_cmd(ALL_SEND_CID)
> >>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> >>             self.assertEqual(oid, "XY") # QEMU default
> >>             self.assertEqual(pnm, "QEMU!") # QEMU default
> >>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> >
> > More importantly, we already have libqos in C with a guest memory
> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
> > that goes into device testing will delay libqos reaching critical mass.
> > Critical mass is where libqos provides all the infrastructure you need
> > to set up a device and focus on your actual test instead of machine,
> > bus, or device initialization.  Starting a Python device testing effort
> > will just lead to duplication and 2 underdeveloped device testing
> > frameworks.
>
> I agree that fragmentation is bad.  However, libqos is small (about 4k
> lines of code, maybe 3k in Python).
>
> I also agree that any qtest written in Python should be written in
> Python 3 from the beginning (in fact we should consider dropping Python
> 2.x support in 2.12).  Doing so should not make binary I/O much
> different than C.
>

Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
which do not provide python 3 yet.

Doing binary io in python 2 and 3 is the same, there is no need to use
python 3 for this. The only advantage is not having to backport code
later from 2 to 3.

Nir


>
> From my point of view, the main advantage of Python is the reflection
> mechanisms.  Those would make it possible to automatically discover the
> set of runnable tests; for example, based on:
>
> - two machine descriptions (including malloc) for ARM -M virt and x86 -M
> q35
>
> - a driver for the generic PCIe host bridge
>
> - a driver for virtio-pci
>
> - a driver for virtio-scsi
>
> - a set of SCSI tests
>
> ... it should be possible to run the same tests as either ARM or x86
> qtests.  This would be a very different framework compared to the C libqos.
>
> Thanks,
>
> Paolo
>
> > Is there a specific reason why adding SD Card support to libqos is not
> > possible in C?
>
>
>

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Philippe Mathieu-Daudé]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

Hi Nir,

On 12/14/2017 12:39 PM, Nir Soffer wrote:
> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com
[...]
>     > Device qtests are better done in C than Python.  Python is not good at
>     > binary I/O and porting this to Python 3 will be extra work later
>     (Python
>     > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>     >
>     > More importantly, we already have libqos in C with a guest memory
>     > allocator, PCI, and virtio support.  Fragmenting the small amount
>     effort
>     > that goes into device testing will delay libqos reaching critical
>     mass.
>     > Critical mass is where libqos provides all the infrastructure you need
>     > to set up a device and focus on your actual test instead of machine,
>     > bus, or device initialization.  Starting a Python device testing
>     effort
>     > will just lead to duplication and 2 underdeveloped device testing
>     > frameworks.
> 
>     I agree that fragmentation is bad.  However, libqos is small (about 4k
>     lines of code, maybe 3k in Python).
> 
>     I also agree that any qtest written in Python should be written in
>     Python 3 from the beginning (in fact we should consider dropping Python
>     2.x support in 2.12).  Doing so should not make binary I/O much
>     different than C.
> 
> 
> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
> which do not provide python 3 yet.

Restricting it to python 2 also make it hard to develop and test QEMU on
Arch Linux which default come with python 3 ;)

> Doing binary io in python 2 and 3 is the same, there is no need to use
> python 3 for this. The only advantage is not having to backport code
> later from 2 to 3.
> 
> Nir


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Markus Armbruster]

Nir Soffer <nirsof@gmail.com> writes:

> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>> >>             # verify Card ID
>> >>             data = self.bus.do_cmd(ALL_SEND_CID)
>> >>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>> >>             self.assertEqual(oid, "XY") # QEMU default
>> >>             self.assertEqual(pnm, "QEMU!") # QEMU default
>> >>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>> > Device qtests are better done in C than Python.  Python is not good at
>> > binary I/O and porting this to Python 3 will be extra work later (Python
>> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>> >
>> > More importantly, we already have libqos in C with a guest memory
>> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
>> > that goes into device testing will delay libqos reaching critical mass.
>> > Critical mass is where libqos provides all the infrastructure you need
>> > to set up a device and focus on your actual test instead of machine,
>> > bus, or device initialization.  Starting a Python device testing effort
>> > will just lead to duplication and 2 underdeveloped device testing
>> > frameworks.
>>
>> I agree that fragmentation is bad.  However, libqos is small (about 4k
>> lines of code, maybe 3k in Python).
>>
>> I also agree that any qtest written in Python should be written in
>> Python 3 from the beginning (in fact we should consider dropping Python
>> 2.x support in 2.12).  Doing so should not make binary I/O much
>> different than C.
>>
>
> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
> which do not provide python 3 yet.

s/hard/slightly inconvenient/: Python 3 is available in EPEL.  

> Doing binary io in python 2 and 3 is the same, there is no need to use
> python 3 for this. The only advantage is not having to backport code
> later from 2 to 3.

Re: [Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

On Thu, Dec 14, 2017 at 11:33:03AM -0300, Philippe Mathieu-Daudé wrote:
> Hi Stefan,
> 
> On 12/14/2017 06:39 AM, Stefan Hajnoczi wrote:
> [...]
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> > 
> > More importantly, we already have libqos in C with a guest memory
> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
> > that goes into device testing will delay libqos reaching critical mass.
> > Critical mass is where libqos provides all the infrastructure you need
> > to set up a device and focus on your actual test instead of machine,
> > bus, or device initialization.  Starting a Python device testing effort
> > will just lead to duplication and 2 underdeveloped device testing
> > frameworks.
> > 
> > Is there a specific reason why adding SD Card support to libqos is not
> > possible in C?
> 
> Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)

041 is not a device-level test.  It doesn't poke device registers, it's
a functional test.  In my email I said I support using Python for those
types of tests.

> Now thinking about the specific reasons...
> 
> What I intend to do is add qtests for the SDHCI implementation.
> There are different revisions of the standard specs.

It would be great to have SDHCI support in libqos.  PCI and virtio are
covered today and support for more busses will help reach that critical
mass where tests can be written for most QEMU device models without
first writing a new device driver framework.

Please post your SDHCI test code, so we can discuss the details and
consider whether it's more like 041 or more like tests/ide-test.c.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Paolo Bonzini]

On 14/12/2017 17:05, Markus Armbruster wrote:
> Nir Soffer <nirsof@gmail.com> writes:
> 
>> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>>>>>             # verify Card ID
>>>>>             data = self.bus.do_cmd(ALL_SEND_CID)
>>>>>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>>>>>             self.assertEqual(oid, "XY") # QEMU default
>>>>>             self.assertEqual(pnm, "QEMU!") # QEMU default
>>>>>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>>>> Device qtests are better done in C than Python.  Python is not good at
>>>> binary I/O and porting this to Python 3 will be extra work later (Python
>>>> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>>>>
>>>> More importantly, we already have libqos in C with a guest memory
>>>> allocator, PCI, and virtio support.  Fragmenting the small amount effort
>>>> that goes into device testing will delay libqos reaching critical mass.
>>>> Critical mass is where libqos provides all the infrastructure you need
>>>> to set up a device and focus on your actual test instead of machine,
>>>> bus, or device initialization.  Starting a Python device testing effort
>>>> will just lead to duplication and 2 underdeveloped device testing
>>>> frameworks.
>>>
>>> I agree that fragmentation is bad.  However, libqos is small (about 4k
>>> lines of code, maybe 3k in Python).
>>>
>>> I also agree that any qtest written in Python should be written in
>>> Python 3 from the beginning (in fact we should consider dropping Python
>>> 2.x support in 2.12).  Doing so should not make binary I/O much
>>> different than C.
>>
>> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
>> which do not provide python 3 yet.
> 
> s/hard/slightly inconvenient/: Python 3 is available in EPEL.  

And also in software collections.  For 2.12 (as part of the above "drop
Python 2.x support" plan) my idea is to set up a virtual Python
environment as part of configure, so that you can just do

	scl enable rh-python36 ./configure
	make

if you don't want to use EPEL.

Paolo

>> Doing binary io in python 2 and 3 is the same, there is no need to use
>> python 3 for this. The only advantage is not having to backport code
>> later from 2 to 3.

Re: [Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Philippe Mathieu-Daudé]

>> > Is there a specific reason why adding SD Card support to libqos is not
>> > possible in C?
>>
>> Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)
>
> 041 is not a device-level test.  It doesn't poke device registers, it's
> a functional test.  In my email I said I support using Python for those
> types of tests.
>
>> Now thinking about the specific reasons...
>>
>> What I intend to do is add qtests for the SDHCI implementation.
>> There are different revisions of the standard specs.
>
> It would be great to have SDHCI support in libqos.  PCI and virtio are
> covered today and support for more busses will help reach that critical
> mass where tests can be written for most QEMU device models without
> first writing a new device driver framework.
>
> Please post your SDHCI test code, so we can discuss the details and
> consider whether it's more like 041 or more like tests/ide-test.c.

The SDHCI test is like ide-test, poking register.

Being not a 'generic' series but only SDHCI I didn't Cc'd you,
you can find the test here:

http://lists.nongnu.org/archive/html/qemu-devel/2017-12/msg02391.html

and the SDCard python test here (you got this one):

http://lists.nongnu.org/archive/html/qemu-devel/2017-12/msg02350.html

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Peter Maydell]

On 14 December 2017 at 17:33, Paolo Bonzini <pbonzini@redhat.com> wrote:
> And also in software collections.  For 2.12 (as part of the above "drop
> Python 2.x support" plan)

I'm reluctant to require a non-system python on OSX to build.

thanks
-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Paolo Bonzini]

On 14/12/2017 20:08, Peter Maydell wrote:
> On 14 December 2017 at 17:33, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> And also in software collections.  For 2.12 (as part of the above "drop
>> Python 2.x support" plan)
> 
> I'm reluctant to require a non-system python on OSX to build.

Oh well.  I guess I can post the infrastructure to simplify using a
non-system Python, without actually requiring it.

Paolo

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3189 bytes --]

On Thu, Dec 14, 2017 at 03:35:07PM +0000, Nir Soffer wrote:
> On Thu, Dec 14, 2017 at 11:39 AM Stefan Hajnoczi <stefanha@redhat.com>
> wrote:
> 
> > On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > With this series we can now write tests using Python rather than C.
> > > For complex tests this can reduce the test development time, we can
> > focus on
> > > the test purposes instead of his implementation details.
> > >
> > > - 1,2: we already have Python classes to run Block tests, move all the
> > >   non Block-related methods to qtest.py,
> > > - 3: default TEST_DIR to TMPDIR,
> > > - 4: add a method to restrict tests to a list of supported machines,
> > > - 5: since the Block tests are output sensitive, do not mess with their
> > >   current tuned iotests::main(unittest), add a more generic one in
> > qtest.py,
> > > - 6: finally modify the tests Makefile to run C/Python tests with the
> > same
> > >   rule.
> >
> > Python tests are great for functional tests of qemu-system-* that
> > interact via the command-line and QMP monitor.  From this perspective I
> > think the series is good.
> >
> > > to have a better idea, here is a snippet from the next series:
> > >
> > >     class TestSdCardSpecV2(qtest.QMPTestCase):
> > >         [...]
> > >         def test_power_on_spec_v2(self):
> > >             self.bus.do_cmd(GO_IDLE_STATE)
> > >             [...]
> > >             # verify Card ID
> > >             data = self.bus.do_cmd(ALL_SEND_CID)
> > >             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> > >             self.assertEqual(oid, "XY") # QEMU default
> > >             self.assertEqual(pnm, "QEMU!") # QEMU default
> > >             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> >
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O
> 
> 
> Why do you think that? what is missing in python 2 for binary io?

C code can reuse existing header files for struct definitions instead
of doing oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data).

In general, Python's model for binary data is a marshalling (copy) model
rather than a type casting (in-place access) model.  It's fine for some
things but programs that deal with a lot of raw data may use third-party
unboxed data primitives like numpy's instead.  I definitely wouldn't say
that binary I/O is a strength of Python.

> > and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> >
> 
> You can write today code that work with both python 2 and 3. For binary
> io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
> and CStringIO.

Yes, it's possible to write the code carefully and test under both
Python 2 & 3.  We have to do that for Python code in QEMU since the
world is currently transitioning and both Python versions are in use.

So in this case, where it's questionable to start a new device-level
testing framework in the first place, the extra trouble of dealing with
Python 2 & 3 makes it even less appealing.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH 0/6] QTests: use Python to run complex tests

[Paolo Bonzini]

> C code can reuse existing header files for struct definitions instead
> of doing oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data).

On the other hand, C code risks having the same bug in tests and
device models, and also endianness is less apparent in C.  Overall
I'd say it's a wash---if I were to start now I'd use Python but I
agree with you that fragmenting the tests is not a good idea.

For the purpose of testing the SD card code, I'd use the SD-over-SSI
device.  Unfortunately there is no board that uses bit-banging SSI,
but STM32F405 is as close as it can get to it.

Paolo

> In general, Python's model for binary data is a marshalling (copy) model
> rather than a type casting (in-place access) model.  It's fine for some
> things but programs that deal with a lot of raw data may use third-party
> unboxed data primitives like numpy's instead.  I definitely wouldn't say
> that binary I/O is a strength of Python.
> 
> > > and porting this to Python 3 will be extra work later (Python
> > > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> > >
> > 
> > You can write today code that work with both python 2 and 3. For binary
> > io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
> > and CStringIO.
> 
> Yes, it's possible to write the code carefully and test under both
> Python 2 & 3.  We have to do that for Python code in QEMU since the
> world is currently transitioning and both Python versions are in use.
> 
> So in this case, where it's questionable to start a new device-level
> testing framework in the first place, the extra trouble of dealing with
> Python 2 & 3 makes it even less appealing.
> 
> Stefan
>