* [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early
@ 2019-05-23 17:06 John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 1/5] " John Snow
` (6 more replies)
0 siblings, 7 replies; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, qemu-stable, Markus Armbruster, Max Reitz, John Snow
See patch one's commit message for justification.
Patches 2-5 are for testing, because that's always how these things go.
001/5:[----] [--] 'blockdev-backup: don't check aio_context too early'
002/5:[0004] [FC] 'iotests.py: do not use infinite waits'
003/5:[down] 'QEMUMachine: add events_wait method'
004/5:[0022] [FC] 'iotests.py: rewrite run_job to be pickier'
005/5:[0017] [FC] 'iotests: add iotest 250 for testing blockdev-backup
across iothread contexts'
v3: Rebased on Max's staging branch:
Rebase patch 2
added patch 3, to add events_wait.
Rework patch 4 to make run_job consume legacy events, too
Minorly edit patch 5 due to the two above.
v2: added patch 4, with iotest framework adjustments in patches 2/3.
John Snow (5):
blockdev-backup: don't check aio_context too early
iotests.py: do not use infinite waits
QEMUMachine: add events_wait method
iotests.py: rewrite run_job to be pickier
iotests: add iotest 250 for testing blockdev-backup across iothread
contexts
blockdev.c | 4 --
python/qemu/__init__.py | 69 +++++++++++++------
tests/qemu-iotests/250 | 122 ++++++++++++++++++++++++++++++++++
tests/qemu-iotests/250.out | 119 +++++++++++++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 60 ++++++++++-------
6 files changed, 326 insertions(+), 49 deletions(-)
create mode 100755 tests/qemu-iotests/250
create mode 100644 tests/qemu-iotests/250.out
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v3 1/5] blockdev-backup: don't check aio_context too early
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
@ 2019-05-23 17:06 ` John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 2/5] iotests.py: do not use infinite waits John Snow
` (5 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, aihua liang, qemu-stable, Markus Armbruster,
Max Reitz, John Snow
in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.
Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.
Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.
Reported-by: aihua liang <aliang@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <jsnow@redhat.com>
---
blockdev.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index e856ca4be9..01a48a2a5a 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1872,10 +1872,6 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
}
aio_context = bdrv_get_aio_context(bs);
- if (aio_context != bdrv_get_aio_context(target)) {
- error_setg(errp, "Backup between two IO threads is not implemented");
- return;
- }
aio_context_acquire(aio_context);
state->bs = bs;
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v3 2/5] iotests.py: do not use infinite waits
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 1/5] " John Snow
@ 2019-05-23 17:06 ` John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method John Snow
` (4 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, qemu-stable, Markus Armbruster, Max Reitz, John Snow
Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/qemu-iotests/iotests.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 6bcddd8870..6e17c040dc 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -521,7 +521,7 @@ class VM(qtest.QEMUQtestMachine):
output_list += [key + '=' + obj[key]]
return ','.join(output_list)
- def get_qmp_events_filtered(self, wait=True):
+ def get_qmp_events_filtered(self, wait=60.0):
result = []
for ev in self.get_qmp_events(wait=wait):
result.append(filter_qmp_event(ev))
@@ -539,10 +539,10 @@ class VM(qtest.QEMUQtestMachine):
# Returns None on success, and an error string on failure
def run_job(self, job, auto_finalize=True, auto_dismiss=False,
- pre_finalize=None):
+ pre_finalize=None, wait=60.0):
error = None
while True:
- for ev in self.get_qmp_events_filtered(wait=True):
+ for ev in self.get_qmp_events_filtered(wait=wait):
if ev['event'] == 'JOB_STATUS_CHANGE':
status = ev['data']['status']
if status == 'aborting':
@@ -647,7 +647,7 @@ class QMPTestCase(unittest.TestCase):
self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
self.vm.flatten_qmp_object(reference))
- def cancel_and_wait(self, drive='drive0', force=False, resume=False):
+ def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
'''Cancel a block job and wait for it to finish, returning the event'''
result = self.vm.qmp('block-job-cancel', device=drive, force=force)
self.assert_qmp(result, 'return', {})
@@ -658,7 +658,7 @@ class QMPTestCase(unittest.TestCase):
cancelled = False
result = None
while not cancelled:
- for event in self.vm.get_qmp_events(wait=True):
+ for event in self.vm.get_qmp_events(wait=wait):
if event['event'] == 'BLOCK_JOB_COMPLETED' or \
event['event'] == 'BLOCK_JOB_CANCELLED':
self.assert_qmp(event, 'data/device', drive)
@@ -671,10 +671,10 @@ class QMPTestCase(unittest.TestCase):
self.assert_no_active_block_jobs()
return result
- def wait_until_completed(self, drive='drive0', check_offset=True):
+ def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
'''Wait for a block job to finish, returning the event'''
while True:
- for event in self.vm.get_qmp_events(wait=True):
+ for event in self.vm.get_qmp_events(wait=wait):
if event['event'] == 'BLOCK_JOB_COMPLETED':
self.assert_qmp(event, 'data/device', drive)
self.assert_qmp_absent(event, 'data/error')
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 1/5] " John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 2/5] iotests.py: do not use infinite waits John Snow
@ 2019-05-23 17:06 ` John Snow
2019-05-23 17:49 ` Max Reitz
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 4/5] iotests.py: rewrite run_job to be pickier John Snow
` (3 subsequent siblings)
6 siblings, 1 reply; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, qemu-stable, Markus Armbruster, Max Reitz, John Snow
Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
1 file changed, 49 insertions(+), 20 deletions(-)
diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index 81d9657ec0..98ed8a2e28 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -402,42 +402,71 @@ class QEMUMachine(object):
self._qmp.clear_events()
return events
- def event_wait(self, name, timeout=60.0, match=None):
+ @staticmethod
+ def event_match(event, match=None):
"""
- Wait for specified timeout on named event in QMP; optionally filter
- results by match.
+ Check if an event matches optional match criteria.
- The 'match' is checked to be a recursive subset of the 'event'; skips
- branch processing on match's value None
- {"foo": {"bar": 1}} matches {"foo": None}
- {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
+ The match criteria takes the form of a matching subdict. The event is
+ checked to be a superset of the subdict, recursively, with matching
+ values whenever those values are not None.
+
+ Examples, with the subdict queries on the left:
+ - None matches any object.
+ - {"foo": None} matches {"foo": {"bar": 1}}
+ - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
+ - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
"""
- def event_match(event, match=None):
- if match is None:
- return True
+ if match is None:
+ return True
- for key in match:
- if key in event:
- if isinstance(event[key], dict):
- if not event_match(event[key], match[key]):
- return False
- elif event[key] != match[key]:
+ for key in match:
+ if key in event:
+ if isinstance(event[key], dict):
+ if not QEMUMachine.event_match(event[key], match[key]):
return False
- else:
+ elif event[key] != match[key]:
return False
+ else:
+ return False
+ return True
- return True
+ def event_wait(self, name, timeout=60.0, match=None):
+ """
+ event_wait waits for and returns a named event from QMP with a timeout.
+
+ name: The event to wait for.
+ timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+ match: Optional match criteria. See event_match for details.
+ """
+ return self.events_wait([(name, match)], timeout)
+
+ def events_wait(self, events, timeout=60.0):
+ """
+ events_wait waits for and returns a named event from QMP with a timeout.
+
+ events: a sequence of (name, match_criteria) tuples.
+ The match criteria are optional and may be None.
+ See event_match for details.
+ timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+ """
+ def _match(event):
+ for name, match in events:
+ if (event['event'] == name and
+ self.event_match(event, match)):
+ return True
+ return False
# Search cached events
for event in self._events:
- if (event['event'] == name) and event_match(event, match):
+ if _match(event):
self._events.remove(event)
return event
# Poll for new events
while True:
event = self._qmp.pull_event(wait=timeout)
- if (event['event'] == name) and event_match(event, match):
+ if _match(event):
return event
self._events.append(event)
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v3 4/5] iotests.py: rewrite run_job to be pickier
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
` (2 preceding siblings ...)
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method John Snow
@ 2019-05-23 17:06 ` John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 5/5] iotests: add iotest 250 for testing blockdev-backup across iothread contexts John Snow
` (2 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, qemu-stable, Markus Armbruster, Max Reitz, John Snow
Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 19 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 6e17c040dc..dc77d3fba0 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -540,27 +540,37 @@ class VM(qtest.QEMUQtestMachine):
# Returns None on success, and an error string on failure
def run_job(self, job, auto_finalize=True, auto_dismiss=False,
pre_finalize=None, wait=60.0):
+ match_device = {'data': {'device': job}}
+ match_id = {'data': {'id': job}}
+ events = [
+ ('BLOCK_JOB_COMPLETED', match_device),
+ ('BLOCK_JOB_CANCELLED', match_device),
+ ('BLOCK_JOB_ERROR', match_device),
+ ('BLOCK_JOB_READY', match_device),
+ ('BLOCK_JOB_PENDING', match_id),
+ ('JOB_STATUS_CHANGE', match_id)
+ ]
error = None
while True:
- for ev in self.get_qmp_events_filtered(wait=wait):
- if ev['event'] == 'JOB_STATUS_CHANGE':
- status = ev['data']['status']
- if status == 'aborting':
- result = self.qmp('query-jobs')
- for j in result['return']:
- if j['id'] == job:
- error = j['error']
- log('Job failed: %s' % (j['error']))
- elif status == 'pending' and not auto_finalize:
- if pre_finalize:
- pre_finalize()
- self.qmp_log('job-finalize', id=job)
- elif status == 'concluded' and not auto_dismiss:
- self.qmp_log('job-dismiss', id=job)
- elif status == 'null':
- return error
- else:
- log(ev)
+ ev = filter_qmp_event(self.events_wait(events))
+ if ev['event'] != 'JOB_STATUS_CHANGE':
+ log(ev)
+ continue
+ status = ev['data']['status']
+ if status == 'aborting':
+ result = self.qmp('query-jobs')
+ for j in result['return']:
+ if j['id'] == job:
+ error = j['error']
+ log('Job failed: %s' % (j['error']))
+ elif status == 'pending' and not auto_finalize:
+ if pre_finalize:
+ pre_finalize()
+ self.qmp_log('job-finalize', id=job)
+ elif status == 'concluded' and not auto_dismiss:
+ self.qmp_log('job-dismiss', id=job)
+ elif status == 'null':
+ return error
def node_info(self, node_name):
nodes = self.qmp('query-named-block-nodes')
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Qemu-devel] [PATCH v3 5/5] iotests: add iotest 250 for testing blockdev-backup across iothread contexts
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
` (3 preceding siblings ...)
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 4/5] iotests.py: rewrite run_job to be pickier John Snow
@ 2019-05-23 17:06 ` John Snow
2019-05-23 17:51 ` [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early Max Reitz
2019-05-27 13:12 ` Max Reitz
6 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2019-05-23 17:06 UTC (permalink / raw)
To: qemu-devel, qemu-block
Cc: Kevin Wolf, qemu-stable, Markus Armbruster, Max Reitz, John Snow
Signed-off-by: John Snow <jsnow@redhat.com>
---
tests/qemu-iotests/250 | 122 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/250.out | 119 ++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 242 insertions(+)
create mode 100755 tests/qemu-iotests/250
create mode 100644 tests/qemu-iotests/250.out
diff --git a/tests/qemu-iotests/250 b/tests/qemu-iotests/250
new file mode 100755
index 0000000000..c594a43205
--- /dev/null
+++ b/tests/qemu-iotests/250
@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+#
+# Test incremental/backup across iothread contexts
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+import os
+import iotests
+from iotests import log
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+size = 64 * 1024 * 1024
+
+with iotests.FilePath('img0') as img0_path, \
+ iotests.FilePath('img1') as img1_path, \
+ iotests.FilePath('img0-full') as img0_full_path, \
+ iotests.FilePath('img1-full') as img1_full_path, \
+ iotests.FilePath('img0-incr') as img0_incr_path, \
+ iotests.FilePath('img1-incr') as img1_incr_path, \
+ iotests.VM() as vm:
+
+ def create_target(filepath, name, size):
+ basename = os.path.basename(filepath)
+ nodename = "file_{}".format(basename)
+ log(vm.command('blockdev-create', job_id='job1',
+ options={
+ 'driver': 'file',
+ 'filename': filepath,
+ 'size': 0,
+ }))
+ vm.run_job('job1')
+ log(vm.command('blockdev-add', driver='file',
+ node_name=nodename, filename=filepath))
+ log(vm.command('blockdev-create', job_id='job2',
+ options={
+ 'driver': iotests.imgfmt,
+ 'file': nodename,
+ 'size': size,
+ }))
+ vm.run_job('job2')
+ log(vm.command('blockdev-add', driver=iotests.imgfmt,
+ node_name=name,
+ file=nodename))
+
+ log('--- Preparing images & VM ---\n')
+ vm.add_object('iothread,id=iothread0')
+ vm.add_object('iothread,id=iothread1')
+ vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
+ vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
+ iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
+ iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+ vm.add_drive(img0_path, interface='none')
+ vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
+ vm.add_drive(img1_path, interface='none')
+ vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
+
+ log('--- Starting VM ---\n')
+ vm.launch()
+
+ log('--- Create Targets & Full Backups ---\n')
+ create_target(img0_full_path, 'img0-full', size)
+ create_target(img1_full_path, 'img1-full', size)
+ ret = vm.qmp_log('transaction', indent=2, actions=[
+ { 'type': 'block-dirty-bitmap-add',
+ 'data': { 'node': 'drive0', 'name': 'bitmap0' }},
+ { 'type': 'block-dirty-bitmap-add',
+ 'data': { 'node': 'drive1', 'name': 'bitmap1' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive0',
+ 'target': 'img0-full',
+ 'sync': 'full',
+ 'job-id': 'j0' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive1',
+ 'target': 'img1-full',
+ 'sync': 'full',
+ 'job-id': 'j1' }}
+ ])
+ if "error" in ret:
+ raise Exception(ret['error']['desc'])
+ vm.run_job('j0', auto_dismiss=True)
+ vm.run_job('j1', auto_dismiss=True)
+
+ log('\n--- Create Targets & Incremental Backups ---\n')
+ create_target(img0_incr_path, 'img0-incr', size)
+ create_target(img1_incr_path, 'img1-incr', size)
+ ret = vm.qmp_log('transaction', indent=2, actions=[
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive0',
+ 'target': 'img0-incr',
+ 'sync': 'incremental',
+ 'bitmap': 'bitmap0',
+ 'job-id': 'j2' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive1',
+ 'target': 'img1-incr',
+ 'sync': 'incremental',
+ 'bitmap': 'bitmap1',
+ 'job-id': 'j3' }}
+ ])
+ if "error" in ret:
+ raise Exception(ret['error']['desc'])
+ vm.run_job('j2', auto_dismiss=True)
+ vm.run_job('j3', auto_dismiss=True)
+
+ log('\n--- Done ---')
+ vm.shutdown()
diff --git a/tests/qemu-iotests/250.out b/tests/qemu-iotests/250.out
new file mode 100644
index 0000000000..eec38614ec
--- /dev/null
+++ b/tests/qemu-iotests/250.out
@@ -0,0 +1,119 @@
+--- Preparing images & VM ---
+
+--- Starting VM ---
+
+--- Create Targets & Full Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "data": {
+ "name": "bitmap0",
+ "node": "drive0"
+ },
+ "type": "block-dirty-bitmap-add"
+ },
+ {
+ "data": {
+ "name": "bitmap1",
+ "node": "drive1"
+ },
+ "type": "block-dirty-bitmap-add"
+ },
+ {
+ "data": {
+ "device": "drive0",
+ "job-id": "j0",
+ "sync": "full",
+ "target": "img0-full"
+ },
+ "type": "blockdev-backup"
+ },
+ {
+ "data": {
+ "device": "drive1",
+ "job-id": "j1",
+ "sync": "full",
+ "target": "img1-full"
+ },
+ "type": "blockdev-backup"
+ }
+ ]
+ }
+}
+{
+ "return": {}
+}
+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Create Targets & Incremental Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "data": {
+ "bitmap": "bitmap0",
+ "device": "drive0",
+ "job-id": "j2",
+ "sync": "incremental",
+ "target": "img0-incr"
+ },
+ "type": "blockdev-backup"
+ },
+ {
+ "data": {
+ "bitmap": "bitmap1",
+ "device": "drive1",
+ "job-id": "j3",
+ "sync": "incremental",
+ "target": "img1-incr"
+ },
+ "type": "blockdev-backup"
+ }
+ ]
+ }
+}
+{
+ "return": {}
+}
+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Done ---
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 2758f48143..ea52e8660d 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -249,6 +249,7 @@
247 rw auto quick
248 rw auto quick
249 rw auto quick
+250 rw auto quick
252 rw auto backing quick
253 rw auto quick
255 rw auto quick
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method John Snow
@ 2019-05-23 17:49 ` Max Reitz
2019-05-23 18:03 ` John Snow
0 siblings, 1 reply; 12+ messages in thread
From: Max Reitz @ 2019-05-23 17:49 UTC (permalink / raw)
To: John Snow, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]
On 23.05.19 19:06, John Snow wrote:
> Instead of event_wait which looks for a single event, add an events_wait
> which can look for any number of events simultaneously. However, it
> will still only return one at a time, whichever happens first.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
> python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
> 1 file changed, 49 insertions(+), 20 deletions(-)
>
> diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
> index 81d9657ec0..98ed8a2e28 100644
> --- a/python/qemu/__init__.py
> +++ b/python/qemu/__init__.py
> @@ -402,42 +402,71 @@ class QEMUMachine(object):
> self._qmp.clear_events()
> return events
>
> - def event_wait(self, name, timeout=60.0, match=None):
> + @staticmethod
> + def event_match(event, match=None):
> """
> - Wait for specified timeout on named event in QMP; optionally filter
> - results by match.
> + Check if an event matches optional match criteria.
>
> - The 'match' is checked to be a recursive subset of the 'event'; skips
> - branch processing on match's value None
> - {"foo": {"bar": 1}} matches {"foo": None}
> - {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
> + The match criteria takes the form of a matching subdict. The event is
> + checked to be a superset of the subdict, recursively, with matching
> + values whenever those values are not None.
> +
> + Examples, with the subdict queries on the left:
> + - None matches any object.
> + - {"foo": None} matches {"foo": {"bar": 1}}
> + - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
Pre-existing, but the difference between “bar” and “baz” confused me
quite a bit.
Also, I wonder... {"foo": None} would not match {"foo": 1}, right?
Does that make sense? Shouldn’t None be the wildcard here in general?
(Also pre-existing of course.)
But this patch doesn’t make things worse, so:
Reviewed-by: Max Reitz <mreitz@redhat.com>
(I’d still like your opinion.)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
` (4 preceding siblings ...)
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 5/5] iotests: add iotest 250 for testing blockdev-backup across iothread contexts John Snow
@ 2019-05-23 17:51 ` Max Reitz
2019-05-27 13:12 ` Max Reitz
6 siblings, 0 replies; 12+ messages in thread
From: Max Reitz @ 2019-05-23 17:51 UTC (permalink / raw)
To: John Snow, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]
On 23.05.19 19:06, John Snow wrote:
> See patch one's commit message for justification.
> Patches 2-5 are for testing, because that's always how these things go.
>
> 001/5:[----] [--] 'blockdev-backup: don't check aio_context too early'
> 002/5:[0004] [FC] 'iotests.py: do not use infinite waits'
> 003/5:[down] 'QEMUMachine: add events_wait method'
> 004/5:[0022] [FC] 'iotests.py: rewrite run_job to be pickier'
> 005/5:[0017] [FC] 'iotests: add iotest 250 for testing blockdev-backup
> across iothread contexts'
>
> v3: Rebased on Max's staging branch:
> Rebase patch 2
> added patch 3, to add events_wait.
> Rework patch 4 to make run_job consume legacy events, too
> Minorly edit patch 5 due to the two above.
> v2: added patch 4, with iotest framework adjustments in patches 2/3.
>
> John Snow (5):
> blockdev-backup: don't check aio_context too early
> iotests.py: do not use infinite waits
> QEMUMachine: add events_wait method
> iotests.py: rewrite run_job to be pickier
> iotests: add iotest 250 for testing blockdev-backup across iothread
> contexts
>
> blockdev.c | 4 --
> python/qemu/__init__.py | 69 +++++++++++++------
> tests/qemu-iotests/250 | 122 ++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/250.out | 119 +++++++++++++++++++++++++++++++++
> tests/qemu-iotests/group | 1 +
> tests/qemu-iotests/iotests.py | 60 ++++++++++-------
> 6 files changed, 326 insertions(+), 49 deletions(-)
> create mode 100755 tests/qemu-iotests/250
> create mode 100644 tests/qemu-iotests/250.out
Looks good to me (if it helps:
Reviewed-by: Max Reitz <mreitz@redhat.com>
), just a question on patch 3 on pre-existing quirks.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method
2019-05-23 17:49 ` Max Reitz
@ 2019-05-23 18:03 ` John Snow
2019-05-24 12:38 ` Max Reitz
0 siblings, 1 reply; 12+ messages in thread
From: John Snow @ 2019-05-23 18:03 UTC (permalink / raw)
To: Max Reitz, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
On 5/23/19 1:49 PM, Max Reitz wrote:
> On 23.05.19 19:06, John Snow wrote:
>> Instead of event_wait which looks for a single event, add an events_wait
>> which can look for any number of events simultaneously. However, it
>> will still only return one at a time, whichever happens first.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>> python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
>> 1 file changed, 49 insertions(+), 20 deletions(-)
>>
>> diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
>> index 81d9657ec0..98ed8a2e28 100644
>> --- a/python/qemu/__init__.py
>> +++ b/python/qemu/__init__.py
>> @@ -402,42 +402,71 @@ class QEMUMachine(object):
>> self._qmp.clear_events()
>> return events
>>
>> - def event_wait(self, name, timeout=60.0, match=None):
>> + @staticmethod
>> + def event_match(event, match=None):
>> """
>> - Wait for specified timeout on named event in QMP; optionally filter
>> - results by match.
>> + Check if an event matches optional match criteria.
>>
>> - The 'match' is checked to be a recursive subset of the 'event'; skips
>> - branch processing on match's value None
>> - {"foo": {"bar": 1}} matches {"foo": None}
>> - {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
>> + The match criteria takes the form of a matching subdict. The event is
>> + checked to be a superset of the subdict, recursively, with matching
>> + values whenever those values are not None.
>> +
>> + Examples, with the subdict queries on the left:
>> + - None matches any object.
>> + - {"foo": None} matches {"foo": {"bar": 1}}
>> + - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
>
> Pre-existing, but the difference between “bar” and “baz” confused me
> quite a bit.
>
> Also, I wonder... {"foo": None} would not match {"foo": 1}, right?
> Does that make sense? Shouldn’t None be the wildcard here in general?
> (Also pre-existing of course.)
>
> But this patch doesn’t make things worse, so:
>
> Reviewed-by: Max Reitz <mreitz@redhat.com>
>
> (I’d still like your opinion.)
>
I knew I was inviting trouble by trying to re-document this.
The intention I had when writing the docs, which I think are wrong now,
was for {"foo": None} to match {"foo": 1}, but I think you're right that
it won't because '1' isn't a dict, so it tests for equality instead.
So I need to fix this one up a little bit, but I'll take the review as a
sign that this approach seems workable.
--js
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method
2019-05-23 18:03 ` John Snow
@ 2019-05-24 12:38 ` Max Reitz
2019-05-24 17:57 ` John Snow
0 siblings, 1 reply; 12+ messages in thread
From: Max Reitz @ 2019-05-24 12:38 UTC (permalink / raw)
To: John Snow, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
[-- Attachment #1: Type: text/plain, Size: 3169 bytes --]
On 23.05.19 20:03, John Snow wrote:
>
>
> On 5/23/19 1:49 PM, Max Reitz wrote:
>> On 23.05.19 19:06, John Snow wrote:
>>> Instead of event_wait which looks for a single event, add an events_wait
>>> which can look for any number of events simultaneously. However, it
>>> will still only return one at a time, whichever happens first.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>> python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
>>> 1 file changed, 49 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
>>> index 81d9657ec0..98ed8a2e28 100644
>>> --- a/python/qemu/__init__.py
>>> +++ b/python/qemu/__init__.py
>>> @@ -402,42 +402,71 @@ class QEMUMachine(object):
>>> self._qmp.clear_events()
>>> return events
>>>
>>> - def event_wait(self, name, timeout=60.0, match=None):
>>> + @staticmethod
>>> + def event_match(event, match=None):
>>> """
>>> - Wait for specified timeout on named event in QMP; optionally filter
>>> - results by match.
>>> + Check if an event matches optional match criteria.
>>>
>>> - The 'match' is checked to be a recursive subset of the 'event'; skips
>>> - branch processing on match's value None
>>> - {"foo": {"bar": 1}} matches {"foo": None}
>>> - {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
>>> + The match criteria takes the form of a matching subdict. The event is
>>> + checked to be a superset of the subdict, recursively, with matching
>>> + values whenever those values are not None.
>>> +
>>> + Examples, with the subdict queries on the left:
>>> + - None matches any object.
>>> + - {"foo": None} matches {"foo": {"bar": 1}}
>>> + - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
>>
>> Pre-existing, but the difference between “bar” and “baz” confused me
>> quite a bit.
>>
>> Also, I wonder... {"foo": None} would not match {"foo": 1}, right?
>> Does that make sense? Shouldn’t None be the wildcard here in general?
>> (Also pre-existing of course.)
>>
>> But this patch doesn’t make things worse, so:
>>
>> Reviewed-by: Max Reitz <mreitz@redhat.com>
>>
>> (I’d still like your opinion.)
>>
>
> I knew I was inviting trouble by trying to re-document this.
>
> The intention I had when writing the docs, which I think are wrong now,
> was for {"foo": None} to match {"foo": 1}, but I think you're right that
> it won't because '1' isn't a dict, so it tests for equality instead.
>
> So I need to fix this one up a little bit, but I'll take the review as a
> sign that this approach seems workable.
I think the comment is technically completely correct. It’s just that
(1) it’s hard to discern “bar” from “baz”, and (2) if {"foo": None}
intentionally does not match {"foo": 1}, we may want to document that,
because it isn’t intuitively clear from the description. If it’s a bug,
the code should be fixed (and it should still be documented).
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method
2019-05-24 12:38 ` Max Reitz
@ 2019-05-24 17:57 ` John Snow
0 siblings, 0 replies; 12+ messages in thread
From: John Snow @ 2019-05-24 17:57 UTC (permalink / raw)
To: Max Reitz, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
On 5/24/19 8:38 AM, Max Reitz wrote:
> On 23.05.19 20:03, John Snow wrote:
>>
>>
>> On 5/23/19 1:49 PM, Max Reitz wrote:
>>> On 23.05.19 19:06, John Snow wrote:
>>>> Instead of event_wait which looks for a single event, add an events_wait
>>>> which can look for any number of events simultaneously. However, it
>>>> will still only return one at a time, whichever happens first.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>> python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
>>>> 1 file changed, 49 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
>>>> index 81d9657ec0..98ed8a2e28 100644
>>>> --- a/python/qemu/__init__.py
>>>> +++ b/python/qemu/__init__.py
>>>> @@ -402,42 +402,71 @@ class QEMUMachine(object):
>>>> self._qmp.clear_events()
>>>> return events
>>>>
>>>> - def event_wait(self, name, timeout=60.0, match=None):
>>>> + @staticmethod
>>>> + def event_match(event, match=None):
>>>> """
>>>> - Wait for specified timeout on named event in QMP; optionally filter
>>>> - results by match.
>>>> + Check if an event matches optional match criteria.
>>>>
>>>> - The 'match' is checked to be a recursive subset of the 'event'; skips
>>>> - branch processing on match's value None
>>>> - {"foo": {"bar": 1}} matches {"foo": None}
>>>> - {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
>>>> + The match criteria takes the form of a matching subdict. The event is
>>>> + checked to be a superset of the subdict, recursively, with matching
>>>> + values whenever those values are not None.
>>>> +
>>>> + Examples, with the subdict queries on the left:
>>>> + - None matches any object.
>>>> + - {"foo": None} matches {"foo": {"bar": 1}}
>>>> + - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
>>>
>>> Pre-existing, but the difference between “bar” and “baz” confused me
>>> quite a bit.
>>>
>>> Also, I wonder... {"foo": None} would not match {"foo": 1}, right?
>>> Does that make sense? Shouldn’t None be the wildcard here in general?
>>> (Also pre-existing of course.)
>>>
>>> But this patch doesn’t make things worse, so:
>>>
>>> Reviewed-by: Max Reitz <mreitz@redhat.com>
>>>
>>> (I’d still like your opinion.)
>>>
>>
>> I knew I was inviting trouble by trying to re-document this.
>>
>> The intention I had when writing the docs, which I think are wrong now,
>> was for {"foo": None} to match {"foo": 1}, but I think you're right that
>> it won't because '1' isn't a dict, so it tests for equality instead.
>>
>> So I need to fix this one up a little bit, but I'll take the review as a
>> sign that this approach seems workable.
>
> I think the comment is technically completely correct. It’s just that
> (1) it’s hard to discern “bar” from “baz”, and (2) if {"foo": None}
> intentionally does not match {"foo": 1}, we may want to document that,
> because it isn’t intuitively clear from the description. If it’s a bug,
> the code should be fixed (and it should still be documented).
>
> Max
>
Yeah, I see. OK; I have prepared a patch that can be applied
independently after this series, if you'd like to stage this as-is. The
new patch changes behavior of this function a little, but is backwards
compatible with no changes because nobody was using these edge cases.
--js
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
` (5 preceding siblings ...)
2019-05-23 17:51 ` [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early Max Reitz
@ 2019-05-27 13:12 ` Max Reitz
6 siblings, 0 replies; 12+ messages in thread
From: Max Reitz @ 2019-05-27 13:12 UTC (permalink / raw)
To: John Snow, qemu-devel, qemu-block
Cc: Kevin Wolf, Markus Armbruster, qemu-stable
[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]
On 23.05.19 19:06, John Snow wrote:
> See patch one's commit message for justification.
> Patches 2-5 are for testing, because that's always how these things go.
>
> 001/5:[----] [--] 'blockdev-backup: don't check aio_context too early'
> 002/5:[0004] [FC] 'iotests.py: do not use infinite waits'
> 003/5:[down] 'QEMUMachine: add events_wait method'
> 004/5:[0022] [FC] 'iotests.py: rewrite run_job to be pickier'
> 005/5:[0017] [FC] 'iotests: add iotest 250 for testing blockdev-backup
> across iothread contexts'
>
> v3: Rebased on Max's staging branch:
> Rebase patch 2
> added patch 3, to add events_wait.
> Rework patch 4 to make run_job consume legacy events, too
> Minorly edit patch 5 due to the two above.
> v2: added patch 4, with iotest framework adjustments in patches 2/3.
Thanks, applied to my block branch:
https://git.xanclic.moe/XanClic/qemu/commits/branch/block
https://github.com/XanClic/qemu/commits/block
(:-P)
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-05-27 13:14 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-23 17:06 [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 1/5] " John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 2/5] iotests.py: do not use infinite waits John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 3/5] QEMUMachine: add events_wait method John Snow
2019-05-23 17:49 ` Max Reitz
2019-05-23 18:03 ` John Snow
2019-05-24 12:38 ` Max Reitz
2019-05-24 17:57 ` John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 4/5] iotests.py: rewrite run_job to be pickier John Snow
2019-05-23 17:06 ` [Qemu-devel] [PATCH v3 5/5] iotests: add iotest 250 for testing blockdev-backup across iothread contexts John Snow
2019-05-23 17:51 ` [Qemu-devel] [PATCH v3 0/5] blockdev-backup: don't check aio_context too early Max Reitz
2019-05-27 13:12 ` Max Reitz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.