* [PATCH 01/97] qcow2: Avoid COW during metadata preallocation
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 02/97] qcow2: Add errp to preallocate_co() Michael Roth
` (100 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
Limiting the allocation to INT_MAX bytes isn't particularly clever
because it means that the final cluster will be a partial cluster which
will be completed through a COW operation. This results in unnecessary
data read and write requests which lead to an unwanted non-sparse
filesystem block for metadata preallocation.
Align the maximum allocation size down to the cluster size to avoid this
situation.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit f29fbf7c6b1c9a84f6931c1c222716fbe073e6e4)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/qcow2.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 3ace3b2209..dfac74c264 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2734,7 +2734,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
bytes = new_length - offset;
while (bytes) {
- cur_bytes = MIN(bytes, INT_MAX);
+ cur_bytes = MIN(bytes, QEMU_ALIGN_DOWN(INT_MAX, s->cluster_size));
ret = qcow2_alloc_cluster_offset(bs, offset, &cur_bytes,
&host_offset, &meta);
if (ret < 0) {
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 02/97] qcow2: Add errp to preallocate_co()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
2019-10-01 23:44 ` [PATCH 01/97] qcow2: Avoid COW during metadata preallocation Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 03/97] qcow2: Fix full preallocation with external data file Michael Roth
` (99 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
We'll add a bdrv_co_truncate() call in the next patch which can return
an Error that we don't want to discard. So add an errp parameter to
preallocate_co().
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit 360bd07471dfd1830246e8403ffdc9ba9d82f9d4)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/qcow2.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index dfac74c264..b4f9f5a240 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2721,7 +2721,7 @@ static int qcow2_set_up_encryption(BlockDriverState *bs,
* Returns: 0 on success, -errno on failure.
*/
static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
- uint64_t new_length)
+ uint64_t new_length, Error **errp)
{
BDRVQcow2State *s = bs->opaque;
uint64_t bytes;
@@ -2738,6 +2738,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
ret = qcow2_alloc_cluster_offset(bs, offset, &cur_bytes,
&host_offset, &meta);
if (ret < 0) {
+ error_setg_errno(errp, -ret, "Allocating clusters failed");
return ret;
}
@@ -2746,6 +2747,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
ret = qcow2_alloc_cluster_link_l2(bs, meta);
if (ret < 0) {
+ error_setg_errno(errp, -ret, "Mapping clusters failed");
qcow2_free_any_clusters(bs, meta->alloc_offset,
meta->nb_clusters, QCOW2_DISCARD_NEVER);
return ret;
@@ -2775,6 +2777,7 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
ret = bdrv_pwrite(s->data_file, (host_offset + cur_bytes) - 1,
&data, 1);
if (ret < 0) {
+ error_setg_errno(errp, -ret, "Writing to EOF failed");
return ret;
}
}
@@ -3748,9 +3751,8 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
break;
case PREALLOC_MODE_METADATA:
- ret = preallocate_co(bs, old_length, offset);
+ ret = preallocate_co(bs, old_length, offset, errp);
if (ret < 0) {
- error_setg_errno(errp, -ret, "Preallocation failed");
goto fail;
}
break;
@@ -3766,9 +3768,8 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
/* With a data file, preallocation means just allocating the metadata
* and forwarding the truncate request to the data file */
if (has_data_file(bs)) {
- ret = preallocate_co(bs, old_length, offset);
+ ret = preallocate_co(bs, old_length, offset, errp);
if (ret < 0) {
- error_setg_errno(errp, -ret, "Preallocation failed");
goto fail;
}
break;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 03/97] qcow2: Fix full preallocation with external data file
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
2019-10-01 23:44 ` [PATCH 01/97] qcow2: Avoid COW during metadata preallocation Michael Roth
2019-10-01 23:44 ` [PATCH 02/97] qcow2: Add errp to preallocate_co() Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 04/97] megasas: fix mapped frame size Michael Roth
` (98 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
preallocate_co() already gave the data file the full size without
forwarding the requested preallocation mode to the protocol. When
bdrv_co_truncate() was called later with the preallocation mode, the
file didn't actually grow any more, so the data file stayed unallocated
even if full preallocation was requested.
Pass the right preallocation mode to preallocate_co() and remove the
second bdrv_co_truncate() to fix this. As a side effect, the ugly
one-byte write in preallocate_co() is replaced with a truncate call,
now leaving the last block unallocated on the protocol level as it
should be.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit 718c0fce2f56755a8d8f737607779a98aa6e7cc4)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/qcow2.c | 41 +++++++++++++++++++++++------------------
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index b4f9f5a240..7fbef97aab 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2721,11 +2721,13 @@ static int qcow2_set_up_encryption(BlockDriverState *bs,
* Returns: 0 on success, -errno on failure.
*/
static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
- uint64_t new_length, Error **errp)
+ uint64_t new_length, PreallocMode mode,
+ Error **errp)
{
BDRVQcow2State *s = bs->opaque;
uint64_t bytes;
uint64_t host_offset = 0;
+ int64_t file_length;
unsigned int cur_bytes;
int ret;
QCowL2Meta *meta;
@@ -2772,12 +2774,19 @@ static int coroutine_fn preallocate_co(BlockDriverState *bs, uint64_t offset,
* all of the allocated clusters (otherwise we get failing reads after
* EOF). Extend the image to the last allocated sector.
*/
- if (host_offset != 0) {
- uint8_t data = 0;
- ret = bdrv_pwrite(s->data_file, (host_offset + cur_bytes) - 1,
- &data, 1);
+ file_length = bdrv_getlength(s->data_file->bs);
+ if (file_length < 0) {
+ error_setg_errno(errp, -file_length, "Could not get file size");
+ return file_length;
+ }
+
+ if (host_offset + cur_bytes > file_length) {
+ if (mode == PREALLOC_MODE_METADATA) {
+ mode = PREALLOC_MODE_OFF;
+ }
+ ret = bdrv_co_truncate(s->data_file, host_offset + cur_bytes, mode,
+ errp);
if (ret < 0) {
- error_setg_errno(errp, -ret, "Writing to EOF failed");
return ret;
}
}
@@ -3748,10 +3757,16 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
switch (prealloc) {
case PREALLOC_MODE_OFF:
+ if (has_data_file(bs)) {
+ ret = bdrv_co_truncate(s->data_file, offset, prealloc, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
break;
case PREALLOC_MODE_METADATA:
- ret = preallocate_co(bs, old_length, offset, errp);
+ ret = preallocate_co(bs, old_length, offset, prealloc, errp);
if (ret < 0) {
goto fail;
}
@@ -3768,7 +3783,7 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
/* With a data file, preallocation means just allocating the metadata
* and forwarding the truncate request to the data file */
if (has_data_file(bs)) {
- ret = preallocate_co(bs, old_length, offset, errp);
+ ret = preallocate_co(bs, old_length, offset, prealloc, errp);
if (ret < 0) {
goto fail;
}
@@ -3883,16 +3898,6 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
bs->total_sectors = offset / BDRV_SECTOR_SIZE;
- if (has_data_file(bs)) {
- if (prealloc == PREALLOC_MODE_METADATA) {
- prealloc = PREALLOC_MODE_OFF;
- }
- ret = bdrv_co_truncate(s->data_file, offset, prealloc, errp);
- if (ret < 0) {
- goto fail;
- }
- }
-
/* write updated header.size */
offset = cpu_to_be64(offset);
ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size),
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 04/97] megasas: fix mapped frame size
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (2 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 03/97] qcow2: Fix full preallocation with external data file Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 05/97] qcow2: Fix qcow2_make_empty() with external data file Michael Roth
` (97 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, Peter Lieven, qemu-stable
From: Peter Lieven <pl@kamp.de>
the current value of 1024 bytes (16 * MFI_FRAME_SIZE) we map is not enough to hold
the maximum number of scatter gather elements we advertise. We actually need a
maximum of 2048 bytes. This is 128 max sg elements * 16 bytes (sizeof (union mfi_sgl)).
Cc: qemu-stable@nongnu.org
Signed-off-by: Peter Lieven <pl@kamp.de>
Message-Id: <20190404121015.28634-1-pl@kamp.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
(cherry picked from commit 2e56fbc87f6ec3cd56c37b01d313abd502b80d61)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/scsi/megasas.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/scsi/megasas.c b/hw/scsi/megasas.c
index a56317e026..5ad762de23 100644
--- a/hw/scsi/megasas.c
+++ b/hw/scsi/megasas.c
@@ -477,7 +477,7 @@ static MegasasCmd *megasas_enqueue_frame(MegasasState *s,
{
PCIDevice *pcid = PCI_DEVICE(s);
MegasasCmd *cmd = NULL;
- int frame_size = MFI_FRAME_SIZE * 16;
+ int frame_size = MEGASAS_MAX_SGE * sizeof(union mfi_sgl);
hwaddr frame_size_p = frame_size;
unsigned long index;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 05/97] qcow2: Fix qcow2_make_empty() with external data file
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (3 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 04/97] megasas: fix mapped frame size Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 06/97] block: Fix AioContext switch for bs->drv == NULL Michael Roth
` (96 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
make_completely_empty() is an optimisated path for bdrv_make_empty()
where completely new metadata is created inside the image file instead
of going through all clusters and discarding them. For an external data
file, however, we actually need to do discard operations on the data
file; just overwriting the qcow2 file doesn't get rid of the data.
The necessary slow path with an explicit discard operation already
exists for other cases. Use it for external data files, too.
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
(cherry picked from commit db04524f820582ebf1189223b6378de238511da1)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/qcow2.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 7fbef97aab..840f289a48 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -4384,14 +4384,17 @@ static int qcow2_make_empty(BlockDriverState *bs)
if (s->qcow_version >= 3 && !s->snapshots && !s->nb_bitmaps &&
3 + l1_clusters <= s->refcount_block_size &&
- s->crypt_method_header != QCOW_CRYPT_LUKS) {
+ s->crypt_method_header != QCOW_CRYPT_LUKS &&
+ !has_data_file(bs)) {
/* The following function only works for qcow2 v3 images (it
* requires the dirty flag) and only as long as there are no
* features that reserve extra clusters (such as snapshots,
* LUKS header, or persistent bitmaps), because it completely
* empties the image. Furthermore, the L1 table and three
* additional clusters (image header, refcount table, one
- * refcount block) have to fit inside one refcount block. */
+ * refcount block) have to fit inside one refcount block. It
+ * only resets the image file, i.e. does not work with an
+ * external data file. */
return make_completely_empty(bs);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 06/97] block: Fix AioContext switch for bs->drv == NULL
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (4 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 05/97] qcow2: Fix qcow2_make_empty() with external data file Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 07/97] cutils: Fix size_to_str() on 32-bit platforms Michael Roth
` (95 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
Even for block nodes with bs->drv == NULL, we can't just ignore a
bdrv_set_aio_context() call. Leaving the node in its old context can
mean that it's still in an iothread context in bdrv_close_all() during
shutdown, resulting in an attempted unlock of the AioContext lock which
we don't hold.
This is an example stack trace of a related crash:
#0 0x00007ffff59da57f in raise () at /lib64/libc.so.6
#1 0x00007ffff59c4895 in abort () at /lib64/libc.so.6
#2 0x0000555555b97b1e in error_exit (err=<optimized out>, msg=msg@entry=0x555555d386d0 <__func__.19059> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36
#3 0x0000555555b97f7f in qemu_mutex_unlock_impl (mutex=mutex@entry=0x5555568002f0, file=file@entry=0x555555d378df "util/async.c", line=line@entry=507) at util/qemu-thread-posix.c:97
#4 0x0000555555b92f55 in aio_context_release (ctx=ctx@entry=0x555556800290) at util/async.c:507
#5 0x0000555555b05cf8 in bdrv_prwv_co (child=child@entry=0x7fffc80012f0, offset=offset@entry=131072, qiov=qiov@entry=0x7fffffffd4f0, is_write=is_write@entry=true, flags=flags@entry=0)
at block/io.c:833
#6 0x0000555555b060a9 in bdrv_pwritev (qiov=0x7fffffffd4f0, offset=131072, child=0x7fffc80012f0) at block/io.c:990
#7 0x0000555555b060a9 in bdrv_pwrite (child=0x7fffc80012f0, offset=131072, buf=<optimized out>, bytes=<optimized out>) at block/io.c:990
#8 0x0000555555ae172b in qcow2_cache_entry_flush (bs=bs@entry=0x555556810680, c=c@entry=0x5555568cc740, i=i@entry=0) at block/qcow2-cache.c:51
#9 0x0000555555ae18dd in qcow2_cache_write (bs=bs@entry=0x555556810680, c=0x5555568cc740) at block/qcow2-cache.c:248
#10 0x0000555555ae15de in qcow2_cache_flush (bs=0x555556810680, c=<optimized out>) at block/qcow2-cache.c:259
#11 0x0000555555ae16b1 in qcow2_cache_flush_dependency (c=0x5555568a1700, c=0x5555568a1700, bs=0x555556810680) at block/qcow2-cache.c:194
#12 0x0000555555ae16b1 in qcow2_cache_entry_flush (bs=bs@entry=0x555556810680, c=c@entry=0x5555568a1700, i=i@entry=0) at block/qcow2-cache.c:194
#13 0x0000555555ae18dd in qcow2_cache_write (bs=bs@entry=0x555556810680, c=0x5555568a1700) at block/qcow2-cache.c:248
#14 0x0000555555ae15de in qcow2_cache_flush (bs=bs@entry=0x555556810680, c=<optimized out>) at block/qcow2-cache.c:259
#15 0x0000555555ad242c in qcow2_inactivate (bs=bs@entry=0x555556810680) at block/qcow2.c:2124
#16 0x0000555555ad2590 in qcow2_close (bs=0x555556810680) at block/qcow2.c:2153
#17 0x0000555555ab0c62 in bdrv_close (bs=0x555556810680) at block.c:3358
#18 0x0000555555ab0c62 in bdrv_delete (bs=0x555556810680) at block.c:3542
#19 0x0000555555ab0c62 in bdrv_unref (bs=0x555556810680) at block.c:4598
#20 0x0000555555af4d72 in blk_remove_bs (blk=blk@entry=0x5555568103d0) at block/block-backend.c:785
#21 0x0000555555af4dbb in blk_remove_all_bs () at block/block-backend.c:483
#22 0x0000555555aae02f in bdrv_close_all () at block.c:3412
#23 0x00005555557f9796 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4776
The reproducer I used is a qcow2 image on gluster volume, where the
virtual disk size (4 GB) is larger than the gluster volume size (64M),
so we can easily trigger an ENOSPC. This backend is assigned to a
virtio-blk device using an iothread, and then from the guest a
'dd if=/dev/zero of=/dev/vda bs=1G count=1' causes the VM to stop
because of an I/O error. qemu_gluster_co_flush_to_disk() sets
bs->drv = NULL on error, so when virtio-blk stops the dataplane, the
block nodes stay in the iothread AioContext. A 'quit' monitor command
issued from this paused state crashes the process.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1631227
Cc: qemu-stable@nongnu.org
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
(cherry picked from commit 1bffe1ae7a7b707c3a14ea2ccd00d3609d3ce4d8)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block.c | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/block.c b/block.c
index 16615bc876..9ae5c0ed2f 100644
--- a/block.c
+++ b/block.c
@@ -5672,10 +5672,6 @@ void bdrv_detach_aio_context(BlockDriverState *bs)
BdrvAioNotifier *baf, *baf_tmp;
BdrvChild *child;
- if (!bs->drv) {
- return;
- }
-
assert(!bs->walking_aio_notifiers);
bs->walking_aio_notifiers = true;
QLIST_FOREACH_SAFE(baf, &bs->aio_notifiers, list, baf_tmp) {
@@ -5690,7 +5686,7 @@ void bdrv_detach_aio_context(BlockDriverState *bs)
*/
bs->walking_aio_notifiers = false;
- if (bs->drv->bdrv_detach_aio_context) {
+ if (bs->drv && bs->drv->bdrv_detach_aio_context) {
bs->drv->bdrv_detach_aio_context(bs);
}
QLIST_FOREACH(child, &bs->children, next) {
@@ -5709,10 +5705,6 @@ void bdrv_attach_aio_context(BlockDriverState *bs,
BdrvAioNotifier *ban, *ban_tmp;
BdrvChild *child;
- if (!bs->drv) {
- return;
- }
-
if (bs->quiesce_counter) {
aio_disable_external(new_context);
}
@@ -5722,7 +5714,7 @@ void bdrv_attach_aio_context(BlockDriverState *bs,
QLIST_FOREACH(child, &bs->children, next) {
bdrv_attach_aio_context(child->bs, new_context);
}
- if (bs->drv->bdrv_attach_aio_context) {
+ if (bs->drv && bs->drv->bdrv_attach_aio_context) {
bs->drv->bdrv_attach_aio_context(bs, new_context);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 07/97] cutils: Fix size_to_str() on 32-bit platforms
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (5 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 06/97] block: Fix AioContext switch for bs->drv == NULL Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 08/97] Makefile: add nit-picky mode to sphinx-build Michael Roth
` (94 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Eric Blake <eblake@redhat.com>
When extracting a human-readable size formatter, we changed 'uint64_t
div' pre-patch to 'unsigned long div' post-patch. Which breaks on
32-bit platforms, resulting in 'inf' instead of intended values larger
than 999GB.
Fixes: 22951aaa
CC: qemu-stable@nongnu.org
Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 754da86714d550c3f995f11a2587395081362e0a)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
util/cutils.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/util/cutils.c b/util/cutils.c
index e098debdc0..d682c90901 100644
--- a/util/cutils.c
+++ b/util/cutils.c
@@ -825,7 +825,7 @@ const char *qemu_ether_ntoa(const MACAddr *mac)
char *size_to_str(uint64_t val)
{
static const char *suffixes[] = { "", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei" };
- unsigned long div;
+ uint64_t div;
int i;
/*
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 08/97] Makefile: add nit-picky mode to sphinx-build
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (6 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 07/97] cutils: Fix size_to_str() on 32-bit platforms Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 09/97] docs/interop/bitmaps: rewrite and modernize doc Michael Roth
` (93 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
If we add references that don't resolve (or accidentally remove them),
it will be helpful to have warning messages alerting us to that.
Further, turn those warnings into errors so we can be alerted to these
problems sooner rather than later.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190426221528.30293-2-jsnow@redhat.com
[adjusted commit message. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index 04a0d45050..d4c5750256 100644
--- a/Makefile
+++ b/Makefile
@@ -899,7 +899,7 @@ docs/version.texi: $(SRC_PATH)/VERSION
sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index.html
# Canned command to build a single manual
-build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
+build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -W -n -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
# We assume all RST files in the manual's directory are used in it
manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 09/97] docs/interop/bitmaps: rewrite and modernize doc
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (7 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 08/97] Makefile: add nit-picky mode to sphinx-build Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 10/97] spapr/xive: fix EQ page addresses above 64GB Michael Roth
` (92 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
This just about rewrites the entirety of the bitmaps.rst document to
make it consistent with the 4.0 release. I have added new features seen
in the 4.0 release, as well as tried to clarify some points that keep
coming up when discussing this feature both in-house and upstream.
It does not yet cover pull backups or migration details, but I intend to
keep extending this document to cover those cases.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190426221528.30293-3-jsnow@redhat.com
[Adjusted commit message. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 90edef80a0852cf8a3d2668898ee40e8970e4314)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
docs/interop/bitmaps.rst | 1573 ++++++++++++++++++++++++++++++--------
1 file changed, 1251 insertions(+), 322 deletions(-)
diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
index 7bcfe7f461..510e8809a9 100644
--- a/docs/interop/bitmaps.rst
+++ b/docs/interop/bitmaps.rst
@@ -1,5 +1,5 @@
..
- Copyright 2015 John Snow <jsnow@redhat.com> and Red Hat, Inc.
+ Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc.
All rights reserved.
This file is licensed via The FreeBSD Documentation License, the full
@@ -9,547 +9,1476 @@
Dirty Bitmaps and Incremental Backup
====================================
-- Dirty Bitmaps are objects that track which data needs to be backed up
- for the next incremental backup.
+Dirty Bitmaps are in-memory objects that track writes to block devices. They
+can be used in conjunction with various block job operations to perform
+incremental or differential backup regimens.
-- Dirty bitmaps can be created at any time and attached to any node
- (not just complete drives).
+This document explains the conceptual mechanisms, as well as up-to-date,
+complete and comprehensive documentation on the API to manipulate them.
+(Hopefully, the "why", "what", and "how".)
+
+The intended audience for this document is developers who are adding QEMU
+backup features to management applications, or power users who run and
+administer QEMU directly via QMP.
.. contents::
+Overview
+--------
+
+Bitmaps are bit vectors where each '1' bit in the vector indicates a modified
+("dirty") segment of the corresponding block device. The size of the segment
+that is tracked is the granularity of the bitmap. If the granularity of a
+bitmap is 64K, each '1' bit means that a 64K region as a whole may have
+changed in some way, possibly by as little as one byte.
+
+Smaller granularities mean more accurate tracking of modified disk data, but
+requires more computational overhead and larger bitmap sizes. Larger
+granularities mean smaller bitmap sizes, but less targeted backups.
+
+The size of a bitmap (in bytes) can be computed as such:
+ ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8)
+
+e.g. the size of a 64KiB granularity bitmap on a 2TiB image is:
+ ``size`` = ((2147483648K / 64K) / 8)
+ = 4194304B = 4MiB.
+
+QEMU uses these bitmaps when making incremental backups to know which sections
+of the file to copy out. They are not enabled by default and must be
+explicitly added in order to begin tracking writes.
+
+Bitmaps can be created at any time and can be attached to any arbitrary block
+node in the storage graph, but are most useful conceptually when attached to
+the root node attached to the guest's storage device model.
+
+That is to say: It's likely most useful to track the guest's writes to disk,
+but you could theoretically track things like qcow2 metadata changes by
+attaching the bitmap elsewhere in the storage graph. This is beyond the scope
+of this document.
+
+QEMU supports persisting these bitmaps to disk via the qcow2 image format.
+Bitmaps which are stored or loaded in this way are called "persistent",
+whereas bitmaps that are not are called "transient".
+
+QEMU also supports the migration of both transient bitmaps (tracking any
+arbitrary image format) or persistent bitmaps (qcow2) via live migration.
+
+Supported Image Formats
+-----------------------
+
+QEMU supports all documented features below on the qcow2 image format.
+
+However, qcow2 is only strictly necessary for the persistence feature, which
+writes bitmap data to disk upon close. If persistence is not required for a
+specific use case, all bitmap features excepting persistence are available for
+any arbitrary image format.
+
+For example, Dirty Bitmaps can be combined with the 'raw' image format, but
+any changes to the bitmap will be discarded upon exit.
+
+.. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent
+ bitmaps are available only on qcow2 images.
+
Dirty Bitmap Names
------------------
-- A dirty bitmap's name is unique to the node, but bitmaps attached to
- different nodes can share the same name.
+Bitmap objects need a method to reference them in the API. All API-created and
+managed bitmaps have a human-readable name chosen by the user at creation
+time.
-- Dirty bitmaps created for internal use by QEMU may be anonymous and
- have no name, but any user-created bitmaps must have a name. There
- can be any number of anonymous bitmaps per node.
+- A bitmap's name is unique to the node, but bitmaps attached to different
+ nodes can share the same name. Therefore, all bitmaps are addressed via
+ their (node, name) pair.
-- The name of a user-created bitmap must not be empty ("").
+- The name of a user-created bitmap cannot be empty ("").
-Bitmap Modes
-------------
+- Transient bitmaps can have JSON unicode names that are effectively not
+ length limited. (QMP protocol may restrict messages to less than 64MiB.)
+
+- Persistent storage formats may impose their own requirements on bitmap names
+ and namespaces. Presently, only qcow2 supports persistent bitmaps. See
+ docs/interop/qcow2.txt for more details on restrictions. Notably:
+
+ - qcow2 bitmap names are limited to between 1 and 1023 bytes long.
+
+ - No two bitmaps saved to the same qcow2 file may share the same name.
+
+- QEMU occasionally uses bitmaps for internal use which have no name. They are
+ hidden from API query calls, cannot be manipulated by the external API, are
+ never persistent, nor ever migrated.
+
+Bitmap Status
+-------------
+
+Dirty Bitmap objects can be queried with the QMP command `query-block
+<qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the
+`BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure.
+
+This struct shows the name, granularity, and dirty byte count for each bitmap.
+Additionally, it shows several boolean status indicators:
+
+- ``recording``: This bitmap is recording writes.
+- ``busy``: This bitmap is in-use by an operation.
+- ``persistent``: This bitmap is a persistent type.
+- ``inconsistent``: This bitmap is corrupted and cannot be used.
+
+The ``+busy`` status prohibits you from deleting, clearing, or otherwise
+modifying a bitmap, and happens when the bitmap is being used for a backup
+operation or is in the process of being loaded from a migration. Many of the
+commands documented below will refuse to work on such bitmaps.
+
+The ``+inconsistent`` status similarly prohibits almost all operations,
+notably allowing only the ``block-dirty-bitmap-remove`` operation.
+
+There is also a deprecated ``status`` field of type `DirtyBitmapStatus
+<qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had
+five visible states:
+
+ #. ``Frozen``: This bitmap is currently in-use by an operation and is
+ immutable. It can't be deleted, renamed, reset, etc.
+
+ (This is now ``+busy``.)
+
+ #. ``Disabled``: This bitmap is not recording new writes.
+
+ (This is now ``-recording -busy``.)
+
+ #. ``Active``: This bitmap is recording new writes.
+
+ (This is now ``+recording -busy``.)
-- A bitmap can be "frozen," which means that it is currently in-use by
- a backup operation and cannot be deleted, renamed, written to, reset,
- etc.
+ #. ``Locked``: This bitmap is in-use by an operation, and is immutable.
+ The difference from "Frozen" was primarily implementation details.
-- The normal operating mode for a bitmap is "active."
+ (This is now ``+busy``.)
+
+ #. ``Inconsistent``: This persistent bitmap was not saved to disk
+ correctly, and can no longer be used. It remains in memory to serve as
+ an indicator of failure.
+
+ (This is now ``+inconsistent``.)
+
+These states are directly replaced by the status indicators and should not be
+used. The difference between ``Frozen`` and ``Locked`` is an implementation
+detail and should not be relevant to external users.
Basic QMP Usage
---------------
+The primary interface to manipulating bitmap objects is via the QMP
+interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad
+overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all
+QMP commands.
+
Supported Commands
~~~~~~~~~~~~~~~~~~
+There are six primary bitmap-management API commands:
+
- ``block-dirty-bitmap-add``
- ``block-dirty-bitmap-remove``
- ``block-dirty-bitmap-clear``
+- ``block-dirty-bitmap-disable``
+- ``block-dirty-bitmap-enable``
+- ``block-dirty-bitmap-merge``
-Creation
-~~~~~~~~
+And one related query command:
-- To create a new bitmap, enabled, on the drive with id=drive0:
+- ``query-block``
-.. code:: json
+Creation: block-dirty-bitmap-add
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- { "execute": "block-dirty-bitmap-add",
- "arguments": {
- "node": "drive0",
- "name": "bitmap0"
- }
- }
+`block-dirty-bitmap-add
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_:
-- This bitmap will have a default granularity that matches the cluster
- size of its associated drive, if available, clamped to between [4KiB,
- 64KiB]. The current default for qcow2 is 64KiB.
+Creates a new bitmap that tracks writes to the specified node. granularity,
+persistence, and recording state can be adjusted at creation time.
-- To create a new bitmap that tracks changes in 32KiB segments:
+.. admonition:: Example
-.. code:: json
+ to create a new, actively recording persistent bitmap:
- { "execute": "block-dirty-bitmap-add",
- "arguments": {
- "node": "drive0",
- "name": "bitmap0",
- "granularity": 32768
- }
- }
+ .. code:: json
-Deletion
-~~~~~~~~
+ -> { "execute": "block-dirty-bitmap-add",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0",
+ "persistent": true,
+ }
+ }
-- Bitmaps that are frozen cannot be deleted.
+ <- { "return": {} }
-- Deleting the bitmap does not impact any other bitmaps attached to the
- same node, nor does it affect any backups already created from this
- node.
+- This bitmap will have a default granularity that matches the cluster size of
+ its associated drive, if available, clamped to between [4KiB, 64KiB]. The
+ current default for qcow2 is 64KiB.
-- Because bitmaps are only unique to the node to which they are
- attached, you must specify the node/drive name here, too.
+.. admonition:: Example
-.. code:: json
+ To create a new, disabled (``-recording``), transient bitmap that tracks
+ changes in 32KiB segments:
- { "execute": "block-dirty-bitmap-remove",
- "arguments": {
- "node": "drive0",
- "name": "bitmap0"
- }
- }
+ .. code:: json
-Resetting
-~~~~~~~~~
+ -> { "execute": "block-dirty-bitmap-add",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap1",
+ "granularity": 32768,
+ "disabled": true
+ }
+ }
-- Resetting a bitmap will clear all information it holds.
+ <- { "return": {} }
-- An incremental backup created from an empty bitmap will copy no data,
- as if nothing has changed.
+Deletion: block-dirty-bitmap-remove
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. code:: json
+`block-dirty-bitmap-remove
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_:
+
+Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed.
+
+- Deleting a bitmap does not impact any other bitmaps attached to the same
+ node, nor does it affect any backups already created from this bitmap or
+ node.
+
+- Because bitmaps are only unique to the node to which they are attached, you
+ must specify the node/drive name here, too.
+
+- Deleting a persistent bitmap will remove it from the qcow2 file.
+
+.. admonition:: Example
+
+ Remove a bitmap named ``bitmap0`` from node ``drive0``:
+
+ .. code:: json
+
+ -> { "execute": "block-dirty-bitmap-remove",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ }
+
+ <- { "return": {} }
+
+Resetting: block-dirty-bitmap-clear
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`block-dirty-bitmap-clear
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_:
+
+Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared.
+
+- An incremental backup created from an empty bitmap will copy no data, as if
+ nothing has changed.
+
+.. admonition:: Example
+
+ Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``:
+
+ .. code:: json
+
+ -> { "execute": "block-dirty-bitmap-clear",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ }
+
+ <- { "return": {} }
+
+Enabling: block-dirty-bitmap-enable
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`block-dirty-bitmap-enable
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_:
- { "execute": "block-dirty-bitmap-clear",
- "arguments": {
- "node": "drive0",
- "name": "bitmap0"
- }
- }
+"Enables" a bitmap, setting the ``recording`` bit to true, causing writes to
+begin being recorded. ``+busy`` bitmaps cannot be enabled.
+
+- Bitmaps default to being enabled when created, unless configured otherwise.
+
+- Persistent enabled bitmaps will remember their ``+recording`` status on
+ load.
+
+.. admonition:: Example
+
+ To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``:
+
+ .. code:: json
+
+ -> { "execute": "block-dirty-bitmap-enable",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ }
+
+ <- { "return": {} }
+
+Enabling: block-dirty-bitmap-disable
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`block-dirty-bitmap-disable
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_:
+
+"Disables" a bitmap, setting the ``recording`` bit to false, causing further
+writes to begin being ignored. ``+busy`` bitmaps cannot be disabled.
+
+.. warning::
+
+ This is potentially dangerous: QEMU makes no effort to stop any writes if
+ there are disabled bitmaps on a node, and will not mark any disabled bitmaps
+ as ``+inconsistent`` if any such writes do happen. Backups made from such
+ bitmaps will not be able to be used to reconstruct a coherent image.
+
+- Disabling a bitmap may be useful for examining which sectors of a disk
+ changed during a specific time period, or for explicit management of
+ differential backup windows.
+
+- Persistent disabled bitmaps will remember their ``-recording`` status on
+ load.
+
+.. admonition:: Example
+
+ To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``:
+
+ .. code:: json
+
+ -> { "execute": "block-dirty-bitmap-disable",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ }
+
+ <- { "return": {} }
+
+Merging, Copying: block-dirty-bitmap-merge
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`block-dirty-bitmap-merge
+<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_:
+
+Merges one or more bitmaps into a target bitmap. For any segment that is dirty
+in any one source bitmap, the target bitmap will mark that segment dirty.
+
+- Merge takes one or more bitmaps as a source and merges them together into a
+ single destination, such that any segment marked as dirty in any source
+ bitmap(s) will be marked dirty in the destination bitmap.
+
+- Merge does not create the destination bitmap if it does not exist. A blank
+ bitmap can be created beforehand to achieve the same effect.
+
+- The destination is not cleared prior to merge, so subsequent merge
+ operations will continue to cumulatively mark more segments as dirty.
+
+- If the merge operation should fail, the destination bitmap is guaranteed to
+ be unmodified. The operation may fail if the source or destination bitmaps
+ are busy, or have different granularities.
+
+- Bitmaps can only be merged on the same node. There is only one "node"
+ argument, so all bitmaps must be attached to that same node.
+
+- Copy can be achieved by merging from a single source to an empty
+ destination.
+
+.. admonition:: Example
+
+ Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node
+ ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves
+ a copy.
+
+ .. code:: json
+
+ -> { "execute": "block-dirty-bitmap-merge",
+ "arguments": {
+ "node": "drive0",
+ "target": "new_bitmap",
+ "bitmaps: [ "bitmap0" ]
+ }
+ }
+
+ <- { "return": {} }
+
+Querying: query-block
+~~~~~~~~~~~~~~~~~~~~~
+
+`query-block
+<qemu-qmp-ref.html#index-query_002dblock>`_:
+
+Not strictly a bitmaps command, but will return information about any bitmaps
+attached to nodes serving as the root for guest devices.
+
+- The "inconsistent" bit will not appear when it is false, appearing only when
+ the value is true to indicate there is a problem.
+
+.. admonition:: Example
+
+ Query the block sub-system of QEMU. The following json has trimmed irrelevant
+ keys from the response to highlight only the bitmap-relevant portions of the
+ API. This result highlights a bitmap ``bitmap0`` attached to the root node of
+ device ``drive0``.
+
+ .. code:: json
+
+ -> {
+ "execute": "query-block",
+ "arguments": {}
+ }
+
+ <- {
+ "return": [ {
+ "dirty-bitmaps": [ {
+ "status": "active",
+ "count": 0,
+ "busy": false,
+ "name": "bitmap0",
+ "persistent": false,
+ "recording": true,
+ "granularity": 65536
+ } ],
+ "device": "drive0",
+ } ]
+ }
+
+Bitmap Persistence
+------------------
+
+As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2
+files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing
+``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to
+disk.
+
+Persistent bitmaps will be automatically loaded into memory upon load, and
+will be written back to disk upon close. Their usage should be mostly
+transparent.
+
+However, if QEMU does not get a chance to close the file cleanly, the bitmap
+will be marked as ``+inconsistent`` at next load and considered unsafe to use
+for any operation. At this point, the only valid operation on such bitmaps is
+``block-dirty-bitmap-remove``.
+
+Losing a bitmap in this way does not invalidate any existing backups that have
+been made from this bitmap, but no further backups will be able to be issued
+for this chain.
Transactions
------------
+Transactions are a QMP feature that allows you to submit multiple QMP commands
+at once, being guaranteed that they will all succeed or fail atomically,
+together. The interaction of bitmaps and transactions are demonstrated below.
+
+See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference
+for more details.
+
Justification
~~~~~~~~~~~~~
-Bitmaps can be safely modified when the VM is paused or halted by using
-the basic QMP commands. For instance, you might perform the following
-actions:
+Bitmaps can generally be modified at any time, but certain operations often
+only make sense when paired directly with other commands. When a VM is paused,
+it's easy to ensure that no guest writes occur between individual QMP
+commands. When a VM is running, this is difficult to accomplish with
+individual QMP commands that may allow guest writes to occur inbetween each
+command.
-1. Boot the VM in a paused state.
-2. Create a full drive backup of drive0.
-3. Create a new bitmap attached to drive0.
-4. Resume execution of the VM.
-5. Incremental backups are ready to be created.
+For example, using only individual QMP commands, we could:
-At this point, the bitmap and drive backup would be correctly in sync,
-and incremental backups made from this point forward would be correctly
-aligned to the full drive backup.
+#. Boot the VM in a paused state.
+#. Create a full drive backup of drive0.
+#. Create a new bitmap attached to drive0, confident that nothing has been
+ written to drive0 in the meantime.
+#. Resume execution of the VM.
+#. At a later point, issue incremental backups from ``bitmap0``.
-This is not particularly useful if we decide we want to start
-incremental backups after the VM has been running for a while, for which
-we will need to perform actions such as the following:
+At this point, the bitmap and drive backup would be correctly in sync, and
+incremental backups made from this point forward would be correctly aligned to
+the full drive backup.
-1. Boot the VM and begin execution.
-2. Using a single transaction, perform the following operations:
+This is not particularly useful if we decide we want to start incremental
+backups after the VM has been running for a while, for which we would want to
+perform actions such as the following:
+
+#. Boot the VM and begin execution.
+#. Using a single transaction, perform the following operations:
- Create ``bitmap0``.
- Create a full drive backup of ``drive0``.
-3. Incremental backups are now ready to be created.
+#. At a later point, issue incremental backups from ``bitmap0``.
+
+.. note:: As a consideration, if ``bitmap0`` is created prior to the full
+ drive backup, incremental backups can still be authored from this
+ bitmap, but they will copy extra segments reflecting writes that
+ occurred prior to the backup operation. Transactions allow us to
+ narrow critical points in time to reduce waste, or, in the other
+ direction, to ensure that no segments are omitted.
Supported Bitmap Transactions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``block-dirty-bitmap-add``
- ``block-dirty-bitmap-clear``
+- ``block-dirty-bitmap-enable``
+- ``block-dirty-bitmap-disable``
+- ``block-dirty-bitmap-merge``
-The usages are identical to their respective QMP commands, but see below
-for examples.
+The usages for these commands are identical to their respective QMP commands,
+but see the sections below for concrete examples.
-Example: New Incremental Backup
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Incremental Backups - Push Model
+--------------------------------
-As outlined in the justification, perhaps we want to create a new
-incremental backup chain attached to a drive.
+Incremental backups are simply partial disk images that can be combined with
+other partial disk images on top of a base image to reconstruct a full backup
+from the point in time at which the incremental backup was issued.
-.. code:: json
+The "Push Model" here references the fact that QEMU is "pushing" the modified
+blocks out to a destination. We will be using the `drive-backup
+<qemu-qmp-ref.html#index-drive_002dbackup>`_ and `blockdev-backup
+<qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP commands to create both
+full and incremental backups.
- { "execute": "transaction",
- "arguments": {
- "actions": [
- {"type": "block-dirty-bitmap-add",
- "data": {"node": "drive0", "name": "bitmap0"} },
- {"type": "drive-backup",
- "data": {"device": "drive0", "target": "/path/to/full_backup.img",
- "sync": "full", "format": "qcow2"} }
- ]
- }
- }
+Both of these commands are jobs, which have their own QMP API for querying and
+management documented in `Background jobs
+<qemu-qmp-ref.html#Background-jobs>`_.
Example: New Incremental Backup Anchor Point
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Maybe we just want to create a new full backup with an existing bitmap
-and want to reset the bitmap to track the new chain.
+As outlined in the Transactions - `Justification`_ section, perhaps we want to
+create a new incremental backup chain attached to a drive.
+
+This example creates a new, full backup of "drive0" and accompanies it with a
+new, empty bitmap that records writes from this point in time forward.
+
+.. note:: Any new writes that happen after this command is issued, even while
+ the backup job runs, will be written locally and not to the backup
+ destination. These writes will be recorded in the bitmap
+ accordingly.
+
+.. code:: json
+
+ -> {
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "type": "block-dirty-bitmap-add",
+ "data": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "target": "/path/to/drive0.full.qcow2",
+ "sync": "full",
+ "format": "qcow2"
+ }
+ }
+ ]
+ }
+ }
+
+ <- { "return": {} }
+
+ <- {
+ "timestamp": {
+ "seconds": 1555436945,
+ "microseconds": 179620
+ },
+ "data": {
+ "status": "created",
+ "id": "drive0"
+ },
+ "event": "JOB_STATUS_CHANGE"
+ }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "status": "concluded",
+ "id": "drive0"
+ },
+ "event": "JOB_STATUS_CHANGE"
+ }
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "status": "null",
+ "id": "drive0"
+ },
+ "event": "JOB_STATUS_CHANGE"
+ }
+
+A full explanation of the job transition semantics and the JOB_STATUS_CHANGE
+event are beyond the scope of this document and will be omitted in all
+subsequent examples; above, several more events have been omitted for brevity.
+
+.. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED
+ except where necessary to illustrate workflow differences.
+
+ Omitted events and json objects will be represented by ellipses:
+ ``...``
+
+Example: Resetting an Incremental Backup Anchor Point
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If we want to start a new backup chain with an existing bitmap, we can also
+use a transaction to reset the bitmap while making a new full backup:
.. code:: json
- { "execute": "transaction",
- "arguments": {
- "actions": [
- {"type": "block-dirty-bitmap-clear",
- "data": {"node": "drive0", "name": "bitmap0"} },
- {"type": "drive-backup",
- "data": {"device": "drive0", "target": "/path/to/new_full_backup.img",
- "sync": "full", "format": "qcow2"} }
- ]
- }
- }
-
-Incremental Backups
--------------------
-
-The star of the show.
-
-**Nota Bene!** Only incremental backups of entire drives are supported
-for now. So despite the fact that you can attach a bitmap to any
-arbitrary node, they are only currently useful when attached to the root
-node. This is because drive-backup only supports drives/devices instead
-of arbitrary nodes.
+ -> {
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "type": "block-dirty-bitmap-clear",
+ "data": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "target": "/path/to/drive0.new_full.qcow2",
+ "sync": "full",
+ "format": "qcow2"
+ }
+ }
+ ]
+ }
+ }
+
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+The result of this example is identical to the first, but we clear an existing
+bitmap instead of adding a new one.
+
+.. tip:: In both of these examples, "bitmap0" is tied conceptually to the
+ creation of new, full backups. This relationship is not saved or
+ remembered by QEMU; it is up to the operator or management layer to
+ remember which bitmaps are associated with which backups.
Example: First Incremental Backup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-1. Create a full backup and sync it to the dirty bitmap, as in the
- transactional examples above; or with the VM offline, manually create
- a full copy and then create a new bitmap before the VM begins
- execution.
+#. Create a full backup and sync it to a dirty bitmap using any method:
- - Let's assume the full backup is named ``full_backup.img``.
- - Let's assume the bitmap you created is ``bitmap0`` attached to
- ``drive0``.
+ - Either of the two live backup method demonstrated above,
+ - Using QMP commands with the VM paused as in the `Justification`_ section,
+ or
+ - With the VM offline, manually copy the image and start the VM in a paused
+ state, careful to add a new bitmap before the VM begins execution.
-2. Create a destination image for the incremental backup that utilizes
- the full backup as a backing image.
+ Whichever method is chosen, let's assume that at the end of this step:
- - Let's assume the new incremental image is named
- ``incremental.0.img``.
+ - The full backup is named ``drive0.full.qcow2``.
+ - The bitmap we created is named ``bitmap0``, attached to ``drive0``.
+
+#. Create a destination image for the incremental backup that utilizes the
+ full backup as a backing image.
+
+ - Let's assume the new incremental image is named ``drive0.inc0.qcow2``:
.. code:: bash
- $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
+ -b drive0.full.qcow2 -F qcow2
-3. Issue the incremental backup command:
+#. Issue an incremental backup command:
.. code:: json
- { "execute": "drive-backup",
+ -> {
+ "execute": "drive-backup",
"arguments": {
"device": "drive0",
"bitmap": "bitmap0",
- "target": "incremental.0.img",
+ "target": "drive0.inc0.qcow2",
"format": "qcow2",
"sync": "incremental",
"mode": "existing"
}
}
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+This copies any blocks modified since the full backup was created into the
+``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked
+``+busy``. If the operation is successful, ``bitmap0`` will be cleared to
+reflect the "incremental" backup regimen, which only copies out new changes
+from each incremental backup.
+
+.. note:: Any new writes that occur after the backup operation starts do not
+ get copied to the destination. The backup's "point in time" is when
+ the backup starts, not when it ends. These writes are recorded in a
+ special bitmap that gets re-added to bitmap0 when the backup ends so
+ that the next incremental backup can copy them out.
+
Example: Second Incremental Backup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-1. Create a new destination image for the incremental backup that points
- to the previous one, e.g.: ``incremental.1.img``
+#. Create a new destination image for the incremental backup that points to
+ the previous one, e.g.: ``drive0.inc1.qcow2``
.. code:: bash
- $ qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2
+ $ qemu-img create -f qcow2 drive0.inc1.qcow2 \
+ -b drive0.inc0.qcow2 -F qcow2
-2. Issue a new incremental backup command. The only difference here is
- that we have changed the target image below.
+#. Issue a new incremental backup command. The only difference here is that we
+ have changed the target image below.
.. code:: json
- { "execute": "drive-backup",
+ -> {
+ "execute": "drive-backup",
"arguments": {
"device": "drive0",
"bitmap": "bitmap0",
- "target": "incremental.1.img",
+ "target": "drive0.inc1.qcow2",
"format": "qcow2",
"sync": "incremental",
"mode": "existing"
}
}
-Errors
-------
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+Because the first incremental backup from the previous example completed
+successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here,
+we use ``bitmap0`` again to create a new incremental backup that targets the
+previous one, creating a chain of three images:
+
+.. admonition:: Diagram
+
+ .. code:: text
+
+ +-------------------+ +-------------------+ +-------------------+
+ | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 |
+ +-------------------+ +-------------------+ +-------------------+
+
+Each new incremental backup re-synchronizes the bitmap to the latest backup
+authored, allowing a user to continue to "consume" it to create new backups on
+top of an existing chain.
+
+In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are
+complete images by themselves, but rely on their backing chain to reconstruct
+a full image. The dependency terminates with each full backup.
+
+Each backup in this chain remains independent, and is unchanged by new entries
+made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly
+valid backup of the disk as it was when that backup was issued.
+
+Example: Incremental Push Backups without Backing Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Backup images are best kept off-site, so we often will not have the preceding
+backups in a chain available to link against. This is not a problem at backup
+time; we simply do not set the backing image when creating the destination
+image:
+
+#. Create a new destination image with no backing file set. We will need to
+ specify the size of the base image, because the backing file isn't
+ available for QEMU to use to determine it.
+
+ .. code:: bash
+
+ $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G
+
+ .. note:: Alternatively, you can omit ``mode: "existing"`` from the push
+ backup commands to have QEMU create an image without a backing
+ file for you, but you lose control over format options like
+ compatibility and preallocation presets.
+
+#. Issue a new incremental backup command. Apart from the new destination
+ image, there is no difference from the last two examples.
+
+ .. code:: json
+
+ -> {
+ "execute": "drive-backup",
+ "arguments": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "target": "drive0.inc2.qcow2",
+ "format": "qcow2",
+ "sync": "incremental",
+ "mode": "existing"
+ }
+ }
+
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+The only difference from the perspective of the user is that you will need to
+set the backing image when attempting to restore the backup:
+
+.. code:: bash
+
+ $ qemu-img rebase drive0.inc2.qcow2 \
+ -u -b drive0.inc1.qcow2
+
+This uses the "unsafe" rebase mode to simply set the backing file to a file
+that isn't present.
+
+It is also possible to use ``--image-opts`` to specify the entire backing
+chain by hand as an ephemeral property at runtime, but that is beyond the
+scope of this document.
+
+Example: Multi-drive Incremental Backup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back
+both of them up such that the two backups represent the same crash-consistent
+point in time.
+
+#. For each drive, create an empty image:
+
+ .. code:: bash
+
+ $ qemu-img create -f qcow2 drive0.full.qcow2 64G
+ $ qemu-img create -f qcow2 drive1.full.qcow2 64G
+
+#. Create a full (anchor) backup for each drive, with accompanying bitmaps:
+
+ .. code:: json
+
+ -> {
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "type": "block-dirty-bitmap-add",
+ "data": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+ },
+ {
+ "type": "block-dirty-bitmap-add",
+ "data": {
+ "node": "drive1",
+ "name": "bitmap0"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "target": "/path/to/drive0.full.qcow2",
+ "sync": "full",
+ "format": "qcow2"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive1",
+ "target": "/path/to/drive1.full.qcow2",
+ "sync": "full",
+ "format": "qcow2"
+ }
+ }
+ ]
+ }
+ }
+
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive1",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+#. Later, create new destination images for each of the incremental backups
+ that point to their respective full backups:
+
+ .. code:: bash
+
+ $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
+ -b drive0.full.qcow2 -F qcow2
+ $ qemu-img create -f qcow2 drive1.inc0.qcow2 \
+ -b drive1.full.qcow2 -F qcow2
+
+#. Issue a multi-drive incremental push backup transaction:
+
+ .. code:: json
+
+ -> {
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive0.inc0.qcow2"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive1",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive1.inc0.qcow2"
+ }
+ },
+ ]
+ }
+ }
+
+ <- { "return": {} }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive1",
+ "type": "backup",
+ "speed": 0,
+ "len": 68719476736,
+ "offset": 68719476736
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
+
+ ...
-- In the event of an error that occurs after a backup job is
- successfully launched, either by a direct QMP command or a QMP
- transaction, the user will receive a ``BLOCK_JOB_COMPLETE`` event with
- a failure message, accompanied by a ``BLOCK_JOB_ERROR`` event.
+Push Backup Errors & Recovery
+-----------------------------
-- In the case of an event being cancelled, the user will receive a
- ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR
- events.
+In the event of an error that occurs after a push backup job is successfully
+launched, either by an individual QMP command or a QMP transaction, the user
+will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message,
+accompanied by a ``BLOCK_JOB_ERROR`` event.
-- In either case, the incremental backup data contained within the
- bitmap is safely rolled back, and the data within the bitmap is not
- lost. The image file created for the failed attempt can be safely
- deleted.
+In the case of a job being cancelled, the user will receive a
+``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR
+events.
-- Once the underlying problem is fixed (e.g. more storage space is
- freed up), you can simply retry the incremental backup command with
- the same bitmap.
+In either failure case, the bitmap used for the failed operation is not
+cleared. It will contain all of the dirty bits it did at the start of the
+operation, plus any new bits that got marked during the operation.
-Example
-~~~~~~~
+Effectively, the "point in time" that a bitmap is recording differences
+against is kept at the issuance of the last successful incremental backup,
+instead of being moved forward to the start of this now-failed backup.
-1. Create a target image:
+Once the underlying problem is addressed (e.g. more storage space is allocated
+on the destination), the incremental backup command can be retried with the
+same bitmap.
+
+Example: Individual Failures
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Incremental Push Backup jobs that fail individually behave simply as
+described above. This example demonstrates the single-job failure case:
+
+#. Create a target image:
.. code:: bash
- $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
+ -b drive0.full.qcow2 -F qcow2
-2. Attempt to create an incremental backup via QMP:
+#. Attempt to create an incremental backup via QMP:
.. code:: json
- { "execute": "drive-backup",
+ -> {
+ "execute": "drive-backup",
"arguments": {
"device": "drive0",
"bitmap": "bitmap0",
- "target": "incremental.0.img",
+ "target": "drive0.inc0.qcow2",
"format": "qcow2",
"sync": "incremental",
"mode": "existing"
}
}
-3. Receive an event notifying us of failure:
+ <- { "return": {} }
+
+#. Receive a pair of events indicating failure:
.. code:: json
- { "timestamp": { "seconds": 1424709442, "microseconds": 844524 },
- "data": { "speed": 0, "offset": 0, "len": 67108864,
- "error": "No space left on device",
- "device": "drive1", "type": "backup" },
- "event": "BLOCK_JOB_COMPLETED" }
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "action": "report",
+ "operation": "write"
+ },
+ "event": "BLOCK_JOB_ERROR"
+ }
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "speed": 0,
+ "offset": 0,
+ "len": 67108864,
+ "error": "No space left on device",
+ "device": "drive0",
+ "type": "backup"
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
-4. Delete the failed incremental, and re-create the image.
+#. Delete the failed image, and re-create it.
.. code:: bash
- $ rm incremental.0.img
- $ qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ $ rm drive0.inc0.qcow2
+ $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
+ -b drive0.full.qcow2 -F qcow2
-5. Retry the command after fixing the underlying problem, such as
+#. Retry the command after fixing the underlying problem, such as
freeing up space on the backup volume:
.. code:: json
- { "execute": "drive-backup",
+ -> {
+ "execute": "drive-backup",
"arguments": {
"device": "drive0",
"bitmap": "bitmap0",
- "target": "incremental.0.img",
+ "target": "drive0.inc0.qcow2",
"format": "qcow2",
"sync": "incremental",
"mode": "existing"
}
}
-6. Receive confirmation that the job completed successfully:
+ <- { "return": {} }
+
+#. Receive confirmation that the job completed successfully:
.. code:: json
- { "timestamp": { "seconds": 1424709668, "microseconds": 526525 },
- "data": { "device": "drive1", "type": "backup",
- "speed": 0, "len": 67108864, "offset": 67108864},
- "event": "BLOCK_JOB_COMPLETED" }
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 67108864,
+ "offset": 67108864
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
-Partial Transactional Failures
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Example: Partial Transactional Failures
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Sometimes, a transaction will succeed in launching and return
- success, but then later the backup jobs themselves may fail. It is
- possible that a management application may have to deal with a
- partial backup failure after a successful transaction.
+QMP commands like `drive-backup <qemu-qmp-ref.html#index-drive_002dbackup>`_
+conceptually only start a job, and so transactions containing these commands
+may succeed even if the job it created later fails. This might have surprising
+interactions with notions of how a "transaction" ought to behave.
-- If multiple backup jobs are specified in a single transaction, when
- one of them fails, it will not interact with the other backup jobs in
- any way.
+This distinction means that on occasion, a transaction containing such job
+launching commands may appear to succeed and return success, but later
+individual jobs associated with the transaction may fail. It is possible that
+a management application may have to deal with a partial backup failure after
+a "successful" transaction.
-- The job(s) that succeeded will clear the dirty bitmap associated with
- the operation, but the job(s) that failed will not. It is not "safe"
- to delete any incremental backups that were created successfully in
- this scenario, even though others failed.
+If multiple backup jobs are specified in a single transaction, if one of those
+jobs fails, it will not interact with the other backup jobs in any way by
+default. The job(s) that succeeded will clear the dirty bitmap associated with
+the operation, but the job(s) that failed will not. It is therefore not safe
+to delete any incremental backups that were created successfully in this
+scenario, even though others failed.
-Example
-^^^^^^^
+This example illustrates a transaction with two backup jobs, where one fails
+and one succeeds:
-- QMP example highlighting two backup jobs:
+#. Issue the transaction to start a backup of both drives.
.. code:: json
- { "execute": "transaction",
+ -> {
+ "execute": "transaction",
"arguments": {
"actions": [
- { "type": "drive-backup",
- "data": { "device": "drive0", "bitmap": "bitmap0",
- "format": "qcow2", "mode": "existing",
- "sync": "incremental", "target": "d0-incr-1.qcow2" } },
- { "type": "drive-backup",
- "data": { "device": "drive1", "bitmap": "bitmap1",
- "format": "qcow2", "mode": "existing",
- "sync": "incremental", "target": "d1-incr-1.qcow2" } },
- ]
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive0.inc0.qcow2"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive1",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive1.inc0.qcow2"
+ }
+ }]
}
}
-- QMP example response, highlighting one success and one failure:
+#. Receive notice that the Transaction was accepted, and jobs were
+ launched:
- - Acknowledgement that the Transaction was accepted and jobs were
- launched:
+ .. code:: json
- .. code:: json
+ <- { "return": {} }
- { "return": {} }
+#. Receive notice that the first job has completed:
- - Later, QEMU sends notice that the first job was completed:
+ .. code:: json
- .. code:: json
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 67108864,
+ "offset": 67108864
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
- { "timestamp": { "seconds": 1447192343, "microseconds": 615698 },
- "data": { "device": "drive0", "type": "backup",
- "speed": 0, "len": 67108864, "offset": 67108864 },
- "event": "BLOCK_JOB_COMPLETED"
- }
+#. Receive notice that the second job has failed:
- - Later yet, QEMU sends notice that the second job has failed:
+ .. code:: json
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive1",
+ "action": "report",
+ "operation": "read"
+ },
+ "event": "BLOCK_JOB_ERROR"
+ }
+
+ ...
+
+ <- {
+ "timestamp": {...},
+ "data": {
+ "speed": 0,
+ "offset": 0,
+ "len": 67108864,
+ "error": "Input/output error",
+ "device": "drive1",
+ "type": "backup"
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
- .. code:: json
+At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and
+must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be
+deleted. If a VM-wide incremental backup of all drives at a point-in-time is
+to be made, new backups for both drives will need to be made, taking into
+account that a new incremental backup for drive0 needs to be based on top of
+``drive0.inc0.qcow2``.
- { "timestamp": { "seconds": 1447192399, "microseconds": 683015 },
- "data": { "device": "drive1", "action": "report",
- "operation": "read" },
- "event": "BLOCK_JOB_ERROR" }
+For this example, an incremental backup for ``drive0`` was created, but not
+for ``drive1``. The last VM-wide crash-consistent backup that is available in
+this case is the full backup:
- .. code:: json
+.. code:: text
- { "timestamp": { "seconds": 1447192399, "microseconds":
- 685853 }, "data": { "speed": 0, "offset": 0, "len": 67108864,
- "error": "Input/output error", "device": "drive1", "type":
- "backup" }, "event": "BLOCK_JOB_COMPLETED" }
+ [drive0.full.qcow2] <-- [drive0.inc0.qcow2]
+ [drive1.full.qcow2]
-- In the above example, ``d0-incr-1.qcow2`` is valid and must be kept,
- but ``d1-incr-1.qcow2`` is invalid and should be deleted. If a VM-wide
- incremental backup of all drives at a point-in-time is to be made,
- new backups for both drives will need to be made, taking into account
- that a new incremental backup for drive0 needs to be based on top of
- ``d0-incr-1.qcow2``.
+To repair this, issue a new incremental backup across both drives. The result
+will be backup chains that resemble the following:
-Grouped Completion Mode
-~~~~~~~~~~~~~~~~~~~~~~~
+.. code:: text
-- While jobs launched by transactions normally complete or fail on
- their own, it is possible to instruct them to complete or fail
- together as a group.
+ [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2]
+ [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2]
-- QMP transactions take an optional properties structure that can
- affect the semantics of the transaction.
+Example: Grouped Completion Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- The "completion-mode" transaction property can be either "individual"
- which is the default, legacy behavior described above, or "grouped,"
- a new behavior detailed below.
+While jobs launched by transactions normally complete or fail individually,
+it's possible to instruct them to complete or fail together as a group. QMP
+transactions take an optional properties structure that can affect the
+behavior of the transaction.
-- Delayed Completion: In grouped completion mode, no jobs will report
- success until all jobs are ready to report success.
+The ``completion-mode`` transaction property can be either ``individual``
+which is the default legacy behavior described above, or ``grouped``, detailed
+below.
-- Grouped failure: If any job fails in grouped completion mode, all
- remaining jobs will be cancelled. Any incremental backups will
- restore their dirty bitmap objects as if no backup command was ever
- issued.
+In ``grouped`` completion mode, no jobs will report success until all jobs are
+ready to report success. If any job fails, all other jobs will be cancelled.
- - Regardless of if QEMU reports a particular incremental backup job
- as CANCELLED or as an ERROR, the in-memory bitmap will be
- restored.
+Regardless of if a participating incremental backup job failed or was
+cancelled, their associated bitmaps will all be held at their existing
+points-in-time, as in individual failure cases.
-Example
-^^^^^^^
+Here's the same multi-drive backup scenario from `Example: Partial
+Transactional Failures`_, but with the ``grouped`` completion-mode property
+applied:
-- Here's the same example scenario from above with the new property:
+#. Issue the multi-drive incremental backup transaction:
.. code:: json
- { "execute": "transaction",
+ -> {
+ "execute": "transaction",
"arguments": {
- "actions": [
- { "type": "drive-backup",
- "data": { "device": "drive0", "bitmap": "bitmap0",
- "format": "qcow2", "mode": "existing",
- "sync": "incremental", "target": "d0-incr-1.qcow2" } },
- { "type": "drive-backup",
- "data": { "device": "drive1", "bitmap": "bitmap1",
- "format": "qcow2", "mode": "existing",
- "sync": "incremental", "target": "d1-incr-1.qcow2" } },
- ],
"properties": {
"completion-mode": "grouped"
- }
+ },
+ "actions": [
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive0.inc0.qcow2"
+ }
+ },
+ {
+ "type": "drive-backup",
+ "data": {
+ "device": "drive1",
+ "bitmap": "bitmap0",
+ "format": "qcow2",
+ "mode": "existing",
+ "sync": "incremental",
+ "target": "drive1.inc0.qcow2"
+ }
+ }]
}
}
-- QMP example response, highlighting a failure for ``drive2``:
+#. Receive notice that the Transaction was accepted, and jobs were launched:
- - Acknowledgement that the Transaction was accepted and jobs were
- launched:
+ .. code:: json
+
+ <- { "return": {} }
- .. code:: json
+#. Receive notification that the backup job for ``drive1`` has failed:
- { "return": {} }
+ .. code:: json
- - Later, QEMU sends notice that the second job has errored out, but
- that the first job was also cancelled:
+ <- {
+ "timestamp": {...},
+ "data": {
+ "device": "drive1",
+ "action": "report",
+ "operation": "read"
+ },
+ "event": "BLOCK_JOB_ERROR"
+ }
- .. code:: json
+ <- {
+ "timestamp": {...},
+ "data": {
+ "speed": 0,
+ "offset": 0,
+ "len": 67108864,
+ "error": "Input/output error",
+ "device": "drive1",
+ "type": "backup"
+ },
+ "event": "BLOCK_JOB_COMPLETED"
+ }
- { "timestamp": { "seconds": 1447193702, "microseconds": 632377 },
- "data": { "device": "drive1", "action": "report",
- "operation": "read" },
- "event": "BLOCK_JOB_ERROR" }
+#. Receive notification that the job for ``drive0`` has been cancelled:
- .. code:: json
+ .. code:: json
- { "timestamp": { "seconds": 1447193702, "microseconds": 640074 },
- "data": { "speed": 0, "offset": 0, "len": 67108864,
- "error": "Input/output error",
- "device": "drive1", "type": "backup" },
- "event": "BLOCK_JOB_COMPLETED" }
+ <- {
+ "timestamp": {...}
+ "data": {
+ "device": "drive0",
+ "type": "backup",
+ "speed": 0,
+ "len": 67108864,
+ "offset": 16777216
+ },
+ "event": "BLOCK_JOB_CANCELLED"
+ }
- .. code:: json
+At the conclusion of *this* example, both jobs have been aborted due to a
+failure. Both destination images should be deleted and are no longer of use.
- { "timestamp": { "seconds": 1447193702, "microseconds": 640163 },
- "data": { "device": "drive0", "type": "backup", "speed": 0,
- "len": 67108864, "offset": 16777216 },
- "event": "BLOCK_JOB_CANCELLED" }
+The transaction as a whole can simply be re-issued at a later time.
.. raw:: html
<!--
The FreeBSD Documentation License
- Redistribution and use in source (Markdown) and 'compiled' forms (SGML, HTML,
- PDF, PostScript, RTF and so forth) with or without modification, are permitted
- provided that the following conditions are met:
-
- Redistributions of source code (Markdown) must retain the above copyright
- notice, this list of conditions and the following disclaimer of this file
- unmodified.
-
- Redistributions in compiled form (transformed to other DTDs, converted to PDF,
- PostScript, RTF and other formats) must reproduce the above copyright notice,
- this list of conditions and the following disclaimer in the documentation and/or
- other materials provided with the distribution.
-
- THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
- DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
- FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
- SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
- CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
- OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
- THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML,
+ PDF, PostScript, RTF and so forth) with or without modification, are
+ permitted provided that the following conditions are met:
+
+ Redistributions of source code (ReST) must retain the above copyright notice,
+ this list of conditions and the following disclaimer of this file unmodified.
+
+ Redistributions in compiled form (transformed to other DTDs, converted to
+ PDF, PostScript, RTF and other formats) must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+
+ THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+ IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF
+ THE POSSIBILITY OF SUCH DAMAGE.
-->
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 10/97] spapr/xive: fix EQ page addresses above 64GB
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (8 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 09/97] docs/interop/bitmaps: rewrite and modernize doc Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 11/97] kbd-state: fix autorepeat handling Michael Roth
` (91 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: David Gibson, qemu-stable, Cédric Le Goater
From: Cédric Le Goater <clg@kaod.org>
The high order bits of the address of the OS event queue is stored in
bits [4-31] of word2 of the XIVE END internal structures and the low
order bits in word3. This structure is using Big Endian ordering and
computing the value requires some simple arithmetic which happens to
be wrong. The mask removing bits [0-3] of word2 is applied to the
wrong value and the resulting address is bogus when above 64GB.
Guests with more than 64GB of RAM will allocate pages for the OS event
queues which will reside above the 64GB limit. In this case, the XIVE
device model will wake up the CPUs in case of a notification, such as
IPIs, but the update of the event queue will be written at the wrong
place in memory. The result is uncertain as the guest memory is
trashed and IPI are not delivered.
Introduce a helper xive_end_qaddr() to compute this value correctly in
all places where it is used.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20190508171946.657-3-clg@kaod.org>
Reviewed-by: Greg Kurz <groug@kaod.org>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(cherry picked from commit 13df93244efbd4bb8b4cf4e26104a26033178674)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/intc/spapr_xive.c | 3 +--
hw/intc/xive.c | 9 +++------
include/hw/ppc/xive_regs.h | 6 ++++++
3 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/hw/intc/spapr_xive.c b/hw/intc/spapr_xive.c
index 097f88d460..db75f5d608 100644
--- a/hw/intc/spapr_xive.c
+++ b/hw/intc/spapr_xive.c
@@ -1144,8 +1144,7 @@ static target_ulong h_int_get_queue_config(PowerPCCPU *cpu,
}
if (xive_end_is_enqueue(end)) {
- args[1] = (uint64_t) be32_to_cpu(end->w2 & 0x0fffffff) << 32
- | be32_to_cpu(end->w3);
+ args[1] = xive_end_qaddr(end);
args[2] = xive_get_field32(END_W0_QSIZE, end->w0) + 12;
} else {
args[1] = 0;
diff --git a/hw/intc/xive.c b/hw/intc/xive.c
index a0b87001da..dcf2fcd108 100644
--- a/hw/intc/xive.c
+++ b/hw/intc/xive.c
@@ -1042,8 +1042,7 @@ static const TypeInfo xive_source_info = {
void xive_end_queue_pic_print_info(XiveEND *end, uint32_t width, Monitor *mon)
{
- uint64_t qaddr_base = (uint64_t) be32_to_cpu(end->w2 & 0x0fffffff) << 32
- | be32_to_cpu(end->w3);
+ uint64_t qaddr_base = xive_end_qaddr(end);
uint32_t qsize = xive_get_field32(END_W0_QSIZE, end->w0);
uint32_t qindex = xive_get_field32(END_W1_PAGE_OFF, end->w1);
uint32_t qentries = 1 << (qsize + 10);
@@ -1072,8 +1071,7 @@ void xive_end_queue_pic_print_info(XiveEND *end, uint32_t width, Monitor *mon)
void xive_end_pic_print_info(XiveEND *end, uint32_t end_idx, Monitor *mon)
{
- uint64_t qaddr_base = (uint64_t) be32_to_cpu(end->w2 & 0x0fffffff) << 32
- | be32_to_cpu(end->w3);
+ uint64_t qaddr_base = xive_end_qaddr(end);
uint32_t qindex = xive_get_field32(END_W1_PAGE_OFF, end->w1);
uint32_t qgen = xive_get_field32(END_W1_GENERATION, end->w1);
uint32_t qsize = xive_get_field32(END_W0_QSIZE, end->w0);
@@ -1101,8 +1099,7 @@ void xive_end_pic_print_info(XiveEND *end, uint32_t end_idx, Monitor *mon)
static void xive_end_enqueue(XiveEND *end, uint32_t data)
{
- uint64_t qaddr_base = (uint64_t) be32_to_cpu(end->w2 & 0x0fffffff) << 32
- | be32_to_cpu(end->w3);
+ uint64_t qaddr_base = xive_end_qaddr(end);
uint32_t qsize = xive_get_field32(END_W0_QSIZE, end->w0);
uint32_t qindex = xive_get_field32(END_W1_PAGE_OFF, end->w1);
uint32_t qgen = xive_get_field32(END_W1_GENERATION, end->w1);
diff --git a/include/hw/ppc/xive_regs.h b/include/hw/ppc/xive_regs.h
index bf36678a24..1a8c5b5e64 100644
--- a/include/hw/ppc/xive_regs.h
+++ b/include/hw/ppc/xive_regs.h
@@ -208,6 +208,12 @@ typedef struct XiveEND {
#define xive_end_is_backlog(end) (be32_to_cpu((end)->w0) & END_W0_BACKLOG)
#define xive_end_is_escalate(end) (be32_to_cpu((end)->w0) & END_W0_ESCALATE_CTL)
+static inline uint64_t xive_end_qaddr(XiveEND *end)
+{
+ return ((uint64_t) be32_to_cpu(end->w2) & 0x0fffffff) << 32 |
+ be32_to_cpu(end->w3);
+}
+
/* Notification Virtual Target (NVT) */
typedef struct XiveNVT {
uint32_t w0;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 11/97] kbd-state: fix autorepeat handling
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (9 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 10/97] spapr/xive: fix EQ page addresses above 64GB Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 12/97] usb-tablet: fix serial compat property Michael Roth
` (90 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Gerd Hoffmann
From: Gerd Hoffmann <kraxel@redhat.com>
When allowing multiple down-events in a row (key autorepeat) we can't
use change_bit() any more to update the state, because autorepeat events
don't change the key state. We have to explicitly use set_bit() and
clear_bit() instead.
Cc: qemu-stable@nongnu.org
Fixes: 35921860156e kbd-state: don't block auto-repeat events
Buglink: https://bugs.launchpad.net/qemu/+bug/1828272
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Message-id: 20190514042443.10735-1-kraxel@redhat.com
(cherry picked from commit 5fff13f245cddd3bc260dfe6ebe1b1f05b72116f)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
ui/kbd-state.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/ui/kbd-state.c b/ui/kbd-state.c
index f3ab2d7a66..1668d17dda 100644
--- a/ui/kbd-state.c
+++ b/ui/kbd-state.c
@@ -59,7 +59,11 @@ void qkbd_state_key_event(QKbdState *kbd, QKeyCode qcode, bool down)
}
/* update key and modifier state */
- change_bit(qcode, kbd->keys);
+ if (down) {
+ set_bit(qcode, kbd->keys);
+ } else {
+ clear_bit(qcode, kbd->keys);
+ }
switch (qcode) {
case Q_KEY_CODE_SHIFT:
case Q_KEY_CODE_SHIFT_R:
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 12/97] usb-tablet: fix serial compat property
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (10 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 11/97] kbd-state: fix autorepeat handling Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 13/97] block/file-posix: Unaligned O_DIRECT block-status Michael Roth
` (89 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Gerd Hoffmann
From: Gerd Hoffmann <kraxel@redhat.com>
s/kbd/tablet/, fixes cut+paste bug.
Cc: qemu-stable@nongnu.org
Reported-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190520081805.15019-1-kraxel@redhat.com
(cherry picked from commit 442bac16a6cd708a9f87adb0a263f9d833f03ed5)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/core/machine.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 743fef2898..b6766632d9 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -33,7 +33,7 @@ GlobalProperty hw_compat_3_1[] = {
{ "tpm-tis", "ppi", "false" },
{ "usb-kbd", "serial", "42" },
{ "usb-mouse", "serial", "42" },
- { "usb-kbd", "serial", "42" },
+ { "usb-tablet", "serial", "42" },
{ "virtio-blk-device", "discard", "false" },
{ "virtio-blk-device", "write-zeroes", "false" },
};
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 13/97] block/file-posix: Unaligned O_DIRECT block-status
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (11 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 12/97] usb-tablet: fix serial compat property Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 14/97] iotests: Test unaligned raw images with O_DIRECT Michael Roth
` (88 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Currently, qemu crashes whenever someone queries the block status of an
unaligned image tail of an O_DIRECT image:
$ echo > foo
$ qemu-img map --image-opts driver=file,filename=foo,cache.direct=on
Offset Length Mapped to File
qemu-img: block/io.c:2093: bdrv_co_block_status: Assertion `*pnum &&
QEMU_IS_ALIGNED(*pnum, align) && align > offset - aligned_offset'
failed.
This is because bdrv_co_block_status() checks that the result returned
by the driver's implementation is aligned to the request_alignment, but
file-posix can fail to do so, which is actually mentioned in a comment
there: "[...] possibly including a partial sector at EOF".
Fix this by rounding up those partial sectors.
There are two possible alternative fixes:
(1) We could refuse to open unaligned image files with O_DIRECT
altogether. That sounds reasonable until you realize that qcow2
does necessarily not fill up its metadata clusters, and that nobody
runs qemu-img create with O_DIRECT. Therefore, unpreallocated qcow2
files usually have an unaligned image tail.
(2) bdrv_co_block_status() could ignore unaligned tails. It actually
throws away everything past the EOF already, so that sounds
reasonable.
Unfortunately, the block layer knows file lengths only with a
granularity of BDRV_SECTOR_SIZE, so bdrv_co_block_status() usually
would have to guess whether its file length information is inexact
or whether the driver is broken.
Fixing what raw_co_block_status() returns is the safest thing to do.
There seems to be no other block driver that sets request_alignment and
does not make sure that it always returns aligned values.
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 9c3db310ff0b7473272ae8dce5e04e2f8a825390)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/file-posix.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/block/file-posix.c b/block/file-posix.c
index 1cf4ee49eb..c185f34a2e 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -2475,6 +2475,8 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs,
off_t data = 0, hole = 0;
int ret;
+ assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
+
ret = fd_open(bs);
if (ret < 0) {
return ret;
@@ -2500,6 +2502,20 @@ static int coroutine_fn raw_co_block_status(BlockDriverState *bs,
/* On a data extent, compute bytes to the end of the extent,
* possibly including a partial sector at EOF. */
*pnum = MIN(bytes, hole - offset);
+
+ /*
+ * We are not allowed to return partial sectors, though, so
+ * round up if necessary.
+ */
+ if (!QEMU_IS_ALIGNED(*pnum, bs->bl.request_alignment)) {
+ int64_t file_length = raw_getlength(bs);
+ if (file_length > 0) {
+ /* Ignore errors, this is just a safeguard */
+ assert(hole == file_length);
+ }
+ *pnum = ROUND_UP(*pnum, bs->bl.request_alignment);
+ }
+
ret = BDRV_BLOCK_DATA;
} else {
/* On a hole, compute bytes to the beginning of the next extent. */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 14/97] iotests: Test unaligned raw images with O_DIRECT
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (12 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 13/97] block/file-posix: Unaligned O_DIRECT block-status Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 15/97] s390x/cpumodel: ignore csske for expansion Michael Roth
` (87 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
We already have 221 for accesses through the page cache, but it is
better to create a new file for O_DIRECT instead of integrating those
test cases into 221. This way, we can make use of
_supported_cache_modes (and _default_cache_mode) so the test is
automatically skipped on filesystems that do not support O_DIRECT.
As part of the split, add _supported_cache_modes to 221. With that, it
no longer fails when run with -c none or -c directsync.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 2fab30c80b33cdc6157c7efe6207e54b6835cf92)
Conflicts:
tests/qemu-iotests/group
*fix context deps on test groups not in 4.0
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/221 | 4 ++
tests/qemu-iotests/253 | 84 ++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/253.out | 14 +++++++
tests/qemu-iotests/group | 1 +
4 files changed, 103 insertions(+)
create mode 100755 tests/qemu-iotests/253
create mode 100644 tests/qemu-iotests/253.out
diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221
index 808cd9a289..92c9b13cd8 100755
--- a/tests/qemu-iotests/221
+++ b/tests/qemu-iotests/221
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
#
# Test qemu-img vs. unaligned images
+# (See also 253, which is the O_DIRECT version)
#
# Copyright (C) 2018 Red Hat, Inc.
#
@@ -37,6 +38,9 @@ _supported_fmt raw
_supported_proto file
_supported_os Linux
+_default_cache_mode writeback
+_supported_cache_modes writeback writethrough unsafe
+
echo
echo "=== Check mapping of unaligned raw image ==="
echo
diff --git a/tests/qemu-iotests/253 b/tests/qemu-iotests/253
new file mode 100755
index 0000000000..d88d5afa45
--- /dev/null
+++ b/tests/qemu-iotests/253
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img vs. unaligned images; O_DIRECT version
+# (Originates from 221)
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt raw
+_supported_proto file
+_supported_os Linux
+
+_default_cache_mode none
+_supported_cache_modes none directsync
+
+echo
+echo "=== Check mapping of unaligned raw image ==="
+echo
+
+# We do not know how large a physical sector is, but it is certainly
+# going to be a factor of 1 MB
+size=$((1 * 1024 * 1024 - 1))
+
+# qemu-img create rounds size up to BDRV_SECTOR_SIZE
+_make_test_img $size
+$QEMU_IMG map --output=json --image-opts \
+ "driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG,cache.direct=on" \
+ | _filter_qemu_img_map
+
+# so we resize it and check again
+truncate --size=$size "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+ "driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG,cache.direct=on" \
+ | _filter_qemu_img_map
+
+# qemu-io with O_DIRECT always writes whole physical sectors. Again,
+# we do not know how large a physical sector is, so we just start
+# writing from a 64 kB boundary, which should always be aligned.
+offset=$((1 * 1024 * 1024 - 64 * 1024))
+$QEMU_IO -c "w $offset $((size - offset))" "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG map --output=json --image-opts \
+ "driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG,cache.direct=on" \
+ | _filter_qemu_img_map
+
+# Resize it and check again -- contrary to 221, we may not get partial
+# sectors here, so there should be only two areas (one zero, one
+# data).
+truncate --size=$size "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+ "driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG,cache.direct=on" \
+ | _filter_qemu_img_map
+
+# success, all done
+echo '*** done'
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
new file mode 100644
index 0000000000..607c0baa0b
--- /dev/null
+++ b/tests/qemu-iotests/253.out
@@ -0,0 +1,14 @@
+QA output created by 253
+
+=== Check mapping of unaligned raw image ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+wrote 65535/65535 bytes at offset 983040
+63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index bae7718380..4cd2fe80a6 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -248,3 +248,4 @@
246 rw auto quick
247 rw auto quick
248 rw auto quick
+253 rw auto quick
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 15/97] s390x/cpumodel: ignore csske for expansion
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (13 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 14/97] iotests: Test unaligned raw images with O_DIRECT Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 16/97] blockdev-backup: don't check aio_context too early Michael Roth
` (86 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Christian Borntraeger, Cornelia Huck, qemu-stable
From: Christian Borntraeger <borntraeger@de.ibm.com>
csske will be removed in a future machine. Ignore it for expanding the
cpu model. Otherwise qemu falls back to z9.
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: qemu-stable@nongnu.org
Reviewed-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190429090250.7648-3-borntraeger@de.ibm.com>
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
(cherry picked from commit eaf6f642abf1d4d24791b70728d4068428fc4658)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/s390x/cpu_models.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index eb125d4d0d..e154d53eda 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -1327,6 +1327,8 @@ static void init_ignored_base_feat(void)
S390_FEAT_KM_TDEA_192,
S390_FEAT_KIMD_SHA_1,
S390_FEAT_KLMD_SHA_1,
+ /* CSSKE is deprecated on newer generations */
+ S390_FEAT_CONDITIONAL_SSKE,
};
int i;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 16/97] blockdev-backup: don't check aio_context too early
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (14 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 15/97] s390x/cpumodel: ignore csske for expansion Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 17/97] block: Drain source node in bdrv_replace_node() Michael Roth
` (85 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Max Reitz
From: John Snow <jsnow@redhat.com>
in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.
Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.
Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.
Reported-by: aihua liang <aliang@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-2-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit d81e1efbea7d19c2f142d300df56538c73800590)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
blockdev.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 4775a07d93..d358169995 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1871,10 +1871,6 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
}
aio_context = bdrv_get_aio_context(bs);
- if (aio_context != bdrv_get_aio_context(target)) {
- error_setg(errp, "Backup between two IO threads is not implemented");
- return;
- }
aio_context_acquire(aio_context);
state->bs = bs;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 17/97] block: Drain source node in bdrv_replace_node()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (15 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 16/97] blockdev-backup: don't check aio_context too early Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 18/97] iotests: Test commit job start with concurrent I/O Michael Roth
` (84 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
Instead of just asserting that no requests are in flight in
bdrv_replace_node(), which is a requirement that most callers ignore, we
can just drain the source node right there. This fixes at least starting
a commit job while I/O is active on the backing chain, but probably
other callers, too.
Having requests in flight on the target node isn't a problem because the
target just gets new parents, but the call path of running requests
isn't modified. So we can just drop this assertion without a replacement.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1711643
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit f871abd60f4b67547e62c57c9bec19420052be39)
*prereq for d81e1efb tests
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/block.c b/block.c
index 9ae5c0ed2f..c0e1c86911 100644
--- a/block.c
+++ b/block.c
@@ -3987,13 +3987,13 @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
uint64_t perm = 0, shared = BLK_PERM_ALL;
int ret;
- assert(!atomic_read(&from->in_flight));
- assert(!atomic_read(&to->in_flight));
-
/* Make sure that @from doesn't go away until we have successfully attached
* all of its parents to @to. */
bdrv_ref(from);
+ assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+ bdrv_drained_begin(from);
+
/* Put all parents into @list and calculate their cumulative permissions */
QLIST_FOREACH_SAFE(c, &from->parents, next_parent, next) {
assert(c->bs == from);
@@ -4034,6 +4034,7 @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
out:
g_slist_free(list);
+ bdrv_drained_end(from);
bdrv_unref(from);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 18/97] iotests: Test commit job start with concurrent I/O
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (16 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 17/97] block: Drain source node in bdrv_replace_node() Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 19/97] iotests.py: do not use infinite waits Michael Roth
` (83 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable
From: Kevin Wolf <kwolf@redhat.com>
This tests that concurrent requests are correctly drained before making
graph modifications instead of running into assertions in
bdrv_replace_node().
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit ac6fb43eae1f5029b51e0a3d975fe2111cc8b976)
Conflicts:
tests/qemu-iotests/group
*prereq for d81e1efb tests
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/255 | 83 +++++++++++++++++++++++++++++++++++
tests/qemu-iotests/255.out | 16 +++++++
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 10 ++++-
4 files changed, 109 insertions(+), 1 deletion(-)
create mode 100755 tests/qemu-iotests/255
create mode 100644 tests/qemu-iotests/255.out
diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
new file mode 100755
index 0000000000..c0bb37a9b0
--- /dev/null
+++ b/tests/qemu-iotests/255
@@ -0,0 +1,83 @@
+#!/usr/bin/env python
+#
+# Test commit job graph modifications while requests are active
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# Creator/Owner: Kevin Wolf <kwolf@redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import iotests
+from iotests import imgfmt
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+
+def blockdev_create(vm, options):
+ result = vm.qmp_log('blockdev-create',
+ filters=[iotests.filter_qmp_testfiles],
+ job_id='job0', options=options)
+
+ if 'return' in result:
+ assert result['return'] == {}
+ vm.run_job('job0')
+ iotests.log("")
+
+with iotests.FilePath('t.qcow2') as disk_path, \
+ iotests.FilePath('t.qcow2.mid') as mid_path, \
+ iotests.FilePath('t.qcow2.base') as base_path, \
+ iotests.VM() as vm:
+
+ iotests.log("=== Create backing chain and start VM ===")
+ iotests.log("")
+
+ size = 128 * 1024 * 1024
+ size_str = str(size)
+
+ iotests.create_image(base_path, size)
+ iotests.qemu_img_log('create', '-f', iotests.imgfmt, mid_path, size_str)
+ iotests.qemu_img_log('create', '-f', iotests.imgfmt, disk_path, size_str)
+
+ # Create a backing chain like this:
+ # base <- [throttled: bps-read=4096] <- mid <- overlay
+
+ vm.add_object('throttle-group,x-bps-read=4096,id=throttle0')
+ vm.add_blockdev('file,filename=%s,node-name=base' % (base_path))
+ vm.add_blockdev('throttle,throttle-group=throttle0,file=base,node-name=throttled')
+ vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid_path))
+ vm.add_blockdev('qcow2,file=mid-file,node-name=mid,backing=throttled')
+ vm.add_drive_raw('if=none,id=overlay,driver=qcow2,file=%s,backing=mid' % (disk_path))
+
+ vm.launch()
+
+ iotests.log("=== Start background read requests ===")
+ iotests.log("")
+
+ def start_requests():
+ vm.hmp_qemu_io('overlay', 'aio_read 0 4k')
+ vm.hmp_qemu_io('overlay', 'aio_read 0 4k')
+
+ start_requests()
+
+ iotests.log("=== Run a commit job ===")
+ iotests.log("")
+
+ result = vm.qmp_log('block-commit', job_id='job0', auto_finalize=False,
+ device='overlay', top_node='mid')
+
+ vm.run_job('job0', auto_finalize=False, pre_finalize=start_requests,
+ auto_dismiss=True)
+
+ vm.shutdown()
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
new file mode 100644
index 0000000000..9a2d7cbb77
--- /dev/null
+++ b/tests/qemu-iotests/255.out
@@ -0,0 +1,16 @@
+=== Create backing chain and start VM ===
+
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+
+=== Start background read requests ===
+
+=== Run a commit job ===
+
+{"execute": "block-commit", "arguments": {"auto-finalize": false, "device": "overlay", "job-id": "job0", "top-node": "mid"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "job0"}}
+{"return": {}}
+{"data": {"id": "job0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "job0", "len": 134217728, "offset": 134217728, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 4cd2fe80a6..19e2bb3aec 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -249,3 +249,4 @@
247 rw auto quick
248 rw auto quick
253 rw auto quick
+255 rw auto quick
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 997dc910cb..8b209ea0ee 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -126,6 +126,11 @@ def qemu_img_pipe(*args):
sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
return subp.communicate()[0]
+def qemu_img_log(*args):
+ result = qemu_img_pipe(*args)
+ log(result, filters=[filter_testfiles])
+ return result
+
def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]):
args = [ 'info' ]
if imgopts:
@@ -533,7 +538,8 @@ class VM(qtest.QEMUQtestMachine):
return result
# Returns None on success, and an error string on failure
- def run_job(self, job, auto_finalize=True, auto_dismiss=False):
+ def run_job(self, job, auto_finalize=True, auto_dismiss=False,
+ pre_finalize=None):
error = None
while True:
for ev in self.get_qmp_events_filtered(wait=True):
@@ -546,6 +552,8 @@ class VM(qtest.QEMUQtestMachine):
error = j['error']
log('Job failed: %s' % (j['error']))
elif status == 'pending' and not auto_finalize:
+ if pre_finalize:
+ pre_finalize()
self.qmp_log('job-finalize', id=job)
elif status == 'concluded' and not auto_dismiss:
self.qmp_log('job-dismiss', id=job)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 19/97] iotests.py: do not use infinite waits
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (17 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 18/97] iotests: Test commit job start with concurrent I/O Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:44 ` [PATCH 20/97] QEMUMachine: add events_wait method Michael Roth
` (82 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Max Reitz
From: John Snow <jsnow@redhat.com>
Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-3-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 8b6f5f8b9f3bec5cbeebefab34bae0102a2581b3)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/iotests.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 8b209ea0ee..8061f01ade 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -521,7 +521,7 @@ class VM(qtest.QEMUQtestMachine):
output_list += [key + '=' + obj[key]]
return ','.join(output_list)
- def get_qmp_events_filtered(self, wait=True):
+ def get_qmp_events_filtered(self, wait=60.0):
result = []
for ev in self.get_qmp_events(wait=wait):
result.append(filter_qmp_event(ev))
@@ -539,10 +539,10 @@ class VM(qtest.QEMUQtestMachine):
# Returns None on success, and an error string on failure
def run_job(self, job, auto_finalize=True, auto_dismiss=False,
- pre_finalize=None):
+ pre_finalize=None, wait=60.0):
error = None
while True:
- for ev in self.get_qmp_events_filtered(wait=True):
+ for ev in self.get_qmp_events_filtered(wait=wait):
if ev['event'] == 'JOB_STATUS_CHANGE':
status = ev['data']['status']
if status == 'aborting':
@@ -633,7 +633,7 @@ class QMPTestCase(unittest.TestCase):
self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
self.vm.flatten_qmp_object(reference))
- def cancel_and_wait(self, drive='drive0', force=False, resume=False):
+ def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
'''Cancel a block job and wait for it to finish, returning the event'''
result = self.vm.qmp('block-job-cancel', device=drive, force=force)
self.assert_qmp(result, 'return', {})
@@ -644,7 +644,7 @@ class QMPTestCase(unittest.TestCase):
cancelled = False
result = None
while not cancelled:
- for event in self.vm.get_qmp_events(wait=True):
+ for event in self.vm.get_qmp_events(wait=wait):
if event['event'] == 'BLOCK_JOB_COMPLETED' or \
event['event'] == 'BLOCK_JOB_CANCELLED':
self.assert_qmp(event, 'data/device', drive)
@@ -657,10 +657,10 @@ class QMPTestCase(unittest.TestCase):
self.assert_no_active_block_jobs()
return result
- def wait_until_completed(self, drive='drive0', check_offset=True):
+ def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
'''Wait for a block job to finish, returning the event'''
while True:
- for event in self.vm.get_qmp_events(wait=True):
+ for event in self.vm.get_qmp_events(wait=wait):
if event['event'] == 'BLOCK_JOB_COMPLETED':
self.assert_qmp(event, 'data/device', drive)
self.assert_qmp_absent(event, 'data/error')
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 20/97] QEMUMachine: add events_wait method
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (18 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 19/97] iotests.py: do not use infinite waits Michael Roth
@ 2019-10-01 23:44 ` Michael Roth
2019-10-01 23:45 ` [PATCH 21/97] iotests.py: Fix VM.run_job Michael Roth
` (81 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:44 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Max Reitz
From: John Snow <jsnow@redhat.com>
Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-4-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit f6f4b3f045ea18e3fa93a50cd0462236c428d62e)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
1 file changed, 49 insertions(+), 20 deletions(-)
diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index fd144c0006..4cb8eb2f01 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -408,42 +408,71 @@ class QEMUMachine(object):
self._qmp.clear_events()
return events
- def event_wait(self, name, timeout=60.0, match=None):
+ @staticmethod
+ def event_match(event, match=None):
"""
- Wait for specified timeout on named event in QMP; optionally filter
- results by match.
+ Check if an event matches optional match criteria.
- The 'match' is checked to be a recursive subset of the 'event'; skips
- branch processing on match's value None
- {"foo": {"bar": 1}} matches {"foo": None}
- {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
+ The match criteria takes the form of a matching subdict. The event is
+ checked to be a superset of the subdict, recursively, with matching
+ values whenever those values are not None.
+
+ Examples, with the subdict queries on the left:
+ - None matches any object.
+ - {"foo": None} matches {"foo": {"bar": 1}}
+ - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
+ - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
"""
- def event_match(event, match=None):
- if match is None:
- return True
+ if match is None:
+ return True
- for key in match:
- if key in event:
- if isinstance(event[key], dict):
- if not event_match(event[key], match[key]):
- return False
- elif event[key] != match[key]:
+ for key in match:
+ if key in event:
+ if isinstance(event[key], dict):
+ if not QEMUMachine.event_match(event[key], match[key]):
return False
- else:
+ elif event[key] != match[key]:
return False
+ else:
+ return False
+ return True
- return True
+ def event_wait(self, name, timeout=60.0, match=None):
+ """
+ event_wait waits for and returns a named event from QMP with a timeout.
+
+ name: The event to wait for.
+ timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+ match: Optional match criteria. See event_match for details.
+ """
+ return self.events_wait([(name, match)], timeout)
+
+ def events_wait(self, events, timeout=60.0):
+ """
+ events_wait waits for and returns a named event from QMP with a timeout.
+
+ events: a sequence of (name, match_criteria) tuples.
+ The match criteria are optional and may be None.
+ See event_match for details.
+ timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+ """
+ def _match(event):
+ for name, match in events:
+ if (event['event'] == name and
+ self.event_match(event, match)):
+ return True
+ return False
# Search cached events
for event in self._events:
- if (event['event'] == name) and event_match(event, match):
+ if _match(event):
self._events.remove(event)
return event
# Poll for new events
while True:
event = self._qmp.pull_event(wait=timeout)
- if (event['event'] == name) and event_match(event, match):
+ if _match(event):
return event
self._events.append(event)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 21/97] iotests.py: Fix VM.run_job
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (19 preceding siblings ...)
2019-10-01 23:44 ` [PATCH 20/97] QEMUMachine: add events_wait method Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 22/97] iotests.py: rewrite run_job to be pickier Michael Roth
` (80 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
log() is in the current module, there is no need to prefix it. In fact,
doing so may make VM.run_job() unusable in tests that never use
iotests.log() themselves.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 86a4f599a67b9b709109c7a7c8b7eb91d21c21fd)
*prereq for d6a79af0e6
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/iotests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 8061f01ade..fa00b61892 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -560,7 +560,7 @@ class VM(qtest.QEMUQtestMachine):
elif status == 'null':
return error
else:
- iotests.log(ev)
+ log(ev)
def node_info(self, node_name):
nodes = self.qmp('query-named-block-nodes')
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 22/97] iotests.py: rewrite run_job to be pickier
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (20 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 21/97] iotests.py: Fix VM.run_job Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 23/97] iotests: add iotest 256 for testing blockdev-backup across iothread contexts Michael Roth
` (79 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Max Reitz
From: John Snow <jsnow@redhat.com>
Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-5-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit d6a79af0e641806d6bd6a42a4920e294b5db179c)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 19 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fa00b61892..a7006662ff 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -540,27 +540,37 @@ class VM(qtest.QEMUQtestMachine):
# Returns None on success, and an error string on failure
def run_job(self, job, auto_finalize=True, auto_dismiss=False,
pre_finalize=None, wait=60.0):
+ match_device = {'data': {'device': job}}
+ match_id = {'data': {'id': job}}
+ events = [
+ ('BLOCK_JOB_COMPLETED', match_device),
+ ('BLOCK_JOB_CANCELLED', match_device),
+ ('BLOCK_JOB_ERROR', match_device),
+ ('BLOCK_JOB_READY', match_device),
+ ('BLOCK_JOB_PENDING', match_id),
+ ('JOB_STATUS_CHANGE', match_id)
+ ]
error = None
while True:
- for ev in self.get_qmp_events_filtered(wait=wait):
- if ev['event'] == 'JOB_STATUS_CHANGE':
- status = ev['data']['status']
- if status == 'aborting':
- result = self.qmp('query-jobs')
- for j in result['return']:
- if j['id'] == job:
- error = j['error']
- log('Job failed: %s' % (j['error']))
- elif status == 'pending' and not auto_finalize:
- if pre_finalize:
- pre_finalize()
- self.qmp_log('job-finalize', id=job)
- elif status == 'concluded' and not auto_dismiss:
- self.qmp_log('job-dismiss', id=job)
- elif status == 'null':
- return error
- else:
- log(ev)
+ ev = filter_qmp_event(self.events_wait(events))
+ if ev['event'] != 'JOB_STATUS_CHANGE':
+ log(ev)
+ continue
+ status = ev['data']['status']
+ if status == 'aborting':
+ result = self.qmp('query-jobs')
+ for j in result['return']:
+ if j['id'] == job:
+ error = j['error']
+ log('Job failed: %s' % (j['error']))
+ elif status == 'pending' and not auto_finalize:
+ if pre_finalize:
+ pre_finalize()
+ self.qmp_log('job-finalize', id=job)
+ elif status == 'concluded' and not auto_dismiss:
+ self.qmp_log('job-dismiss', id=job)
+ elif status == 'null':
+ return error
def node_info(self, node_name):
nodes = self.qmp('query-named-block-nodes')
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 23/97] iotests: add iotest 256 for testing blockdev-backup across iothread contexts
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (21 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 22/97] iotests.py: rewrite run_job to be pickier Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 24/97] migration/dirty-bitmaps: change bitmap enumeration method Michael Roth
` (78 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Max Reitz
From: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-6-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
[mreitz: Moved from 250 to 256]
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit ba7704f2228f16ed61b9903801e28e17666c7e38)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/256 | 122 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 242 insertions(+)
create mode 100755 tests/qemu-iotests/256
create mode 100644 tests/qemu-iotests/256.out
diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
new file mode 100755
index 0000000000..c594a43205
--- /dev/null
+++ b/tests/qemu-iotests/256
@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+#
+# Test incremental/backup across iothread contexts
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+import os
+import iotests
+from iotests import log
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+size = 64 * 1024 * 1024
+
+with iotests.FilePath('img0') as img0_path, \
+ iotests.FilePath('img1') as img1_path, \
+ iotests.FilePath('img0-full') as img0_full_path, \
+ iotests.FilePath('img1-full') as img1_full_path, \
+ iotests.FilePath('img0-incr') as img0_incr_path, \
+ iotests.FilePath('img1-incr') as img1_incr_path, \
+ iotests.VM() as vm:
+
+ def create_target(filepath, name, size):
+ basename = os.path.basename(filepath)
+ nodename = "file_{}".format(basename)
+ log(vm.command('blockdev-create', job_id='job1',
+ options={
+ 'driver': 'file',
+ 'filename': filepath,
+ 'size': 0,
+ }))
+ vm.run_job('job1')
+ log(vm.command('blockdev-add', driver='file',
+ node_name=nodename, filename=filepath))
+ log(vm.command('blockdev-create', job_id='job2',
+ options={
+ 'driver': iotests.imgfmt,
+ 'file': nodename,
+ 'size': size,
+ }))
+ vm.run_job('job2')
+ log(vm.command('blockdev-add', driver=iotests.imgfmt,
+ node_name=name,
+ file=nodename))
+
+ log('--- Preparing images & VM ---\n')
+ vm.add_object('iothread,id=iothread0')
+ vm.add_object('iothread,id=iothread1')
+ vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
+ vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
+ iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
+ iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+ vm.add_drive(img0_path, interface='none')
+ vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
+ vm.add_drive(img1_path, interface='none')
+ vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
+
+ log('--- Starting VM ---\n')
+ vm.launch()
+
+ log('--- Create Targets & Full Backups ---\n')
+ create_target(img0_full_path, 'img0-full', size)
+ create_target(img1_full_path, 'img1-full', size)
+ ret = vm.qmp_log('transaction', indent=2, actions=[
+ { 'type': 'block-dirty-bitmap-add',
+ 'data': { 'node': 'drive0', 'name': 'bitmap0' }},
+ { 'type': 'block-dirty-bitmap-add',
+ 'data': { 'node': 'drive1', 'name': 'bitmap1' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive0',
+ 'target': 'img0-full',
+ 'sync': 'full',
+ 'job-id': 'j0' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive1',
+ 'target': 'img1-full',
+ 'sync': 'full',
+ 'job-id': 'j1' }}
+ ])
+ if "error" in ret:
+ raise Exception(ret['error']['desc'])
+ vm.run_job('j0', auto_dismiss=True)
+ vm.run_job('j1', auto_dismiss=True)
+
+ log('\n--- Create Targets & Incremental Backups ---\n')
+ create_target(img0_incr_path, 'img0-incr', size)
+ create_target(img1_incr_path, 'img1-incr', size)
+ ret = vm.qmp_log('transaction', indent=2, actions=[
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive0',
+ 'target': 'img0-incr',
+ 'sync': 'incremental',
+ 'bitmap': 'bitmap0',
+ 'job-id': 'j2' }},
+ { 'type': 'blockdev-backup',
+ 'data': { 'device': 'drive1',
+ 'target': 'img1-incr',
+ 'sync': 'incremental',
+ 'bitmap': 'bitmap1',
+ 'job-id': 'j3' }}
+ ])
+ if "error" in ret:
+ raise Exception(ret['error']['desc'])
+ vm.run_job('j2', auto_dismiss=True)
+ vm.run_job('j3', auto_dismiss=True)
+
+ log('\n--- Done ---')
+ vm.shutdown()
diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
new file mode 100644
index 0000000000..eec38614ec
--- /dev/null
+++ b/tests/qemu-iotests/256.out
@@ -0,0 +1,119 @@
+--- Preparing images & VM ---
+
+--- Starting VM ---
+
+--- Create Targets & Full Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "data": {
+ "name": "bitmap0",
+ "node": "drive0"
+ },
+ "type": "block-dirty-bitmap-add"
+ },
+ {
+ "data": {
+ "name": "bitmap1",
+ "node": "drive1"
+ },
+ "type": "block-dirty-bitmap-add"
+ },
+ {
+ "data": {
+ "device": "drive0",
+ "job-id": "j0",
+ "sync": "full",
+ "target": "img0-full"
+ },
+ "type": "blockdev-backup"
+ },
+ {
+ "data": {
+ "device": "drive1",
+ "job-id": "j1",
+ "sync": "full",
+ "target": "img1-full"
+ },
+ "type": "blockdev-backup"
+ }
+ ]
+ }
+}
+{
+ "return": {}
+}
+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Create Targets & Incremental Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {
+ "data": {
+ "bitmap": "bitmap0",
+ "device": "drive0",
+ "job-id": "j2",
+ "sync": "incremental",
+ "target": "img0-incr"
+ },
+ "type": "blockdev-backup"
+ },
+ {
+ "data": {
+ "bitmap": "bitmap1",
+ "device": "drive1",
+ "job-id": "j3",
+ "sync": "incremental",
+ "target": "img1-incr"
+ },
+ "type": "blockdev-backup"
+ }
+ ]
+ }
+}
+{
+ "return": {}
+}
+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Done ---
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 19e2bb3aec..7aa6dc9875 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -250,3 +250,4 @@
248 rw auto quick
253 rw auto quick
255 rw auto quick
+256 rw auto quick
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 24/97] migration/dirty-bitmaps: change bitmap enumeration method
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (22 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 23/97] iotests: add iotest 256 for testing blockdev-backup across iothread contexts Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 25/97] vhost: fix vhost_log size overflow during migration Michael Roth
` (77 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Stefan Hajnoczi
From: John Snow <jsnow@redhat.com>
Shift from looking at every root BDS to *every* BDS. This will migrate
bitmaps that are attached to blockdev created nodes instead of just ones
attached to emulated storage devices.
Note that this will not migrate anonymous or internal-use bitmaps, as
those are defined as having no name.
This will also fix the Coverity issues Peter Maydell has been asking
about for the past several releases, as well as fixing a real bug.
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Reported-by: Coverity 😅
Reported-by: aihua liang <aliang@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190514201926.10407-1-jsnow@redhat.com
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1652490
Fixes: Coverity CID 1390625
CC: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 592203e7cfbd1ad261178431fcf390adfe8b16df)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
migration/block-dirty-bitmap.c | 14 ++++----------
1 file changed, 4 insertions(+), 10 deletions(-)
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index d1bb863cb6..4a896a09eb 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -273,7 +273,6 @@ static int init_dirty_bitmap_migration(void)
BlockDriverState *bs;
BdrvDirtyBitmap *bitmap;
DirtyBitmapMigBitmapState *dbms;
- BdrvNextIterator it;
Error *local_err = NULL;
dirty_bitmap_mig_state.bulk_completed = false;
@@ -281,13 +280,8 @@ static int init_dirty_bitmap_migration(void)
dirty_bitmap_mig_state.prev_bitmap = NULL;
dirty_bitmap_mig_state.no_bitmaps = false;
- for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
- const char *drive_name = bdrv_get_device_or_node_name(bs);
-
- /* skip automatically inserted nodes */
- while (bs && bs->drv && bs->implicit) {
- bs = backing_bs(bs);
- }
+ for (bs = bdrv_next_all_states(NULL); bs; bs = bdrv_next_all_states(bs)) {
+ const char *name = bdrv_get_device_or_node_name(bs);
for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap;
bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
@@ -296,7 +290,7 @@ static int init_dirty_bitmap_migration(void)
continue;
}
- if (drive_name == NULL) {
+ if (!name || strcmp(name, "") == 0) {
error_report("Found bitmap '%s' in unnamed node %p. It can't "
"be migrated", bdrv_dirty_bitmap_name(bitmap), bs);
goto fail;
@@ -313,7 +307,7 @@ static int init_dirty_bitmap_migration(void)
dbms = g_new0(DirtyBitmapMigBitmapState, 1);
dbms->bs = bs;
- dbms->node_name = drive_name;
+ dbms->node_name = name;
dbms->bitmap = bitmap;
dbms->total_sectors = bdrv_nb_sectors(bs);
dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 25/97] vhost: fix vhost_log size overflow during migration
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (23 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 24/97] migration/dirty-bitmaps: change bitmap enumeration method Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 26/97] target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p Michael Roth
` (76 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Li Hangjing, qemu-stable, Michael S . Tsirkin
From: Li Hangjing <lihangjing@baidu.com>
When a guest which doesn't support multiqueue is migrated with a multi queues
vhost-user-blk deivce, a crash will occur like:
0 qemu_memfd_alloc (name=<value optimized out>, size=562949953421312, seals=<value optimized out>, fd=0x7f87171fe8b4, errp=0x7f87171fe8a8) at util/memfd.c:153
1 0x00007f883559d7cf in vhost_log_alloc (size=70368744177664, share=true) at hw/virtio/vhost.c:186
2 0x00007f88355a0758 in vhost_log_get (listener=0x7f8838bd7940, enable=1) at qemu-2-12/hw/virtio/vhost.c:211
3 vhost_dev_log_resize (listener=0x7f8838bd7940, enable=1) at hw/virtio/vhost.c:263
4 vhost_migration_log (listener=0x7f8838bd7940, enable=1) at hw/virtio/vhost.c:787
5 0x00007f88355463d6 in memory_global_dirty_log_start () at memory.c:2503
6 0x00007f8835550577 in ram_init_bitmaps (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2173
7 ram_init_all (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2192
8 ram_save_setup (f=0x7f88384ce600, opaque=0x7f8836024098) at migration/ram.c:2219
9 0x00007f88357a419d in qemu_savevm_state_setup (f=0x7f88384ce600) at migration/savevm.c:1002
10 0x00007f883579fc3e in migration_thread (opaque=0x7f8837530400) at migration/migration.c:2382
11 0x00007f8832447893 in start_thread () from /lib64/libpthread.so.0
12 0x00007f8832178bfd in clone () from /lib64/libc.so.6
This is because vhost_get_log_size() returns a overflowed vhost-log size.
In this function, it uses the uninitialized variable vqs->used_phys and
vqs->used_size to get the vhost-log size.
Signed-off-by: Li Hangjing <lihangjing@baidu.com>
Reviewed-by: Xie Yongji <xieyongji@baidu.com>
Reviewed-by: Chai Wen <chaiwen@baidu.com>
Message-Id: <20190603061524.24076-1-lihangjing@baidu.com>
Cc: qemu-stable@nongnu.org
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 240e647a14df9677b3a501f7b8b870e40aac3fd5)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/vhost.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 7f61018f2a..6d3a013f49 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -131,6 +131,11 @@ static int vhost_sync_dirty_bitmap(struct vhost_dev *dev,
}
for (i = 0; i < dev->nvqs; ++i) {
struct vhost_virtqueue *vq = dev->vqs + i;
+
+ if (!vq->used_phys && !vq->used_size) {
+ continue;
+ }
+
vhost_dev_sync_region(dev, section, start_addr, end_addr, vq->used_phys,
range_get_last(vq->used_phys, vq->used_size));
}
@@ -168,6 +173,11 @@ static uint64_t vhost_get_log_size(struct vhost_dev *dev)
}
for (i = 0; i < dev->nvqs; ++i) {
struct vhost_virtqueue *vq = dev->vqs + i;
+
+ if (!vq->used_phys && !vq->used_size) {
+ continue;
+ }
+
uint64_t last = vq->used_phys + vq->used_size - 1;
log_size = MAX(log_size, last / VHOST_LOG_CHUNK + 1);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 26/97] target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (24 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 25/97] vhost: fix vhost_log size overflow during migration Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 27/97] target/ppc: Fix xvxsigdp Michael Roth
` (75 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anton Blanchard, qemu-stable, David Gibson
From: Anton Blanchard <anton@ozlabs.org>
We were using set_cpu_vsr*() when we should have used get_cpu_vsr*().
Fixes: 8b3b2d75c7c0 ("introduce get_cpu_vsr{l,h}() and set_cpu_vsr{l,h}() helpers for VSR register access")
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Message-Id: <20190509104912.6b754dff@kryten>
Reviewed-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(upstream commit 77bd8937c03dd55e57cc257951ad07c185303c3e)
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/ppc/translate/vsx-impl.inc.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/target/ppc/translate/vsx-impl.inc.c b/target/ppc/translate/vsx-impl.inc.c
index 489b2436e4..1b483fc27e 100644
--- a/target/ppc/translate/vsx-impl.inc.c
+++ b/target/ppc/translate/vsx-impl.inc.c
@@ -858,8 +858,8 @@ static void glue(gen_, name)(DisasContext * ctx) \
xbh = tcg_temp_new_i64(); \
xbl = tcg_temp_new_i64(); \
sgm = tcg_temp_new_i64(); \
- set_cpu_vsrh(xB(ctx->opcode), xbh); \
- set_cpu_vsrl(xB(ctx->opcode), xbl); \
+ get_cpu_vsrh(xbh, xB(ctx->opcode)); \
+ get_cpu_vsrl(xbl, xB(ctx->opcode)); \
tcg_gen_movi_i64(sgm, sgn_mask); \
switch (op) { \
case OP_ABS: { \
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 27/97] target/ppc: Fix xvxsigdp
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (25 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 26/97] target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 28/97] target/ppc: Fix xxbrq, xxbrw Michael Roth
` (74 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anton Blanchard, qemu-stable, David Gibson
From: Anton Blanchard <anton@ozlabs.org>
Fix a typo in xvxsigdp where we put both results into the lower
doubleword.
Fixes: dd977e4f45cb ("target/ppc: Optimize x[sv]xsigdp using deposit_i64()")
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Message-Id: <20190507004811.29968-1-anton@ozlabs.org>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(upstream commit cf4e9363f7fd889d8d804c8f78e8927782c2aa48)
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/ppc/translate/vsx-impl.inc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/target/ppc/translate/vsx-impl.inc.c b/target/ppc/translate/vsx-impl.inc.c
index 1b483fc27e..96e4ff4411 100644
--- a/target/ppc/translate/vsx-impl.inc.c
+++ b/target/ppc/translate/vsx-impl.inc.c
@@ -1819,7 +1819,7 @@ static void gen_xvxsigdp(DisasContext *ctx)
tcg_gen_movi_i64(t0, 0x0010000000000000);
tcg_gen_movcond_i64(TCG_COND_EQ, t0, exp, zr, zr, t0);
tcg_gen_movcond_i64(TCG_COND_EQ, t0, exp, nan, zr, t0);
- tcg_gen_deposit_i64(xth, t0, xbl, 0, 52);
+ tcg_gen_deposit_i64(xtl, t0, xbl, 0, 52);
set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_temp_free_i64(t0);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 28/97] target/ppc: Fix xxbrq, xxbrw
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (26 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 27/97] target/ppc: Fix xvxsigdp Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 29/97] target/ppc: Fix vsum2sws Michael Roth
` (73 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anton Blanchard, qemu-stable, David Gibson
From: Anton Blanchard <anton@ozlabs.org>
Fix a typo in xxbrq and xxbrw where we put both results into the lower
doubleword.
Fixes: 8b3b2d75c7c0 ("introduce get_cpu_vsr{l,h}() and set_cpu_vsr{l,h}() helpers for VSR register access")
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Message-Id: <20190507004811.29968-3-anton@ozlabs.org>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(upstream commit d47a751adab7833e9831408376077bc8dba41d5d)
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/ppc/translate/vsx-impl.inc.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/target/ppc/translate/vsx-impl.inc.c b/target/ppc/translate/vsx-impl.inc.c
index 96e4ff4411..7778d5d651 100644
--- a/target/ppc/translate/vsx-impl.inc.c
+++ b/target/ppc/translate/vsx-impl.inc.c
@@ -1192,7 +1192,7 @@ static void gen_xxbrq(DisasContext *ctx)
tcg_gen_bswap64_i64(xtl, xbh);
set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_gen_mov_i64(xth, t0);
- set_cpu_vsrl(xT(ctx->opcode), xth);
+ set_cpu_vsrh(xT(ctx->opcode), xth);
tcg_temp_free_i64(t0);
tcg_temp_free_i64(xth);
@@ -1220,7 +1220,7 @@ static void gen_xxbrw(DisasContext *ctx)
get_cpu_vsrl(xbl, xB(ctx->opcode));
gen_bswap32x4(xth, xtl, xbh, xbl);
- set_cpu_vsrl(xT(ctx->opcode), xth);
+ set_cpu_vsrh(xT(ctx->opcode), xth);
set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_temp_free_i64(xth);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 29/97] target/ppc: Fix vsum2sws
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (27 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 28/97] target/ppc: Fix xxbrq, xxbrw Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 30/97] target/ppc: Fix lxvw4x, lxvh8x and lxvb16x Michael Roth
` (72 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anton Blanchard, qemu-stable, David Gibson
From: Anton Blanchard <anton@ozlabs.org>
A recent cleanup changed the pre zeroing of the result from 64 bit
to 32 bit operations:
- result.u64[i] = 0;
+ result.VsrW(i) = 0;
This corrupts the result.
Fixes: 60594fea298d ("target/ppc: remove various HOST_WORDS_BIGENDIAN hacks in int_helper.c")
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Message-Id: <20190507004811.29968-9-anton@ozlabs.org>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(upstream commit 7fa0ddc1d63806769d1b6246a62708d3bde39037)
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/ppc/int_helper.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/target/ppc/int_helper.c b/target/ppc/int_helper.c
index 162add561e..6bd1d32b1d 100644
--- a/target/ppc/int_helper.c
+++ b/target/ppc/int_helper.c
@@ -2030,7 +2030,7 @@ void helper_vsum2sws(CPUPPCState *env, ppc_avr_t *r, ppc_avr_t *a, ppc_avr_t *b)
for (i = 0; i < ARRAY_SIZE(r->u64); i++) {
int64_t t = (int64_t)b->VsrSW(upper + i * 2);
- result.VsrW(i) = 0;
+ result.VsrD(i) = 0;
for (j = 0; j < ARRAY_SIZE(r->u64); j++) {
t += a->VsrSW(2 * i + j);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 30/97] target/ppc: Fix lxvw4x, lxvh8x and lxvb16x
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (28 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 29/97] target/ppc: Fix vsum2sws Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 31/97] q35: Revert to kernel irqchip Michael Roth
` (71 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anton Blanchard, qemu-stable, David Gibson
From: Anton Blanchard <anton@ozlabs.org>
During the conversion these instructions were incorrectly treated as
stores. We need to use set_cpu_vsr* and not get_cpu_vsr*.
Fixes: 8b3b2d75c7c0 ("introduce get_cpu_vsr{l,h}() and set_cpu_vsr{l,h}() helpers for VSR register access")
Signed-off-by: Anton Blanchard <anton@ozlabs.org>
Reviewed-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Tested-by: Greg Kurz <groug@kaod.org>
Reviewed-by: Greg Kurz <groug@kaod.org>
Message-Id: <20190524065345.25591-1-mark.cave-ayland@ilande.co.uk>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
(upstream commit 2a1224359008e23b051b7b45be4789afa0269f8c)
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/ppc/translate/vsx-impl.inc.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/target/ppc/translate/vsx-impl.inc.c b/target/ppc/translate/vsx-impl.inc.c
index 7778d5d651..9e59e2a309 100644
--- a/target/ppc/translate/vsx-impl.inc.c
+++ b/target/ppc/translate/vsx-impl.inc.c
@@ -102,8 +102,7 @@ static void gen_lxvw4x(DisasContext *ctx)
}
xth = tcg_temp_new_i64();
xtl = tcg_temp_new_i64();
- get_cpu_vsrh(xth, xT(ctx->opcode));
- get_cpu_vsrl(xtl, xT(ctx->opcode));
+
gen_set_access_type(ctx, ACCESS_INT);
EA = tcg_temp_new();
@@ -126,6 +125,8 @@ static void gen_lxvw4x(DisasContext *ctx)
tcg_gen_addi_tl(EA, EA, 8);
tcg_gen_qemu_ld_i64(xtl, EA, ctx->mem_idx, MO_BEQ);
}
+ set_cpu_vsrh(xT(ctx->opcode), xth);
+ set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_temp_free(EA);
tcg_temp_free_i64(xth);
tcg_temp_free_i64(xtl);
@@ -185,8 +186,6 @@ static void gen_lxvh8x(DisasContext *ctx)
}
xth = tcg_temp_new_i64();
xtl = tcg_temp_new_i64();
- get_cpu_vsrh(xth, xT(ctx->opcode));
- get_cpu_vsrl(xtl, xT(ctx->opcode));
gen_set_access_type(ctx, ACCESS_INT);
EA = tcg_temp_new();
@@ -197,6 +196,8 @@ static void gen_lxvh8x(DisasContext *ctx)
if (ctx->le_mode) {
gen_bswap16x8(xth, xtl, xth, xtl);
}
+ set_cpu_vsrh(xT(ctx->opcode), xth);
+ set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_temp_free(EA);
tcg_temp_free_i64(xth);
tcg_temp_free_i64(xtl);
@@ -214,14 +215,14 @@ static void gen_lxvb16x(DisasContext *ctx)
}
xth = tcg_temp_new_i64();
xtl = tcg_temp_new_i64();
- get_cpu_vsrh(xth, xT(ctx->opcode));
- get_cpu_vsrl(xtl, xT(ctx->opcode));
gen_set_access_type(ctx, ACCESS_INT);
EA = tcg_temp_new();
gen_addr_reg_index(ctx, EA);
tcg_gen_qemu_ld_i64(xth, EA, ctx->mem_idx, MO_BEQ);
tcg_gen_addi_tl(EA, EA, 8);
tcg_gen_qemu_ld_i64(xtl, EA, ctx->mem_idx, MO_BEQ);
+ set_cpu_vsrh(xT(ctx->opcode), xth);
+ set_cpu_vsrl(xT(ctx->opcode), xtl);
tcg_temp_free(EA);
tcg_temp_free_i64(xth);
tcg_temp_free_i64(xtl);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 31/97] q35: Revert to kernel irqchip
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (29 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 30/97] target/ppc: Fix lxvw4x, lxvh8x and lxvb16x Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 32/97] vl: Fix -drive / -blockdev persistent reservation management Michael Roth
` (70 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Alex Williamson, qemu-stable
From: Alex Williamson <alex.williamson@redhat.com>
Backport of QEMU v4.1 commit for stable v4.0.1 release
commit c87759ce876a7a0b17c2bf4f0b964bd51f0ee871
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue May 14 14:14:41 2019 -0600
q35: Revert to kernel irqchip
Commit b2fc91db8447 ("q35: set split kernel irqchip as default") changed
the default for the pc-q35-4.0 machine type to use split irqchip, which
turned out to have disasterous effects on vfio-pci INTx support. KVM
resampling irqfds are registered for handling these interrupts, but
these are non-functional in split irqchip mode. We can't simply test
for split irqchip in QEMU as userspace handling of this interrupt is a
significant performance regression versus KVM handling (GeForce GPUs
assigned to Windows VMs are non-functional without forcing MSI mode or
re-enabling kernel irqchip).
The resolution is to revert the change in default irqchip mode in the
pc-q35-4.1 machine and create a pc-q35-4.0.1 machine for the 4.0-stable
branch. The qemu-q35-4.0 machine type should not be used in vfio-pci
configurations for devices requiring legacy INTx support without
explicitly modifying the VM configuration to use kernel irqchip.
Link: https://bugs.launchpad.net/qemu/+bug/1826422
Fixes: b2fc91db8447 ("q35: set split kernel irqchip as default")
Cc: qemu-stable@nongnu.org
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
(upstream commit c87759ce876a7a0b17c2bf4f0b964bd51f0ee871)
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
*add comments regarding AML mismatch warnings from
tests/bios-tables-test.c
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/core/machine.c | 3 +++
hw/i386/pc.c | 3 +++
hw/i386/pc_q35.c | 16 ++++++++++++++--
include/hw/boards.h | 3 +++
include/hw/i386/pc.h | 3 +++
tests/bios-tables-test.c | 9 +++++++++
6 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index b6766632d9..eb34f53a85 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -24,6 +24,9 @@
#include "hw/pci/pci.h"
#include "hw/mem/nvdimm.h"
+GlobalProperty hw_compat_4_0[] = {};
+const size_t hw_compat_4_0_len = G_N_ELEMENTS(hw_compat_4_0);
+
GlobalProperty hw_compat_3_1[] = {
{ "pcie-root-port", "x-speed", "2_5" },
{ "pcie-root-port", "x-width", "1" },
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index f2c15bf1f2..d98b737b8f 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -115,6 +115,9 @@ struct hpet_fw_config hpet_cfg = {.count = UINT8_MAX};
/* Physical Address of PVH entry point read from kernel ELF NOTE */
static size_t pvh_start_addr;
+GlobalProperty pc_compat_4_0[] = {};
+const size_t pc_compat_4_0_len = G_N_ELEMENTS(pc_compat_4_0);
+
GlobalProperty pc_compat_3_1[] = {
{ "intel-iommu", "dma-drain", "off" },
{ "Opteron_G3" "-" TYPE_X86_CPU, "rdtscp", "off" },
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 372c6b73be..45cc29d1ad 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
m->units_per_default_bus = 1;
m->default_machine_opts = "firmware=bios-256k.bin";
m->default_display = "std";
- m->default_kernel_irqchip_split = true;
+ m->default_kernel_irqchip_split = false;
m->no_floppy = 1;
machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
@@ -365,12 +365,24 @@ static void pc_q35_machine_options(MachineClass *m)
m->max_cpus = 288;
}
-static void pc_q35_4_0_machine_options(MachineClass *m)
+static void pc_q35_4_0_1_machine_options(MachineClass *m)
{
pc_q35_machine_options(m);
m->alias = "q35";
}
+DEFINE_Q35_MACHINE(v4_0_1, "pc-q35-4.0.1", NULL,
+ pc_q35_4_0_1_machine_options);
+
+static void pc_q35_4_0_machine_options(MachineClass *m)
+{
+ pc_q35_4_0_1_machine_options(m);
+ m->default_kernel_irqchip_split = true;
+ m->alias = NULL;
+ compat_props_add(m->compat_props, hw_compat_4_0, hw_compat_4_0_len);
+ compat_props_add(m->compat_props, pc_compat_4_0, pc_compat_4_0_len);
+}
+
DEFINE_Q35_MACHINE(v4_0, "pc-q35-4.0", NULL,
pc_q35_4_0_machine_options);
diff --git a/include/hw/boards.h b/include/hw/boards.h
index e231860666..fe1885cbff 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -293,6 +293,9 @@ struct MachineState {
} \
type_init(machine_initfn##_register_types)
+extern GlobalProperty hw_compat_4_0[];
+extern const size_t hw_compat_4_0_len;
+
extern GlobalProperty hw_compat_3_1[];
extern const size_t hw_compat_3_1_len;
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index ca65ef18af..43df7230a2 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -293,6 +293,9 @@ int e820_add_entry(uint64_t, uint64_t, uint32_t);
int e820_get_num_entries(void);
bool e820_get_entry(int, uint32_t, uint64_t *, uint64_t *);
+extern GlobalProperty pc_compat_4_0[];
+extern const size_t pc_compat_4_0_len;
+
extern GlobalProperty pc_compat_3_1[];
extern const size_t pc_compat_3_1_len;
diff --git a/tests/bios-tables-test.c b/tests/bios-tables-test.c
index a506dcbb29..996599991e 100644
--- a/tests/bios-tables-test.c
+++ b/tests/bios-tables-test.c
@@ -354,6 +354,15 @@ static void test_acpi_asl(test_data *data)
} else {
sdt->tmp_files_retain = true;
exp_sdt->tmp_files_retain = true;
+ /* NOTE: In QEMU 4.0.1, q35 aliases "pc-q35-4.0.1" instead of
+ * "pc-q35-4.0". This gets encoded into the smbios tables and
+ * due to the extra characters changes the offset of ACPI
+ * pointer values allocated by SeaBIOS, causing some tests to
+ * report AML mismatches. Since ACPI expected file paths rely
+ * on the "q35" alias being used we can't easily change the
+ * tests to use pc-q35-4.0, so for now hopefully this friendly
+ * note will suffice.
+ */
fprintf(stderr,
"acpi-test: Warning! %.4s mismatch. "
"Actual [asl:%s, aml:%s], Expected [asl:%s, aml:%s].\n",
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 32/97] vl: Fix -drive / -blockdev persistent reservation management
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (30 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 31/97] q35: Revert to kernel irqchip Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 33/97] target/i386: add MDS-NO feature Michael Roth
` (69 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, qemu-stable, Markus Armbruster
From: Markus Armbruster <armbru@redhat.com>
qemu-system-FOO's main() acts on command line arguments in its own
idiosyncratic order. There's not much method to its madness.
Whenever we find a case where one kind of command line argument needs
to refer to something created for another kind later, we rejigger the
order.
Recent commit cda4aa9a5a "vl: Create block backends before setting
machine properties" was such a rejigger. Block backends are now
created before "delayed" objects. This broke persistent reservation
management. Reproducer:
$ qemu-system-x86_64 -object pr-manager-helper,id=pr-helper0,path=/tmp/pr-helper0.sock-drive -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-0-2
qemu-system-x86_64: -drive file=/dev/mapper/crypt,file.pr-manager=pr-helper0,format=raw,if=none,id=drive-scsi0-0-0-2: No persistent reservation manager with id 'pr-helper0'
The delayed pr-manager-helper object is created too late for use by
-drive or -blockdev. Normal objects are still created in time.
pr-manager-helper has always been a delayed object (commit 7c9e527659
"scsi, file-posix: add support for persistent reservation
management"). Turns out there's no real reason for that. Make it a
normal object.
Fixes: cda4aa9a5a08777cf13e164c0543bd4888b8adce
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190604151251.9903-2-armbru@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
(cherry picked from commit 9ea18ed25a36527167e9676f25d983df5e7f76e6)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
vl.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/vl.c b/vl.c
index c696ad2a13..c3dee09c9b 100644
--- a/vl.c
+++ b/vl.c
@@ -2825,8 +2825,7 @@ static bool object_create_initial(const char *type, QemuOpts *opts)
exit(0);
}
- if (g_str_equal(type, "rng-egd") ||
- g_str_has_prefix(type, "pr-manager-")) {
+ if (g_str_equal(type, "rng-egd")) {
return false;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 33/97] target/i386: add MDS-NO feature
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (31 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 32/97] vl: Fix -drive / -blockdev persistent reservation management Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 34/97] target/i386: define md-clear bit Michael Roth
` (68 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, Oguz Bektas, qemu-stable, Eduardo Habkost
From: Paolo Bonzini <pbonzini@redhat.com>
Microarchitectural Data Sampling is a hardware vulnerability which allows
unprivileged speculative access to data which is available in various CPU
internal buffers.
Some Intel processors use the ARCH_CAP_MDS_NO bit in the
IA32_ARCH_CAPABILITIES
MSR to report that they are not vulnerable, make it available to guests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20190516185320.28340-1-pbonzini@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
(cherry picked from commit 20140a82c67467f53814ca197403d5e1b561a5e5)
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/i386/cpu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index d6bb57d210..ee4b8b47e2 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1183,7 +1183,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
.type = MSR_FEATURE_WORD,
.feat_names = {
"rdctl-no", "ibrs-all", "rsba", "skip-l1dfl-vmentry",
- "ssb-no", NULL, NULL, NULL,
+ "ssb-no", "mds-no", NULL, NULL,
NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 34/97] target/i386: define md-clear bit
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (32 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 33/97] target/i386: add MDS-NO feature Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 35/97] docs: recommend use of md-clear feature on all Intel CPUs Michael Roth
` (67 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, Oguz Bektas, qemu-stable, Eduardo Habkost
From: Paolo Bonzini <pbonzini@redhat.com>
md-clear is a new CPUID bit which is set when microcode provides the
mechanism to invoke a flush of various exploitable CPU buffers by invoking
the VERW instruction.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20190515141011.5315-2-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
(cherry picked from commit b2ae52101fca7f9547ac2f388085dbc58f8fe1c0)
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/i386/cpu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index ee4b8b47e2..331a364a1b 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1076,7 +1076,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
.feat_names = {
NULL, NULL, "avx512-4vnniw", "avx512-4fmaps",
NULL, NULL, NULL, NULL,
- NULL, NULL, NULL, NULL,
+ NULL, NULL, "md-clear", NULL,
NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 35/97] docs: recommend use of md-clear feature on all Intel CPUs
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (33 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 34/97] target/i386: define md-clear bit Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 36/97] virtio-pci: fix missing device properties Michael Roth
` (66 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel
Cc: Oguz Bektas, Daniel P. Berrangé, qemu-stable, Eduardo Habkost
From: Daniel P. Berrangé <berrange@redhat.com>
Update x86 CPU model guidance to recommend that the md-clear feature is
manually enabled with all Intel CPU models, when supported by the host
microcode.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190515141011.5315-3-berrange@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
(cherry picked from commit 2c7e82a30774730100da9dbe68d2360459030d91)
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
docs/qemu-cpu-models.texi | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/docs/qemu-cpu-models.texi b/docs/qemu-cpu-models.texi
index 23c11dc86f..ad040cfc98 100644
--- a/docs/qemu-cpu-models.texi
+++ b/docs/qemu-cpu-models.texi
@@ -200,6 +200,18 @@ Not included by default in any Intel CPU model.
Should be explicitly turned on for all Intel CPU models.
Note that not all CPU hardware will support this feature.
+
+@item @code{md-clear}
+
+Required to confirm the MDS (CVE-2018-12126, CVE-2018-12127, CVE-2018-12130,
+CVE-2019-11091) fixes.
+
+Not included by default in any Intel CPU model.
+
+Must be explicitly turned on for all Intel CPU models.
+
+Requires the host CPU microcode to support this feature before it
+can be used for guest CPUs.
@end table
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 36/97] virtio-pci: fix missing device properties
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (34 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 35/97] docs: recommend use of md-clear feature on all Intel CPUs Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 37/97] usbredir: fix buffer-overflow on vmload Michael Roth
` (65 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Marc-André Lureau, qemu-stable
From: Marc-André Lureau <marcandre.lureau@redhat.com>
Since commit a4ee4c8baa37154 ("virtio: Helper for registering virtio
device types"), virtio-gpu-pci, virtio-vga, and virtio-crypto-pci lost
some properties: "ioeventfd" and "vectors". This may cause various
issues, such as failing migration or invalid properties.
Since those VirtioPCI devices do not have a base name, their class are
initialized with virtio_pci_generic_base_class_init(). However, if the
VirtioPCIDeviceTypeInfo provided a class_init which sets dc->props,
the properties were overwritten by virtio_pci_generic_class_init().
Instead, introduce an intermediary base-type to register the generic
properties.
Fixes: a4ee4c8baa37154f42b4dc6a13fee79268d15238
Cc: qemu-stable@nongnu.org
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190625232333.30752-1-marcandre.lureau@redhat.com>
(cherry picked from commit 683c1d89efd1eeb111c129a9a91f629b94d90d45)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-pci.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index cb44e19b67..497092e83c 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1905,13 +1905,6 @@ static void virtio_pci_generic_class_init(ObjectClass *klass, void *data)
dc->props = virtio_pci_generic_properties;
}
-/* Used when the generic type and the base type is the same */
-static void virtio_pci_generic_base_class_init(ObjectClass *klass, void *data)
-{
- virtio_pci_base_class_init(klass, data);
- virtio_pci_generic_class_init(klass, NULL);
-}
-
static void virtio_pci_transitional_instance_init(Object *obj)
{
VirtIOPCIProxy *proxy = VIRTIO_PCI(obj);
@@ -1930,14 +1923,13 @@ static void virtio_pci_non_transitional_instance_init(Object *obj)
void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t)
{
+ char *base_name = NULL;
TypeInfo base_type_info = {
.name = t->base_name,
.parent = t->parent ? t->parent : TYPE_VIRTIO_PCI,
.instance_size = t->instance_size,
.instance_init = t->instance_init,
.class_size = t->class_size,
- .class_init = virtio_pci_base_class_init,
- .class_data = (void *)t,
.abstract = true,
};
TypeInfo generic_type_info = {
@@ -1953,13 +1945,20 @@ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t)
if (!base_type_info.name) {
/* No base type -> register a single generic device type */
- base_type_info.name = t->generic_name;
- base_type_info.class_init = virtio_pci_generic_base_class_init;
- base_type_info.interfaces = generic_type_info.interfaces;
- base_type_info.abstract = false;
- generic_type_info.name = NULL;
+ /* use intermediate %s-base-type to add generic device props */
+ base_name = g_strdup_printf("%s-base-type", t->generic_name);
+ base_type_info.name = base_name;
+ base_type_info.class_init = virtio_pci_generic_class_init;
+
+ generic_type_info.parent = base_name;
+ generic_type_info.class_init = virtio_pci_base_class_init;
+ generic_type_info.class_data = (void *)t;
+
assert(!t->non_transitional_name);
assert(!t->transitional_name);
+ } else {
+ base_type_info.class_init = virtio_pci_base_class_init;
+ base_type_info.class_data = (void *)t;
}
type_register(&base_type_info);
@@ -1997,6 +1996,7 @@ void virtio_pci_types_register(const VirtioPCIDeviceTypeInfo *t)
};
type_register(&transitional_type_info);
}
+ g_free(base_name);
}
/* virtio-pci-bus */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 37/97] usbredir: fix buffer-overflow on vmload
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (35 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 36/97] virtio-pci: fix missing device properties Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 38/97] virtio-balloon: fix QEMU 4.0 config size migration incompatibility Michael Roth
` (64 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Marc-André Lureau, qemu-stable, Gerd Hoffmann
From: Marc-André Lureau <marcandre.lureau@redhat.com>
If interface_count is NO_INTERFACE_INFO, let's not access the arrays
out-of-bounds.
==994==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625000243930 at pc 0x5642068086a8 bp 0x7f0b6f9ffa50 sp 0x7f0b6f9ffa40
READ of size 1 at 0x625000243930 thread T0
#0 0x5642068086a7 in usbredir_check_bulk_receiving /home/elmarco/src/qemu/hw/usb/redirect.c:1503
#1 0x56420681301c in usbredir_post_load /home/elmarco/src/qemu/hw/usb/redirect.c:2154
#2 0x5642068a56c2 in vmstate_load_state /home/elmarco/src/qemu/migration/vmstate.c:168
#3 0x56420688e2ac in vmstate_load /home/elmarco/src/qemu/migration/savevm.c:829
#4 0x5642068980cb in qemu_loadvm_section_start_full /home/elmarco/src/qemu/migration/savevm.c:2211
#5 0x564206899645 in qemu_loadvm_state_main /home/elmarco/src/qemu/migration/savevm.c:2395
#6 0x5642068998cf in qemu_loadvm_state /home/elmarco/src/qemu/migration/savevm.c:2467
#7 0x56420685f3e9 in process_incoming_migration_co /home/elmarco/src/qemu/migration/migration.c:449
#8 0x564207106c47 in coroutine_trampoline /home/elmarco/src/qemu/util/coroutine-ucontext.c:115
#9 0x7f0c0604e37f (/lib64/libc.so.6+0x4d37f)
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Liam Merwick <liam.merwick@oracle.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190807084048.4258-1-marcandre.lureau@redhat.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
(cherry picked from commit 7b84b90966568da0e05655ecaa78c209300aae6e)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/usb/redirect.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
index 7cb6b120d4..586c7db025 100644
--- a/hw/usb/redirect.c
+++ b/hw/usb/redirect.c
@@ -1494,6 +1494,11 @@ static void usbredir_check_bulk_receiving(USBRedirDevice *dev)
for (i = EP2I(USB_DIR_IN); i < MAX_ENDPOINTS; i++) {
dev->endpoint[i].bulk_receiving_enabled = 0;
}
+
+ if (dev->interface_info.interface_count == NO_INTERFACE_INFO) {
+ return;
+ }
+
for (i = 0; i < dev->interface_info.interface_count; i++) {
quirks = usb_get_quirks(dev->device_info.vendor_id,
dev->device_info.product_id,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 38/97] virtio-balloon: fix QEMU 4.0 config size migration incompatibility
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (36 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 37/97] usbredir: fix buffer-overflow on vmload Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 39/97] docs/interop/bitmaps.rst: Fix typos Michael Roth
` (63 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Stefan Hajnoczi, Michael S . Tsirkin
From: Stefan Hajnoczi <stefanha@redhat.com>
The virtio-balloon config size changed in QEMU 4.0 even for existing
machine types. Migration from QEMU 3.1 to 4.0 can fail in some
circumstances with the following error:
qemu-system-x86_64: get_pci_config_device: Bad config data: i=0x10 read: a1 device: 1 cmask: ff wmask: c0 w1cmask:0
This happens because the virtio-balloon config size affects the VIRTIO
Legacy I/O Memory PCI BAR size.
Introduce a qdev property called "qemu-4-0-config-size" and enable it
only for the QEMU 4.0 machine types. This way <4.0 machine types use
the old size, 4.0 uses the larger size, and >4.0 machine types use the
appropriate size depending on enabled virtio-balloon features.
Live migration to and from old QEMUs to QEMU 4.1 works again as long as
a versioned machine type is specified (do not use just "pc"!).
Originally-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20190710141440.27635-1-stefanha@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Tested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Tested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 2bbadb08ce272d65e1f78621002008b07d1e0f03)
Conflicts:
hw/core/machine.c
* drop context dep. on 0a71966253c
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/core/machine.c | 5 ++++-
hw/virtio/virtio-balloon.c | 28 +++++++++++++++++++++++++---
include/hw/virtio/virtio-balloon.h | 2 ++
3 files changed, 31 insertions(+), 4 deletions(-)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index eb34f53a85..55b08f1466 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -24,7 +24,9 @@
#include "hw/pci/pci.h"
#include "hw/mem/nvdimm.h"
-GlobalProperty hw_compat_4_0[] = {};
+GlobalProperty hw_compat_4_0[] = {
+ { "virtio-balloon-device", "qemu-4-0-config-size", "true" },
+};
const size_t hw_compat_4_0_len = G_N_ELEMENTS(hw_compat_4_0);
GlobalProperty hw_compat_3_1[] = {
@@ -39,6 +41,7 @@ GlobalProperty hw_compat_3_1[] = {
{ "usb-tablet", "serial", "42" },
{ "virtio-blk-device", "discard", "false" },
{ "virtio-blk-device", "write-zeroes", "false" },
+ { "virtio-balloon-device", "qemu-4-0-config-size", "false" },
};
const size_t hw_compat_3_1_len = G_N_ELEMENTS(hw_compat_3_1);
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 2112874055..5579260fd4 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -615,6 +615,22 @@ virtio_balloon_free_page_report_notify(NotifierWithReturn *n, void *data)
return 0;
}
+static size_t virtio_balloon_config_size(VirtIOBalloon *s)
+{
+ uint64_t features = s->host_features;
+
+ if (s->qemu_4_0_config_size) {
+ return sizeof(struct virtio_balloon_config);
+ }
+ if (virtio_has_feature(features, VIRTIO_BALLOON_F_PAGE_POISON)) {
+ return sizeof(struct virtio_balloon_config);
+ }
+ if (virtio_has_feature(features, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) {
+ return offsetof(struct virtio_balloon_config, poison_val);
+ }
+ return offsetof(struct virtio_balloon_config, free_page_report_cmd_id);
+}
+
static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_data)
{
VirtIOBalloon *dev = VIRTIO_BALLOON(vdev);
@@ -635,7 +651,7 @@ static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_data)
}
trace_virtio_balloon_get_config(config.num_pages, config.actual);
- memcpy(config_data, &config, sizeof(struct virtio_balloon_config));
+ memcpy(config_data, &config, virtio_balloon_config_size(dev));
}
static int build_dimm_list(Object *obj, void *opaque)
@@ -679,7 +695,7 @@ static void virtio_balloon_set_config(VirtIODevice *vdev,
uint32_t oldactual = dev->actual;
ram_addr_t vm_ram_size = get_current_ram_size();
- memcpy(&config, config_data, sizeof(struct virtio_balloon_config));
+ memcpy(&config, config_data, virtio_balloon_config_size(dev));
dev->actual = le32_to_cpu(config.actual);
if (dev->actual != oldactual) {
qapi_event_send_balloon_change(vm_ram_size -
@@ -766,7 +782,7 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
int ret;
virtio_init(vdev, "virtio-balloon", VIRTIO_ID_BALLOON,
- sizeof(struct virtio_balloon_config));
+ virtio_balloon_config_size(s));
ret = qemu_add_balloon_handler(virtio_balloon_to_target,
virtio_balloon_stat, s);
@@ -897,6 +913,12 @@ static Property virtio_balloon_properties[] = {
VIRTIO_BALLOON_F_DEFLATE_ON_OOM, false),
DEFINE_PROP_BIT("free-page-hint", VirtIOBalloon, host_features,
VIRTIO_BALLOON_F_FREE_PAGE_HINT, false),
+ /* QEMU 4.0 accidentally changed the config size even when free-page-hint
+ * is disabled, resulting in QEMU 3.1 migration incompatibility. This
+ * property retains this quirk for QEMU 4.1 machine types.
+ */
+ DEFINE_PROP_BOOL("qemu-4-0-config-size", VirtIOBalloon,
+ qemu_4_0_config_size, false),
DEFINE_PROP_LINK("iothread", VirtIOBalloon, iothread, TYPE_IOTHREAD,
IOThread *),
DEFINE_PROP_END_OF_LIST(),
diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-balloon.h
index 1afafb12f6..5a99293a45 100644
--- a/include/hw/virtio/virtio-balloon.h
+++ b/include/hw/virtio/virtio-balloon.h
@@ -71,6 +71,8 @@ typedef struct VirtIOBalloon {
int64_t stats_poll_interval;
uint32_t host_features;
PartiallyBalloonedPage *pbp;
+
+ bool qemu_4_0_config_size;
} VirtIOBalloon;
#endif
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 39/97] docs/interop/bitmaps.rst: Fix typos
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (37 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 38/97] virtio-balloon: fix QEMU 4.0 config size migration incompatibility Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 40/97] sphinx: add qmp_lexer Michael Roth
` (62 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
Pygments and Sphinx get pickier all the time; Sphinx 2.1+ now catches
these errors.
Signed-off-by: John Snow <jsnow@redhat.com>
Reported-by: Aarushi Mehta <mehta.aaru20@gmail.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190603214653.29369-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 575e6226287072bd0d6eb85d9712d280eb29c392)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
docs/interop/bitmaps.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
index 510e8809a9..c29ac4a854 100644
--- a/docs/interop/bitmaps.rst
+++ b/docs/interop/bitmaps.rst
@@ -399,7 +399,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty.
"arguments": {
"node": "drive0",
"target": "new_bitmap",
- "bitmaps: [ "bitmap0" ]
+ "bitmaps": [ "bitmap0" ]
}
}
@@ -1437,7 +1437,7 @@ applied:
.. code:: json
<- {
- "timestamp": {...}
+ "timestamp": {...},
"data": {
"device": "drive0",
"type": "backup",
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 40/97] sphinx: add qmp_lexer
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (38 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 39/97] docs/interop/bitmaps.rst: Fix typos Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 41/97] docs/bitmaps: use QMP lexer instead of json Michael Roth
` (61 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable, Eduardo Habkost
From: John Snow <jsnow@redhat.com>
Sphinx, through Pygments, does not like annotated json examples very
much. In some versions of Sphinx (1.7), it will render the non-json
portions of code blocks in red, but in newer versions (2.0) it will
throw an exception and not highlight the block at all. Though we can
suppress this warning, it doesn't bring back highlighting on non-strict
json blocks.
We can alleviate this by creating a custom lexer for QMP examples that
allows us to properly highlight these examples in a robust way, keeping
our directionality and elision notations.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reported-by: Aarushi Mehta <mehta.aaru20@gmail.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 20190603214653.29369-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit cd231e13bdcb8d686b014bef940c7d19c6f1e769)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
docs/conf.py | 4 ++--
docs/sphinx/qmp_lexer.py | 43 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 45 insertions(+), 2 deletions(-)
create mode 100644 docs/sphinx/qmp_lexer.py
diff --git a/docs/conf.py b/docs/conf.py
index befbcc6c3e..e46b299b71 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -41,7 +41,7 @@ except NameError:
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use an absolute path starting from qemu_docdir.
#
-# sys.path.insert(0, os.path.join(qemu_docdir, "my_subdir"))
+sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
# -- General configuration ------------------------------------------------
@@ -54,7 +54,7 @@ needs_sphinx = '1.3'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
-extensions = []
+extensions = ['qmp_lexer']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
new file mode 100644
index 0000000000..f7e4c0e198
--- /dev/null
+++ b/docs/sphinx/qmp_lexer.py
@@ -0,0 +1,43 @@
+# QEMU Monitor Protocol Lexer Extension
+#
+# Copyright (C) 2019, Red Hat Inc.
+#
+# Authors:
+# Eduardo Habkost <ehabkost@redhat.com>
+# John Snow <jsnow@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPLv2 or later.
+# See the COPYING file in the top-level directory.
+"""qmp_lexer is a Sphinx extension that provides a QMP lexer for code blocks."""
+
+from pygments.lexer import RegexLexer, DelegatingLexer
+from pygments.lexers.data import JsonLexer
+from pygments import token
+from sphinx import errors
+
+class QMPExampleMarkersLexer(RegexLexer):
+ """
+ QMPExampleMarkersLexer lexes QMP example annotations.
+ This lexer adds support for directionality flow and elision indicators.
+ """
+ tokens = {
+ 'root': [
+ (r'-> ', token.Generic.Prompt),
+ (r'<- ', token.Generic.Prompt),
+ (r' ?\.{3} ?', token.Generic.Prompt),
+ ]
+ }
+
+class QMPExampleLexer(DelegatingLexer):
+ """QMPExampleLexer lexes annotated QMP examples."""
+ def __init__(self, **options):
+ super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer,
+ token.Error, **options)
+
+def setup(sphinx):
+ """For use by the Sphinx extensions API."""
+ try:
+ sphinx.require_sphinx('2.1')
+ sphinx.add_lexer('QMP', QMPExampleLexer)
+ except errors.VersionRequirementError:
+ sphinx.add_lexer('QMP', QMPExampleLexer())
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 41/97] docs/bitmaps: use QMP lexer instead of json
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (39 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 40/97] sphinx: add qmp_lexer Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 42/97] hw/ssi/xilinx_spips: Convert lqspi_read() to read_with_attrs Michael Roth
` (60 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
The annotated style json we use in QMP documentation is not strict json
and depending on the version of Sphinx (2.0+) or Pygments installed,
might cause the build to fail.
Use the new QMP lexer.
Further, some versions of Sphinx can not apply custom lexers to "code"
directives and require the use of "code-block" directives instead, so
make that change at this time as well.
Tested under:
- Sphinx 1.3.6 and Pygments 2.4
- Sphinx 1.7.6 and Pygments 2.2 (Fedora 29 packages)
- Sphinx 2.0.1 and Pygments 2.4
- Sphinx 3.0.0+/f396b3a783 and Pygments 2.4 (From Sphinx git c4f44bdd)
Reported-by: Aarushi Mehta <mehta.aaru20@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-id: 20190603214653.29369-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit a7786bfb0effe0b4b0fc61d8a8cd307c0b739ed7)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
docs/interop/bitmaps.rst | 54 ++++++++++++++++++++--------------------
1 file changed, 27 insertions(+), 27 deletions(-)
diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
index c29ac4a854..c20bd37a79 100644
--- a/docs/interop/bitmaps.rst
+++ b/docs/interop/bitmaps.rst
@@ -199,7 +199,7 @@ persistence, and recording state can be adjusted at creation time.
to create a new, actively recording persistent bitmap:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-add",
"arguments": {
@@ -220,7 +220,7 @@ persistence, and recording state can be adjusted at creation time.
To create a new, disabled (``-recording``), transient bitmap that tracks
changes in 32KiB segments:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-add",
"arguments": {
@@ -254,7 +254,7 @@ Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed.
Remove a bitmap named ``bitmap0`` from node ``drive0``:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-remove",
"arguments": {
@@ -280,7 +280,7 @@ Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared.
Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-clear",
"arguments": {
@@ -309,7 +309,7 @@ begin being recorded. ``+busy`` bitmaps cannot be enabled.
To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-enable",
"arguments": {
@@ -347,7 +347,7 @@ writes to begin being ignored. ``+busy`` bitmaps cannot be disabled.
To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``:
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-disable",
"arguments": {
@@ -393,7 +393,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty.
``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves
a copy.
- .. code:: json
+ .. code-block:: QMP
-> { "execute": "block-dirty-bitmap-merge",
"arguments": {
@@ -424,7 +424,7 @@ attached to nodes serving as the root for guest devices.
API. This result highlights a bitmap ``bitmap0`` attached to the root node of
device ``drive0``.
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "query-block",
@@ -562,7 +562,7 @@ new, empty bitmap that records writes from this point in time forward.
destination. These writes will be recorded in the bitmap
accordingly.
-.. code:: json
+.. code-block:: QMP
-> {
"execute": "transaction",
@@ -650,7 +650,7 @@ Example: Resetting an Incremental Backup Anchor Point
If we want to start a new backup chain with an existing bitmap, we can also
use a transaction to reset the bitmap while making a new full backup:
-.. code:: json
+.. code-block:: QMP
-> {
"execute": "transaction",
@@ -730,7 +730,7 @@ Example: First Incremental Backup
#. Issue an incremental backup command:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "drive-backup",
@@ -788,7 +788,7 @@ Example: Second Incremental Backup
#. Issue a new incremental backup command. The only difference here is that we
have changed the target image below.
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "drive-backup",
@@ -869,7 +869,7 @@ image:
#. Issue a new incremental backup command. Apart from the new destination
image, there is no difference from the last two examples.
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "drive-backup",
@@ -932,7 +932,7 @@ point in time.
#. Create a full (anchor) backup for each drive, with accompanying bitmaps:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "transaction",
@@ -1018,7 +1018,7 @@ point in time.
#. Issue a multi-drive incremental push backup transaction:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "transaction",
@@ -1121,7 +1121,7 @@ described above. This example demonstrates the single-job failure case:
#. Attempt to create an incremental backup via QMP:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "drive-backup",
@@ -1139,7 +1139,7 @@ described above. This example demonstrates the single-job failure case:
#. Receive a pair of events indicating failure:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
@@ -1175,7 +1175,7 @@ described above. This example demonstrates the single-job failure case:
#. Retry the command after fixing the underlying problem, such as
freeing up space on the backup volume:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "drive-backup",
@@ -1193,7 +1193,7 @@ described above. This example demonstrates the single-job failure case:
#. Receive confirmation that the job completed successfully:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
@@ -1233,7 +1233,7 @@ and one succeeds:
#. Issue the transaction to start a backup of both drives.
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "transaction",
@@ -1267,13 +1267,13 @@ and one succeeds:
#. Receive notice that the Transaction was accepted, and jobs were
launched:
- .. code:: json
+ .. code-block:: QMP
<- { "return": {} }
#. Receive notice that the first job has completed:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
@@ -1289,7 +1289,7 @@ and one succeeds:
#. Receive notice that the second job has failed:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
@@ -1365,7 +1365,7 @@ applied:
#. Issue the multi-drive incremental backup transaction:
- .. code:: json
+ .. code-block:: QMP
-> {
"execute": "transaction",
@@ -1401,13 +1401,13 @@ applied:
#. Receive notice that the Transaction was accepted, and jobs were launched:
- .. code:: json
+ .. code-block:: QMP
<- { "return": {} }
#. Receive notification that the backup job for ``drive1`` has failed:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
@@ -1434,7 +1434,7 @@ applied:
#. Receive notification that the job for ``drive0`` has been cancelled:
- .. code:: json
+ .. code-block:: QMP
<- {
"timestamp": {...},
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 42/97] hw/ssi/xilinx_spips: Convert lqspi_read() to read_with_attrs
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (40 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 41/97] docs/bitmaps: use QMP lexer instead of json Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 43/97] hw/ssi/xilinx_spips: Avoid AXI writes to the LQSPI linear memory Michael Roth
` (59 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, Philippe Mathieu-Daudé, qemu-stable
From: Philippe Mathieu-Daudé <philmd@redhat.com>
In the next commit we will implement the write_with_attrs()
handler. To avoid using different APIs, convert the read()
handler first.
Reviewed-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Tested-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
(cherry picked from commit 5937bd50d3841b6ab2592c1ff4233448762a8483)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/ssi/xilinx_spips.c | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/hw/ssi/xilinx_spips.c b/hw/ssi/xilinx_spips.c
index 16f88f7402..9c5dd93b21 100644
--- a/hw/ssi/xilinx_spips.c
+++ b/hw/ssi/xilinx_spips.c
@@ -1199,27 +1199,26 @@ static void lqspi_load_cache(void *opaque, hwaddr addr)
}
}
-static uint64_t
-lqspi_read(void *opaque, hwaddr addr, unsigned int size)
+static MemTxResult lqspi_read(void *opaque, hwaddr addr, uint64_t *value,
+ unsigned size, MemTxAttrs attrs)
{
- XilinxQSPIPS *q = opaque;
- uint32_t ret;
+ XilinxQSPIPS *q = XILINX_QSPIPS(opaque);
if (addr >= q->lqspi_cached_addr &&
addr <= q->lqspi_cached_addr + LQSPI_CACHE_SIZE - 4) {
uint8_t *retp = &q->lqspi_buf[addr - q->lqspi_cached_addr];
- ret = cpu_to_le32(*(uint32_t *)retp);
- DB_PRINT_L(1, "addr: %08x, data: %08x\n", (unsigned)addr,
- (unsigned)ret);
- return ret;
- } else {
- lqspi_load_cache(opaque, addr);
- return lqspi_read(opaque, addr, size);
+ *value = cpu_to_le32(*(uint32_t *)retp);
+ DB_PRINT_L(1, "addr: %08" HWADDR_PRIx ", data: %08" PRIx64 "\n",
+ addr, *value);
+ return MEMTX_OK;
}
+
+ lqspi_load_cache(opaque, addr);
+ return lqspi_read(opaque, addr, value, size, attrs);
}
static const MemoryRegionOps lqspi_ops = {
- .read = lqspi_read,
+ .read_with_attrs = lqspi_read,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid = {
.min_access_size = 1,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 43/97] hw/ssi/xilinx_spips: Avoid AXI writes to the LQSPI linear memory
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (41 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 42/97] hw/ssi/xilinx_spips: Convert lqspi_read() to read_with_attrs Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 44/97] hw/ssi/xilinx_spips: Avoid out-of-bound access to lqspi_buf[] Michael Roth
` (58 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, Philippe Mathieu-Daudé, qemu-stable
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Lei Sun found while auditing the code that a CPU write would
trigger a NULL pointer dereference.
>From UG1085 datasheet [*] AXI writes in this region are ignored
and generates an AXI Slave Error (SLVERR).
Fix by implementing the write_with_attrs() handler.
Return MEMTX_ERROR when the region is accessed (this error maps
to an AXI slave error).
[*] https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
Reported-by: Lei Sun <slei.casper@gmail.com>
Reviewed-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Tested-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
(cherry picked from commit 936a236c4e4b1068ade99220260cd04f68eb0212)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/ssi/xilinx_spips.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/hw/ssi/xilinx_spips.c b/hw/ssi/xilinx_spips.c
index 9c5dd93b21..83ed5ab1e0 100644
--- a/hw/ssi/xilinx_spips.c
+++ b/hw/ssi/xilinx_spips.c
@@ -1217,8 +1217,24 @@ static MemTxResult lqspi_read(void *opaque, hwaddr addr, uint64_t *value,
return lqspi_read(opaque, addr, value, size, attrs);
}
+static MemTxResult lqspi_write(void *opaque, hwaddr offset, uint64_t value,
+ unsigned size, MemTxAttrs attrs)
+{
+ /*
+ * From UG1085, Chapter 24 (Quad-SPI controllers):
+ * - Writes are ignored
+ * - AXI writes generate an external AXI slave error (SLVERR)
+ */
+ qemu_log_mask(LOG_GUEST_ERROR, "%s Unexpected %u-bit access to 0x%" PRIx64
+ " (value: 0x%" PRIx64 "\n",
+ __func__, size << 3, offset, value);
+
+ return MEMTX_ERROR;
+}
+
static const MemoryRegionOps lqspi_ops = {
.read_with_attrs = lqspi_read,
+ .write_with_attrs = lqspi_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid = {
.min_access_size = 1,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 44/97] hw/ssi/xilinx_spips: Avoid out-of-bound access to lqspi_buf[]
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (42 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 43/97] hw/ssi/xilinx_spips: Avoid AXI writes to the LQSPI linear memory Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 45/97] ioapic: kvm: Skip route updates for masked pins Michael Roth
` (57 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, Philippe Mathieu-Daudé, qemu-stable
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Both lqspi_read() and lqspi_load_cache() expect a 32-bit
aligned address.
>From UG1085 datasheet [*] chapter on 'Quad-SPI Controller':
Transfer Size Limitations
Because of the 32-bit wide TX, RX, and generic FIFO, all
APB/AXI transfers must be an integer multiple of 4-bytes.
Shorter transfers are not possible.
Set MemoryRegionOps.impl values to force 32-bit accesses,
this way we are sure we do not access the lqspi_buf[] array
out of bound.
[*] https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
Reviewed-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Tested-by: Francisco Iglesias <frasse.iglesias@gmail.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
(cherry picked from commit 526668c734e6a07f2fedfd378840a61b70c1cbab)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/ssi/xilinx_spips.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/hw/ssi/xilinx_spips.c b/hw/ssi/xilinx_spips.c
index 83ed5ab1e0..b649c464fb 100644
--- a/hw/ssi/xilinx_spips.c
+++ b/hw/ssi/xilinx_spips.c
@@ -1236,6 +1236,10 @@ static const MemoryRegionOps lqspi_ops = {
.read_with_attrs = lqspi_read,
.write_with_attrs = lqspi_write,
.endianness = DEVICE_NATIVE_ENDIAN,
+ .impl = {
+ .min_access_size = 4,
+ .max_access_size = 4,
+ },
.valid = {
.min_access_size = 1,
.max_access_size = 4
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 45/97] ioapic: kvm: Skip route updates for masked pins
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (43 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 44/97] hw/ssi/xilinx_spips: Avoid out-of-bound access to lqspi_buf[] Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 46/97] i386/acpi: show PCI Express bus on pxb-pcie expanders Michael Roth
` (56 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Jan Kiszka, qemu-stable, Michael S . Tsirkin
From: Jan Kiszka <jan.kiszka@siemens.com>
Masked entries will not generate interrupt messages, thus do no need to
be routed by KVM. This is a cosmetic cleanup, just avoiding warnings of
the kind
qemu-system-x86_64: vtd_irte_get: detected non-present IRTE (index=0, high=0xff00, low=0x100)
if the masked entry happens to reference a non-present IRTE.
Cc: qemu-stable@nongnu.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Message-Id: <a84b7e03-f9a8-b577-be27-4d93d1caa1c9@siemens.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
(cherry picked from commit be1927c97e564346cbd409cb17fe611df74b84e5)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/intc/ioapic.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/hw/intc/ioapic.c b/hw/intc/ioapic.c
index 9d75f84d3b..194ccb6a3e 100644
--- a/hw/intc/ioapic.c
+++ b/hw/intc/ioapic.c
@@ -188,9 +188,11 @@ static void ioapic_update_kvm_routes(IOAPICCommonState *s)
MSIMessage msg;
struct ioapic_entry_info info;
ioapic_entry_parse(s->ioredtbl[i], &info);
- msg.address = info.addr;
- msg.data = info.data;
- kvm_irqchip_update_msi_route(kvm_state, i, msg, NULL);
+ if (!info.masked) {
+ msg.address = info.addr;
+ msg.data = info.data;
+ kvm_irqchip_update_msi_route(kvm_state, i, msg, NULL);
+ }
}
kvm_irqchip_commit_routes(kvm_state);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 46/97] i386/acpi: show PCI Express bus on pxb-pcie expanders
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (44 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 45/97] ioapic: kvm: Skip route updates for masked pins Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 47/97] virtio-balloon: Fix wrong sign extension of PFNs Michael Roth
` (55 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Evgeny Yakovlev, Michael S . Tsirkin
From: Evgeny Yakovlev <wrfsh@yandex-team.ru>
Show PCIe host bridge PNP id with PCI host bridge as a compatible id
when expanding a pcie bus.
Cc: qemu-stable@nongnu.org
Signed-off-by: Evgeny Yakovlev <wrfsh@yandex-team.ru>
Message-Id: <1563526469-15588-1-git-send-email-wrfsh@yandex-team.ru>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit ee4b0c8686f781987879508d7c6dd605b5435bac)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/i386/acpi-build.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 416da318ae..ede27ab3c4 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1895,10 +1895,13 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
scope = aml_scope("\\_SB");
dev = aml_device("PC%.02X", bus_num);
aml_append(dev, aml_name_decl("_UID", aml_int(bus_num)));
- aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
if (pci_bus_is_express(bus)) {
+ aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
+ aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
aml_append(dev, build_q35_osc_method());
+ } else {
+ aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
}
if (numa_node != NUMA_NODE_UNASSIGNED) {
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 47/97] virtio-balloon: Fix wrong sign extension of PFNs
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (45 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 46/97] i386/acpi: show PCI Express bus on pxb-pcie expanders Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 48/97] virtio-balloon: Fix QEMU crashes on pagesize > BALLOON_PAGE_SIZE Michael Roth
` (54 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
If we directly cast from int to uint64_t, we will first sign-extend to
an int64_t, which is wrong. We actually want to treat the PFNs like
unsigned values.
As far as I can see, this dates back to the initial virtio-balloon
commit, but wasn't triggered as fairly big guests would be required.
Cc: qemu-stable@nongnu.org
Reported-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-2-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
(cherry picked from commit ffa207d08253ffffb3993a1dbe09e40af4fc91f1)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 5579260fd4..49194f5638 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -343,8 +343,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
}
while (iov_to_buf(elem->out_sg, elem->out_num, offset, &pfn, 4) == 4) {
+ unsigned int p = virtio_ldl_p(vdev, &pfn);
hwaddr pa;
- int p = virtio_ldl_p(vdev, &pfn);
pa = (hwaddr) p << VIRTIO_BALLOON_PFN_SHIFT;
offset += 4;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 48/97] virtio-balloon: Fix QEMU crashes on pagesize > BALLOON_PAGE_SIZE
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (46 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 47/97] virtio-balloon: Fix wrong sign extension of PFNs Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 49/97] virtio-balloon: Simplify deflate with pbp Michael Roth
` (53 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
We are using the wrong functions to set/clear bits, effectively touching
multiple bits, writing out of range of the bitmap, resulting in memory
corruptions. We have to use set_bit()/clear_bit() instead.
Can easily be reproduced by starting a qemu guest on hugetlbfs memory,
inflating the balloon. QEMU crashes. This never could have worked
properly - especially, also pages would have been discarded when the
first sub-page would be inflated (the whole bitmap would be set).
While testing I realized, that on hugetlbfs it is pretty much impossible
to discard a page - the guest just frees the 4k sub-pages in random order
most of the time. I was only able to discard a hugepage a handful of
times - so I hope that now works correctly.
Fixes: ed48c59875b6 ("virtio-balloon: Safely handle BALLOON_PAGE_SIZE < host page size")
Fixes: b27b32391404 ("virtio-balloon: Fix possible guest memory corruption with inflates & deflates")
Cc: qemu-stable@nongnu.org #v4.0.0
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-3-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 483f13524bb2a08b7ff6a7560b846564ed3b0c33)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 49194f5638..038cf1c23a 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -94,9 +94,8 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
balloon->pbp->base = host_page_base;
}
- bitmap_set(balloon->pbp->bitmap,
- (ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
- subpages);
+ set_bit((ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
+ balloon->pbp->bitmap);
if (bitmap_full(balloon->pbp->bitmap, subpages)) {
/* We've accumulated a full host page, we can actually discard
@@ -140,9 +139,8 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,
* for a guest to do this in practice, but handle it anyway,
* since getting it wrong could mean discarding memory the
* guest is still using. */
- bitmap_clear(balloon->pbp->bitmap,
- (ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
- subpages);
+ clear_bit((ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
+ balloon->pbp->bitmap);
if (bitmap_empty(balloon->pbp->bitmap, subpages)) {
g_free(balloon->pbp);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 49/97] virtio-balloon: Simplify deflate with pbp
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (47 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 48/97] virtio-balloon: Fix QEMU crashes on pagesize > BALLOON_PAGE_SIZE Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 50/97] virtio-balloon: Better names for offset variables in inflate/deflate code Michael Roth
` (52 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
Let's simplify this - the case we are optimizing for is very hard to
trigger and not worth the effort. If we're switching from inflation to
deflation, let's reset the pbp.
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-4-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 2ffc49eea1bbd454913a88a0ad872c2649b36950)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 26 +++++---------------------
1 file changed, 5 insertions(+), 21 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 038cf1c23a..4f42d3f2b0 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -117,7 +117,7 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,
void *addr = memory_region_get_ram_ptr(mr) + offset;
RAMBlock *rb;
size_t rb_page_size;
- ram_addr_t ram_offset, host_page_base;
+ ram_addr_t ram_offset;
void *host_addr;
int ret;
@@ -125,27 +125,11 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,
* host address? */
rb = qemu_ram_block_from_host(addr, false, &ram_offset);
rb_page_size = qemu_ram_pagesize(rb);
- host_page_base = ram_offset & ~(rb_page_size - 1);
-
- if (balloon->pbp
- && rb == balloon->pbp->rb
- && host_page_base == balloon->pbp->base) {
- int subpages = rb_page_size / BALLOON_PAGE_SIZE;
- /*
- * This means the guest has asked to discard some of the 4kiB
- * subpages of a host page, but then changed its mind and
- * asked to keep them after all. It's exceedingly unlikely
- * for a guest to do this in practice, but handle it anyway,
- * since getting it wrong could mean discarding memory the
- * guest is still using. */
- clear_bit((ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
- balloon->pbp->bitmap);
-
- if (bitmap_empty(balloon->pbp->bitmap, subpages)) {
- g_free(balloon->pbp);
- balloon->pbp = NULL;
- }
+ if (balloon->pbp) {
+ /* Let's play safe and always reset the pbp on deflation requests. */
+ g_free(balloon->pbp);
+ balloon->pbp = NULL;
}
host_addr = (void *)((uintptr_t)addr & ~(rb_page_size - 1));
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 50/97] virtio-balloon: Better names for offset variables in inflate/deflate code
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (48 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 49/97] virtio-balloon: Simplify deflate with pbp Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 51/97] virtio-balloon: Rework pbp tracking data Michael Roth
` (51 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
"host_page_base" is really confusing, let's make this clearer, also
rename the other offsets to indicate to which base they apply.
offset -> mr_offset
ram_offset -> rb_offset
host_page_base -> rb_aligned_offset
While at it, use QEMU_ALIGN_DOWN() instead of a handcrafted computation
and move the computation to the place where it is needed.
Acked-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-5-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit e6129b271b9dccca22c84870e313c315f2c70063)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 26 +++++++++++++-------------
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 4f42d3f2b0..76b4c58206 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -41,24 +41,23 @@ struct PartiallyBalloonedPage {
};
static void balloon_inflate_page(VirtIOBalloon *balloon,
- MemoryRegion *mr, hwaddr offset)
+ MemoryRegion *mr, hwaddr mr_offset)
{
- void *addr = memory_region_get_ram_ptr(mr) + offset;
+ void *addr = memory_region_get_ram_ptr(mr) + mr_offset;
+ ram_addr_t rb_offset, rb_aligned_offset;
RAMBlock *rb;
size_t rb_page_size;
int subpages;
- ram_addr_t ram_offset, host_page_base;
/* XXX is there a better way to get to the RAMBlock than via a
* host address? */
- rb = qemu_ram_block_from_host(addr, false, &ram_offset);
+ rb = qemu_ram_block_from_host(addr, false, &rb_offset);
rb_page_size = qemu_ram_pagesize(rb);
- host_page_base = ram_offset & ~(rb_page_size - 1);
if (rb_page_size == BALLOON_PAGE_SIZE) {
/* Easy case */
- ram_block_discard_range(rb, ram_offset, rb_page_size);
+ ram_block_discard_range(rb, rb_offset, rb_page_size);
/* We ignore errors from ram_block_discard_range(), because it
* has already reported them, and failing to discard a balloon
* page is not fatal */
@@ -74,11 +73,12 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
warn_report_once(
"Balloon used with backing page size > 4kiB, this may not be reliable");
+ rb_aligned_offset = QEMU_ALIGN_DOWN(rb_offset, rb_page_size);
subpages = rb_page_size / BALLOON_PAGE_SIZE;
if (balloon->pbp
&& (rb != balloon->pbp->rb
- || host_page_base != balloon->pbp->base)) {
+ || rb_aligned_offset != balloon->pbp->base)) {
/* We've partially ballooned part of a host page, but now
* we're trying to balloon part of a different one. Too hard,
* give up on the old partial page */
@@ -91,10 +91,10 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
size_t bitlen = BITS_TO_LONGS(subpages) * sizeof(unsigned long);
balloon->pbp = g_malloc0(sizeof(PartiallyBalloonedPage) + bitlen);
balloon->pbp->rb = rb;
- balloon->pbp->base = host_page_base;
+ balloon->pbp->base = rb_aligned_offset;
}
- set_bit((ram_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
+ set_bit((rb_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
balloon->pbp->bitmap);
if (bitmap_full(balloon->pbp->bitmap, subpages)) {
@@ -112,18 +112,18 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
}
static void balloon_deflate_page(VirtIOBalloon *balloon,
- MemoryRegion *mr, hwaddr offset)
+ MemoryRegion *mr, hwaddr mr_offset)
{
- void *addr = memory_region_get_ram_ptr(mr) + offset;
+ void *addr = memory_region_get_ram_ptr(mr) + mr_offset;
+ ram_addr_t rb_offset;
RAMBlock *rb;
size_t rb_page_size;
- ram_addr_t ram_offset;
void *host_addr;
int ret;
/* XXX is there a better way to get to the RAMBlock than via a
* host address? */
- rb = qemu_ram_block_from_host(addr, false, &ram_offset);
+ rb = qemu_ram_block_from_host(addr, false, &rb_offset);
rb_page_size = qemu_ram_pagesize(rb);
if (balloon->pbp) {
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 51/97] virtio-balloon: Rework pbp tracking data
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (49 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 50/97] virtio-balloon: Better names for offset variables in inflate/deflate code Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 52/97] virtio-balloon: Use temporary PBP only Michael Roth
` (50 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
Using the address of a RAMBlock to test for a matching pbp is not really
safe. Instead, let's use the guest physical address of the base page
along with the page size (via the number of subpages).
Also, let's allocate the bitmap separately. This makes the code
easier to read and maintain - we can reuse bitmap_new().
Prepare the code to move the PBP out of the device.
Fixes: ed48c59875b6 ("virtio-balloon: Safely handle BALLOON_PAGE_SIZE < host page size")
Fixes: b27b32391404 ("virtio-balloon: Fix possible guest memory corruption with inflates & deflates")
Cc: qemu-stable@nongnu.org #v4.0.0
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-6-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 1c5cfc2b7153dd72bf4b8ddc456408eb2b9b66d8)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 69 +++++++++++++++++++++++++-------------
1 file changed, 46 insertions(+), 23 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 76b4c58206..49999d0bbe 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -35,16 +35,44 @@
#define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT)
struct PartiallyBalloonedPage {
- RAMBlock *rb;
- ram_addr_t base;
- unsigned long bitmap[];
+ ram_addr_t base_gpa;
+ long subpages;
+ unsigned long *bitmap;
};
+static void virtio_balloon_pbp_free(PartiallyBalloonedPage *pbp)
+{
+ if (!pbp) {
+ return;
+ }
+ g_free(pbp->bitmap);
+ g_free(pbp);
+}
+
+static PartiallyBalloonedPage *virtio_balloon_pbp_alloc(ram_addr_t base_gpa,
+ long subpages)
+{
+ PartiallyBalloonedPage *pbp = g_new0(PartiallyBalloonedPage, 1);
+
+ pbp->base_gpa = base_gpa;
+ pbp->subpages = subpages;
+ pbp->bitmap = bitmap_new(subpages);
+
+ return pbp;
+}
+
+static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
+ ram_addr_t base_gpa, long subpages)
+{
+ return pbp->subpages == subpages && pbp->base_gpa == base_gpa;
+}
+
static void balloon_inflate_page(VirtIOBalloon *balloon,
MemoryRegion *mr, hwaddr mr_offset)
{
void *addr = memory_region_get_ram_ptr(mr) + mr_offset;
- ram_addr_t rb_offset, rb_aligned_offset;
+ ram_addr_t rb_offset, rb_aligned_offset, base_gpa;
+ PartiallyBalloonedPage **pbp = &balloon->pbp;
RAMBlock *rb;
size_t rb_page_size;
int subpages;
@@ -75,39 +103,34 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
rb_aligned_offset = QEMU_ALIGN_DOWN(rb_offset, rb_page_size);
subpages = rb_page_size / BALLOON_PAGE_SIZE;
+ base_gpa = memory_region_get_ram_addr(mr) + mr_offset -
+ (rb_offset - rb_aligned_offset);
- if (balloon->pbp
- && (rb != balloon->pbp->rb
- || rb_aligned_offset != balloon->pbp->base)) {
+ if (*pbp && !virtio_balloon_pbp_matches(*pbp, base_gpa, subpages)) {
/* We've partially ballooned part of a host page, but now
* we're trying to balloon part of a different one. Too hard,
* give up on the old partial page */
- g_free(balloon->pbp);
- balloon->pbp = NULL;
+ virtio_balloon_pbp_free(*pbp);
+ *pbp = NULL;
}
- if (!balloon->pbp) {
- /* Starting on a new host page */
- size_t bitlen = BITS_TO_LONGS(subpages) * sizeof(unsigned long);
- balloon->pbp = g_malloc0(sizeof(PartiallyBalloonedPage) + bitlen);
- balloon->pbp->rb = rb;
- balloon->pbp->base = rb_aligned_offset;
+ if (!*pbp) {
+ *pbp = virtio_balloon_pbp_alloc(base_gpa, subpages);
}
- set_bit((rb_offset - balloon->pbp->base) / BALLOON_PAGE_SIZE,
- balloon->pbp->bitmap);
+ set_bit((rb_offset - rb_aligned_offset) / BALLOON_PAGE_SIZE,
+ (*pbp)->bitmap);
- if (bitmap_full(balloon->pbp->bitmap, subpages)) {
+ if (bitmap_full((*pbp)->bitmap, subpages)) {
/* We've accumulated a full host page, we can actually discard
* it now */
- ram_block_discard_range(rb, balloon->pbp->base, rb_page_size);
+ ram_block_discard_range(rb, rb_aligned_offset, rb_page_size);
/* We ignore errors from ram_block_discard_range(), because it
* has already reported them, and failing to discard a balloon
* page is not fatal */
-
- g_free(balloon->pbp);
- balloon->pbp = NULL;
+ virtio_balloon_pbp_free(*pbp);
+ *pbp = NULL;
}
}
@@ -128,7 +151,7 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,
if (balloon->pbp) {
/* Let's play safe and always reset the pbp on deflation requests. */
- g_free(balloon->pbp);
+ virtio_balloon_pbp_free(balloon->pbp);
balloon->pbp = NULL;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 52/97] virtio-balloon: Use temporary PBP only
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (50 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 51/97] virtio-balloon: Rework pbp tracking data Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 53/97] virtio-balloon: don't track subpages for the PBP Michael Roth
` (49 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
We still have multiple issues in the current code
- The PBP is not freed during unrealize()
- The PBP is not reset on device resets: After a reset, the PBP is stale.
- We are not indicating VIRTIO_BALLOON_F_MUST_TELL_HOST, therefore
guests (esp. legacy guests) will reuse pages without deflating,
turning the PBP stale. Adding that would require compat handling.
Instead, let's use the PBP only temporarily, when processing one bulk of
inflation requests. This will keep guest_page_size > 4k working (with
Linux guests). There is nothing to do for deflation requests anymore.
The pbp is only used for a limited amount of time.
Fixes: ed48c59875b6 ("virtio-balloon: Safely handle BALLOON_PAGE_SIZE < host page size")
Cc: qemu-stable@nongnu.org #v4.0.0
Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190722134108.22151-7-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: David Gibson <david@gibson.dropbear.id.au>
(cherry picked from commit a8cd64d488325f3be5c4ddec4bf07efb3b8c7330)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 21 +++++++++------------
include/hw/virtio/virtio-balloon.h | 3 ---
2 files changed, 9 insertions(+), 15 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 49999d0bbe..bd54e302de 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -34,11 +34,11 @@
#define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT)
-struct PartiallyBalloonedPage {
+typedef struct PartiallyBalloonedPage {
ram_addr_t base_gpa;
long subpages;
unsigned long *bitmap;
-};
+} PartiallyBalloonedPage;
static void virtio_balloon_pbp_free(PartiallyBalloonedPage *pbp)
{
@@ -68,11 +68,11 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
}
static void balloon_inflate_page(VirtIOBalloon *balloon,
- MemoryRegion *mr, hwaddr mr_offset)
+ MemoryRegion *mr, hwaddr mr_offset,
+ PartiallyBalloonedPage **pbp)
{
void *addr = memory_region_get_ram_ptr(mr) + mr_offset;
ram_addr_t rb_offset, rb_aligned_offset, base_gpa;
- PartiallyBalloonedPage **pbp = &balloon->pbp;
RAMBlock *rb;
size_t rb_page_size;
int subpages;
@@ -149,12 +149,6 @@ static void balloon_deflate_page(VirtIOBalloon *balloon,
rb = qemu_ram_block_from_host(addr, false, &rb_offset);
rb_page_size = qemu_ram_pagesize(rb);
- if (balloon->pbp) {
- /* Let's play safe and always reset the pbp on deflation requests. */
- virtio_balloon_pbp_free(balloon->pbp);
- balloon->pbp = NULL;
- }
-
host_addr = (void *)((uintptr_t)addr & ~(rb_page_size - 1));
/* When a page is deflated, we hint the whole host page it lives
@@ -336,6 +330,7 @@ static void balloon_stats_set_poll_interval(Object *obj, Visitor *v,
static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
{
VirtIOBalloon *s = VIRTIO_BALLOON(vdev);
+ PartiallyBalloonedPage *pbp = NULL;
VirtQueueElement *elem;
MemoryRegionSection section;
@@ -344,7 +339,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
uint32_t pfn;
elem = virtqueue_pop(vq, sizeof(VirtQueueElement));
if (!elem) {
- return;
+ break;
}
while (iov_to_buf(elem->out_sg, elem->out_num, offset, &pfn, 4) == 4) {
@@ -373,7 +368,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
if (!qemu_balloon_is_inhibited()) {
if (vq == s->ivq) {
balloon_inflate_page(s, section.mr,
- section.offset_within_region);
+ section.offset_within_region, &pbp);
} else if (vq == s->dvq) {
balloon_deflate_page(s, section.mr, section.offset_within_region);
} else {
@@ -387,6 +382,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
virtio_notify(vdev, vq);
g_free(elem);
}
+
+ virtio_balloon_pbp_free(pbp);
}
static void virtio_balloon_receive_stats(VirtIODevice *vdev, VirtQueue *vq)
diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-balloon.h
index 5a99293a45..d1c968d237 100644
--- a/include/hw/virtio/virtio-balloon.h
+++ b/include/hw/virtio/virtio-balloon.h
@@ -33,8 +33,6 @@ typedef struct virtio_balloon_stat_modern {
uint64_t val;
} VirtIOBalloonStatModern;
-typedef struct PartiallyBalloonedPage PartiallyBalloonedPage;
-
enum virtio_balloon_free_page_report_status {
FREE_PAGE_REPORT_S_STOP = 0,
FREE_PAGE_REPORT_S_REQUESTED = 1,
@@ -70,7 +68,6 @@ typedef struct VirtIOBalloon {
int64_t stats_last_update;
int64_t stats_poll_interval;
uint32_t host_features;
- PartiallyBalloonedPage *pbp;
bool qemu_4_0_config_size;
} VirtIOBalloon;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 53/97] virtio-balloon: don't track subpages for the PBP
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (51 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 52/97] virtio-balloon: Use temporary PBP only Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 54/97] virtio-balloon: free pbp more aggressively Michael Roth
` (48 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Michael S . Tsirkin, qemu-stable, David Hildenbrand
From: David Hildenbrand <david@redhat.com>
As ramblocks cannot get removed/readded while we are processing a bulk
of inflation requests, there is no more need to track the page size
in form of the number of subpages.
Suggested-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20190725113638.4702-8-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 9a7ca8a7c920360db9dcaf616ca6f1440c025043)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index bd54e302de..4a5a22ca3d 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -36,7 +36,6 @@
typedef struct PartiallyBalloonedPage {
ram_addr_t base_gpa;
- long subpages;
unsigned long *bitmap;
} PartiallyBalloonedPage;
@@ -55,16 +54,15 @@ static PartiallyBalloonedPage *virtio_balloon_pbp_alloc(ram_addr_t base_gpa,
PartiallyBalloonedPage *pbp = g_new0(PartiallyBalloonedPage, 1);
pbp->base_gpa = base_gpa;
- pbp->subpages = subpages;
pbp->bitmap = bitmap_new(subpages);
return pbp;
}
static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
- ram_addr_t base_gpa, long subpages)
+ ram_addr_t base_gpa)
{
- return pbp->subpages == subpages && pbp->base_gpa == base_gpa;
+ return pbp->base_gpa == base_gpa;
}
static void balloon_inflate_page(VirtIOBalloon *balloon,
@@ -106,7 +104,7 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
base_gpa = memory_region_get_ram_addr(mr) + mr_offset -
(rb_offset - rb_aligned_offset);
- if (*pbp && !virtio_balloon_pbp_matches(*pbp, base_gpa, subpages)) {
+ if (*pbp && !virtio_balloon_pbp_matches(*pbp, base_gpa)) {
/* We've partially ballooned part of a host page, but now
* we're trying to balloon part of a different one. Too hard,
* give up on the old partial page */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 54/97] virtio-balloon: free pbp more aggressively
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (52 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 53/97] virtio-balloon: don't track subpages for the PBP Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 55/97] i386/acpi: fix gint overflow in crs_range_compare Michael Roth
` (47 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: David Hildenbrand, qemu-stable, Michael S. Tsirkin
From: "Michael S. Tsirkin" <mst@redhat.com>
Previous patches switched to a temporary pbp but that does not go far
enough: after device uses a buffer, guest is free to reuse it, so
tracking the page and freeing it later is wrong.
Free and reset the pbp after we push each element.
Fixes: ed48c59875b6 ("virtio-balloon: Safely handle BALLOON_PAGE_SIZE < host page size")
Cc: qemu-stable@nongnu.org #v4.0.0
Cc: David Hildenbrand <david@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 1b47b37c33ec01ae1efc527f4c97f97f93723bc4)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/virtio/virtio-balloon.c | 37 ++++++++++++++++---------------------
1 file changed, 16 insertions(+), 21 deletions(-)
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 4a5a22ca3d..adde97fe4b 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -41,22 +41,19 @@ typedef struct PartiallyBalloonedPage {
static void virtio_balloon_pbp_free(PartiallyBalloonedPage *pbp)
{
- if (!pbp) {
+ if (!pbp->bitmap) {
return;
}
g_free(pbp->bitmap);
- g_free(pbp);
+ pbp->bitmap = NULL;
}
-static PartiallyBalloonedPage *virtio_balloon_pbp_alloc(ram_addr_t base_gpa,
- long subpages)
+static void virtio_balloon_pbp_alloc(PartiallyBalloonedPage *pbp,
+ ram_addr_t base_gpa,
+ long subpages)
{
- PartiallyBalloonedPage *pbp = g_new0(PartiallyBalloonedPage, 1);
-
pbp->base_gpa = base_gpa;
pbp->bitmap = bitmap_new(subpages);
-
- return pbp;
}
static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
@@ -67,7 +64,7 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp,
static void balloon_inflate_page(VirtIOBalloon *balloon,
MemoryRegion *mr, hwaddr mr_offset,
- PartiallyBalloonedPage **pbp)
+ PartiallyBalloonedPage *pbp)
{
void *addr = memory_region_get_ram_ptr(mr) + mr_offset;
ram_addr_t rb_offset, rb_aligned_offset, base_gpa;
@@ -104,22 +101,21 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
base_gpa = memory_region_get_ram_addr(mr) + mr_offset -
(rb_offset - rb_aligned_offset);
- if (*pbp && !virtio_balloon_pbp_matches(*pbp, base_gpa)) {
+ if (pbp->bitmap && !virtio_balloon_pbp_matches(pbp, base_gpa)) {
/* We've partially ballooned part of a host page, but now
* we're trying to balloon part of a different one. Too hard,
* give up on the old partial page */
- virtio_balloon_pbp_free(*pbp);
- *pbp = NULL;
+ virtio_balloon_pbp_free(pbp);
}
- if (!*pbp) {
- *pbp = virtio_balloon_pbp_alloc(base_gpa, subpages);
+ if (!pbp->bitmap) {
+ virtio_balloon_pbp_alloc(pbp, base_gpa, subpages);
}
set_bit((rb_offset - rb_aligned_offset) / BALLOON_PAGE_SIZE,
- (*pbp)->bitmap);
+ pbp->bitmap);
- if (bitmap_full((*pbp)->bitmap, subpages)) {
+ if (bitmap_full(pbp->bitmap, subpages)) {
/* We've accumulated a full host page, we can actually discard
* it now */
@@ -127,8 +123,7 @@ static void balloon_inflate_page(VirtIOBalloon *balloon,
/* We ignore errors from ram_block_discard_range(), because it
* has already reported them, and failing to discard a balloon
* page is not fatal */
- virtio_balloon_pbp_free(*pbp);
- *pbp = NULL;
+ virtio_balloon_pbp_free(pbp);
}
}
@@ -328,13 +323,14 @@ static void balloon_stats_set_poll_interval(Object *obj, Visitor *v,
static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
{
VirtIOBalloon *s = VIRTIO_BALLOON(vdev);
- PartiallyBalloonedPage *pbp = NULL;
VirtQueueElement *elem;
MemoryRegionSection section;
for (;;) {
+ PartiallyBalloonedPage pbp = {};
size_t offset = 0;
uint32_t pfn;
+
elem = virtqueue_pop(vq, sizeof(VirtQueueElement));
if (!elem) {
break;
@@ -379,9 +375,8 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq)
virtqueue_push(vq, elem, offset);
virtio_notify(vdev, vq);
g_free(elem);
+ virtio_balloon_pbp_free(&pbp);
}
-
- virtio_balloon_pbp_free(pbp);
}
static void virtio_balloon_receive_stats(VirtIODevice *vdev, VirtQueue *vq)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 55/97] i386/acpi: fix gint overflow in crs_range_compare
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (53 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 54/97] virtio-balloon: free pbp more aggressively Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 56/97] tpm: Exit in reset when backend indicates failure Michael Roth
` (46 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Evgeny Yakovlev
From: Evgeny Yakovlev <wrfsh@yandex-team.ru>
When very large regions (32GB sized in our case, PCI pass-through of GPUs)
are compared substraction result does not fit into gint.
As a result crs_replace_with_free_ranges does not get sorted ranges and
incorrectly computes PCI64 free space regions. Which then makes linux
guest complain about device and PCI64 hole intersection and device
becomes unusable.
Fix that by returning exactly fitting ranges.
Also fix indentation of an entire crs_replace_with_free_ranges to make
checkpatch happy.
Cc: qemu-stable@nongnu.org
Signed-off-by: Evgeny Yakovlev <wrfsh@yandex-team.ru>
Message-Id: <1563466463-26012-1-git-send-email-wrfsh@yandex-team.ru>
Signed-off-by: Evgeny Yakovlev <wrfsh@yandex-team.ru>
(cherry picked from commit 21e2acd583126db94f6d881005cd58e835160582)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/i386/acpi-build.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index ede27ab3c4..bf59c475be 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -743,10 +743,16 @@ static void crs_range_set_free(CrsRangeSet *range_set)
static gint crs_range_compare(gconstpointer a, gconstpointer b)
{
- CrsRangeEntry *entry_a = *(CrsRangeEntry **)a;
- CrsRangeEntry *entry_b = *(CrsRangeEntry **)b;
+ CrsRangeEntry *entry_a = *(CrsRangeEntry **)a;
+ CrsRangeEntry *entry_b = *(CrsRangeEntry **)b;
- return (int64_t)entry_a->base - (int64_t)entry_b->base;
+ if (entry_a->base < entry_b->base) {
+ return -1;
+ } else if (entry_a->base > entry_b->base) {
+ return 1;
+ } else {
+ return 0;
+ }
}
/*
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 56/97] tpm: Exit in reset when backend indicates failure
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (54 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 55/97] i386/acpi: fix gint overflow in crs_range_compare Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 57/97] tpm_emulator: Translate TPM error codes to strings Michael Roth
` (45 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Berger, qemu-stable, Stefan Berger
From: Stefan Berger <stefanb@linux.vnet.ibm.com>
Exit() in the frontend reset function when the backend indicates
intialization failure.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
(cherry picked from commit bcfd16fe26d6bb6eabfd2dfb46b9fda59d5493db)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/tpm/tpm_crb.c | 4 +++-
hw/tpm/tpm_tis.c | 4 +++-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/hw/tpm/tpm_crb.c b/hw/tpm/tpm_crb.c
index 3087acc4ab..55ea74f3ba 100644
--- a/hw/tpm/tpm_crb.c
+++ b/hw/tpm/tpm_crb.c
@@ -273,7 +273,9 @@ static void tpm_crb_reset(void *dev)
s->be_buffer_size = MIN(tpm_backend_get_buffer_size(s->tpmbe),
CRB_CTRL_CMD_SIZE);
- tpm_backend_startup_tpm(s->tpmbe, s->be_buffer_size);
+ if (tpm_backend_startup_tpm(s->tpmbe, s->be_buffer_size) < 0) {
+ exit(1);
+ }
}
static void tpm_crb_realize(DeviceState *dev, Error **errp)
diff --git a/hw/tpm/tpm_tis.c b/hw/tpm/tpm_tis.c
index fd183e8deb..4203da1008 100644
--- a/hw/tpm/tpm_tis.c
+++ b/hw/tpm/tpm_tis.c
@@ -909,7 +909,9 @@ static void tpm_tis_reset(DeviceState *dev)
s->rw_offset = 0;
}
- tpm_backend_startup_tpm(s->be_driver, s->be_buffer_size);
+ if (tpm_backend_startup_tpm(s->be_driver, s->be_buffer_size) < 0) {
+ exit(1);
+ }
}
/* persistent state handling */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 57/97] tpm_emulator: Translate TPM error codes to strings
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (55 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 56/97] tpm: Exit in reset when backend indicates failure Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 58/97] block/backup: simplify backup_incremental_init_copy_bitmap Michael Roth
` (44 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Berger, qemu-stable, Stefan Berger
From: Stefan Berger <stefanb@linux.vnet.ibm.com>
Implement a function to translate TPM error codes to strings so that
at least the most common error codes can be translated to human
readable strings.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
(cherry picked from commit 7e095e84ba0b7c0a1ac45bc6824dace2fd352e56)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/tpm/tpm_emulator.c | 60 +++++++++++++++++++++++++++++++++++--------
hw/tpm/tpm_int.h | 13 ++++++++++
2 files changed, 63 insertions(+), 10 deletions(-)
diff --git a/hw/tpm/tpm_emulator.c b/hw/tpm/tpm_emulator.c
index 70f4b10284..58894dc320 100644
--- a/hw/tpm/tpm_emulator.c
+++ b/hw/tpm/tpm_emulator.c
@@ -81,6 +81,40 @@ typedef struct TPMEmulator {
TPMBlobBuffers state_blobs;
} TPMEmulator;
+struct tpm_error {
+ uint32_t tpm_result;
+ const char *string;
+};
+
+static const struct tpm_error tpm_errors[] = {
+ /* TPM 1.2 error codes */
+ { TPM_BAD_PARAMETER , "a parameter is bad" },
+ { TPM_FAIL , "operation failed" },
+ { TPM_KEYNOTFOUND , "key could not be found" },
+ { TPM_BAD_PARAM_SIZE , "bad parameter size"},
+ { TPM_ENCRYPT_ERROR , "encryption error" },
+ { TPM_DECRYPT_ERROR , "decryption error" },
+ { TPM_BAD_KEY_PROPERTY, "bad key property" },
+ { TPM_BAD_MODE , "bad (encryption) mode" },
+ { TPM_BAD_VERSION , "bad version identifier" },
+ { TPM_BAD_LOCALITY , "bad locality" },
+ /* TPM 2 error codes */
+ { TPM_RC_FAILURE , "operation failed" },
+ { TPM_RC_LOCALITY , "bad locality" },
+ { TPM_RC_INSUFFICIENT, "insufficient amount of data" },
+};
+
+static const char *tpm_emulator_strerror(uint32_t tpm_result)
+{
+ size_t i;
+
+ for (i = 0; i < ARRAY_SIZE(tpm_errors); i++) {
+ if (tpm_errors[i].tpm_result == tpm_result) {
+ return tpm_errors[i].string;
+ }
+ }
+ return "";
+}
static int tpm_emulator_ctrlcmd(TPMEmulator *tpm, unsigned long cmd, void *msg,
size_t msg_len_in, size_t msg_len_out)
@@ -263,7 +297,8 @@ static int tpm_emulator_stop_tpm(TPMBackend *tb)
res = be32_to_cpu(res);
if (res) {
- error_report("tpm-emulator: TPM result for CMD_STOP: 0x%x", res);
+ error_report("tpm-emulator: TPM result for CMD_STOP: 0x%x %s", res,
+ tpm_emulator_strerror(res));
return -1;
}
@@ -292,8 +327,9 @@ static int tpm_emulator_set_buffer_size(TPMBackend *tb,
psbs.u.resp.tpm_result = be32_to_cpu(psbs.u.resp.tpm_result);
if (psbs.u.resp.tpm_result != 0) {
- error_report("tpm-emulator: TPM result for set buffer size : 0x%x",
- psbs.u.resp.tpm_result);
+ error_report("tpm-emulator: TPM result for set buffer size : 0x%x %s",
+ psbs.u.resp.tpm_result,
+ tpm_emulator_strerror(psbs.u.resp.tpm_result));
return -1;
}
@@ -338,7 +374,8 @@ static int tpm_emulator_startup_tpm_resume(TPMBackend *tb, size_t buffersize,
res = be32_to_cpu(init.u.resp.tpm_result);
if (res) {
- error_report("tpm-emulator: TPM result for CMD_INIT: 0x%x", res);
+ error_report("tpm-emulator: TPM result for CMD_INIT: 0x%x %s", res,
+ tpm_emulator_strerror(res));
goto err_exit;
}
return 0;
@@ -398,8 +435,9 @@ static int tpm_emulator_reset_tpm_established_flag(TPMBackend *tb,
res = be32_to_cpu(reset_est.u.resp.tpm_result);
if (res) {
- error_report("tpm-emulator: TPM result for rest establixhed flag: 0x%x",
- res);
+ error_report(
+ "tpm-emulator: TPM result for rest established flag: 0x%x %s",
+ res, tpm_emulator_strerror(res));
return -1;
}
@@ -637,7 +675,8 @@ static int tpm_emulator_get_state_blob(TPMEmulator *tpm_emu,
res = be32_to_cpu(pgs.u.resp.tpm_result);
if (res != 0 && (res & 0x800) == 0) {
error_report("tpm-emulator: Getting the stateblob (type %d) failed "
- "with a TPM error 0x%x", type, res);
+ "with a TPM error 0x%x %s", type, res,
+ tpm_emulator_strerror(res));
return -1;
}
@@ -757,7 +796,8 @@ static int tpm_emulator_set_state_blob(TPMEmulator *tpm_emu,
tpm_result = be32_to_cpu(pss.u.resp.tpm_result);
if (tpm_result != 0) {
error_report("tpm-emulator: Setting the stateblob (type %d) failed "
- "with a TPM error 0x%x", type, tpm_result);
+ "with a TPM error 0x%x %s", type, tpm_result,
+ tpm_emulator_strerror(tpm_result));
return -1;
}
@@ -887,8 +927,8 @@ static void tpm_emulator_shutdown(TPMEmulator *tpm_emu)
error_report("tpm-emulator: Could not cleanly shutdown the TPM: %s",
strerror(errno));
} else if (res != 0) {
- error_report("tpm-emulator: TPM result for sutdown: 0x%x",
- be32_to_cpu(res));
+ error_report("tpm-emulator: TPM result for shutdown: 0x%x %s",
+ be32_to_cpu(res), tpm_emulator_strerror(be32_to_cpu(res)));
}
}
diff --git a/hw/tpm/tpm_int.h b/hw/tpm/tpm_int.h
index a4c77fbd7e..3fb28a9d6c 100644
--- a/hw/tpm/tpm_int.h
+++ b/hw/tpm/tpm_int.h
@@ -39,7 +39,16 @@ struct tpm_resp_hdr {
#define TPM_TAG_RSP_AUTH1_COMMAND 0xc5
#define TPM_TAG_RSP_AUTH2_COMMAND 0xc6
+#define TPM_BAD_PARAMETER 3
#define TPM_FAIL 9
+#define TPM_KEYNOTFOUND 13
+#define TPM_BAD_PARAM_SIZE 25
+#define TPM_ENCRYPT_ERROR 32
+#define TPM_DECRYPT_ERROR 33
+#define TPM_BAD_KEY_PROPERTY 40
+#define TPM_BAD_MODE 44
+#define TPM_BAD_VERSION 46
+#define TPM_BAD_LOCALITY 61
#define TPM_ORD_ContinueSelfTest 0x53
#define TPM_ORD_GetTicks 0xf1
@@ -59,4 +68,8 @@ struct tpm_resp_hdr {
#define TPM2_PT_MAX_COMMAND_SIZE 0x11e
+#define TPM_RC_INSUFFICIENT 0x9a
+#define TPM_RC_FAILURE 0x101
+#define TPM_RC_LOCALITY 0x907
+
#endif /* TPM_TPM_INT_H */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 58/97] block/backup: simplify backup_incremental_init_copy_bitmap
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (56 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 57/97] tpm_emulator: Translate TPM error codes to strings Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 59/97] block/backup: move to copy_bitmap with granularity Michael Roth
` (43 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Simplify backup_incremental_init_copy_bitmap using the function
bdrv_dirty_bitmap_next_dirty_area.
Note: move to job->len instead of bitmap size: it should not matter but
less code.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190429090842.57910-2-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit c2da3413c021398152e98022261bb1643276a2fe)
*prereq for 4a5b91ca
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 40 ++++++++++++----------------------------
1 file changed, 12 insertions(+), 28 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index 9988753249..d9f5db18ac 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -403,43 +403,27 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
/* init copy_bitmap from sync_bitmap */
static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
{
- BdrvDirtyBitmapIter *dbi;
- int64_t offset;
- int64_t end = DIV_ROUND_UP(bdrv_dirty_bitmap_size(job->sync_bitmap),
- job->cluster_size);
-
- dbi = bdrv_dirty_iter_new(job->sync_bitmap);
- while ((offset = bdrv_dirty_iter_next(dbi)) != -1) {
- int64_t cluster = offset / job->cluster_size;
- int64_t next_cluster;
-
- offset += bdrv_dirty_bitmap_granularity(job->sync_bitmap);
- if (offset >= bdrv_dirty_bitmap_size(job->sync_bitmap)) {
- hbitmap_set(job->copy_bitmap, cluster, end - cluster);
- break;
- }
+ uint64_t offset = 0;
+ uint64_t bytes = job->len;
- offset = bdrv_dirty_bitmap_next_zero(job->sync_bitmap, offset,
- UINT64_MAX);
- if (offset == -1) {
- hbitmap_set(job->copy_bitmap, cluster, end - cluster);
- break;
- }
+ while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap,
+ &offset, &bytes))
+ {
+ uint64_t cluster = offset / job->cluster_size;
+ uint64_t end_cluster = DIV_ROUND_UP(offset + bytes, job->cluster_size);
- next_cluster = DIV_ROUND_UP(offset, job->cluster_size);
- hbitmap_set(job->copy_bitmap, cluster, next_cluster - cluster);
- if (next_cluster >= end) {
+ hbitmap_set(job->copy_bitmap, cluster, end_cluster - cluster);
+
+ offset = end_cluster * job->cluster_size;
+ if (offset >= job->len) {
break;
}
-
- bdrv_set_dirty_iter(dbi, next_cluster * job->cluster_size);
+ bytes = job->len - offset;
}
/* TODO job_progress_set_remaining() would make more sense */
job_progress_update(&job->common.job,
job->len - hbitmap_count(job->copy_bitmap) * job->cluster_size);
-
- bdrv_dirty_iter_free(dbi);
}
static int coroutine_fn backup_run(Job *job, Error **errp)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 59/97] block/backup: move to copy_bitmap with granularity
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (57 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 58/97] block/backup: simplify backup_incremental_init_copy_bitmap Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 60/97] block/backup: refactor and tolerate unallocated cluster skipping Michael Roth
` (42 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
We are going to share this bitmap between backup and backup-top filter
driver, so let's share something more meaningful. It also simplifies
some calculations.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190429090842.57910-3-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit a8389e315ef71913ae99cf8f3b1f89e84631f599)
*prereq for 4a5b91ca
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 48 +++++++++++++++++++++++-------------------------
1 file changed, 23 insertions(+), 25 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index d9f5db18ac..510fc54f98 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -113,7 +113,8 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,
int read_flags = is_write_notifier ? BDRV_REQ_NO_SERIALISING : 0;
int write_flags = job->serialize_target_writes ? BDRV_REQ_SERIALISING : 0;
- hbitmap_reset(job->copy_bitmap, start / job->cluster_size, 1);
+ assert(QEMU_IS_ALIGNED(start, job->cluster_size));
+ hbitmap_reset(job->copy_bitmap, start, job->cluster_size);
nbytes = MIN(job->cluster_size, job->len - start);
if (!*bounce_buffer) {
*bounce_buffer = blk_blockalign(blk, job->cluster_size);
@@ -147,7 +148,7 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,
return nbytes;
fail:
- hbitmap_set(job->copy_bitmap, start / job->cluster_size, 1);
+ hbitmap_set(job->copy_bitmap, start, job->cluster_size);
return ret;
}
@@ -167,16 +168,15 @@ static int coroutine_fn backup_cow_with_offload(BackupBlockJob *job,
int write_flags = job->serialize_target_writes ? BDRV_REQ_SERIALISING : 0;
assert(QEMU_IS_ALIGNED(job->copy_range_size, job->cluster_size));
+ assert(QEMU_IS_ALIGNED(start, job->cluster_size));
nbytes = MIN(job->copy_range_size, end - start);
nr_clusters = DIV_ROUND_UP(nbytes, job->cluster_size);
- hbitmap_reset(job->copy_bitmap, start / job->cluster_size,
- nr_clusters);
+ hbitmap_reset(job->copy_bitmap, start, job->cluster_size * nr_clusters);
ret = blk_co_copy_range(blk, start, job->target, start, nbytes,
read_flags, write_flags);
if (ret < 0) {
trace_backup_do_cow_copy_range_fail(job, start, ret);
- hbitmap_set(job->copy_bitmap, start / job->cluster_size,
- nr_clusters);
+ hbitmap_set(job->copy_bitmap, start, job->cluster_size * nr_clusters);
return ret;
}
@@ -204,7 +204,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
cow_request_begin(&cow_request, job, start, end);
while (start < end) {
- if (!hbitmap_get(job->copy_bitmap, start / job->cluster_size)) {
+ if (!hbitmap_get(job->copy_bitmap, start)) {
trace_backup_do_cow_skip(job, start);
start += job->cluster_size;
continue; /* already copied */
@@ -300,6 +300,11 @@ static void backup_clean(Job *job)
assert(s->target);
blk_unref(s->target);
s->target = NULL;
+
+ if (s->copy_bitmap) {
+ hbitmap_free(s->copy_bitmap);
+ s->copy_bitmap = NULL;
+ }
}
static void backup_attached_aio_context(BlockJob *job, AioContext *aio_context)
@@ -312,7 +317,6 @@ static void backup_attached_aio_context(BlockJob *job, AioContext *aio_context)
void backup_do_checkpoint(BlockJob *job, Error **errp)
{
BackupBlockJob *backup_job = container_of(job, BackupBlockJob, common);
- int64_t len;
assert(block_job_driver(job) == &backup_job_driver);
@@ -322,8 +326,7 @@ void backup_do_checkpoint(BlockJob *job, Error **errp)
return;
}
- len = DIV_ROUND_UP(backup_job->len, backup_job->cluster_size);
- hbitmap_set(backup_job->copy_bitmap, 0, len);
+ hbitmap_set(backup_job->copy_bitmap, 0, backup_job->len);
}
static void backup_drain(BlockJob *job)
@@ -378,16 +381,16 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
{
int ret;
bool error_is_read;
- int64_t cluster;
+ int64_t offset;
HBitmapIter hbi;
hbitmap_iter_init(&hbi, job->copy_bitmap, 0);
- while ((cluster = hbitmap_iter_next(&hbi)) != -1) {
+ while ((offset = hbitmap_iter_next(&hbi)) != -1) {
do {
if (yield_and_check(job)) {
return 0;
}
- ret = backup_do_cow(job, cluster * job->cluster_size,
+ ret = backup_do_cow(job, offset,
job->cluster_size, &error_is_read, false);
if (ret < 0 && backup_error_action(job, error_is_read, -ret) ==
BLOCK_ERROR_ACTION_REPORT)
@@ -409,12 +412,9 @@ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap,
&offset, &bytes))
{
- uint64_t cluster = offset / job->cluster_size;
- uint64_t end_cluster = DIV_ROUND_UP(offset + bytes, job->cluster_size);
+ hbitmap_set(job->copy_bitmap, offset, bytes);
- hbitmap_set(job->copy_bitmap, cluster, end_cluster - cluster);
-
- offset = end_cluster * job->cluster_size;
+ offset += bytes;
if (offset >= job->len) {
break;
}
@@ -423,30 +423,27 @@ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
/* TODO job_progress_set_remaining() would make more sense */
job_progress_update(&job->common.job,
- job->len - hbitmap_count(job->copy_bitmap) * job->cluster_size);
+ job->len - hbitmap_count(job->copy_bitmap));
}
static int coroutine_fn backup_run(Job *job, Error **errp)
{
BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
BlockDriverState *bs = blk_bs(s->common.blk);
- int64_t offset, nb_clusters;
+ int64_t offset;
int ret = 0;
QLIST_INIT(&s->inflight_reqs);
qemu_co_rwlock_init(&s->flush_rwlock);
- nb_clusters = DIV_ROUND_UP(s->len, s->cluster_size);
job_progress_set_remaining(job, s->len);
- s->copy_bitmap = hbitmap_alloc(nb_clusters, 0);
if (s->sync_mode == MIRROR_SYNC_MODE_INCREMENTAL) {
backup_incremental_init_copy_bitmap(s);
} else {
- hbitmap_set(s->copy_bitmap, 0, nb_clusters);
+ hbitmap_set(s->copy_bitmap, 0, s->len);
}
-
s->before_write.notify = backup_before_write_notify;
bdrv_add_before_write_notifier(bs, &s->before_write);
@@ -527,7 +524,6 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
/* wait until pending backup_do_cow() calls have completed */
qemu_co_rwlock_wrlock(&s->flush_rwlock);
qemu_co_rwlock_unlock(&s->flush_rwlock);
- hbitmap_free(s->copy_bitmap);
return ret;
}
@@ -678,6 +674,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
} else {
job->cluster_size = MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size);
}
+
+ job->copy_bitmap = hbitmap_alloc(len, ctz32(job->cluster_size));
job->use_copy_range = true;
job->copy_range_size = MIN_NON_ZERO(blk_get_max_transfer(job->common.blk),
blk_get_max_transfer(job->target));
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 60/97] block/backup: refactor and tolerate unallocated cluster skipping
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (58 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 59/97] block/backup: move to copy_bitmap with granularity Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 61/97] block/backup: unify different modes code path Michael Roth
` (41 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Split allocation checking to separate function and reduce nesting.
Consider bdrv_is_allocated() fail as allocated area, as copying more
than needed is not wrong (and we do it anyway) and seems better than
fail the whole job. And, most probably we will fail on the next read,
if there are real problem with source.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190429090842.57910-4-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 9eb5a248f3e50c1f034bc6ff4b2f25c8c56515a5)
*prereq for 110571be4e
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 60 +++++++++++++++++++-------------------------------
1 file changed, 23 insertions(+), 37 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index 510fc54f98..298e85f1a9 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -377,6 +377,22 @@ static bool coroutine_fn yield_and_check(BackupBlockJob *job)
return false;
}
+static bool bdrv_is_unallocated_range(BlockDriverState *bs,
+ int64_t offset, int64_t bytes)
+{
+ int64_t end = offset + bytes;
+
+ while (offset < end && !bdrv_is_allocated(bs, offset, bytes, &bytes)) {
+ if (bytes == 0) {
+ return true;
+ }
+ offset += bytes;
+ bytes = end - offset;
+ }
+
+ return offset >= end;
+}
+
static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
{
int ret;
@@ -462,49 +478,19 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
for (offset = 0; offset < s->len;
offset += s->cluster_size) {
bool error_is_read;
- int alloced = 0;
if (yield_and_check(s)) {
break;
}
- if (s->sync_mode == MIRROR_SYNC_MODE_TOP) {
- int i;
- int64_t n;
-
- /* Check to see if these blocks are already in the
- * backing file. */
-
- for (i = 0; i < s->cluster_size;) {
- /* bdrv_is_allocated() only returns true/false based
- * on the first set of sectors it comes across that
- * are are all in the same state.
- * For that reason we must verify each sector in the
- * backup cluster length. We end up copying more than
- * needed but at some point that is always the case. */
- alloced =
- bdrv_is_allocated(bs, offset + i,
- s->cluster_size - i, &n);
- i += n;
-
- if (alloced || n == 0) {
- break;
- }
- }
-
- /* If the above loop never found any sectors that are in
- * the topmost image, skip this backup. */
- if (alloced == 0) {
- continue;
- }
- }
- /* FULL sync mode we copy the whole drive. */
- if (alloced < 0) {
- ret = alloced;
- } else {
- ret = backup_do_cow(s, offset, s->cluster_size,
- &error_is_read, false);
+ if (s->sync_mode == MIRROR_SYNC_MODE_TOP &&
+ bdrv_is_unallocated_range(bs, offset, s->cluster_size))
+ {
+ continue;
}
+
+ ret = backup_do_cow(s, offset, s->cluster_size,
+ &error_is_read, false);
if (ret < 0) {
/* Depending on error action, fail now or retry cluster */
BlockErrorAction action =
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 61/97] block/backup: unify different modes code path
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (59 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 60/97] block/backup: refactor and tolerate unallocated cluster skipping Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 62/97] block/backup: refactor: split out backup_calculate_cluster_size Michael Roth
` (40 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Do full, top and incremental mode copying all in one place. This
unifies the code path and helps further improvements.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190429090842.57910-5-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit c334e897d08eea1f5a3a95f6a2208afe6757c103)
*prereq for 110571be4e
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 43 ++++++++++---------------------------------
1 file changed, 10 insertions(+), 33 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index 298e85f1a9..b54386b699 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -393,15 +393,23 @@ static bool bdrv_is_unallocated_range(BlockDriverState *bs,
return offset >= end;
}
-static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
+static int coroutine_fn backup_loop(BackupBlockJob *job)
{
int ret;
bool error_is_read;
int64_t offset;
HBitmapIter hbi;
+ BlockDriverState *bs = blk_bs(job->common.blk);
hbitmap_iter_init(&hbi, job->copy_bitmap, 0);
while ((offset = hbitmap_iter_next(&hbi)) != -1) {
+ if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
+ bdrv_is_unallocated_range(bs, offset, job->cluster_size))
+ {
+ hbitmap_reset(job->copy_bitmap, offset, job->cluster_size);
+ continue;
+ }
+
do {
if (yield_and_check(job)) {
return 0;
@@ -446,7 +454,6 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
{
BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
BlockDriverState *bs = blk_bs(s->common.blk);
- int64_t offset;
int ret = 0;
QLIST_INIT(&s->inflight_reqs);
@@ -471,38 +478,8 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
* notify callback service CoW requests. */
job_yield(job);
}
- } else if (s->sync_mode == MIRROR_SYNC_MODE_INCREMENTAL) {
- ret = backup_run_incremental(s);
} else {
- /* Both FULL and TOP SYNC_MODE's require copying.. */
- for (offset = 0; offset < s->len;
- offset += s->cluster_size) {
- bool error_is_read;
-
- if (yield_and_check(s)) {
- break;
- }
-
- if (s->sync_mode == MIRROR_SYNC_MODE_TOP &&
- bdrv_is_unallocated_range(bs, offset, s->cluster_size))
- {
- continue;
- }
-
- ret = backup_do_cow(s, offset, s->cluster_size,
- &error_is_read, false);
- if (ret < 0) {
- /* Depending on error action, fail now or retry cluster */
- BlockErrorAction action =
- backup_error_action(s, error_is_read, -ret);
- if (action == BLOCK_ERROR_ACTION_REPORT) {
- break;
- } else {
- offset -= s->cluster_size;
- continue;
- }
- }
- }
+ ret = backup_loop(s);
}
notifier_with_return_remove(&s->before_write);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 62/97] block/backup: refactor: split out backup_calculate_cluster_size
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (60 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 61/97] block/backup: unify different modes code path Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 63/97] backup: Copy only dirty areas Michael Roth
` (39 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Split out cluster_size calculation. Move copy-bitmap creation above
block-job creation, as we are going to share it with upcoming
backup-top filter, which also should be created before actual block job
creation.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190429090842.57910-6-vsementsov@virtuozzo.com
[mreitz: Dropped a paragraph from the commit message that was left over
from a previous version]
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit ae6b12fa4cf7d54add35531c790aaf2bd6d833f3)
*prereq for 110571be4e
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 82 ++++++++++++++++++++++++++++++++------------------
1 file changed, 52 insertions(+), 30 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index b54386b699..d1b94a6dbe 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -507,6 +507,42 @@ static const BlockJobDriver backup_job_driver = {
.drain = backup_drain,
};
+static int64_t backup_calculate_cluster_size(BlockDriverState *target,
+ Error **errp)
+{
+ int ret;
+ BlockDriverInfo bdi;
+
+ /*
+ * If there is no backing file on the target, we cannot rely on COW if our
+ * backup cluster size is smaller than the target cluster size. Even for
+ * targets with a backing file, try to avoid COW if possible.
+ */
+ ret = bdrv_get_info(target, &bdi);
+ if (ret == -ENOTSUP && !target->backing) {
+ /* Cluster size is not defined */
+ warn_report("The target block device doesn't provide "
+ "information about the block size and it doesn't have a "
+ "backing file. The default block size of %u bytes is "
+ "used. If the actual block size of the target exceeds "
+ "this default, the backup may be unusable",
+ BACKUP_CLUSTER_SIZE_DEFAULT);
+ return BACKUP_CLUSTER_SIZE_DEFAULT;
+ } else if (ret < 0 && !target->backing) {
+ error_setg_errno(errp, -ret,
+ "Couldn't determine the cluster size of the target image, "
+ "which has no backing file");
+ error_append_hint(errp,
+ "Aborting, since this may create an unusable destination image\n");
+ return ret;
+ } else if (ret < 0 && target->backing) {
+ /* Not fatal; just trudge on ahead. */
+ return BACKUP_CLUSTER_SIZE_DEFAULT;
+ }
+
+ return MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size);
+}
+
BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
BlockDriverState *target, int64_t speed,
MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap,
@@ -518,9 +554,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
JobTxn *txn, Error **errp)
{
int64_t len;
- BlockDriverInfo bdi;
BackupBlockJob *job = NULL;
int ret;
+ int64_t cluster_size;
+ HBitmap *copy_bitmap = NULL;
assert(bs);
assert(target);
@@ -582,6 +619,13 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
goto error;
}
+ cluster_size = backup_calculate_cluster_size(target, errp);
+ if (cluster_size < 0) {
+ goto error;
+ }
+
+ copy_bitmap = hbitmap_alloc(len, ctz32(cluster_size));
+
/* job->len is fixed, so we can't allow resize */
job = block_job_create(job_id, &backup_job_driver, txn, bs,
BLK_PERM_CONSISTENT_READ,
@@ -610,35 +654,9 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
/* Detect image-fleecing (and similar) schemes */
job->serialize_target_writes = bdrv_chain_contains(target, bs);
-
- /* If there is no backing file on the target, we cannot rely on COW if our
- * backup cluster size is smaller than the target cluster size. Even for
- * targets with a backing file, try to avoid COW if possible. */
- ret = bdrv_get_info(target, &bdi);
- if (ret == -ENOTSUP && !target->backing) {
- /* Cluster size is not defined */
- warn_report("The target block device doesn't provide "
- "information about the block size and it doesn't have a "
- "backing file. The default block size of %u bytes is "
- "used. If the actual block size of the target exceeds "
- "this default, the backup may be unusable",
- BACKUP_CLUSTER_SIZE_DEFAULT);
- job->cluster_size = BACKUP_CLUSTER_SIZE_DEFAULT;
- } else if (ret < 0 && !target->backing) {
- error_setg_errno(errp, -ret,
- "Couldn't determine the cluster size of the target image, "
- "which has no backing file");
- error_append_hint(errp,
- "Aborting, since this may create an unusable destination image\n");
- goto error;
- } else if (ret < 0 && target->backing) {
- /* Not fatal; just trudge on ahead. */
- job->cluster_size = BACKUP_CLUSTER_SIZE_DEFAULT;
- } else {
- job->cluster_size = MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size);
- }
-
- job->copy_bitmap = hbitmap_alloc(len, ctz32(job->cluster_size));
+ job->cluster_size = cluster_size;
+ job->copy_bitmap = copy_bitmap;
+ copy_bitmap = NULL;
job->use_copy_range = true;
job->copy_range_size = MIN_NON_ZERO(blk_get_max_transfer(job->common.blk),
blk_get_max_transfer(job->target));
@@ -654,6 +672,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
return &job->common;
error:
+ if (copy_bitmap) {
+ assert(!job || !job->copy_bitmap);
+ hbitmap_free(copy_bitmap);
+ }
if (sync_bitmap) {
bdrv_reclaim_dirty_bitmap(bs, sync_bitmap, NULL);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 63/97] backup: Copy only dirty areas
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (61 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 62/97] block/backup: refactor: split out backup_calculate_cluster_size Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 64/97] iotests: Test backup job with two guest writes Michael Roth
` (38 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
The backup job must only copy areas that the copy_bitmap reports as
dirty. This is always the case when using traditional non-offloading
backup, because it copies each cluster separately. When offloading the
copy operation, we sometimes copy more than one cluster at a time, but
we only check whether the first one is dirty.
Therefore, whenever copy offloading is possible, the backup job
currently produces wrong output when the guest writes to an area of
which an inner part has already been backed up, because that inner part
will be re-copied.
Fixes: 9ded4a0114968e98b41494fc035ba14f84cdf700
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190801173900.23851-2-mreitz@redhat.com
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 4a5b91ca024fc6fd87021c54655af76a35f2ef1e)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index d1b94a6dbe..f67c208cf0 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -204,22 +204,31 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
cow_request_begin(&cow_request, job, start, end);
while (start < end) {
+ int64_t dirty_end;
+
if (!hbitmap_get(job->copy_bitmap, start)) {
trace_backup_do_cow_skip(job, start);
start += job->cluster_size;
continue; /* already copied */
}
+ dirty_end = hbitmap_next_zero(job->copy_bitmap, start, (end - start));
+ if (dirty_end < 0) {
+ dirty_end = end;
+ }
+
trace_backup_do_cow_process(job, start);
if (job->use_copy_range) {
- ret = backup_cow_with_offload(job, start, end, is_write_notifier);
+ ret = backup_cow_with_offload(job, start, dirty_end,
+ is_write_notifier);
if (ret < 0) {
job->use_copy_range = false;
}
}
if (!job->use_copy_range) {
- ret = backup_cow_with_bounce_buffer(job, start, end, is_write_notifier,
+ ret = backup_cow_with_bounce_buffer(job, start, dirty_end,
+ is_write_notifier,
error_is_read, &bounce_buffer);
}
if (ret < 0) {
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 64/97] iotests: Test backup job with two guest writes
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (62 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 63/97] backup: Copy only dirty areas Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 65/97] util/hbitmap: update orig_size on truncate Michael Roth
` (37 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Perform two guest writes to not yet backed up areas of an image, where
the former touches an inner area of the latter.
Before HEAD^, copy offloading broke this in two ways:
(1) The target image differs from the reference image (what the source
was when the backup started).
(2) But you will not see that in the failing output, because the job
offset is reported as being greater than the job length. This is
because one cluster is copied twice, and thus accounted for twice,
but of course the job length does not increase.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190801173900.23851-3-mreitz@redhat.com
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 5f594a2e99f19ca0f7744d333bcd556f5976b78f)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/056 | 39 ++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/056.out | 4 ++--
2 files changed, 41 insertions(+), 2 deletions(-)
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 3df323984d..e5ac25127b 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase):
self.vm = iotests.VM()
self.test_img = img_create('test')
self.dest_img = img_create('dest')
+ self.ref_img = img_create('ref')
self.vm.add_drive(self.test_img)
self.vm.launch()
@@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase):
self.vm.shutdown()
try_remove(self.test_img)
try_remove(self.dest_img)
+ try_remove(self.ref_img)
def hmp_io_writes(self, drive, patterns):
for pattern in patterns:
@@ -177,6 +179,43 @@ class BackupTest(iotests.QMPTestCase):
self.assert_qmp(event, 'data/error', qerror)
return False
+ def test_overlapping_writes(self):
+ # Write something to back up
+ self.hmp_io_writes('drive0', [('42', '0M', '2M')])
+
+ # Create a reference backup
+ self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt,
+ sync='full', target=self.ref_img,
+ auto_dismiss=False)
+ res = self.vm.qmp('block-job-dismiss', id='drive0')
+ self.assert_qmp(res, 'return', {})
+
+ # Now to the test backup: We simulate the following guest
+ # writes:
+ # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that
+ # area should be in the target image, and we must not copy
+ # it again (because the source image has changed now)
+ # (64k is the job's cluster size)
+ # (2) [1M, 2M): The backup job must not get overeager. It
+ # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately,
+ # but not the area in between.
+
+ self.qmp_backup(device='drive0', format=iotests.imgfmt, sync='full',
+ target=self.dest_img, speed=1, auto_dismiss=False)
+
+ self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'),
+ ('66', '1M', '1M')])
+
+ # Let the job complete
+ res = self.vm.qmp('block-job-set-speed', device='drive0', speed=0)
+ self.assert_qmp(res, 'return', {})
+ self.qmp_backup_wait('drive0')
+ res = self.vm.qmp('block-job-dismiss', id='drive0')
+ self.assert_qmp(res, 'return', {})
+
+ self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img),
+ 'target image does not match reference image')
+
def test_dismiss_false(self):
res = self.vm.qmp('query-block-jobs')
self.assert_qmp(res, 'return', [])
diff --git a/tests/qemu-iotests/056.out b/tests/qemu-iotests/056.out
index dae404e278..36376bed87 100644
--- a/tests/qemu-iotests/056.out
+++ b/tests/qemu-iotests/056.out
@@ -1,5 +1,5 @@
-.........
+..........
----------------------------------------------------------------------
-Ran 9 tests
+Ran 10 tests
OK
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 65/97] util/hbitmap: update orig_size on truncate
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (63 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 64/97] iotests: Test backup job with two guest writes Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 66/97] iotests: Test incremental backup after truncation Michael Roth
` (36 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Without this, hbitmap_next_zero and hbitmap_next_dirty_area are broken
after truncate. So, orig_size is broken since it's introduction in
76d570dc495c56bb.
Fixes: 76d570dc495c56bb
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190805120120.23585-1-vsementsov@virtuozzo.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 4e4de222799634d8159ee7b9303b9e1b45c6be2c)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
util/hbitmap.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 7905212a8b..bcc0acdc6a 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -53,7 +53,9 @@
*/
struct HBitmap {
- /* Size of the bitmap, as requested in hbitmap_alloc. */
+ /*
+ * Size of the bitmap, as requested in hbitmap_alloc or in hbitmap_truncate.
+ */
uint64_t orig_size;
/* Number of total bits in the bottom level. */
@@ -732,6 +734,8 @@ void hbitmap_truncate(HBitmap *hb, uint64_t size)
uint64_t num_elements = size;
uint64_t old;
+ hb->orig_size = size;
+
/* Size comes in as logical elements, adjust for granularity. */
size = (size + (1ULL << hb->granularity) - 1) >> hb->granularity;
assert(size <= ((uint64_t)1 << HBITMAP_LOG_MAX_SIZE));
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 66/97] iotests: Test incremental backup after truncation
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (64 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 65/97] util/hbitmap: update orig_size on truncate Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 67/97] mirror: Only mirror granularity-aligned chunks Michael Roth
` (35 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190805152840.32190-1-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 8a9cb864086269af14bbd13f395472703cf99f8c)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/124 | 38 ++++++++++++++++++++++++++++++++++----
tests/qemu-iotests/124.out | 4 ++--
2 files changed, 36 insertions(+), 6 deletions(-)
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 80b356f7bb..3440f54781 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -212,25 +212,28 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
return bitmap
- def prepare_backup(self, bitmap=None, parent=None):
+ def prepare_backup(self, bitmap=None, parent=None, **kwargs):
if bitmap is None:
bitmap = self.bitmaps[-1]
if parent is None:
parent, _ = bitmap.last_target()
target, _ = bitmap.new_target()
- self.img_create(target, bitmap.drive['fmt'], parent=parent)
+ self.img_create(target, bitmap.drive['fmt'], parent=parent,
+ **kwargs)
return target
def create_incremental(self, bitmap=None, parent=None,
- parentFormat=None, validate=True):
+ parentFormat=None, validate=True,
+ target=None):
if bitmap is None:
bitmap = self.bitmaps[-1]
if parent is None:
parent, _ = bitmap.last_target()
- target = self.prepare_backup(bitmap, parent)
+ if target is None:
+ target = self.prepare_backup(bitmap, parent)
res = self.do_qmp_backup(job_id=bitmap.drive['id'],
device=bitmap.drive['id'],
sync='incremental', bitmap=bitmap.name,
@@ -572,6 +575,33 @@ class TestIncrementalBackup(TestIncrementalBackupBase):
'bitmap0', self.drives[0],
granularity=64000)
+ def test_growing_before_backup(self):
+ '''
+ Test: Add a bitmap, truncate the image, write past the old
+ end, do a backup.
+
+ Incremental backup should not ignore dirty bits past the old
+ image end.
+ '''
+ self.assert_no_active_block_jobs()
+
+ self.create_anchor_backup()
+
+ self.add_bitmap('bitmap0', self.drives[0])
+
+ res = self.vm.qmp('block_resize', device=self.drives[0]['id'],
+ size=(65 * 1048576))
+ self.assert_qmp(res, 'return', {})
+
+ # Dirty the image past the old end
+ self.vm.hmp_qemu_io(self.drives[0]['id'], 'write 64M 64k')
+
+ target = self.prepare_backup(size='65M')
+ self.create_incremental(target=target)
+
+ self.vm.shutdown()
+ self.check_backups()
+
class TestIncrementalBackupBlkdebug(TestIncrementalBackupBase):
'''Incremental backup tests that utilize a BlkDebug filter on drive0.'''
diff --git a/tests/qemu-iotests/124.out b/tests/qemu-iotests/124.out
index 281b69efea..fa16b5ccef 100644
--- a/tests/qemu-iotests/124.out
+++ b/tests/qemu-iotests/124.out
@@ -1,5 +1,5 @@
-............
+.............
----------------------------------------------------------------------
-Ran 12 tests
+Ran 13 tests
OK
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 67/97] mirror: Only mirror granularity-aligned chunks
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (65 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 66/97] iotests: Test incremental backup after truncation Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 68/97] iotests: Test unaligned blocking mirror write Michael Roth
` (34 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
In write-blocking mode, all writes to the top node directly go to the
target. We must only mirror chunks of data that are aligned to the
job's granularity, because that is how the dirty bitmap works.
Therefore, the request alignment for writes must be the job's
granularity (in write-blocking mode).
Unfortunately, this forces all reads and writes to have the same
granularity (we only need this alignment for writes to the target, not
the source), but that is something to be fixed another time.
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190805153308.2657-1-mreitz@redhat.com
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Fixes: d06107ade0ce74dc39739bac80de84b51ec18546
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 9adc1cb49af8d4e54f57980b1eed5c0a4b2dafa6)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/mirror.c | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/block/mirror.c b/block/mirror.c
index ff15cfb197..062dc42867 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1477,6 +1477,15 @@ static void bdrv_mirror_top_child_perm(BlockDriverState *bs, BdrvChild *c,
*nshared = BLK_PERM_ALL;
}
+static void bdrv_mirror_top_refresh_limits(BlockDriverState *bs, Error **errp)
+{
+ MirrorBDSOpaque *s = bs->opaque;
+
+ if (s && s->job && s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING) {
+ bs->bl.request_alignment = s->job->granularity;
+ }
+}
+
/* Dummy node that provides consistent read to its users without requiring it
* from its backing file and that allows writes on the backing file chain. */
static BlockDriver bdrv_mirror_top = {
@@ -1489,6 +1498,7 @@ static BlockDriver bdrv_mirror_top = {
.bdrv_co_block_status = bdrv_co_block_status_from_backing,
.bdrv_refresh_filename = bdrv_mirror_top_refresh_filename,
.bdrv_child_perm = bdrv_mirror_top_child_perm,
+ .bdrv_refresh_limits = bdrv_mirror_top_refresh_limits,
};
static void mirror_start_job(const char *job_id, BlockDriverState *bs,
@@ -1627,6 +1637,25 @@ static void mirror_start_job(const char *job_id, BlockDriverState *bs,
s->should_complete = true;
}
+ /*
+ * Must be called before we start tracking writes, but after
+ *
+ * ((MirrorBlockJob *)
+ * ((MirrorBDSOpaque *)
+ * mirror_top_bs->opaque
+ * )->job
+ * )->copy_mode
+ *
+ * has the correct value.
+ * (We start tracking writes as of the following
+ * bdrv_create_dirty_bitmap() call.)
+ */
+ bdrv_refresh_limits(mirror_top_bs, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ goto fail;
+ }
+
s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp);
if (!s->dirty_bitmap) {
goto fail;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 68/97] iotests: Test unaligned blocking mirror write
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (66 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 67/97] mirror: Only mirror granularity-aligned chunks Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 69/97] block/backup: disable copy_range for compressed backup Michael Roth
` (33 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190805113526.20319-1-mreitz@redhat.com
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 19ba4651fe2d17cc49adae29acbb4a8cc29db8d1)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/151 | 25 +++++++++++++++++++++++++
tests/qemu-iotests/151.out | 4 ++--
2 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index 1bb74d67c4..ad7359fc8d 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -114,6 +114,31 @@ class TestActiveMirror(iotests.QMPTestCase):
def testActiveIOFlushed(self):
self.doActiveIO(True)
+ def testUnalignedActiveIO(self):
+ # Fill the source image
+ result = self.vm.hmp_qemu_io('source', 'write -P 1 0 2M')
+
+ # Start the block job (very slowly)
+ result = self.vm.qmp('blockdev-mirror',
+ job_id='mirror',
+ filter_node_name='mirror-node',
+ device='source-node',
+ target='target-node',
+ sync='full',
+ copy_mode='write-blocking',
+ buf_size=(1048576 // 4),
+ speed=1)
+ self.assert_qmp(result, 'return', {})
+
+ # Start an unaligned request to a dirty area
+ result = self.vm.hmp_qemu_io('source', 'write -P 2 %i 1' % (1048576 + 42))
+
+ # Let the job finish
+ result = self.vm.qmp('block-job-set-speed', device='mirror', speed=0)
+ self.assert_qmp(result, 'return', {})
+ self.complete_and_wait(drive='mirror')
+
+ self.potential_writes_in_flight = False
if __name__ == '__main__':
diff --git a/tests/qemu-iotests/151.out b/tests/qemu-iotests/151.out
index fbc63e62f8..8d7e996700 100644
--- a/tests/qemu-iotests/151.out
+++ b/tests/qemu-iotests/151.out
@@ -1,5 +1,5 @@
-..
+...
----------------------------------------------------------------------
-Ran 2 tests
+Ran 3 tests
OK
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 69/97] block/backup: disable copy_range for compressed backup
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (67 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 68/97] iotests: Test unaligned blocking mirror write Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 70/97] Revert "ide/ahci: Check for -ECANCELED in aio callbacks" Michael Roth
` (32 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy, qemu-stable, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Enabled by default copy_range ignores compress option. It's definitely
unexpected for user.
It's broken since introduction of copy_range usage in backup in
9ded4a011496.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190730163251.755248-3-vsementsov@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 110571be4e70ac015628e76d2731f96dd8d1998c)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/backup.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/backup.c b/block/backup.c
index f67c208cf0..381659d5ef 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -666,7 +666,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
job->cluster_size = cluster_size;
job->copy_bitmap = copy_bitmap;
copy_bitmap = NULL;
- job->use_copy_range = true;
+ job->use_copy_range = !compress; /* compression isn't supported for it */
job->copy_range_size = MIN_NON_ZERO(blk_get_max_transfer(job->common.blk),
blk_get_max_transfer(job->target));
job->copy_range_size = MAX(job->cluster_size,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 70/97] Revert "ide/ahci: Check for -ECANCELED in aio callbacks"
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (68 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 69/97] block/backup: disable copy_range for compressed backup Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 71/97] qcow2: Fix the calculation of the maximum L2 cache size Michael Roth
` (31 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
This reverts commit 0d910cfeaf2076b116b4517166d5deb0fea76394.
It's not correct to just ignore an error code in a callback; we need to
handle that error and possible report failure to the guest so that they
don't wait indefinitely for an operation that will now never finish.
This ought to help cases reported by Nutanix where iSCSI returns a
legitimate -ECANCELED for certain operations which should be propagated
normally.
Reported-by: Shaju Abraham <shaju.abraham@nutanix.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190729223605.7163-1-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 8ec41c4265714255d5a138f8b538faf3583dcff6)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/ide/ahci.c | 3 ---
hw/ide/core.c | 14 --------------
2 files changed, 17 deletions(-)
diff --git a/hw/ide/ahci.c b/hw/ide/ahci.c
index d700ca973b..bda9583c8b 100644
--- a/hw/ide/ahci.c
+++ b/hw/ide/ahci.c
@@ -1022,9 +1022,6 @@ static void ncq_cb(void *opaque, int ret)
IDEState *ide_state = &ncq_tfs->drive->port.ifs[0];
ncq_tfs->aiocb = NULL;
- if (ret == -ECANCELED) {
- return;
- }
if (ret < 0) {
bool is_read = ncq_tfs->cmd == READ_FPDMA_QUEUED;
diff --git a/hw/ide/core.c b/hw/ide/core.c
index 6afadf894f..8e1624f7ce 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -722,9 +722,6 @@ static void ide_sector_read_cb(void *opaque, int ret)
s->pio_aiocb = NULL;
s->status &= ~BUSY_STAT;
- if (ret == -ECANCELED) {
- return;
- }
if (ret != 0) {
if (ide_handle_rw_error(s, -ret, IDE_RETRY_PIO |
IDE_RETRY_READ)) {
@@ -840,10 +837,6 @@ static void ide_dma_cb(void *opaque, int ret)
uint64_t offset;
bool stay_active = false;
- if (ret == -ECANCELED) {
- return;
- }
-
if (ret == -EINVAL) {
ide_dma_error(s);
return;
@@ -975,10 +968,6 @@ static void ide_sector_write_cb(void *opaque, int ret)
IDEState *s = opaque;
int n;
- if (ret == -ECANCELED) {
- return;
- }
-
s->pio_aiocb = NULL;
s->status &= ~BUSY_STAT;
@@ -1058,9 +1047,6 @@ static void ide_flush_cb(void *opaque, int ret)
s->pio_aiocb = NULL;
- if (ret == -ECANCELED) {
- return;
- }
if (ret < 0) {
/* XXX: What sector number to set here? */
if (ide_handle_rw_error(s, -ret, IDE_RETRY_FLUSH)) {
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 71/97] qcow2: Fix the calculation of the maximum L2 cache size
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (69 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 70/97] Revert "ide/ahci: Check for -ECANCELED in aio callbacks" Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 72/97] dma-helpers: ensure AIO callback is invoked after cancellation Michael Roth
` (30 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Alberto Garcia, qemu-stable
From: Alberto Garcia <berto@igalia.com>
The size of the qcow2 L2 cache defaults to 32 MB, which can be easily
larger than the maximum amount of L2 metadata that the image can have.
For example: with 64 KB clusters the user would need a qcow2 image
with a virtual size of 256 GB in order to have 32 MB of L2 metadata.
Because of that, since commit b749562d9822d14ef69c9eaa5f85903010b86c30
we forbid the L2 cache to become larger than the maximum amount of L2
metadata for the image, calculated using this formula:
uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
The problem with this formula is that the result should be rounded up
to the cluster size because an L2 table on disk always takes one full
cluster.
For example, a 1280 MB qcow2 image with 64 KB clusters needs exactly
160 KB of L2 metadata, but we need 192 KB on disk (3 clusters) even if
the last 32 KB of those are not going to be used.
However QEMU rounds the numbers down and only creates 2 cache tables
(128 KB), which is not enough for the image.
A quick test doing 4KB random writes on a 1280 MB image gives me
around 500 IOPS, while with the correct cache size I get 16K IOPS.
Cc: qemu-stable@nongnu.org
Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit b70d08205b2e4044c529eefc21df2c8ab61b473b)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/qcow2.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 840f289a48..c80f48a02b 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -831,7 +831,11 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
bool l2_cache_entry_size_set;
int min_refcount_cache = MIN_REFCOUNT_CACHE_SIZE * s->cluster_size;
uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
- uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
+ uint64_t max_l2_entries = DIV_ROUND_UP(virtual_disk_size, s->cluster_size);
+ /* An L2 table is always one cluster in size so the max cache size
+ * should be a multiple of the cluster size. */
+ uint64_t max_l2_cache = ROUND_UP(max_l2_entries * sizeof(uint64_t),
+ s->cluster_size);
combined_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_CACHE_SIZE);
l2_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_SIZE);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 72/97] dma-helpers: ensure AIO callback is invoked after cancellation
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (70 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 71/97] qcow2: Fix the calculation of the maximum L2 cache size Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 73/97] target/arm: Don't abort on M-profile exception return in linux-user mode Michael Roth
` (29 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, John Snow, qemu-stable
From: Paolo Bonzini <pbonzini@redhat.com>
dma_aio_cancel unschedules the BH if there is one, which corresponds
to the reschedule_dma case of dma_blk_cb. This can stall the DMA
permanently, because dma_complete will never get invoked and therefore
nobody will ever invoke the original AIO callback in dbs->common.cb.
Fix this by invoking the callback (which is ensured to happen after
a bdrv_aio_cancel_async, or done manually in the dbs->bh case), and
add assertions to check that the DMA state machine is indeed waiting
for dma_complete or reschedule_dma, but never both.
Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20190729213416.1972-1-pbonzini@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 539343c0a47e19d5dd64d846d64d084d9793681f)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
dma-helpers.c | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/dma-helpers.c b/dma-helpers.c
index 2d7e02d35e..d3871dc61e 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -90,6 +90,7 @@ static void reschedule_dma(void *opaque)
{
DMAAIOCB *dbs = (DMAAIOCB *)opaque;
+ assert(!dbs->acb && dbs->bh);
qemu_bh_delete(dbs->bh);
dbs->bh = NULL;
dma_blk_cb(dbs, 0);
@@ -111,15 +112,12 @@ static void dma_complete(DMAAIOCB *dbs, int ret)
{
trace_dma_complete(dbs, ret, dbs->common.cb);
+ assert(!dbs->acb && !dbs->bh);
dma_blk_unmap(dbs);
if (dbs->common.cb) {
dbs->common.cb(dbs->common.opaque, ret);
}
qemu_iovec_destroy(&dbs->iov);
- if (dbs->bh) {
- qemu_bh_delete(dbs->bh);
- dbs->bh = NULL;
- }
qemu_aio_unref(dbs);
}
@@ -179,14 +177,21 @@ static void dma_aio_cancel(BlockAIOCB *acb)
trace_dma_aio_cancel(dbs);
+ assert(!(dbs->acb && dbs->bh));
if (dbs->acb) {
+ /* This will invoke dma_blk_cb. */
blk_aio_cancel_async(dbs->acb);
+ return;
}
+
if (dbs->bh) {
cpu_unregister_map_client(dbs->bh);
qemu_bh_delete(dbs->bh);
dbs->bh = NULL;
}
+ if (dbs->common.cb) {
+ dbs->common.cb(dbs->common.opaque, -ECANCELED);
+ }
}
static AioContext *dma_get_aio_context(BlockAIOCB *acb)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 73/97] target/arm: Don't abort on M-profile exception return in linux-user mode
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (71 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 72/97] dma-helpers: ensure AIO callback is invoked after cancellation Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 74/97] xen-bus: Fix backend state transition on device reset Michael Roth
` (28 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, qemu-stable
From: Peter Maydell <peter.maydell@linaro.org>
An attempt to do an exception-return (branch to one of the magic
addresses) in linux-user mode for M-profile should behave like
a normal branch, because linux-user mode is always going to be
in 'handler' mode. This used to work, but we broke it when we added
support for the M-profile security extension in commit d02a8698d7ae2bfed.
In that commit we allowed even handler-mode calls to magic return
values to be checked for and dealt with by causing an
EXCP_EXCEPTION_EXIT exception to be taken, because this is
needed for the FNC_RETURN return-from-non-secure-function-call
handling. For system mode we added a check in do_v7m_exception_exit()
to make any spurious calls from Handler mode behave correctly, but
forgot that linux-user mode would also be affected.
How an attempted return-from-non-secure-function-call in linux-user
mode should be handled is not clear -- on real hardware it would
result in return to secure code (not to the Linux kernel) which
could then handle the error in any way it chose. For QEMU we take
the simple approach of treating this erroneous return the same way
it would be handled on a CPU without the security extensions --
treat it as a normal branch.
The upshot of all this is that for linux-user mode we should never
do any of the bx_excret magic, so the code change is simple.
This ought to be a weird corner case that only affects broken guest
code (because Linux user processes should never be attempting to do
exception returns or NS function returns), except that the code that
assigns addresses in RAM for the process and stack in our linux-user
code does not attempt to avoid this magic address range, so
legitimate code attempting to return to a trampoline routine on the
stack can fall into this case. This change fixes those programs,
but we should also look at restricting the range of memory we
use for M-profile linux-user guests to the area that would be
real RAM in hardware.
Cc: qemu-stable@nongnu.org
Reported-by: Christophe Lyon <christophe.lyon@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 20190822131534.16602-1-peter.maydell@linaro.org
Fixes: https://bugs.launchpad.net/qemu/+bug/1840922
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
(cherry picked from commit 5e5584c89f36b302c666bc6db535fd3f7ff35ad2)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
target/arm/translate.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/target/arm/translate.c b/target/arm/translate.c
index d408e4d7ef..d9d4e765ca 100644
--- a/target/arm/translate.c
+++ b/target/arm/translate.c
@@ -964,10 +964,27 @@ static inline void gen_bx(DisasContext *s, TCGv_i32 var)
store_cpu_field(var, thumb);
}
-/* Set PC and Thumb state from var. var is marked as dead.
+/*
+ * Set PC and Thumb state from var. var is marked as dead.
* For M-profile CPUs, include logic to detect exception-return
* branches and handle them. This is needed for Thumb POP/LDM to PC, LDR to PC,
* and BX reg, and no others, and happens only for code in Handler mode.
+ * The Security Extension also requires us to check for the FNC_RETURN
+ * which signals a function return from non-secure state; this can happen
+ * in both Handler and Thread mode.
+ * To avoid having to do multiple comparisons in inline generated code,
+ * we make the check we do here loose, so it will match for EXC_RETURN
+ * in Thread mode. For system emulation do_v7m_exception_exit() checks
+ * for these spurious cases and returns without doing anything (giving
+ * the same behaviour as for a branch to a non-magic address).
+ *
+ * In linux-user mode it is unclear what the right behaviour for an
+ * attempted FNC_RETURN should be, because in real hardware this will go
+ * directly to Secure code (ie not the Linux kernel) which will then treat
+ * the error in any way it chooses. For QEMU we opt to make the FNC_RETURN
+ * attempt behave the way it would on a CPU without the security extension,
+ * which is to say "like a normal branch". That means we can simply treat
+ * all branches as normal with no magic address behaviour.
*/
static inline void gen_bx_excret(DisasContext *s, TCGv_i32 var)
{
@@ -975,10 +992,12 @@ static inline void gen_bx_excret(DisasContext *s, TCGv_i32 var)
* s->base.is_jmp that we need to do the rest of the work later.
*/
gen_bx(s, var);
+#ifndef CONFIG_USER_ONLY
if (arm_dc_feature(s, ARM_FEATURE_M_SECURITY) ||
(s->v7m_handler_mode && arm_dc_feature(s, ARM_FEATURE_M))) {
s->base.is_jmp = DISAS_BX_EXCRET;
}
+#endif
}
static inline void gen_bx_excret_final_code(DisasContext *s)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 74/97] xen-bus: Fix backend state transition on device reset
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (72 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 73/97] target/arm: Don't abort on M-profile exception return in linux-user mode Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 75/97] pr-manager: Fix invalid g_free() crash bug Michael Roth
` (27 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Anthony PERARD, qemu-stable
From: Anthony PERARD <anthony.perard@citrix.com>
When a frontend wants to reset its state and the backend one, it
starts with setting "Closing", then waits for the backend (QEMU) to do
the same.
But when QEMU is setting "Closing" to its state, it triggers an event
(xenstore watch) that re-execute xen_device_backend_changed() and set
the backend state to "Closed". QEMU should wait for the frontend to
set "Closed" before doing the same.
Before setting "Closed" to the backend_state, we are also going to
check if there is a frontend. If that the case, when the backend state
is set to "Closing" the frontend should react and sets its state to
"Closing" then "Closed". The backend should wait for that to happen.
Fixes: b6af8926fb858c4f1426e5acb2cfc1f0580ec98a
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Reviewed-by: Paul Durrant <paul.durrant@citrix.com>
Message-Id: <20190823101534.465-2-anthony.perard@citrix.com>
(cherry picked from commit cb3231460747552d70af9d546dc53d8195bcb796)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/xen/xen-bus.c | 23 ++++++++++++++++++++---
1 file changed, 20 insertions(+), 3 deletions(-)
diff --git a/hw/xen/xen-bus.c b/hw/xen/xen-bus.c
index 49a725e8c7..0a1033d9a9 100644
--- a/hw/xen/xen-bus.c
+++ b/hw/xen/xen-bus.c
@@ -515,6 +515,23 @@ static void xen_device_backend_set_online(XenDevice *xendev, bool online)
xen_device_backend_printf(xendev, "online", "%u", online);
}
+/*
+ * Tell from the state whether the frontend is likely alive,
+ * i.e. it will react to a change of state of the backend.
+ */
+static bool xen_device_state_is_active(enum xenbus_state state)
+{
+ switch (state) {
+ case XenbusStateInitWait:
+ case XenbusStateInitialised:
+ case XenbusStateConnected:
+ case XenbusStateClosing:
+ return true;
+ default:
+ return false;
+ }
+}
+
static void xen_device_backend_changed(void *opaque)
{
XenDevice *xendev = opaque;
@@ -538,11 +555,11 @@ static void xen_device_backend_changed(void *opaque)
/*
* If the toolstack (or unplug request callback) has set the backend
- * state to Closing, but there is no active frontend (i.e. the
- * state is not Connected) then set the backend state to Closed.
+ * state to Closing, but there is no active frontend then set the
+ * backend state to Closed.
*/
if (xendev->backend_state == XenbusStateClosing &&
- xendev->frontend_state != XenbusStateConnected) {
+ !xen_device_state_is_active(state)) {
xen_device_backend_set_state(xendev, XenbusStateClosed);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 75/97] pr-manager: Fix invalid g_free() crash bug
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (73 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 74/97] xen-bus: Fix backend state transition on device reset Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 76/97] iotests: add testing shim for script-style python tests Michael Roth
` (26 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Markus Armbruster
From: Markus Armbruster <armbru@redhat.com>
pr_manager_worker() passes its @opaque argument to g_free(). Wrong;
it points to pr_manager_worker()'s automatic @data. Broken when
commit 2f3a7ab39be converted @data from heap- to stack-allocated. Fix
by deleting the g_free().
Fixes: 2f3a7ab39bec4ba8022dc4d42ea641165b004e3e
Cc: qemu-stable@nongnu.org
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 6b9d62c2a9e83bbad73fb61406f0ff69b46ff6f3)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
scsi/pr-manager.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/scsi/pr-manager.c b/scsi/pr-manager.c
index d9f4e8c3ad..227bdfaad2 100644
--- a/scsi/pr-manager.c
+++ b/scsi/pr-manager.c
@@ -38,7 +38,6 @@ static int pr_manager_worker(void *opaque)
int fd = data->fd;
int r;
- g_free(data);
trace_pr_manager_run(fd, hdr->cmdp[0], hdr->cmdp[1]);
/* The reference was taken in pr_manager_execute. */
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 76/97] iotests: add testing shim for script-style python tests
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (74 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 75/97] pr-manager: Fix invalid g_free() crash bug Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 77/97] vpc: Return 0 from vpc_co_create() on success Michael Roth
` (25 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, qemu-stable
From: John Snow <jsnow@redhat.com>
Because the new-style python tests don't use the iotests.main() test
launcher, we don't turn on the debugger logging for these scripts
when invoked via ./check -d.
Refactor the launcher shim into new and old style shims so that they
share environmental configuration.
Two cleanup notes: debug was not actually used as a global, and there
was no reason to create a class in an inner scope just to achieve
default variables; we can simply create an instance of the runner with
the values we want instead.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190709232550.10724-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
(cherry picked from commit 456a2d5ac7641c7e75c76328a561b528a8607a8e)
*prereq for 1a37e31244/88d2aa533a
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/iotests.py | 40 +++++++++++++++++++++++------------
1 file changed, 26 insertions(+), 14 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index a7006662ff..5efc21d801 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -61,7 +61,6 @@ cachemode = os.environ.get('CACHEMODE')
qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
-debug = False
luks_default_secret_object = 'secret,id=keysec0,data=' + \
os.environ.get('IMGKEYSECRET', '')
@@ -817,11 +816,22 @@ def skip_if_unsupported(required_formats=[], read_only=False):
return func_wrapper
return skip_test_decorator
-def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
- unsupported_fmts=[]):
- '''Run tests'''
+def execute_unittest(output, verbosity, debug):
+ runner = unittest.TextTestRunner(stream=output, descriptions=True,
+ verbosity=verbosity)
+ try:
+ # unittest.main() will use sys.exit(); so expect a SystemExit
+ # exception
+ unittest.main(testRunner=runner)
+ finally:
+ if not debug:
+ sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s',
+ r'Ran \1 tests', output.getvalue()))
- global debug
+def execute_test(test_function=None,
+ supported_fmts=[], supported_oses=['linux'],
+ supported_cache_modes=[], unsupported_fmts=[]):
+ """Run either unittest or script-style tests."""
# We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
# indicate that we're not being run via "check". There may be
@@ -853,13 +863,15 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
- class MyTestRunner(unittest.TextTestRunner):
- def __init__(self, stream=output, descriptions=True, verbosity=verbosity):
- unittest.TextTestRunner.__init__(self, stream, descriptions, verbosity)
+ if not test_function:
+ execute_unittest(output, verbosity, debug)
+ else:
+ test_function()
+
+def script_main(test_function, *args, **kwargs):
+ """Run script-style tests outside of the unittest framework"""
+ execute_test(test_function, *args, **kwargs)
- # unittest.main() will use sys.exit() so expect a SystemExit exception
- try:
- unittest.main(testRunner=MyTestRunner)
- finally:
- if not debug:
- sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', output.getvalue()))
+def main(*args, **kwargs):
+ """Run tests using the unittest framework"""
+ execute_test(None, *args, **kwargs)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 77/97] vpc: Return 0 from vpc_co_create() on success
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (75 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 76/97] iotests: add testing shim for script-style python tests Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 78/97] iotests: Add supported protocols to execute_test() Michael Roth
` (24 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
blockdev_create_run() directly uses .bdrv_co_create()'s return value as
the job's return value. Jobs must return 0 on success, not just any
nonnegative value. Therefore, using blockdev-create for VPC images may
currently fail as the vpc driver may return a positive integer.
Because there is no point in returning a positive integer anywhere in
the block layer (all non-negative integers are generally treated as
complete success), we probably do not want to add more such cases.
Therefore, fix this problem by making the vpc driver always return 0 in
case of success.
Suggested-by: Kevin Wolf <kwolf@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 1a37e3124407b5a145d44478d3ecbdb89c63789f)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/vpc.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/block/vpc.c b/block/vpc.c
index a902a4c54d..dbc3da2bd5 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -881,6 +881,7 @@ static int create_dynamic_disk(BlockBackend *blk, uint8_t *buf,
goto fail;
}
+ ret = 0;
fail:
return ret;
}
@@ -904,7 +905,7 @@ static int create_fixed_disk(BlockBackend *blk, uint8_t *buf,
return ret;
}
- return ret;
+ return 0;
}
static int calculate_rounded_image_size(BlockdevCreateOptionsVpc *vpc_opts,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 78/97] iotests: Add supported protocols to execute_test()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (76 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 77/97] vpc: Return 0 from vpc_co_create() on success Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 79/97] iotests: Restrict file Python tests to file Michael Roth
` (23 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 88d2aa533a4a1aad44a27c2e6cd5bc5fbcbce7ed)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/iotests.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 5efc21d801..069cb20be9 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -830,7 +830,8 @@ def execute_unittest(output, verbosity, debug):
def execute_test(test_function=None,
supported_fmts=[], supported_oses=['linux'],
- supported_cache_modes=[], unsupported_fmts=[]):
+ supported_cache_modes=[], unsupported_fmts=[],
+ supported_protocols=[], unsupported_protocols=[]):
"""Run either unittest or script-style tests."""
# We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
@@ -844,6 +845,7 @@ def execute_test(test_function=None,
debug = '-d' in sys.argv
verbosity = 1
verify_image_format(supported_fmts, unsupported_fmts)
+ verify_protocol(supported_protocols, unsupported_protocols)
verify_platform(supported_oses)
verify_cache_mode(supported_cache_modes)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 79/97] iotests: Restrict file Python tests to file
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (77 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 78/97] iotests: Add supported protocols to execute_test() Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:45 ` [PATCH 80/97] iotests: Restrict nbd Python tests to nbd Michael Roth
` (22 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Most of our Python unittest-style tests only support the file protocol.
You can run them with any other protocol, but the test will simply
ignore your choice and use file anyway.
We should let them signal that they require the file protocol so they
are skipped when you want to test some other protocol.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 103cbc771e5660d1f5bb458be80aa9e363547ae0)
Conflicts:
tests/qemu-iotests/257
*drop changes for test 257, not present in 4.0
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/030 | 3 ++-
tests/qemu-iotests/040 | 3 ++-
tests/qemu-iotests/041 | 3 ++-
tests/qemu-iotests/044 | 3 ++-
tests/qemu-iotests/045 | 3 ++-
tests/qemu-iotests/055 | 3 ++-
tests/qemu-iotests/056 | 3 ++-
tests/qemu-iotests/057 | 3 ++-
tests/qemu-iotests/065 | 3 ++-
tests/qemu-iotests/096 | 3 ++-
tests/qemu-iotests/118 | 3 ++-
tests/qemu-iotests/124 | 3 ++-
tests/qemu-iotests/129 | 3 ++-
tests/qemu-iotests/132 | 3 ++-
tests/qemu-iotests/139 | 3 ++-
tests/qemu-iotests/148 | 3 ++-
tests/qemu-iotests/151 | 3 ++-
tests/qemu-iotests/152 | 3 ++-
tests/qemu-iotests/155 | 3 ++-
tests/qemu-iotests/163 | 3 ++-
tests/qemu-iotests/165 | 3 ++-
tests/qemu-iotests/169 | 3 ++-
tests/qemu-iotests/196 | 3 ++-
tests/qemu-iotests/199 | 3 ++-
tests/qemu-iotests/245 | 3 ++-
25 files changed, 50 insertions(+), 25 deletions(-)
diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index c6311d1825..cb550812a7 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -865,4 +865,5 @@ class TestSetSpeed(iotests.QMPTestCase):
self.cancel_and_wait(resume=True)
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2', 'qed'])
+ iotests.main(supported_fmts=['qcow2', 'qed'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index b81133a474..e2ca706117 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -395,4 +395,5 @@ class TestReopenOverlay(ImageCommitTestCase):
self.run_commit_test(self.img1, self.img0)
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2', 'qed'])
+ iotests.main(supported_fmts=['qcow2', 'qed'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index 26bf1701eb..ae6ed952c6 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -1068,4 +1068,5 @@ class TestOrphanedSource(iotests.QMPTestCase):
self.assert_qmp(result, 'error/class', 'GenericError')
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2', 'qed'])
+ iotests.main(supported_fmts=['qcow2', 'qed'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 9ec3dba734..05ea1f49c5 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -118,4 +118,5 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
pass
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index d5484a0ee1..01cc038884 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -175,4 +175,5 @@ class TestSCMFd(iotests.QMPTestCase):
"File descriptor named '%s' not found" % fdname)
if __name__ == '__main__':
- iotests.main(supported_fmts=['raw'])
+ iotests.main(supported_fmts=['raw'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055
index 3437c11507..c732a112d6 100755
--- a/tests/qemu-iotests/055
+++ b/tests/qemu-iotests/055
@@ -563,4 +563,5 @@ class TestDriveCompression(iotests.QMPTestCase):
target='drive1')
if __name__ == '__main__':
- iotests.main(supported_fmts=['raw', 'qcow2'])
+ iotests.main(supported_fmts=['raw', 'qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index e5ac25127b..ce7adf9174 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -335,4 +335,5 @@ class BackupTest(iotests.QMPTestCase):
self.dismissal_failure(True)
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2', 'qed'])
+ iotests.main(supported_fmts=['qcow2', 'qed'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/057 b/tests/qemu-iotests/057
index 9f0a5a3057..9fbba759b6 100755
--- a/tests/qemu-iotests/057
+++ b/tests/qemu-iotests/057
@@ -256,4 +256,5 @@ class TestSnapshotDelete(ImageSnapshotTestCase):
self.assert_qmp(result, 'error/class', 'GenericError')
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 8bac383ea7..5b21eb96bd 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -129,4 +129,5 @@ TestQemuImgInfo = None
TestQMP = None
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/096 b/tests/qemu-iotests/096
index a69439602d..ab9cb47822 100755
--- a/tests/qemu-iotests/096
+++ b/tests/qemu-iotests/096
@@ -67,4 +67,5 @@ class TestLiveSnapshot(iotests.QMPTestCase):
self.checkConfig('target')
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index 603e10e8a2..c854378281 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -703,4 +703,5 @@ if __name__ == '__main__':
iotests.qemu_default_machine)
# Need to support image creation
iotests.main(supported_fmts=['vpc', 'parallels', 'qcow', 'vdi', 'qcow2',
- 'vmdk', 'raw', 'vhdx', 'qed'])
+ 'vmdk', 'raw', 'vhdx', 'qed'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3440f54781..ca40ba3be2 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -779,4 +779,5 @@ class TestIncrementalBackupBlkdebug(TestIncrementalBackupBase):
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index 9e87e1c8d9..cd6b9e9ce7 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -83,4 +83,5 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
self.do_test_stop("block-commit", device="drive0")
if __name__ == '__main__':
- iotests.main(supported_fmts=["qcow2"])
+ iotests.main(supported_fmts=["qcow2"],
+ supported_protocols=["file"])
diff --git a/tests/qemu-iotests/132 b/tests/qemu-iotests/132
index f53ef6e391..0f2a106c81 100755
--- a/tests/qemu-iotests/132
+++ b/tests/qemu-iotests/132
@@ -56,4 +56,5 @@ class TestSingleDrive(iotests.QMPTestCase):
'target image does not match source after mirroring')
if __name__ == '__main__':
- iotests.main(supported_fmts=['raw', 'qcow2'])
+ iotests.main(supported_fmts=['raw', 'qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index 933b45121a..8dc839ec7e 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -361,4 +361,5 @@ class TestBlockdevDel(iotests.QMPTestCase):
if __name__ == '__main__':
- iotests.main(supported_fmts=["qcow2"])
+ iotests.main(supported_fmts=["qcow2"],
+ supported_protocols=["file"])
diff --git a/tests/qemu-iotests/148 b/tests/qemu-iotests/148
index e01b061fe7..8c11c53cba 100755
--- a/tests/qemu-iotests/148
+++ b/tests/qemu-iotests/148
@@ -137,4 +137,5 @@ class TestFifoQuorumEvents(TestQuorumEvents):
if __name__ == '__main__':
iotests.verify_quorum()
- iotests.main(supported_fmts=["raw"])
+ iotests.main(supported_fmts=["raw"],
+ supported_protocols=["file"])
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index ad7359fc8d..76ae265cc1 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -142,4 +142,5 @@ class TestActiveMirror(iotests.QMPTestCase):
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2', 'raw'])
+ iotests.main(supported_fmts=['qcow2', 'raw'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/152 b/tests/qemu-iotests/152
index fec546d033..732bf5f062 100755
--- a/tests/qemu-iotests/152
+++ b/tests/qemu-iotests/152
@@ -59,4 +59,5 @@ class TestUnaligned(iotests.QMPTestCase):
if __name__ == '__main__':
- iotests.main(supported_fmts=['raw', 'qcow2'])
+ iotests.main(supported_fmts=['raw', 'qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155
index 63a5b5e2c0..e19485911c 100755
--- a/tests/qemu-iotests/155
+++ b/tests/qemu-iotests/155
@@ -258,4 +258,5 @@ BaseClass = None
MirrorBaseClass = None
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 158ba5d092..081ccc8ac1 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -170,4 +170,5 @@ class TestShrink1M(ShrinkBaseClass):
ShrinkBaseClass = None
if __name__ == '__main__':
- iotests.main(supported_fmts=['raw', 'qcow2'])
+ iotests.main(supported_fmts=['raw', 'qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index 88f62d3c6d..5650dc7c87 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -103,4 +103,5 @@ class TestPersistentDirtyBitmap(iotests.QMPTestCase):
self.vm.shutdown()
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index 527aebd0cb..25924d68fc 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -222,4 +222,5 @@ for cmb in list(itertools.product((True, False), repeat=2)):
'do_test_migration_resume_source', *list(cmb))
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/196 b/tests/qemu-iotests/196
index 4116ebc92b..92fe9244f8 100755
--- a/tests/qemu-iotests/196
+++ b/tests/qemu-iotests/196
@@ -63,4 +63,5 @@ class TestInvalidateAutoclear(iotests.QMPTestCase):
self.assertEqual(f.read(1), b'\x00')
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'])
+ iotests.main(supported_fmts=['qcow2'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
index 651e8df5d9..a2c8ecab5a 100755
--- a/tests/qemu-iotests/199
+++ b/tests/qemu-iotests/199
@@ -115,4 +115,5 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
self.assert_qmp(result, 'return/sha256', sha256);
if __name__ == '__main__':
- iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'])
+ iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index a04c6235c1..fbfd624878 100644
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -994,4 +994,5 @@ class TestBlockdevReopen(iotests.QMPTestCase):
self.reopen(opts, {'backing': 'hd2'})
if __name__ == '__main__':
- iotests.main(supported_fmts=["qcow2"])
+ iotests.main(supported_fmts=["qcow2"],
+ supported_protocols=["file"])
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 80/97] iotests: Restrict nbd Python tests to nbd
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (78 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 79/97] iotests: Restrict file Python tests to file Michael Roth
@ 2019-10-01 23:45 ` Michael Roth
2019-10-01 23:46 ` [PATCH 81/97] iotests: Test blockdev-create for vpc Michael Roth
` (21 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:45 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
We have two Python unittest-style tests that test NBD. As such, they
should specify supported_protocols=['nbd'] so they are skipped when the
user wants to test some other protocol.
Furthermore, we should restrict their choice of formats to 'raw'. The
idea of a protocol/format combination is to use some format over some
protocol; but we always use the raw format over NBD. It does not really
matter what the NBD server uses on its end, and it is not a useful test
of the respective format driver anyway.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 7c932a1d69a6d6ac5c0b615c11d191da3bbe9aa8)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/147 | 5 ++---
tests/qemu-iotests/205 | 3 ++-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index 82513279b0..ce1f5b459d 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -287,6 +287,5 @@ class BuiltinNBD(NBDBlockdevAddBase):
if __name__ == '__main__':
- # Need to support image creation
- iotests.main(supported_fmts=['vpc', 'parallels', 'qcow', 'vdi', 'qcow2',
- 'vmdk', 'raw', 'vhdx', 'qed'])
+ iotests.main(supported_fmts=['raw'],
+ supported_protocols=['nbd'])
diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
index 31b2f5707a..1cf2215730 100755
--- a/tests/qemu-iotests/205
+++ b/tests/qemu-iotests/205
@@ -153,4 +153,5 @@ class TestNbdServerRemove(iotests.QMPTestCase):
if __name__ == '__main__':
- iotests.main(supported_fmts=['generic'])
+ iotests.main(supported_fmts=['raw'],
+ supported_protocols=['nbd'])
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 81/97] iotests: Test blockdev-create for vpc
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (79 preceding siblings ...)
2019-10-01 23:45 ` [PATCH 80/97] iotests: Restrict nbd Python tests to nbd Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 82/97] libvhost-user: fix SLAVE_SEND_FD handling Michael Roth
` (20 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit cb73747e1a47b93d3dfdc3f769c576b053916938)
Conflicts:
tests/qemu-iotests/group
*drop context dep. on tests not in 4.0
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
tests/qemu-iotests/266 | 153 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/266.out | 137 +++++++++++++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 291 insertions(+)
create mode 100755 tests/qemu-iotests/266
create mode 100644 tests/qemu-iotests/266.out
diff --git a/tests/qemu-iotests/266 b/tests/qemu-iotests/266
new file mode 100755
index 0000000000..5b35cd67e4
--- /dev/null
+++ b/tests/qemu-iotests/266
@@ -0,0 +1,153 @@
+#!/usr/bin/env python
+#
+# Test VPC and file image creation
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import iotests
+from iotests import imgfmt
+
+
+def blockdev_create(vm, options):
+ result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+ filters=[iotests.filter_qmp_testfiles])
+
+ if 'return' in result:
+ assert result['return'] == {}
+ vm.run_job('job0')
+
+
+# Successful image creation (defaults)
+def implicit_defaults(vm, file_path):
+ iotests.log("=== Successful image creation (defaults) ===")
+ iotests.log("")
+
+ # 8 heads, 964 cyls/head, 17 secs/cyl
+ # (Close to 64 MB)
+ size = 8 * 964 * 17 * 512
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': size })
+
+
+# Successful image creation (explicit defaults)
+def explicit_defaults(vm, file_path):
+ iotests.log("=== Successful image creation (explicit defaults) ===")
+ iotests.log("")
+
+ # 16 heads, 964 cyls/head, 17 secs/cyl
+ # (Close to 128 MB)
+ size = 16 * 964 * 17 * 512
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': size,
+ 'subformat': 'dynamic',
+ 'force-size': False })
+
+
+# Successful image creation (non-default options)
+def non_defaults(vm, file_path):
+ iotests.log("=== Successful image creation (non-default options) ===")
+ iotests.log("")
+
+ # Not representable in CHS (fine with force-size=True)
+ size = 1048576
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': size,
+ 'subformat': 'fixed',
+ 'force-size': True })
+
+
+# Size not representable in CHS with force-size=False
+def non_chs_size_without_force(vm, file_path):
+ iotests.log("=== Size not representable in CHS ===")
+ iotests.log("")
+
+ # Not representable in CHS (will not work with force-size=False)
+ size = 1048576
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': size,
+ 'force-size': False })
+
+
+# Zero size
+def zero_size(vm, file_path):
+ iotests.log("=== Zero size===")
+ iotests.log("")
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': 0 })
+
+
+# Maximum CHS size
+def maximum_chs_size(vm, file_path):
+ iotests.log("=== Maximum CHS size===")
+ iotests.log("")
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': 16 * 65535 * 255 * 512 })
+
+
+# Actual maximum size
+def maximum_size(vm, file_path):
+ iotests.log("=== Actual maximum size===")
+ iotests.log("")
+
+ blockdev_create(vm, { 'driver': imgfmt,
+ 'file': 'protocol-node',
+ 'size': 0xff000000 * 512,
+ 'force-size': True })
+
+
+def main():
+ for test_func in [implicit_defaults, explicit_defaults, non_defaults,
+ non_chs_size_without_force, zero_size, maximum_chs_size,
+ maximum_size]:
+
+ with iotests.FilePath('t.vpc') as file_path, \
+ iotests.VM() as vm:
+
+ vm.launch()
+
+ iotests.log('--- Creating empty file ---')
+ blockdev_create(vm, { 'driver': 'file',
+ 'filename': file_path,
+ 'size': 0 })
+
+ vm.qmp_log('blockdev-add', driver='file', filename=file_path,
+ node_name='protocol-node',
+ filters=[iotests.filter_qmp_testfiles])
+ iotests.log('')
+
+ print_info = test_func(vm, file_path)
+ iotests.log('')
+
+ vm.shutdown()
+ iotests.img_info_log(file_path)
+
+
+iotests.script_main(main,
+ supported_fmts=['vpc'],
+ supported_protocols=['file'])
diff --git a/tests/qemu-iotests/266.out b/tests/qemu-iotests/266.out
new file mode 100644
index 0000000000..b11953e81f
--- /dev/null
+++ b/tests/qemu-iotests/266.out
@@ -0,0 +1,137 @@
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Successful image creation (defaults) ===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "size": 67125248}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 64 MiB (67125248 bytes)
+cluster_size: 2097152
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Successful image creation (explicit defaults) ===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "force-size": false, "size": 134250496, "subformat": "dynamic"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 128 MiB (134250496 bytes)
+cluster_size: 2097152
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Successful image creation (non-default options) ===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "force-size": true, "size": 1048576, "subformat": "fixed"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 1 MiB (1048576 bytes)
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Size not representable in CHS ===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "force-size": false, "size": 1048576}}}
+{"return": {}}
+Job failed: The requested image size cannot be represented in CHS geometry
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+qemu-img: Could not open 'TEST_IMG': File too small for a VHD header
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Zero size===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 0 B (0 bytes)
+cluster_size: 2097152
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Maximum CHS size===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "size": 136899993600}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 127 GiB (136899993600 bytes)
+cluster_size: 2097152
+
+--- Creating empty file ---
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vpc", "node-name": "protocol-node"}}
+{"return": {}}
+
+=== Actual maximum size===
+
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vpc", "file": "protocol-node", "force-size": true, "size": 2190433320960}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+image: TEST_IMG
+file format: IMGFMT
+virtual size: 1.99 TiB (2190433320960 bytes)
+cluster_size: 2097152
+
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 7aa6dc9875..3605796bb2 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -251,3 +251,4 @@
253 rw auto quick
255 rw auto quick
256 rw auto quick
+266 rw quick
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 82/97] libvhost-user: fix SLAVE_SEND_FD handling
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (80 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 81/97] iotests: Test blockdev-create for vpc Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 83/97] block/create: Do not abort if a block driver is not available Michael Roth
` (19 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Johannes Berg, Michael S . Tsirkin
From: Johannes Berg <johannes.berg@intel.com>
It doesn't look like this could possibly work properly since
VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD is defined to 10, but the
dev->protocol_features has a bitmap. I suppose the peer this
was tested with also supported VHOST_USER_PROTOCOL_F_LOG_SHMFD,
in which case the test would always be false, but nevertheless
the code seems wrong.
Use has_feature() to fix this.
Fixes: d84599f56c82 ("libvhost-user: support host notifier")
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Message-Id: <20190903200422.11693-1-johannes@sipsolutions.net>
Reviewed-by: Tiwei Bie <tiwei.bie@intel.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
(cherry picked from commit 8726b70b449896f1211f869ec4f608904f027207)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
contrib/libvhost-user/libvhost-user.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/libvhost-user.c
index e08d6c7b97..516d0de7bb 100644
--- a/contrib/libvhost-user/libvhost-user.c
+++ b/contrib/libvhost-user/libvhost-user.c
@@ -1085,7 +1085,8 @@ bool vu_set_queue_host_notifier(VuDev *dev, VuVirtq *vq, int fd,
vmsg.fd_num = fd_num;
- if ((dev->protocol_features & VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) == 0) {
+ if (!has_feature(dev->protocol_features,
+ VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD)) {
return false;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 83/97] block/create: Do not abort if a block driver is not available
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (81 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 82/97] libvhost-user: fix SLAVE_SEND_FD handling Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 84/97] block/nfs: tear down aio before nfs_close Michael Roth
` (18 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Philippe Mathieu-Daudé, qemu-stable
From: Philippe Mathieu-Daudé <philmd@redhat.com>
The 'blockdev-create' QMP command was introduced as experimental
feature in commit b0292b851b8, using the assert() debug call.
It got promoted to 'stable' command in 3fb588a0f2c, but the
assert call was not removed.
Some block drivers are optional, and bdrv_find_format() might
return a NULL value, triggering the assertion.
Stable code is not expected to abort, so return an error instead.
This is easily reproducible when libnfs is not installed:
./configure
[...]
module support no
Block whitelist (rw)
Block whitelist (ro)
libiscsi support yes
libnfs support no
[...]
Start QEMU:
$ qemu-system-x86_64 -S -qmp unix:/tmp/qemu.qmp,server,nowait
Send the 'blockdev-create' with the 'nfs' driver:
$ ( cat << 'EOF'
{'execute': 'qmp_capabilities'}
{'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'}
EOF
) | socat STDIO UNIX:/tmp/qemu.qmp
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 4}, "package": "v4.1.0-733-g89ea03a7dc"}, "capabilities": ["oob"]}}
{"return": {}}
QEMU crashes:
$ gdb qemu-system-x86_64 core
Program received signal SIGSEGV, Segmentation fault.
(gdb) bt
#0 0x00007ffff510957f in raise () at /lib64/libc.so.6
#1 0x00007ffff50f3895 in abort () at /lib64/libc.so.6
#2 0x00007ffff50f3769 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3 0x00007ffff5101a26 in .annobin_assert.c_end () at /lib64/libc.so.6
#4 0x0000555555d7e1f1 in qmp_blockdev_create (job_id=0x555556baee40 "x", options=0x555557666610, errp=0x7fffffffc770) at block/create.c:69
#5 0x0000555555c96b52 in qmp_marshal_blockdev_create (args=0x7fffdc003830, ret=0x7fffffffc7f8, errp=0x7fffffffc7f0) at qapi/qapi-commands-block-core.c:1314
#6 0x0000555555deb0a0 in do_qmp_dispatch (cmds=0x55555645de70 <qmp_commands>, request=0x7fffdc005c70, allow_oob=false, errp=0x7fffffffc898) at qapi/qmp-dispatch.c:131
#7 0x0000555555deb2a1 in qmp_dispatch (cmds=0x55555645de70 <qmp_commands>, request=0x7fffdc005c70, allow_oob=false) at qapi/qmp-dispatch.c:174
With this patch applied, QEMU returns a QMP error:
{'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'}
{"id": "x", "error": {"class": "GenericError", "desc": "Block driver 'nfs' not found or not supported"}}
Cc: qemu-stable@nongnu.org
Reported-by: Xu Tian <xutian@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit d90d5cae2b10efc0e8d0b3cc91ff16201853d3ba)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/create.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/block/create.c b/block/create.c
index 95341219ef..de5e97bb18 100644
--- a/block/create.c
+++ b/block/create.c
@@ -63,9 +63,13 @@ void qmp_blockdev_create(const char *job_id, BlockdevCreateOptions *options,
const char *fmt = BlockdevDriver_str(options->driver);
BlockDriver *drv = bdrv_find_format(fmt);
+ if (!drv) {
+ error_setg(errp, "Block driver '%s' not found or not supported", fmt);
+ return;
+ }
+
/* If the driver is in the schema, we know that it exists. But it may not
* be whitelisted. */
- assert(drv);
if (bdrv_uses_whitelist() && !bdrv_is_whitelisted(drv, false)) {
error_setg(errp, "Driver is not whitelisted");
return;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 84/97] block/nfs: tear down aio before nfs_close
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (82 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 83/97] block/create: Do not abort if a block driver is not available Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 85/97] blockjob: update nodes head while removing all bdrv Michael Roth
` (17 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Peter Lieven, qemu-stable
From: Peter Lieven <pl@kamp.de>
nfs_close is a sync call from libnfs and has its own event
handler polling on the nfs FD. Avoid that both QEMU and libnfs
are intefering here.
CC: qemu-stable@nongnu.org
Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit 601dc6559725f7a614b6f893611e17ff0908e914)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/nfs.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/block/nfs.c b/block/nfs.c
index 531903610b..f04f675c63 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -389,12 +389,14 @@ static void nfs_attach_aio_context(BlockDriverState *bs,
static void nfs_client_close(NFSClient *client)
{
if (client->context) {
+ qemu_mutex_lock(&client->mutex);
+ aio_set_fd_handler(client->aio_context, nfs_get_fd(client->context),
+ false, NULL, NULL, NULL, NULL);
+ qemu_mutex_unlock(&client->mutex);
if (client->fh) {
nfs_close(client->context, client->fh);
client->fh = NULL;
}
- aio_set_fd_handler(client->aio_context, nfs_get_fd(client->context),
- false, NULL, NULL, NULL, NULL);
nfs_destroy_context(client->context);
client->context = NULL;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 85/97] blockjob: update nodes head while removing all bdrv
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (83 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 84/97] block/nfs: tear down aio before nfs_close Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 86/97] curl: Keep pointer to the CURLState in CURLSocket Michael Roth
` (16 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Sergio Lopez, Max Reitz
From: Sergio Lopez <slp@redhat.com>
block_job_remove_all_bdrv() iterates through job->nodes, calling
bdrv_root_unref_child() for each entry. The call to the latter may
reach child_job_[can_]set_aio_ctx(), which will also attempt to
traverse job->nodes, potentially finding entries that where freed
on previous iterations.
To avoid this situation, update job->nodes head on each iteration to
ensure that already freed entries are no longer linked to the list.
RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746631
Signed-off-by: Sergio Lopez <slp@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190911100316.32282-1-mreitz@redhat.com
Reviewed-by: Sergio Lopez <slp@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit d876bf676f5e7c6aa9ac64555e48cba8734ecb2f)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
blockjob.c | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index 730101d282..d770144fd6 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -193,14 +193,23 @@ static const BdrvChildRole child_job = {
void block_job_remove_all_bdrv(BlockJob *job)
{
- GSList *l;
- for (l = job->nodes; l; l = l->next) {
+ /*
+ * bdrv_root_unref_child() may reach child_job_[can_]set_aio_ctx(),
+ * which will also traverse job->nodes, so consume the list one by
+ * one to make sure that such a concurrent access does not attempt
+ * to process an already freed BdrvChild.
+ */
+ while (job->nodes) {
+ GSList *l = job->nodes;
BdrvChild *c = l->data;
+
+ job->nodes = l->next;
+
bdrv_op_unblock_all(c->bs, job->blocker);
bdrv_root_unref_child(c);
+
+ g_slist_free_1(l);
}
- g_slist_free(job->nodes);
- job->nodes = NULL;
}
int block_job_add_bdrv(BlockJob *job, const char *name, BlockDriverState *bs,
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 86/97] curl: Keep pointer to the CURLState in CURLSocket
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (84 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 85/97] blockjob: update nodes head while removing all bdrv Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 87/97] curl: Keep *socket until the end of curl_sock_cb() Michael Roth
` (15 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
A follow-up patch will make curl_multi_do() and curl_multi_read() take a
CURLSocket instead of the CURLState. They still need the latter,
though, so add a pointer to it to the former.
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190910124136.10565-2-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 0487861685294660b23bc146e1ebd5304aa8bbe0)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/block/curl.c b/block/curl.c
index 606709fea4..4eaae9e211 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -79,6 +79,7 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
#define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5
struct BDRVCURLState;
+struct CURLState;
static bool libcurl_initialized;
@@ -96,6 +97,7 @@ typedef struct CURLAIOCB {
typedef struct CURLSocket {
int fd;
+ struct CURLState *state;
QLIST_ENTRY(CURLSocket) next;
} CURLSocket;
@@ -179,6 +181,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
if (!socket) {
socket = g_new0(CURLSocket, 1);
socket->fd = fd;
+ socket->state = state;
QLIST_INSERT_HEAD(&state->sockets, socket, next);
}
socket = NULL;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 87/97] curl: Keep *socket until the end of curl_sock_cb()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (85 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 86/97] curl: Keep pointer to the CURLState in CURLSocket Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 88/97] curl: Check completion in curl_multi_do() Michael Roth
` (14 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
This does not really change anything, but it makes the code a bit easier
to follow once we use @socket as the opaque pointer for
aio_set_fd_handler().
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-3-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 007f339b1099af46a008dac438ca0943e31dba72)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/block/curl.c b/block/curl.c
index 4eaae9e211..a0381ae0b4 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -171,10 +171,6 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
QLIST_FOREACH(socket, &state->sockets, next) {
if (socket->fd == fd) {
- if (action == CURL_POLL_REMOVE) {
- QLIST_REMOVE(socket, next);
- g_free(socket);
- }
break;
}
}
@@ -184,7 +180,6 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
socket->state = state;
QLIST_INSERT_HEAD(&state->sockets, socket, next);
}
- socket = NULL;
trace_curl_sock_cb(action, (int)fd);
switch (action) {
@@ -206,6 +201,11 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
break;
}
+ if (action == CURL_POLL_REMOVE) {
+ QLIST_REMOVE(socket, next);
+ g_free(socket);
+ }
+
return 0;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 88/97] curl: Check completion in curl_multi_do()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (86 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 87/97] curl: Keep *socket until the end of curl_sock_cb() Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 89/97] curl: Pass CURLSocket to curl_multi_do() Michael Roth
` (13 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
While it is more likely that transfers complete after some file
descriptor has data ready to read, we probably should not rely on it.
Better be safe than sorry and call curl_multi_check_completion() in
curl_multi_do(), too, just like it is done in curl_multi_read().
With this change, curl_multi_do() and curl_multi_read() are actually the
same, so drop curl_multi_read() and use curl_multi_do() as the sole FD
handler.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-4-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 948403bcb1c7e71dcbe8ab8479cf3934a0efcbb5)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 14 ++------------
1 file changed, 2 insertions(+), 12 deletions(-)
diff --git a/block/curl.c b/block/curl.c
index a0381ae0b4..bf64c2a0db 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -138,7 +138,6 @@ typedef struct BDRVCURLState {
static void curl_clean_state(CURLState *s);
static void curl_multi_do(void *arg);
-static void curl_multi_read(void *arg);
#ifdef NEED_CURL_TIMER_CALLBACK
/* Called from curl_multi_do_locked, with s->mutex held. */
@@ -185,7 +184,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
switch (action) {
case CURL_POLL_IN:
aio_set_fd_handler(s->aio_context, fd, false,
- curl_multi_read, NULL, NULL, state);
+ curl_multi_do, NULL, NULL, state);
break;
case CURL_POLL_OUT:
aio_set_fd_handler(s->aio_context, fd, false,
@@ -193,7 +192,7 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
break;
case CURL_POLL_INOUT:
aio_set_fd_handler(s->aio_context, fd, false,
- curl_multi_read, curl_multi_do, NULL, state);
+ curl_multi_do, curl_multi_do, NULL, state);
break;
case CURL_POLL_REMOVE:
aio_set_fd_handler(s->aio_context, fd, false,
@@ -415,15 +414,6 @@ static void curl_multi_do(void *arg)
{
CURLState *s = (CURLState *)arg;
- qemu_mutex_lock(&s->s->mutex);
- curl_multi_do_locked(s);
- qemu_mutex_unlock(&s->s->mutex);
-}
-
-static void curl_multi_read(void *arg)
-{
- CURLState *s = (CURLState *)arg;
-
qemu_mutex_lock(&s->s->mutex);
curl_multi_do_locked(s);
curl_multi_check_completion(s->s);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 89/97] curl: Pass CURLSocket to curl_multi_do()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (87 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 88/97] curl: Check completion in curl_multi_do() Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 90/97] curl: Report only ready sockets Michael Roth
` (12 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
curl_multi_do_locked() currently marks all sockets as ready. That is
not only inefficient, but in fact unsafe (the loop is). A follow-up
patch will change that, but to do so, curl_multi_do_locked() needs to
know exactly which socket is ready; and that is accomplished by this
patch here.
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-5-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 9dbad87d25587ff640ef878f7b6159fc368ff541)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/block/curl.c b/block/curl.c
index bf64c2a0db..5b163d71dc 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -184,15 +184,15 @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
switch (action) {
case CURL_POLL_IN:
aio_set_fd_handler(s->aio_context, fd, false,
- curl_multi_do, NULL, NULL, state);
+ curl_multi_do, NULL, NULL, socket);
break;
case CURL_POLL_OUT:
aio_set_fd_handler(s->aio_context, fd, false,
- NULL, curl_multi_do, NULL, state);
+ NULL, curl_multi_do, NULL, socket);
break;
case CURL_POLL_INOUT:
aio_set_fd_handler(s->aio_context, fd, false,
- curl_multi_do, curl_multi_do, NULL, state);
+ curl_multi_do, curl_multi_do, NULL, socket);
break;
case CURL_POLL_REMOVE:
aio_set_fd_handler(s->aio_context, fd, false,
@@ -391,9 +391,10 @@ static void curl_multi_check_completion(BDRVCURLState *s)
}
/* Called with s->mutex held. */
-static void curl_multi_do_locked(CURLState *s)
+static void curl_multi_do_locked(CURLSocket *ready_socket)
{
CURLSocket *socket, *next_socket;
+ CURLState *s = ready_socket->state;
int running;
int r;
@@ -412,12 +413,13 @@ static void curl_multi_do_locked(CURLState *s)
static void curl_multi_do(void *arg)
{
- CURLState *s = (CURLState *)arg;
+ CURLSocket *socket = arg;
+ BDRVCURLState *s = socket->state->s;
- qemu_mutex_lock(&s->s->mutex);
- curl_multi_do_locked(s);
- curl_multi_check_completion(s->s);
- qemu_mutex_unlock(&s->s->mutex);
+ qemu_mutex_lock(&s->mutex);
+ curl_multi_do_locked(socket);
+ curl_multi_check_completion(s);
+ qemu_mutex_unlock(&s->mutex);
}
static void curl_multi_timeout_do(void *arg)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 90/97] curl: Report only ready sockets
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (88 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 89/97] curl: Pass CURLSocket to curl_multi_do() Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 91/97] curl: Handle success in multi_check_completion Michael Roth
` (11 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Instead of reporting all sockets to cURL, only report the one that has
caused curl_multi_do_locked() to be called. This lets us get rid of the
QLIST_FOREACH_SAFE() list, which was actually wrong: SAFE foreaches are
only safe when the current element is removed in each iteration. If it
possible for the list to be concurrently modified, we cannot guarantee
that only the current element will be removed. Therefore, we must not
use QLIST_FOREACH_SAFE() here.
Fixes: ff5ca1664af85b24a4180d595ea6873fd3deac57
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-6-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit 9abaf9fc474c3dd53e8e119326abc774c977c331)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 17 ++++++-----------
1 file changed, 6 insertions(+), 11 deletions(-)
diff --git a/block/curl.c b/block/curl.c
index 5b163d71dc..de0cebd361 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -391,24 +391,19 @@ static void curl_multi_check_completion(BDRVCURLState *s)
}
/* Called with s->mutex held. */
-static void curl_multi_do_locked(CURLSocket *ready_socket)
+static void curl_multi_do_locked(CURLSocket *socket)
{
- CURLSocket *socket, *next_socket;
- CURLState *s = ready_socket->state;
+ BDRVCURLState *s = socket->state->s;
int running;
int r;
- if (!s->s->multi) {
+ if (!s->multi) {
return;
}
- /* Need to use _SAFE because curl_multi_socket_action() may trigger
- * curl_sock_cb() which might modify this list */
- QLIST_FOREACH_SAFE(socket, &s->sockets, next, next_socket) {
- do {
- r = curl_multi_socket_action(s->s->multi, socket->fd, 0, &running);
- } while (r == CURLM_CALL_MULTI_PERFORM);
- }
+ do {
+ r = curl_multi_socket_action(s->multi, socket->fd, 0, &running);
+ } while (r == CURLM_CALL_MULTI_PERFORM);
}
static void curl_multi_do(void *arg)
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 91/97] curl: Handle success in multi_check_completion
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (89 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 90/97] curl: Report only ready sockets Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 92/97] curl: Check curl_multi_add_handle()'s return code Michael Roth
` (10 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
Background: As of cURL 7.59.0, it verifies that several functions are
not called from within a callback. Among these functions is
curl_multi_add_handle().
curl_read_cb() is a callback from cURL and not a coroutine. Waking up
acb->co will lead to entering it then and there, which means the current
request will settle and the caller (if it runs in the same coroutine)
may then issue the next request. In such a case, we will enter
curl_setup_preadv() effectively from within curl_read_cb().
Calling curl_multi_add_handle() will then fail and the new request will
not be processed.
Fix this by not letting curl_read_cb() wake up acb->co. Instead, leave
the whole business of settling the AIOCB objects to
curl_multi_check_completion() (which is called from our timer callback
and our FD handler, so not from any cURL callbacks).
Reported-by: Natalie Gavrielov <ngavrilo@redhat.com>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1740193
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit bfb23b480a49114315877aacf700b49453e0f9d9)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 69 ++++++++++++++++++++++------------------------------
1 file changed, 29 insertions(+), 40 deletions(-)
diff --git a/block/curl.c b/block/curl.c
index de0cebd361..af40203711 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -228,7 +228,6 @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
{
CURLState *s = ((CURLState*)opaque);
size_t realsize = size * nmemb;
- int i;
trace_curl_read_cb(realsize);
@@ -244,32 +243,6 @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
memcpy(s->orig_buf + s->buf_off, ptr, realsize);
s->buf_off += realsize;
- for(i=0; i<CURL_NUM_ACB; i++) {
- CURLAIOCB *acb = s->acb[i];
-
- if (!acb)
- continue;
-
- if ((s->buf_off >= acb->end)) {
- size_t request_length = acb->bytes;
-
- qemu_iovec_from_buf(acb->qiov, 0, s->orig_buf + acb->start,
- acb->end - acb->start);
-
- if (acb->end - acb->start < request_length) {
- size_t offset = acb->end - acb->start;
- qemu_iovec_memset(acb->qiov, offset, 0,
- request_length - offset);
- }
-
- acb->ret = 0;
- s->acb[i] = NULL;
- qemu_mutex_unlock(&s->s->mutex);
- aio_co_wake(acb->co);
- qemu_mutex_lock(&s->s->mutex);
- }
- }
-
read_end:
/* curl will error out if we do not return this value */
return size * nmemb;
@@ -350,13 +323,14 @@ static void curl_multi_check_completion(BDRVCURLState *s)
break;
if (msg->msg == CURLMSG_DONE) {
+ int i;
CURLState *state = NULL;
+ bool error = msg->data.result != CURLE_OK;
+
curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE,
(char **)&state);
- /* ACBs for successful messages get completed in curl_read_cb */
- if (msg->data.result != CURLE_OK) {
- int i;
+ if (error) {
static int errcount = 100;
/* Don't lose the original error message from curl, since
@@ -368,20 +342,35 @@ static void curl_multi_check_completion(BDRVCURLState *s)
error_report("curl: further errors suppressed");
}
}
+ }
- for (i = 0; i < CURL_NUM_ACB; i++) {
- CURLAIOCB *acb = state->acb[i];
+ for (i = 0; i < CURL_NUM_ACB; i++) {
+ CURLAIOCB *acb = state->acb[i];
- if (acb == NULL) {
- continue;
- }
+ if (acb == NULL) {
+ continue;
+ }
+
+ if (!error) {
+ /* Assert that we have read all data */
+ assert(state->buf_off >= acb->end);
+
+ qemu_iovec_from_buf(acb->qiov, 0,
+ state->orig_buf + acb->start,
+ acb->end - acb->start);
- acb->ret = -EIO;
- state->acb[i] = NULL;
- qemu_mutex_unlock(&s->mutex);
- aio_co_wake(acb->co);
- qemu_mutex_lock(&s->mutex);
+ if (acb->end - acb->start < acb->bytes) {
+ size_t offset = acb->end - acb->start;
+ qemu_iovec_memset(acb->qiov, offset, 0,
+ acb->bytes - offset);
+ }
}
+
+ acb->ret = error ? -EIO : 0;
+ state->acb[i] = NULL;
+ qemu_mutex_unlock(&s->mutex);
+ aio_co_wake(acb->co);
+ qemu_mutex_lock(&s->mutex);
}
curl_clean_state(state);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 92/97] curl: Check curl_multi_add_handle()'s return code
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (90 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 91/97] curl: Handle success in multi_check_completion Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 93/97] slirp: Fix heap overflow in ip_reass on big packet input Michael Roth
` (9 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-stable, Max Reitz
From: Max Reitz <mreitz@redhat.com>
If we had done that all along, debugging would have been much simpler.
(Also, I/O errors are better than hangs.)
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-8-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
(cherry picked from commit c34dc07f9f01cf686e512f939aece744723072cd)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
block/curl.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/block/curl.c b/block/curl.c
index af40203711..7a29b40954 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -881,7 +881,13 @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
trace_curl_setup_preadv(acb->bytes, start, state->range);
curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
- curl_multi_add_handle(s->multi, state->curl);
+ if (curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
+ state->acb[0] = NULL;
+ acb->ret = -EIO;
+
+ curl_clean_state(state);
+ goto out;
+ }
/* Tell curl it needs to kick things off */
curl_multi_socket_action(s->multi, CURL_SOCKET_TIMEOUT, 0, &running);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 93/97] slirp: Fix heap overflow in ip_reass on big packet input
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (91 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 92/97] curl: Check curl_multi_add_handle()'s return code Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 94/97] slirp: ip_reass: Fix use after free Michael Roth
` (8 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Samuel Thibault, qemu-stable
When the first fragment does not fit in the preallocated buffer, q will
already be pointing to the ext buffer, so we mustn't try to update it.
Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
(from libslirp.git commit 126c04acbabd7ad32c2b018fe10dfac2a3bc1210)
(from libslirp.git commit e0be80430c390bce181ea04dfcdd6ea3dfa97de1)
*squash in e0be80 (clarifying comments)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
slirp/src/ip_input.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/slirp/src/ip_input.c b/slirp/src/ip_input.c
index a714fecd58..68a99de5b5 100644
--- a/slirp/src/ip_input.c
+++ b/slirp/src/ip_input.c
@@ -331,6 +331,8 @@ insert:
q = fp->frag_link.next;
m = dtom(slirp, q);
+ int was_ext = m->m_flags & M_EXT;
+
q = (struct ipasfrag *) q->ipf_next;
while (q != (struct ipasfrag*)&fp->frag_link) {
struct mbuf *t = dtom(slirp, q);
@@ -347,13 +349,12 @@ insert:
q = fp->frag_link.next;
/*
- * If the fragments concatenated to an mbuf that's
- * bigger than the total size of the fragment, then and
- * m_ext buffer was alloced. But fp->ipq_next points to
- * the old buffer (in the mbuf), so we must point ip
- * into the new buffer.
+ * If the fragments concatenated to an mbuf that's bigger than the total
+ * size of the fragment and the mbuf was not already using an m_ext buffer,
+ * then an m_ext buffer was alloced. But fp->ipq_next points to the old
+ * buffer (in the mbuf), so we must point ip into the new buffer.
*/
- if (m->m_flags & M_EXT) {
+ if (!was_ext && m->m_flags & M_EXT) {
int delta = (char *)q - m->m_dat;
q = (struct ipasfrag *)(m->m_ext + delta);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 94/97] slirp: ip_reass: Fix use after free
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (92 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 93/97] slirp: Fix heap overflow in ip_reass on big packet input Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 95/97] s390: PCI: fix IOMMU region init Michael Roth
` (7 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Samuel Thibault, qemu-stable
Using ip_deq after m_free might read pointers from an allocation reuse.
This would be difficult to exploit, but that is still related with
CVE-2019-14378 which generates fragmented IP packets that would trigger this
issue and at least produce a DoS.
Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
(from libslirp.git commit c59279437eda91841b9d26079c70b8a540d41204)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
slirp/src/ip_input.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/slirp/src/ip_input.c b/slirp/src/ip_input.c
index 68a99de5b5..89ae04e0c1 100644
--- a/slirp/src/ip_input.c
+++ b/slirp/src/ip_input.c
@@ -297,6 +297,7 @@ ip_reass(Slirp *slirp, struct ip *ip, struct ipq *fp)
*/
while (q != (struct ipasfrag*)&fp->frag_link &&
ip->ip_off + ip->ip_len > q->ipf_off) {
+ struct ipasfrag *prev;
i = (ip->ip_off + ip->ip_len) - q->ipf_off;
if (i < q->ipf_len) {
q->ipf_len -= i;
@@ -304,9 +305,10 @@ ip_reass(Slirp *slirp, struct ip *ip, struct ipq *fp)
m_adj(dtom(slirp, q), i);
break;
}
+ prev = q;
q = q->ipf_next;
- m_free(dtom(slirp, q->ipf_prev));
- ip_deq(q->ipf_prev);
+ ip_deq(prev);
+ m_free(dtom(slirp, prev));
}
insert:
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 95/97] s390: PCI: fix IOMMU region init
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (93 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 94/97] slirp: ip_reass: Fix use after free Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 96/97] hw/core/loader: Fix possible crash in rom_copy() Michael Roth
` (6 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Christian Borntraeger, qemu-stable, Matthew Rosato
From: Matthew Rosato <mjrosato@linux.ibm.com>
The fix in dbe9cf606c shrinks the IOMMU memory region to a size
that seems reasonable on the surface, however is actually too
small as it is based against a 0-mapped address space. This
causes breakage with small guests as they can overrun the IOMMU window.
Let's go back to the prior method of initializing iommu for now.
Fixes: dbe9cf606c ("s390x/pci: Set the iommu region size mpcifc request")
Cc: qemu-stable@nongnu.org
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Reported-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
Tested-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
Reported-by: Stefan Zimmerman <stzi@linux.ibm.com>
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
Message-Id: <1569507036-15314-1-git-send-email-mjrosato@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
(cherry picked from commit 7df1dac5f1c85312474df9cb3a8fcae72303da62)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/s390x/s390-pci-bus.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c
index 2d0a28d544..a6be6305b8 100644
--- a/hw/s390x/s390-pci-bus.c
+++ b/hw/s390x/s390-pci-bus.c
@@ -694,10 +694,15 @@ static const MemoryRegionOps s390_msi_ctrl_ops = {
void s390_pci_iommu_enable(S390PCIIOMMU *iommu)
{
+ /*
+ * The iommu region is initialized against a 0-mapped address space,
+ * so the smallest IOMMU region we can define runs from 0 to the end
+ * of the PCI address space.
+ */
char *name = g_strdup_printf("iommu-s390-%04x", iommu->pbdev->uid);
memory_region_init_iommu(&iommu->iommu_mr, sizeof(iommu->iommu_mr),
TYPE_S390_IOMMU_MEMORY_REGION, OBJECT(&iommu->mr),
- name, iommu->pal - iommu->pba + 1);
+ name, iommu->pal + 1);
iommu->enabled = true;
memory_region_add_subregion(&iommu->mr, 0, MEMORY_REGION(&iommu->iommu_mr));
g_free(name);
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 96/97] hw/core/loader: Fix possible crash in rom_copy()
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (94 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 95/97] s390: PCI: fix IOMMU region init Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-01 23:46 ` [PATCH 97/97] scsi: lsi: exit infinite loop while executing script (CVE-2019-12068) Michael Roth
` (5 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Thomas Huth, qemu-stable
From: Thomas Huth <thuth@redhat.com>
Both, "rom->addr" and "addr" are derived from the binary image
that can be loaded with the "-kernel" paramer. The code in
rom_copy() then calculates:
d = dest + (rom->addr - addr);
and uses "d" as destination in a memcpy() some lines later. Now with
bad kernel images, it is possible that rom->addr is smaller than addr,
thus "rom->addr - addr" gets negative and the memcpy() then tries to
copy contents from the image to a bad memory location. This could
maybe be used to inject code from a kernel image into the QEMU binary,
so we better fix it with an additional sanity check here.
Cc: qemu-stable@nongnu.org
Reported-by: Guangming Liu
Buglink: https://bugs.launchpad.net/qemu/+bug/1844635
Message-Id: <20190925130331.27825-1-thuth@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
(cherry picked from commit e423455c4f23a1a828901c78fe6d03b7dde79319)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/core/loader.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/core/loader.c b/hw/core/loader.c
index fe5cb24122..4ef2095247 100644
--- a/hw/core/loader.c
+++ b/hw/core/loader.c
@@ -1240,7 +1240,7 @@ int rom_copy(uint8_t *dest, hwaddr addr, size_t size)
if (rom->addr + rom->romsize < addr) {
continue;
}
- if (rom->addr > end) {
+ if (rom->addr > end || rom->addr < addr) {
break;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* [PATCH 97/97] scsi: lsi: exit infinite loop while executing script (CVE-2019-12068)
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (95 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 96/97] hw/core/loader: Fix possible crash in rom_copy() Michael Roth
@ 2019-10-01 23:46 ` Michael Roth
2019-10-02 4:40 ` [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Thomas Huth
` (4 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-01 23:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Paolo Bonzini, qemu-stable, Prasad J Pandit
From: Paolo Bonzini <pbonzini@redhat.com>
When executing script in lsi_execute_script(), the LSI scsi adapter
emulator advances 's->dsp' index to read next opcode. This can lead
to an infinite loop if the next opcode is empty. Move the existing
loop exit after 10k iterations so that it covers no-op opcodes as
well.
Reported-by: Bugs SysSec <bugs-syssec@rub.de>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
(cherry picked from commit de594e47659029316bbf9391efb79da0a1a08e08)
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
---
hw/scsi/lsi53c895a.c | 41 +++++++++++++++++++++++++++--------------
1 file changed, 27 insertions(+), 14 deletions(-)
diff --git a/hw/scsi/lsi53c895a.c b/hw/scsi/lsi53c895a.c
index da7239d94f..d3380b6f95 100644
--- a/hw/scsi/lsi53c895a.c
+++ b/hw/scsi/lsi53c895a.c
@@ -184,6 +184,9 @@ static const char *names[] = {
/* Flag set if this is a tagged command. */
#define LSI_TAG_VALID (1 << 16)
+/* Maximum instructions to process. */
+#define LSI_MAX_INSN 10000
+
typedef struct lsi_request {
SCSIRequest *req;
uint32_t tag;
@@ -1131,7 +1134,21 @@ static void lsi_execute_script(LSIState *s)
s->istat1 |= LSI_ISTAT1_SRUN;
again:
- insn_processed++;
+ if (++insn_processed > LSI_MAX_INSN) {
+ /* Some windows drivers make the device spin waiting for a memory
+ location to change. If we have been executed a lot of code then
+ assume this is the case and force an unexpected device disconnect.
+ This is apparently sufficient to beat the drivers into submission.
+ */
+ if (!(s->sien0 & LSI_SIST0_UDC)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "lsi_scsi: inf. loop with UDC masked");
+ }
+ lsi_script_scsi_interrupt(s, LSI_SIST0_UDC, 0);
+ lsi_disconnect(s);
+ trace_lsi_execute_script_stop();
+ return;
+ }
insn = read_dword(s, s->dsp);
if (!insn) {
/* If we receive an empty opcode increment the DSP by 4 bytes
@@ -1568,19 +1585,7 @@ again:
}
}
}
- if (insn_processed > 10000 && s->waiting == LSI_NOWAIT) {
- /* Some windows drivers make the device spin waiting for a memory
- location to change. If we have been executed a lot of code then
- assume this is the case and force an unexpected device disconnect.
- This is apparently sufficient to beat the drivers into submission.
- */
- if (!(s->sien0 & LSI_SIST0_UDC)) {
- qemu_log_mask(LOG_GUEST_ERROR,
- "lsi_scsi: inf. loop with UDC masked");
- }
- lsi_script_scsi_interrupt(s, LSI_SIST0_UDC, 0);
- lsi_disconnect(s);
- } else if (s->istat1 & LSI_ISTAT1_SRUN && s->waiting == LSI_NOWAIT) {
+ if (s->istat1 & LSI_ISTAT1_SRUN && s->waiting == LSI_NOWAIT) {
if (s->dcntl & LSI_DCNTL_SSM) {
lsi_script_dma_interrupt(s, LSI_DSTAT_SSI);
} else {
@@ -1968,6 +1973,10 @@ static void lsi_reg_writeb(LSIState *s, int offset, uint8_t val)
case 0x2f: /* DSP[24:31] */
s->dsp &= 0x00ffffff;
s->dsp |= val << 24;
+ /*
+ * FIXME: if s->waiting != LSI_NOWAIT, this will only execute one
+ * instruction. Is this correct?
+ */
if ((s->dmode & LSI_DMODE_MAN) == 0
&& (s->istat1 & LSI_ISTAT1_SRUN) == 0)
lsi_execute_script(s);
@@ -1986,6 +1995,10 @@ static void lsi_reg_writeb(LSIState *s, int offset, uint8_t val)
break;
case 0x3b: /* DCNTL */
s->dcntl = val & ~(LSI_DCNTL_PFF | LSI_DCNTL_STD);
+ /*
+ * FIXME: if s->waiting != LSI_NOWAIT, this will only execute one
+ * instruction. Is this correct?
+ */
if ((val & LSI_DCNTL_STD) && (s->istat1 & LSI_ISTAT1_SRUN) == 0)
lsi_execute_script(s);
break;
--
2.17.1
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (96 preceding siblings ...)
2019-10-01 23:46 ` [PATCH 97/97] scsi: lsi: exit infinite loop while executing script (CVE-2019-12068) Michael Roth
@ 2019-10-02 4:40 ` Thomas Huth
2019-10-03 18:56 ` Michael Roth
2019-10-03 22:43 ` Philippe Mathieu-Daudé
` (3 subsequent siblings)
101 siblings, 1 reply; 107+ messages in thread
From: Thomas Huth @ 2019-10-02 4:40 UTC (permalink / raw)
To: Michael Roth, qemu-devel
Cc: Michal Privoznik, Marc-André Lureau,
Philippe Mathieu-Daudé,
qemu-stable, Igor Mammedov
On 02/10/2019 01.44, Michael Roth wrote:
> Hi everyone,
>
> The following new patches are queued for QEMU stable v4.0.1:
>
> https://github.com/mdroth/qemu/commits/stable-4.0-staging
>
> The release is planned for 2019-10-17:
>
> https://wiki.qemu.org/Planning/4.0
>
> Please respond here or CC qemu-stable@nongnu.org on any patches you
> think should be included in the release.
>
Would it make sense to include the slirp update:
e1a4a24d262ba5ac74ea1795adb3ab1cd574c7fb
"slirp: update with CVE-2019-14378 fix"
?
And maybe these commits:
22235bb609c18547cf6b215bad1f9d2ec56ad371
"pc-dimm: fix crash when invalid slot number is used"
95667c3be0c9f5fc62f58fe845879250f63f7d32
"nvme: Set number of queues later in nvme_init()"
c0bccee9b40ec58c9d165b406ae3d4f63652ce53
"hw/ssi/mss-spi: Avoid crash when reading empty RX FIFO"
a09ef5040477643a7026703199d8781fe048d3a8
"hw/display/xlnx_dp: Avoid crash when reading empty RX FIFO"
Thomas
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-02 4:40 ` [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Thomas Huth
@ 2019-10-03 18:56 ` Michael Roth
0 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-03 18:56 UTC (permalink / raw)
To: Thomas Huth, qemu-devel
Cc: Michal Privoznik, Igor Mammedov, Philippe Mathieu-Daudé,
qemu-stable, Marc-André Lureau
Quoting Thomas Huth (2019-10-01 23:40:49)
> On 02/10/2019 01.44, Michael Roth wrote:
> > Hi everyone,
> >
> > The following new patches are queued for QEMU stable v4.0.1:
> >
> > https://github.com/mdroth/qemu/commits/stable-4.0-staging
> >
> > The release is planned for 2019-10-17:
> >
> > https://wiki.qemu.org/Planning/4.0
> >
> > Please respond here or CC qemu-stable@nongnu.org on any patches you
> > think should be included in the release.
> >
> Would it make sense to include the slirp update:
>
> e1a4a24d262ba5ac74ea1795adb3ab1cd574c7fb
> "slirp: update with CVE-2019-14378 fix"
4.0 hadn't yet moved to using slirp submodule, but the following
patches are included here and should provide the equivalent fixes:
slirp: ip_reass: Fix use after free
slirp: Fix heap overflow in ip_reass on big packet input
>
> ?
>
> And maybe these commits:
>
> 22235bb609c18547cf6b215bad1f9d2ec56ad371
> "pc-dimm: fix crash when invalid slot number is used"
>
> 95667c3be0c9f5fc62f58fe845879250f63f7d32
> "nvme: Set number of queues later in nvme_init()"
>
> c0bccee9b40ec58c9d165b406ae3d4f63652ce53
> "hw/ssi/mss-spi: Avoid crash when reading empty RX FIFO"
>
> a09ef5040477643a7026703199d8781fe048d3a8
> "hw/display/xlnx_dp: Avoid crash when reading empty RX FIFO"
Applied, thanks!
>
> Thomas
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (97 preceding siblings ...)
2019-10-02 4:40 ` [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Thomas Huth
@ 2019-10-03 22:43 ` Philippe Mathieu-Daudé
2019-10-07 15:57 ` Alexandr Iarygin
` (2 subsequent siblings)
101 siblings, 0 replies; 107+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-10-03 22:43 UTC (permalink / raw)
To: Michael Roth, qemu-devel; +Cc: qemu-stable
On 10/2/19 1:44 AM, Michael Roth wrote:
> Hi everyone,
>
> The following new patches are queued for QEMU stable v4.0.1:
>
> https://github.com/mdroth/qemu/commits/stable-4.0-staging
>
> The release is planned for 2019-10-17:
>
> https://wiki.qemu.org/Planning/4.0
>
> Please respond here or CC qemu-stable@nongnu.org on any patches you
> think should be included in the release.
>
> Note that this update falls outside the normal stable release support
> window (~1 development cycle), but is being released now since it was
> delayed from its intended release date.
This commit fix data corruption introduced by ee5d0f89de3:
commit 04da973501b591525ce68c2925c61c8886badd4d
Author: Li Qiang <liq3ea@163.com>
Date: Wed Apr 24 07:06:41 2019 -0700
hw/nvram/fw_cfg: Store 'reboot-timeout' as little endian
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (98 preceding siblings ...)
2019-10-03 22:43 ` Philippe Mathieu-Daudé
@ 2019-10-07 15:57 ` Alexandr Iarygin
2019-10-08 13:04 ` Philippe Mathieu-Daudé
2019-10-11 9:51 ` Anthony PERARD
101 siblings, 0 replies; 107+ messages in thread
From: Alexandr Iarygin @ 2019-10-07 15:57 UTC (permalink / raw)
To: Michael Roth, qemu-devel; +Cc: qemu-stable
Michael Roth <mdroth@linux.vnet.ibm.com> writes:
> Hi everyone,
>
> The following new patches are queued for QEMU stable v4.0.1:
>
> https://github.com/mdroth/qemu/commits/stable-4.0-staging
>
> The release is planned for 2019-10-17:
>
> https://wiki.qemu.org/Planning/4.0
Hi,
How about
commit fd392cfa8e6fb0dc34bd0327fc356dfbf6edf1fd
Author: Yury Kotov <yury-kotov@yandex-team.ru>
Date: Mon Apr 8 14:33:43 2019 +0300
migration: Fix use-after-free during process exit
commit ebb6ff25cd888a52a64a9adc3692541c6d1d9a42
Author: Philippe Mathieu-Daudé <philmd@redhat.com>
Date: Fri Aug 16 19:15:03 2019 +0200
virtio-blk: Cancel the pending BH when the dataplane is reset
and https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg04618.html
?
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (99 preceding siblings ...)
2019-10-07 15:57 ` Alexandr Iarygin
@ 2019-10-08 13:04 ` Philippe Mathieu-Daudé
2019-10-09 14:17 ` Michael Roth
2019-10-11 9:51 ` Anthony PERARD
101 siblings, 1 reply; 107+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-10-08 13:04 UTC (permalink / raw)
To: Michael Roth, qemu-devel; +Cc: qemu-stable
Hi Michael,
On 10/2/19 1:44 AM, Michael Roth wrote:
> Hi everyone,
>
> The following new patches are queued for QEMU stable v4.0.1:
>
> https://github.com/mdroth/qemu/commits/stable-4.0-staging
>
> The release is planned for 2019-10-17:
>
> https://wiki.qemu.org/Planning/4.0
>
> Please respond here or CC qemu-stable@nongnu.org on any patches you
> think should be included in the release.
Since it is a "release", these probably fit:
commit 45c61c6c23918e3b05ed9ecac5b2328ebae5f774
Author: Michael Roth <mdroth@linux.vnet.ibm.com>
Date: Thu Sep 12 18:12:01 2019 -0500
make-release: pull in edk2 submodules so we can build it from tarballs
commit f3e330e3c319160ac04954399b5a10afc965098c
Author: Michael Roth <mdroth@linux.vnet.ibm.com>
Date: Thu Sep 12 18:12:02 2019 -0500
roms/Makefile.edk2: don't pull in submodules when building from tarball
When is the next qemu-stable release scheduled? Do we care about Python2
use for this one?
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-08 13:04 ` Philippe Mathieu-Daudé
@ 2019-10-09 14:17 ` Michael Roth
2019-10-09 14:23 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 107+ messages in thread
From: Michael Roth @ 2019-10-09 14:17 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel; +Cc: qemu-stable
Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
> Hi Michael,
>
> On 10/2/19 1:44 AM, Michael Roth wrote:
> > Hi everyone,
> >
> > The following new patches are queued for QEMU stable v4.0.1:
> >
> > https://github.com/mdroth/qemu/commits/stable-4.0-staging
> >
> > The release is planned for 2019-10-17:
> >
> > https://wiki.qemu.org/Planning/4.0
> >
> > Please respond here or CC qemu-stable@nongnu.org on any patches you
> > think should be included in the release.
>
> Since it is a "release", these probably fit:
>
> commit 45c61c6c23918e3b05ed9ecac5b2328ebae5f774
> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
> Date: Thu Sep 12 18:12:01 2019 -0500
>
> make-release: pull in edk2 submodules so we can build it from tarballs
>
> commit f3e330e3c319160ac04954399b5a10afc965098c
> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
> Date: Thu Sep 12 18:12:02 2019 -0500
>
> roms/Makefile.edk2: don't pull in submodules when building from tarball
>
> When is the next qemu-stable release scheduled? Do we care about Python2
> use for this one?
4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay
since presumably downstreams would've crossed that bridge with 4.1.0, assuming
you're thinking of the changes in your recent edk2 pull.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-09 14:17 ` Michael Roth
@ 2019-10-09 14:23 ` Philippe Mathieu-Daudé
2019-10-17 22:19 ` Michael Roth
0 siblings, 1 reply; 107+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-10-09 14:23 UTC (permalink / raw)
To: Michael Roth, qemu-devel; +Cc: qemu-stable
On 10/9/19 4:17 PM, Michael Roth wrote:
> Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
>> Hi Michael,
>>
>> On 10/2/19 1:44 AM, Michael Roth wrote:
>>> Hi everyone,
>>>
>>> The following new patches are queued for QEMU stable v4.0.1:
>>>
>>> https://github.com/mdroth/qemu/commits/stable-4.0-staging
>>>
>>> The release is planned for 2019-10-17:
>>>
>>> https://wiki.qemu.org/Planning/4.0
>>>
>>> Please respond here or CC qemu-stable@nongnu.org on any patches you
>>> think should be included in the release.
>>
>> Since it is a "release", these probably fit:
>>
>> commit 45c61c6c23918e3b05ed9ecac5b2328ebae5f774
>> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
>> Date: Thu Sep 12 18:12:01 2019 -0500
>>
>> make-release: pull in edk2 submodules so we can build it from tarballs
>>
>> commit f3e330e3c319160ac04954399b5a10afc965098c
>> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
>> Date: Thu Sep 12 18:12:02 2019 -0500
>>
>> roms/Makefile.edk2: don't pull in submodules when building from tarball
>>
>> When is the next qemu-stable release scheduled? Do we care about Python2
>> use for this one?
>
> 4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay
> since presumably downstreams would've crossed that bridge with 4.1.0, assuming
> you're thinking of the changes in your recent edk2 pull.
Yes.
Do you mind adding these dates to https://wiki.qemu.org/Planning/4.1?
Thanks,
Phil.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-09 14:23 ` Philippe Mathieu-Daudé
@ 2019-10-17 22:19 ` Michael Roth
0 siblings, 0 replies; 107+ messages in thread
From: Michael Roth @ 2019-10-17 22:19 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel; +Cc: qemu-stable
Quoting Philippe Mathieu-Daudé (2019-10-09 09:23:22)
> On 10/9/19 4:17 PM, Michael Roth wrote:
> > Quoting Philippe Mathieu-Daudé (2019-10-08 08:04:52)
> >> Hi Michael,
> >>
> >> On 10/2/19 1:44 AM, Michael Roth wrote:
> >>> Hi everyone,
> >>>
> >>> The following new patches are queued for QEMU stable v4.0.1:
> >>>
> >>> https://github.com/mdroth/qemu/commits/stable-4.0-staging
> >>>
> >>> The release is planned for 2019-10-17:
> >>>
> >>> https://wiki.qemu.org/Planning/4.0
> >>>
> >>> Please respond here or CC qemu-stable@nongnu.org on any patches you
> >>> think should be included in the release.
> >>
> >> Since it is a "release", these probably fit:
> >>
> >> commit 45c61c6c23918e3b05ed9ecac5b2328ebae5f774
> >> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
> >> Date: Thu Sep 12 18:12:01 2019 -0500
> >>
> >> make-release: pull in edk2 submodules so we can build it from tarballs
> >>
> >> commit f3e330e3c319160ac04954399b5a10afc965098c
> >> Author: Michael Roth <mdroth@linux.vnet.ibm.com>
> >> Date: Thu Sep 12 18:12:02 2019 -0500
> >>
> >> roms/Makefile.edk2: don't pull in submodules when building from tarball
> >>
> >> When is the next qemu-stable release scheduled? Do we care about Python2
> >> use for this one?
> >
> > 4.1.1, likely early November during 4.2 hard-freeze. I think Python2 is okay
> > since presumably downstreams would've crossed that bridge with 4.1.0, assuming
> > you're thinking of the changes in your recent edk2 pull.
>
> Yes.
>
> Do you mind adding these dates to https://wiki.qemu.org/Planning/4.1?
Sure, I've set the freeze date for November 18th, and the release for
the 21st.
>
> Thanks,
>
> Phil.
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10
2019-10-01 23:44 [PATCH 00/97] Patch Round-up for stable 4.0.1, freeze on 2019-10-10 Michael Roth
` (100 preceding siblings ...)
2019-10-08 13:04 ` Philippe Mathieu-Daudé
@ 2019-10-11 9:51 ` Anthony PERARD
101 siblings, 0 replies; 107+ messages in thread
From: Anthony PERARD @ 2019-10-11 9:51 UTC (permalink / raw)
To: Michael Roth; +Cc: qemu-devel, qemu-stable
On Tue, Oct 01, 2019 at 06:44:39PM -0500, Michael Roth wrote:
> Hi everyone,
Hi Michael,
> Please respond here or CC qemu-stable@nongnu.org on any patches you
> think should be included in the release.
[...]
> Anthony PERARD (1):
> xen-bus: Fix backend state transition on device reset
Could you cherry-pick df6180bb56cd03949c2c64083da58755fed81a61
"xen-bus: check whether the frontend is active during device reset..."
as well, to fix the fix?
Thanks,
--
Anthony PERARD
^ permalink raw reply [flat|nested] 107+ messages in thread