[Qemu-devel] [PULL 0/7] Block patches

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 0ab4537f08e09b13788db67efd760592fb7db769:

  Merge remote-tracking branch 'remotes/stefanberger/tags/pull-tpm-2018-03-07-1' into staging (2018-03-08 12:56:39 +0000)

are available in the Git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 4486e89c219c0d1b9bd8dfa0b1dd5b0d51ff2268:

  vl: introduce vm_shutdown() (2018-03-08 17:38:51 +0000)

----------------------------------------------------------------

----------------------------------------------------------------

Deepa Srinivasan (1):
  block: Fix qemu crash when using scsi-block

Fam Zheng (1):
  README: Fix typo 'git-publish'

Sergio Lopez (1):
  virtio-blk: dataplane: Don't batch notifications if EVENT_IDX is
    present

Stefan Hajnoczi (4):
  block: add aio_wait_bh_oneshot()
  virtio-blk: fix race between .ioeventfd_stop() and vq handler
  virtio-scsi: fix race between .ioeventfd_stop() and vq handler
  vl: introduce vm_shutdown()

 include/block/aio-wait.h        | 13 +++++++++++
 include/sysemu/iothread.h       |  1 -
 include/sysemu/sysemu.h         |  1 +
 block/block-backend.c           | 51 ++++++++++++++++++++---------------------
 cpus.c                          | 16 ++++++++++---
 hw/block/dataplane/virtio-blk.c | 39 +++++++++++++++++++++++--------
 hw/scsi/virtio-scsi-dataplane.c |  9 ++++----
 iothread.c                      | 31 -------------------------
 util/aio-wait.c                 | 31 +++++++++++++++++++++++++
 vl.c                            | 13 +++--------
 README                          |  2 +-
 11 files changed, 122 insertions(+), 85 deletions(-)

-- 
2.14.3

[Qemu-devel] [PULL 1/7] block: Fix qemu crash when using scsi-block

[Stefan Hajnoczi]

From: Deepa Srinivasan <deepa.srinivasan@oracle.com>

Starting qemu with the following arguments causes qemu to segfault:
... -device lsi,id=lsi0 -drive file=iscsi:<...>,format=raw,if=none,node-name=
iscsi1 -device scsi-block,bus=lsi0.0,id=<...>,drive=iscsi1

This patch fixes blk_aio_ioctl() so it does not pass stack addresses to
blk_aio_ioctl_entry() which may be invoked after blk_aio_ioctl() returns. More
details about the bug follow.

blk_aio_ioctl() invokes blk_aio_prwv() with blk_aio_ioctl_entry as the
coroutine parameter. blk_aio_prwv() ultimately calls aio_co_enter().

When blk_aio_ioctl() is executed from within a coroutine context (e.g.
iscsi_bh_cb()), aio_co_enter() adds the coroutine (blk_aio_ioctl_entry) to
the current coroutine's wakeup queue. blk_aio_ioctl() then returns.

When blk_aio_ioctl_entry() executes later, it accesses an invalid pointer:
....
    BlkRwCo *rwco = &acb->rwco;

    rwco->ret = blk_co_ioctl(rwco->blk, rwco->offset,
                             rwco->qiov->iov[0].iov_base);  <--- qiov is
                                                                 invalid here
...

In the case when blk_aio_ioctl() is called from a non-coroutine context,
blk_aio_ioctl_entry() executes immediately. But if bdrv_co_ioctl() calls
qemu_coroutine_yield(), blk_aio_ioctl() will return. When the coroutine
execution is complete, control returns to blk_aio_ioctl_entry() after the call
to blk_co_ioctl(). There is no invalid reference after this point, but the
function is still holding on to invalid pointers.

The fix is to change blk_aio_prwv() to accept a void pointer for the IO buffer
rather than a QEMUIOVector. blk_aio_prwv() passes this through in BlkRwCo and the
coroutine function casts it to QEMUIOVector or uses the void pointer directly.

Signed-off-by: Deepa Srinivasan <deepa.srinivasan@oracle.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Mark Kanda <mark.kanda@oracle.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/block-backend.c | 51 +++++++++++++++++++++++++--------------------------
 1 file changed, 25 insertions(+), 26 deletions(-)

diff --git a/block/block-backend.c b/block/block-backend.c
index b3c790e2bd..f2e0a855ff 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -1150,7 +1150,7 @@ int coroutine_fn blk_co_pwritev(BlockBackend *blk, int64_t offset,
 typedef struct BlkRwCo {
     BlockBackend *blk;
     int64_t offset;
-    QEMUIOVector *qiov;
+    void *iobuf;
     int ret;
     BdrvRequestFlags flags;
 } BlkRwCo;
@@ -1158,17 +1158,19 @@ typedef struct BlkRwCo {
 static void blk_read_entry(void *opaque)
 {
     BlkRwCo *rwco = opaque;
+    QEMUIOVector *qiov = rwco->iobuf;
 
-    rwco->ret = blk_co_preadv(rwco->blk, rwco->offset, rwco->qiov->size,
-                              rwco->qiov, rwco->flags);
+    rwco->ret = blk_co_preadv(rwco->blk, rwco->offset, qiov->size,
+                              qiov, rwco->flags);
 }
 
 static void blk_write_entry(void *opaque)
 {
     BlkRwCo *rwco = opaque;
+    QEMUIOVector *qiov = rwco->iobuf;
 
-    rwco->ret = blk_co_pwritev(rwco->blk, rwco->offset, rwco->qiov->size,
-                               rwco->qiov, rwco->flags);
+    rwco->ret = blk_co_pwritev(rwco->blk, rwco->offset, qiov->size,
+                               qiov, rwco->flags);
 }
 
 static int blk_prw(BlockBackend *blk, int64_t offset, uint8_t *buf,
@@ -1188,7 +1190,7 @@ static int blk_prw(BlockBackend *blk, int64_t offset, uint8_t *buf,
     rwco = (BlkRwCo) {
         .blk    = blk,
         .offset = offset,
-        .qiov   = &qiov,
+        .iobuf  = &qiov,
         .flags  = flags,
         .ret    = NOT_DONE,
     };
@@ -1296,7 +1298,7 @@ static void blk_aio_complete_bh(void *opaque)
 }
 
 static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes,
-                                QEMUIOVector *qiov, CoroutineEntry co_entry,
+                                void *iobuf, CoroutineEntry co_entry,
                                 BdrvRequestFlags flags,
                                 BlockCompletionFunc *cb, void *opaque)
 {
@@ -1308,7 +1310,7 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes,
     acb->rwco = (BlkRwCo) {
         .blk    = blk,
         .offset = offset,
-        .qiov   = qiov,
+        .iobuf  = iobuf,
         .flags  = flags,
         .ret    = NOT_DONE,
     };
@@ -1331,10 +1333,11 @@ static void blk_aio_read_entry(void *opaque)
 {
     BlkAioEmAIOCB *acb = opaque;
     BlkRwCo *rwco = &acb->rwco;
+    QEMUIOVector *qiov = rwco->iobuf;
 
-    assert(rwco->qiov->size == acb->bytes);
+    assert(qiov->size == acb->bytes);
     rwco->ret = blk_co_preadv(rwco->blk, rwco->offset, acb->bytes,
-                              rwco->qiov, rwco->flags);
+                              qiov, rwco->flags);
     blk_aio_complete(acb);
 }
 
@@ -1342,10 +1345,11 @@ static void blk_aio_write_entry(void *opaque)
 {
     BlkAioEmAIOCB *acb = opaque;
     BlkRwCo *rwco = &acb->rwco;
+    QEMUIOVector *qiov = rwco->iobuf;
 
-    assert(!rwco->qiov || rwco->qiov->size == acb->bytes);
+    assert(!qiov || qiov->size == acb->bytes);
     rwco->ret = blk_co_pwritev(rwco->blk, rwco->offset, acb->bytes,
-                               rwco->qiov, rwco->flags);
+                               qiov, rwco->flags);
     blk_aio_complete(acb);
 }
 
@@ -1474,8 +1478,10 @@ int blk_co_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
 static void blk_ioctl_entry(void *opaque)
 {
     BlkRwCo *rwco = opaque;
+    QEMUIOVector *qiov = rwco->iobuf;
+
     rwco->ret = blk_co_ioctl(rwco->blk, rwco->offset,
-                             rwco->qiov->iov[0].iov_base);
+                             qiov->iov[0].iov_base);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
@@ -1488,24 +1494,15 @@ static void blk_aio_ioctl_entry(void *opaque)
     BlkAioEmAIOCB *acb = opaque;
     BlkRwCo *rwco = &acb->rwco;
 
-    rwco->ret = blk_co_ioctl(rwco->blk, rwco->offset,
-                             rwco->qiov->iov[0].iov_base);
+    rwco->ret = blk_co_ioctl(rwco->blk, rwco->offset, rwco->iobuf);
+
     blk_aio_complete(acb);
 }
 
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque)
 {
-    QEMUIOVector qiov;
-    struct iovec iov;
-
-    iov = (struct iovec) {
-        .iov_base = buf,
-        .iov_len = 0,
-    };
-    qemu_iovec_init_external(&qiov, &iov, 1);
-
-    return blk_aio_prwv(blk, req, 0, &qiov, blk_aio_ioctl_entry, 0, cb, opaque);
+    return blk_aio_prwv(blk, req, 0, buf, blk_aio_ioctl_entry, 0, cb, opaque);
 }
 
 int blk_co_pdiscard(BlockBackend *blk, int64_t offset, int bytes)
@@ -1949,7 +1946,9 @@ int blk_truncate(BlockBackend *blk, int64_t offset, PreallocMode prealloc,
 static void blk_pdiscard_entry(void *opaque)
 {
     BlkRwCo *rwco = opaque;
-    rwco->ret = blk_co_pdiscard(rwco->blk, rwco->offset, rwco->qiov->size);
+    QEMUIOVector *qiov = rwco->iobuf;
+
+    rwco->ret = blk_co_pdiscard(rwco->blk, rwco->offset, qiov->size);
 }
 
 int blk_pdiscard(BlockBackend *blk, int64_t offset, int bytes)
-- 
2.14.3

[Qemu-devel] [PULL 2/7] README: Fix typo 'git-publish'

[Stefan Hajnoczi]

From: Fam Zheng <famz@redhat.com>

Reported-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-id: 20180306024328.19195-1-famz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 README | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README b/README
index 7833b97365..49a9fd09cd 100644
--- a/README
+++ b/README
@@ -73,7 +73,7 @@ The QEMU website is also maintained under source control.
   git clone git://git.qemu.org/qemu-web.git
   https://www.qemu.org/2017/02/04/the-new-qemu-website-is-up/
 
-A 'git-profile' utility was created to make above process less
+A 'git-publish' utility was created to make above process less
 cumbersome, and is highly recommended for making regular contributions,
 or even just for sending consecutive patch series revisions. It also
 requires a working 'git send-email' setup, and by default doesn't
-- 
2.14.3

[Qemu-devel] [PULL 3/7] virtio-blk: dataplane: Don't batch notifications if EVENT_IDX is present

[Stefan Hajnoczi]

From: Sergio Lopez <slp@redhat.com>

Commit 5b2ffbe4d99843fd8305c573a100047a8c962327 ("virtio-blk: dataplane:
notify guest as a batch") deferred guest notification to a BH in order
batch notifications, with purpose of avoiding flooding the guest with
interruptions.

This optimization came with a cost. The average latency perceived in the
guest is increased by a few microseconds, but also when multiple IO
operations finish at the same time, the guest won't be notified until
all completions from each operation has been run. On the contrary,
virtio-scsi issues the notification at the end of each completion.

On the other hand, nowadays we have the EVENT_IDX feature that allows a
better coordination between QEMU and the Guest OS to avoid sending
unnecessary interruptions.

With this change, virtio-blk/dataplane only batches notifications if the
EVENT_IDX feature is not present.

Some numbers obtained with fio (ioengine=sync, iodepth=1, direct=1):
 - Test specs:
   * fio-3.4 (ioengine=sync, iodepth=1, direct=1)
   * qemu master
   * virtio-blk with a dedicated iothread (default poll-max-ns)
   * backend: null_blk nr_devices=1 irqmode=2 completion_nsec=280000
   * 8 vCPUs pinned to isolated physical cores
   * Emulator and iothread also pinned to separate isolated cores
   * variance between runs < 1%

 - Not patched
   * numjobs=1:  lat_avg=327.32  irqs=29998
   * numjobs=4:  lat_avg=337.89  irqs=29073
   * numjobs=8:  lat_avg=342.98  irqs=28643

 - Patched:
   * numjobs=1:  lat_avg=323.92  irqs=30262
   * numjobs=4:  lat_avg=332.65  irqs=29520
   * numjobs=8:  lat_avg=335.54  irqs=29323

Signed-off-by: Sergio Lopez <slp@redhat.com>
Message-id: 20180307114459.26636-1-slp@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/dataplane/virtio-blk.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c
index 2cb990997e..c46253a924 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -34,6 +34,7 @@ struct VirtIOBlockDataPlane {
     VirtIODevice *vdev;
     QEMUBH *bh;                     /* bh for guest notification */
     unsigned long *batch_notify_vqs;
+    bool batch_notifications;
 
     /* Note that these EventNotifiers are assigned by value.  This is
      * fine as long as you do not call event_notifier_cleanup on them
@@ -47,8 +48,12 @@ struct VirtIOBlockDataPlane {
 /* Raise an interrupt to signal guest, if necessary */
 void virtio_blk_data_plane_notify(VirtIOBlockDataPlane *s, VirtQueue *vq)
 {
-    set_bit(virtio_get_queue_index(vq), s->batch_notify_vqs);
-    qemu_bh_schedule(s->bh);
+    if (s->batch_notifications) {
+        set_bit(virtio_get_queue_index(vq), s->batch_notify_vqs);
+        qemu_bh_schedule(s->bh);
+    } else {
+        virtio_notify_irqfd(s->vdev, vq);
+    }
 }
 
 static void notify_guest_bh(void *opaque)
@@ -177,6 +182,12 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev)
 
     s->starting = true;
 
+    if (!virtio_vdev_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        s->batch_notifications = true;
+    } else {
+        s->batch_notifications = false;
+    }
+
     /* Set up guest notifier (irq) */
     r = k->set_guest_notifiers(qbus->parent, nvqs, true);
     if (r != 0) {
-- 
2.14.3

[Qemu-devel] [PULL 4/7] block: add aio_wait_bh_oneshot()

[Stefan Hajnoczi]

Sometimes it's necessary for the main loop thread to run a BH in an
IOThread and wait for its completion.  This primitive is useful during
startup/shutdown to synchronize and avoid race conditions.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20180307144205.20619-2-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/block/aio-wait.h | 13 +++++++++++++
 util/aio-wait.c          | 31 +++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/include/block/aio-wait.h b/include/block/aio-wait.h
index a48c744fa8..f7a3972200 100644
--- a/include/block/aio-wait.h
+++ b/include/block/aio-wait.h
@@ -113,4 +113,17 @@ typedef struct {
  */
 void aio_wait_kick(AioWait *wait);
 
+/**
+ * aio_wait_bh_oneshot:
+ * @ctx: the aio context
+ * @cb: the BH callback function
+ * @opaque: user data for the BH callback function
+ *
+ * Run a BH in @ctx and wait for it to complete.
+ *
+ * Must be called from the main loop thread with @ctx acquired exactly once.
+ * Note that main loop event processing may occur.
+ */
+void aio_wait_bh_oneshot(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
+
 #endif /* QEMU_AIO_WAIT */
diff --git a/util/aio-wait.c b/util/aio-wait.c
index a487cdb852..975afddf4c 100644
--- a/util/aio-wait.c
+++ b/util/aio-wait.c
@@ -38,3 +38,34 @@ void aio_wait_kick(AioWait *wait)
         aio_bh_schedule_oneshot(qemu_get_aio_context(), dummy_bh_cb, NULL);
     }
 }
+
+typedef struct {
+    AioWait wait;
+    bool done;
+    QEMUBHFunc *cb;
+    void *opaque;
+} AioWaitBHData;
+
+/* Context: BH in IOThread */
+static void aio_wait_bh(void *opaque)
+{
+    AioWaitBHData *data = opaque;
+
+    data->cb(data->opaque);
+
+    data->done = true;
+    aio_wait_kick(&data->wait);
+}
+
+void aio_wait_bh_oneshot(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
+{
+    AioWaitBHData data = {
+        .cb = cb,
+        .opaque = opaque,
+    };
+
+    assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+    aio_bh_schedule_oneshot(ctx, aio_wait_bh, &data);
+    AIO_WAIT_WHILE(&data.wait, ctx, !data.done);
+}
-- 
2.14.3

[Qemu-devel] [PULL 5/7] virtio-blk: fix race between .ioeventfd_stop() and vq handler

[Stefan Hajnoczi]

If the main loop thread invokes .ioeventfd_stop() just as the vq handler
function begins in the IOThread then the handler may lose the race for
the AioContext lock.  By the time the vq handler is able to acquire the
AioContext lock the ioeventfd has already been removed and the handler
isn't supposed to run anymore!

Use the new aio_wait_bh_oneshot() function to perform ioeventfd removal
from within the IOThread.  This way no races with the vq handler are
possible.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20180307144205.20619-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/block/dataplane/virtio-blk.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c
index c46253a924..101f32cf66 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -240,6 +240,22 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev)
     return -ENOSYS;
 }
 
+/* Stop notifications for new requests from guest.
+ *
+ * Context: BH in IOThread
+ */
+static void virtio_blk_data_plane_stop_bh(void *opaque)
+{
+    VirtIOBlockDataPlane *s = opaque;
+    unsigned i;
+
+    for (i = 0; i < s->conf->num_queues; i++) {
+        VirtQueue *vq = virtio_get_queue(s->vdev, i);
+
+        virtio_queue_aio_set_host_notifier_handler(vq, s->ctx, NULL);
+    }
+}
+
 /* Context: QEMU global mutex held */
 void virtio_blk_data_plane_stop(VirtIODevice *vdev)
 {
@@ -264,13 +280,7 @@ void virtio_blk_data_plane_stop(VirtIODevice *vdev)
     trace_virtio_blk_data_plane_stop(s);
 
     aio_context_acquire(s->ctx);
-
-    /* Stop notifications for new requests from guest */
-    for (i = 0; i < nvqs; i++) {
-        VirtQueue *vq = virtio_get_queue(s->vdev, i);
-
-        virtio_queue_aio_set_host_notifier_handler(vq, s->ctx, NULL);
-    }
+    aio_wait_bh_oneshot(s->ctx, virtio_blk_data_plane_stop_bh, s);
 
     /* Drain and switch bs back to the QEMU main loop */
     blk_set_aio_context(s->conf->conf.blk, qemu_get_aio_context());
-- 
2.14.3

[Qemu-devel] [PULL 6/7] virtio-scsi: fix race between .ioeventfd_stop() and vq handler

[Stefan Hajnoczi]

If the main loop thread invokes .ioeventfd_stop() just as the vq handler
function begins in the IOThread then the handler may lose the race for
the AioContext lock.  By the time the vq handler is able to acquire the
AioContext lock the ioeventfd has already been removed and the handler
isn't supposed to run anymore!

Use the new aio_wait_bh_oneshot() function to perform ioeventfd removal
from within the IOThread.  This way no races with the vq handler are
possible.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20180307144205.20619-4-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/scsi/virtio-scsi-dataplane.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/hw/scsi/virtio-scsi-dataplane.c b/hw/scsi/virtio-scsi-dataplane.c
index 1c33322ba6..912e5005d8 100644
--- a/hw/scsi/virtio-scsi-dataplane.c
+++ b/hw/scsi/virtio-scsi-dataplane.c
@@ -107,9 +107,10 @@ static int virtio_scsi_vring_init(VirtIOSCSI *s, VirtQueue *vq, int n,
     return 0;
 }
 
-/* assumes s->ctx held */
-static void virtio_scsi_clear_aio(VirtIOSCSI *s)
+/* Context: BH in IOThread */
+static void virtio_scsi_dataplane_stop_bh(void *opaque)
 {
+    VirtIOSCSI *s = opaque;
     VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(s);
     int i;
 
@@ -171,7 +172,7 @@ int virtio_scsi_dataplane_start(VirtIODevice *vdev)
     return 0;
 
 fail_vrings:
-    virtio_scsi_clear_aio(s);
+    aio_wait_bh_oneshot(s->ctx, virtio_scsi_dataplane_stop_bh, s);
     aio_context_release(s->ctx);
     for (i = 0; i < vs->conf.num_queues + 2; i++) {
         virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false);
@@ -207,7 +208,7 @@ void virtio_scsi_dataplane_stop(VirtIODevice *vdev)
     s->dataplane_stopping = true;
 
     aio_context_acquire(s->ctx);
-    virtio_scsi_clear_aio(s);
+    aio_wait_bh_oneshot(s->ctx, virtio_scsi_dataplane_stop_bh, s);
     aio_context_release(s->ctx);
 
     blk_drain_all(); /* ensure there are no in-flight requests */
-- 
2.14.3

[Qemu-devel] [PULL 7/7] vl: introduce vm_shutdown()

[Stefan Hajnoczi]

Commit 00d09fdbbae5f7864ce754913efc84c12fdf9f1a ("vl: pause vcpus before
stopping iothreads") and commit dce8921b2baaf95974af8176406881872067adfa
("iothread: Stop threads before main() quits") tried to work around the
fact that emulation was still active during termination by stopping
iothreads.  They suffer from race conditions:
1. virtio_scsi_handle_cmd_vq() racing with iothread_stop_all() hits the
   virtio_scsi_ctx_check() assertion failure because the BDS AioContext
   has been modified by iothread_stop_all().
2. Guest vq kick racing with main loop termination leaves a readable
   ioeventfd that is handled by the next aio_poll() when external
   clients are enabled again, resulting in unwanted emulation activity.

This patch obsoletes those commits by fully disabling emulation activity
when vcpus are stopped.

Use the new vm_shutdown() function instead of pause_all_vcpus() so that
vm change state handlers are invoked too.  Virtio devices will now stop
their ioeventfds, preventing further emulation activity after vm_stop().

Note that vm_stop(RUN_STATE_SHUTDOWN) cannot be used because it emits a
QMP STOP event that may affect existing clients.

It is no longer necessary to call replay_disable_events() directly since
vm_shutdown() does so already.

Drop iothread_stop_all() since it is no longer used.

Cc: Fam Zheng <famz@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-id: 20180307144205.20619-5-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/sysemu/iothread.h |  1 -
 include/sysemu/sysemu.h   |  1 +
 cpus.c                    | 16 +++++++++++++---
 iothread.c                | 31 -------------------------------
 vl.c                      | 13 +++----------
 5 files changed, 17 insertions(+), 45 deletions(-)

diff --git a/include/sysemu/iothread.h b/include/sysemu/iothread.h
index 799614ffd2..8a7ac2c528 100644
--- a/include/sysemu/iothread.h
+++ b/include/sysemu/iothread.h
@@ -45,7 +45,6 @@ typedef struct {
 char *iothread_get_id(IOThread *iothread);
 IOThread *iothread_by_id(const char *id);
 AioContext *iothread_get_aio_context(IOThread *iothread);
-void iothread_stop_all(void);
 GMainContext *iothread_get_g_main_context(IOThread *iothread);
 
 /*
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index d24ad09f37..356bfdc1c1 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -56,6 +56,7 @@ void vm_start(void);
 int vm_prepare_start(void);
 int vm_stop(RunState state);
 int vm_stop_force_state(RunState state);
+int vm_shutdown(void);
 
 typedef enum WakeupReason {
     /* Always keep QEMU_WAKEUP_REASON_NONE = 0 */
diff --git a/cpus.c b/cpus.c
index 9bcff7d63c..d8fe90eafe 100644
--- a/cpus.c
+++ b/cpus.c
@@ -993,7 +993,7 @@ void cpu_synchronize_all_pre_loadvm(void)
     }
 }
 
-static int do_vm_stop(RunState state)
+static int do_vm_stop(RunState state, bool send_stop)
 {
     int ret = 0;
 
@@ -1002,7 +1002,9 @@ static int do_vm_stop(RunState state)
         pause_all_vcpus();
         runstate_set(state);
         vm_state_notify(0, state);
-        qapi_event_send_stop(&error_abort);
+        if (send_stop) {
+            qapi_event_send_stop(&error_abort);
+        }
     }
 
     bdrv_drain_all();
@@ -1012,6 +1014,14 @@ static int do_vm_stop(RunState state)
     return ret;
 }
 
+/* Special vm_stop() variant for terminating the process.  Historically clients
+ * did not expect a QMP STOP event and so we need to retain compatibility.
+ */
+int vm_shutdown(void)
+{
+    return do_vm_stop(RUN_STATE_SHUTDOWN, false);
+}
+
 static bool cpu_can_run(CPUState *cpu)
 {
     if (cpu->stop) {
@@ -1994,7 +2004,7 @@ int vm_stop(RunState state)
         return 0;
     }
 
-    return do_vm_stop(state);
+    return do_vm_stop(state, true);
 }
 
 /**
diff --git a/iothread.c b/iothread.c
index 2ec5a3bffe..1b3463cb00 100644
--- a/iothread.c
+++ b/iothread.c
@@ -101,18 +101,6 @@ void iothread_stop(IOThread *iothread)
     qemu_thread_join(&iothread->thread);
 }
 
-static int iothread_stop_iter(Object *object, void *opaque)
-{
-    IOThread *iothread;
-
-    iothread = (IOThread *)object_dynamic_cast(object, TYPE_IOTHREAD);
-    if (!iothread) {
-        return 0;
-    }
-    iothread_stop(iothread);
-    return 0;
-}
-
 static void iothread_instance_init(Object *obj)
 {
     IOThread *iothread = IOTHREAD(obj);
@@ -333,25 +321,6 @@ IOThreadInfoList *qmp_query_iothreads(Error **errp)
     return head;
 }
 
-void iothread_stop_all(void)
-{
-    Object *container = object_get_objects_root();
-    BlockDriverState *bs;
-    BdrvNextIterator it;
-
-    for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
-        AioContext *ctx = bdrv_get_aio_context(bs);
-        if (ctx == qemu_get_aio_context()) {
-            continue;
-        }
-        aio_context_acquire(ctx);
-        bdrv_set_aio_context(bs, qemu_get_aio_context());
-        aio_context_release(ctx);
-    }
-
-    object_child_foreach(container, iothread_stop_iter, NULL);
-}
-
 static gpointer iothread_g_main_context_init(gpointer opaque)
 {
     AioContext *ctx;
diff --git a/vl.c b/vl.c
index dae986b352..3ef04ce991 100644
--- a/vl.c
+++ b/vl.c
@@ -4722,17 +4722,10 @@ int main(int argc, char **argv, char **envp)
     os_setup_post();
 
     main_loop();
-    replay_disable_events();
 
-    /* The ordering of the following is delicate.  Stop vcpus to prevent new
-     * I/O requests being queued by the guest.  Then stop IOThreads (this
-     * includes a drain operation and completes all request processing).  At
-     * this point emulated devices are still associated with their IOThreads
-     * (if any) but no longer have any work to do.  Only then can we close
-     * block devices safely because we know there is no more I/O coming.
-     */
-    pause_all_vcpus();
-    iothread_stop_all();
+    /* No more vcpu or device emulation activity beyond this point */
+    vm_shutdown();
+
     bdrv_close_all();
 
     res_free();
-- 
2.14.3

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 9 March 2018 at 13:19, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 0ab4537f08e09b13788db67efd760592fb7db769:
>
>   Merge remote-tracking branch 'remotes/stefanberger/tags/pull-tpm-2018-03-07-1' into staging (2018-03-08 12:56:39 +0000)
>
> are available in the Git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 4486e89c219c0d1b9bd8dfa0b1dd5b0d51ff2268:
>
>   vl: introduce vm_shutdown() (2018-03-08 17:38:51 +0000)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM

Re: [Qemu-devel] [PULL 7/7] vl: introduce vm_shutdown()

[John Snow]

On 03/09/2018 08:19 AM, Stefan Hajnoczi wrote:
> Commit 00d09fdbbae5f7864ce754913efc84c12fdf9f1a ("vl: pause vcpus before
> stopping iothreads") and commit dce8921b2baaf95974af8176406881872067adfa
> ("iothread: Stop threads before main() quits") tried to work around the
> fact that emulation was still active during termination by stopping
> iothreads.  They suffer from race conditions:
> 1. virtio_scsi_handle_cmd_vq() racing with iothread_stop_all() hits the
>    virtio_scsi_ctx_check() assertion failure because the BDS AioContext
>    has been modified by iothread_stop_all().
> 2. Guest vq kick racing with main loop termination leaves a readable
>    ioeventfd that is handled by the next aio_poll() when external
>    clients are enabled again, resulting in unwanted emulation activity.
> 
> This patch obsoletes those commits by fully disabling emulation activity
> when vcpus are stopped.
> 
> Use the new vm_shutdown() function instead of pause_all_vcpus() so that
> vm change state handlers are invoked too.  Virtio devices will now stop
> their ioeventfds, preventing further emulation activity after vm_stop().
> 
> Note that vm_stop(RUN_STATE_SHUTDOWN) cannot be used because it emits a
> QMP STOP event that may affect existing clients.
> 
> It is no longer necessary to call replay_disable_events() directly since
> vm_shutdown() does so already.
> 
> Drop iothread_stop_all() since it is no longer used.
> 
> Cc: Fam Zheng <famz@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
> Message-id: 20180307144205.20619-5-stefanha@redhat.com
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  include/sysemu/iothread.h |  1 -
>  include/sysemu/sysemu.h   |  1 +
>  cpus.c                    | 16 +++++++++++++---
>  iothread.c                | 31 -------------------------------
>  vl.c                      | 13 +++----------
>  5 files changed, 17 insertions(+), 45 deletions(-)
> 
> diff --git a/include/sysemu/iothread.h b/include/sysemu/iothread.h
> index 799614ffd2..8a7ac2c528 100644
> --- a/include/sysemu/iothread.h
> +++ b/include/sysemu/iothread.h
> @@ -45,7 +45,6 @@ typedef struct {
>  char *iothread_get_id(IOThread *iothread);
>  IOThread *iothread_by_id(const char *id);
>  AioContext *iothread_get_aio_context(IOThread *iothread);
> -void iothread_stop_all(void);
>  GMainContext *iothread_get_g_main_context(IOThread *iothread);
>  
>  /*
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index d24ad09f37..356bfdc1c1 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -56,6 +56,7 @@ void vm_start(void);
>  int vm_prepare_start(void);
>  int vm_stop(RunState state);
>  int vm_stop_force_state(RunState state);
> +int vm_shutdown(void);
>  
>  typedef enum WakeupReason {
>      /* Always keep QEMU_WAKEUP_REASON_NONE = 0 */
> diff --git a/cpus.c b/cpus.c
> index 9bcff7d63c..d8fe90eafe 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -993,7 +993,7 @@ void cpu_synchronize_all_pre_loadvm(void)
>      }
>  }
>  
> -static int do_vm_stop(RunState state)
> +static int do_vm_stop(RunState state, bool send_stop)
>  {
>      int ret = 0;
>  
> @@ -1002,7 +1002,9 @@ static int do_vm_stop(RunState state)
>          pause_all_vcpus();
>          runstate_set(state);
>          vm_state_notify(0, state);
> -        qapi_event_send_stop(&error_abort);
> +        if (send_stop) {
> +            qapi_event_send_stop(&error_abort);
> +        }
>      }
>  
>      bdrv_drain_all();
> @@ -1012,6 +1014,14 @@ static int do_vm_stop(RunState state)
>      return ret;
>  }
>  
> +/* Special vm_stop() variant for terminating the process.  Historically clients
> + * did not expect a QMP STOP event and so we need to retain compatibility.
> + */
> +int vm_shutdown(void)
> +{
> +    return do_vm_stop(RUN_STATE_SHUTDOWN, false);
> +}
> +
>  static bool cpu_can_run(CPUState *cpu)
>  {
>      if (cpu->stop) {
> @@ -1994,7 +2004,7 @@ int vm_stop(RunState state)
>          return 0;
>      }
>  
> -    return do_vm_stop(state);
> +    return do_vm_stop(state, true);
>  }
>  
>  /**
> diff --git a/iothread.c b/iothread.c
> index 2ec5a3bffe..1b3463cb00 100644
> --- a/iothread.c
> +++ b/iothread.c
> @@ -101,18 +101,6 @@ void iothread_stop(IOThread *iothread)
>      qemu_thread_join(&iothread->thread);
>  }
>  
> -static int iothread_stop_iter(Object *object, void *opaque)
> -{
> -    IOThread *iothread;
> -
> -    iothread = (IOThread *)object_dynamic_cast(object, TYPE_IOTHREAD);
> -    if (!iothread) {
> -        return 0;
> -    }
> -    iothread_stop(iothread);
> -    return 0;
> -}
> -
>  static void iothread_instance_init(Object *obj)
>  {
>      IOThread *iothread = IOTHREAD(obj);
> @@ -333,25 +321,6 @@ IOThreadInfoList *qmp_query_iothreads(Error **errp)
>      return head;
>  }
>  
> -void iothread_stop_all(void)
> -{
> -    Object *container = object_get_objects_root();
> -    BlockDriverState *bs;
> -    BdrvNextIterator it;
> -
> -    for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
> -        AioContext *ctx = bdrv_get_aio_context(bs);
> -        if (ctx == qemu_get_aio_context()) {
> -            continue;
> -        }
> -        aio_context_acquire(ctx);
> -        bdrv_set_aio_context(bs, qemu_get_aio_context());
> -        aio_context_release(ctx);
> -    }
> -
> -    object_child_foreach(container, iothread_stop_iter, NULL);
> -}
> -
>  static gpointer iothread_g_main_context_init(gpointer opaque)
>  {
>      AioContext *ctx;
> diff --git a/vl.c b/vl.c
> index dae986b352..3ef04ce991 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4722,17 +4722,10 @@ int main(int argc, char **argv, char **envp)
>      os_setup_post();
>  
>      main_loop();
> -    replay_disable_events();
>  
> -    /* The ordering of the following is delicate.  Stop vcpus to prevent new
> -     * I/O requests being queued by the guest.  Then stop IOThreads (this
> -     * includes a drain operation and completes all request processing).  At
> -     * this point emulated devices are still associated with their IOThreads
> -     * (if any) but no longer have any work to do.  Only then can we close
> -     * block devices safely because we know there is no more I/O coming.
> -     */
> -    pause_all_vcpus();
> -    iothread_stop_all();
> +    /* No more vcpu or device emulation activity beyond this point */
> +    vm_shutdown();
> +
>      bdrv_close_all();
>  
>      res_free();
> 

This appears to cause a regression in qemu-iotest 185:

--- 185.out.bad	2018-03-12 14:54:25.692884537 -0400
+++ ../../../../tests/qemu-iotests/185.out	2017-12-21 16:15:50.879455552
-0500
@@ -20,7 +20,7 @@
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 1048576, "speed": 65536, "type": "commit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 524288, "speed": 65536, "type": "commit"}}

 === Start active commit job and exit qemu ===

@@ -28,8 +28,7 @@
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_READY", "data": {"device": "disk", "len": 4194304,
"offset": 4194304, "speed": 65536, "type": "commit"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_COMPLETED", "data": {"device": "disk", "len":
4194304, "offset": 4194304, "speed": 65536, "type": "commit"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
4194304, "offset": 4194304, "speed": 65536, "type": "commit"}}

 === Start mirror job and exit qemu ===

@@ -38,8 +37,7 @@
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_READY", "data": {"device": "disk", "len": 4194304,
"offset": 4194304, "speed": 65536, "type": "mirror"}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_COMPLETED", "data": {"device": "disk", "len":
4194304, "offset": 4194304, "speed": 65536, "type": "mirror"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
4194304, "offset": 4194304, "speed": 65536, "type": "mirror"}}

 === Start backup job and exit qemu ===

@@ -48,7 +46,7 @@
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 131072, "speed": 65536, "type": "backup"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 65536, "speed": 65536, "type": "backup"}}

 === Start streaming job and exit qemu ===

@@ -56,6 +54,6 @@
 {"return": {}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}
-{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 1048576, "speed": 65536, "type": "stream"}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "BLOCK_JOB_CANCELLED", "data": {"device": "disk", "len":
67108864, "offset": 524288, "speed": 65536, "type": "stream"}}
 No errors were found on the image.
 *** done

Re: [Qemu-devel] [PULL 7/7] vl: introduce vm_shutdown()

[Christian Borntraeger]

On 03/12/2018 08:05 PM, John Snow wrote:
> 
> 
> On 03/09/2018 08:19 AM, Stefan Hajnoczi wrote:
>> Commit 00d09fdbbae5f7864ce754913efc84c12fdf9f1a ("vl: pause vcpus before
>> stopping iothreads") and commit dce8921b2baaf95974af8176406881872067adfa
>> ("iothread: Stop threads before main() quits") tried to work around the
>> fact that emulation was still active during termination by stopping
>> iothreads.  They suffer from race conditions:
>> 1. virtio_scsi_handle_cmd_vq() racing with iothread_stop_all() hits the
>>    virtio_scsi_ctx_check() assertion failure because the BDS AioContext
>>    has been modified by iothread_stop_all().
>> 2. Guest vq kick racing with main loop termination leaves a readable
>>    ioeventfd that is handled by the next aio_poll() when external
>>    clients are enabled again, resulting in unwanted emulation activity.
>>
>> This patch obsoletes those commits by fully disabling emulation activity
>> when vcpus are stopped.
>>
>> Use the new vm_shutdown() function instead of pause_all_vcpus() so that
>> vm change state handlers are invoked too.  Virtio devices will now stop
>> their ioeventfds, preventing further emulation activity after vm_stop().
>>
>> Note that vm_stop(RUN_STATE_SHUTDOWN) cannot be used because it emits a
>> QMP STOP event that may affect existing clients.
>>
>> It is no longer necessary to call replay_disable_events() directly since
>> vm_shutdown() does so already.
>>
>> Drop iothread_stop_all() since it is no longer used.
>>
>> Cc: Fam Zheng <famz@redhat.com>
>> Cc: Kevin Wolf <kwolf@redhat.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Reviewed-by: Fam Zheng <famz@redhat.com>
>> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
>> Message-id: 20180307144205.20619-5-stefanha@redhat.com
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

Stefan, I see the same iotest regression that was reported by John Snow also
on s390. I can confirm that it works with this patch reverted.

Christian

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On Fri, 8 Mar 2019 at 16:53, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 6cb4f6db4f4367faa33da85b15f75bbbd2bed2a6:
>
>   Merge remote-tracking branch 'remotes/cleber/tags/python-next-pull-request' into staging (2019-03-07 16:16:02 +0000)
>
> are available in the Git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 6ca206204fa773c8626d59caf2a5676d6cc35f52:
>
>   iothread: document about why we need explicit aio_poll() (2019-03-08 10:20:57 +0000)
>
> ----------------------------------------------------------------
> Pull request
>
> ----------------------------------------------------------------

Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.0
for any user-visible changes.

-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 6cb4f6db4f4367faa33da85b15f75bbbd2bed2a6:

  Merge remote-tracking branch 'remotes/cleber/tags/python-next-pull-request' into staging (2019-03-07 16:16:02 +0000)

are available in the Git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 6ca206204fa773c8626d59caf2a5676d6cc35f52:

  iothread: document about why we need explicit aio_poll() (2019-03-08 10:20:57 +0000)

----------------------------------------------------------------
Pull request

----------------------------------------------------------------

Anastasiia Rusakova (1):
  hw/block/virtio-blk: Clean req->dev repetitions

Peter Xu (5):
  iothread: replace init_done_cond with a semaphore
  iothread: create the gcontext unconditionally
  iothread: create main loop unconditionally
  iothread: push gcontext earlier in the thread_fn
  iothread: document about why we need explicit aio_poll()

Stefan Hajnoczi (1):
  MAINTAINERS: add missing support status fields

 MAINTAINERS               |  3 ++
 include/sysemu/iothread.h |  5 +--
 hw/block/virtio-blk.c     | 16 ++++---
 iothread.c                | 90 +++++++++++++++++++--------------------
 4 files changed, 57 insertions(+), 57 deletions(-)

-- 
2.20.1

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 30 June 2017 at 12:46, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 464588675455afda2899e20a0b120e4075de50c7:
>
>   Merge remote-tracking branch 'remotes/sstabellini/tags/xen-20170627-tag' into staging (2017-06-29 11:45:01 +0100)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to c324fd0a39cee43c13f6d1fb34f74fc5e2fb007b:
>
>   virtio-pci: use ioeventfd even when KVM is disabled (2017-06-30 11:03:45 +0100)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------
>
> Stefan Hajnoczi (7):
>   virtio-blk: trace vdev so devices can be distinguished
>   libqos: fix typo in virtio.h QVirtQueue->used comment
>   libqos: add virtio used ring support
>   tests: fix virtio-scsi-test ISR dependence
>   tests: fix virtio-blk-test ISR dependence
>   tests: fix virtio-net-test ISR dependence
>   virtio-pci: use ioeventfd even when KVM is disabled

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 464588675455afda2899e20a0b120e4075de50c7:

  Merge remote-tracking branch 'remotes/sstabellini/tags/xen-20170627-tag' into staging (2017-06-29 11:45:01 +0100)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to c324fd0a39cee43c13f6d1fb34f74fc5e2fb007b:

  virtio-pci: use ioeventfd even when KVM is disabled (2017-06-30 11:03:45 +0100)

----------------------------------------------------------------

----------------------------------------------------------------

Stefan Hajnoczi (7):
  virtio-blk: trace vdev so devices can be distinguished
  libqos: fix typo in virtio.h QVirtQueue->used comment
  libqos: add virtio used ring support
  tests: fix virtio-scsi-test ISR dependence
  tests: fix virtio-blk-test ISR dependence
  tests: fix virtio-net-test ISR dependence
  virtio-pci: use ioeventfd even when KVM is disabled

 tests/libqos/virtio.h    |  8 ++++++-
 hw/block/virtio-blk.c    | 12 ++++++----
 hw/virtio/virtio-pci.c   |  2 +-
 tests/libqos/virtio.c    | 60 ++++++++++++++++++++++++++++++++++++++++++++++++
 tests/virtio-blk-test.c  | 27 ++++++++++++++--------
 tests/virtio-net-test.c  |  6 ++---
 tests/virtio-scsi-test.c |  2 +-
 hw/block/trace-events    | 10 ++++----
 8 files changed, 101 insertions(+), 26 deletions(-)

-- 
2.9.4

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 28 September 2016 at 11:15, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 25930ed60aad49f1fdd7de05272317c86ce1275b:
>
>   Merge remote-tracking branch 'remotes/ehabkost/tags/x86-pull-request' into staging (2016-09-27 23:10:12 +0100)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to fe121b9d3c4258e41f7efa4976bf79151b2d5dbb:
>
>   linux-aio: fix re-entrant completion processing (2016-09-28 17:11:23 +0100)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------
>
> Laurent Vivier (1):
>   libqos: fix qvring_init()
>
> Lin Ma (1):
>   iothread: check iothread->ctx before aio_context_unref to avoid
>     assertion
>
> Stefan Hajnoczi (3):
>   coroutine: add qemu_coroutine_entered() function
>   test-coroutine: test qemu_coroutine_entered()
>   linux-aio: fix re-entrant completion processing
>
> Yaowei Bai (2):
>   block: mirror: fix wrong comment of mirror_start
>   aio-posix: avoid unnecessary aio_epoll_enabled() calls

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 25930ed60aad49f1fdd7de05272317c86ce1275b:

  Merge remote-tracking branch 'remotes/ehabkost/tags/x86-pull-request' into staging (2016-09-27 23:10:12 +0100)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to fe121b9d3c4258e41f7efa4976bf79151b2d5dbb:

  linux-aio: fix re-entrant completion processing (2016-09-28 17:11:23 +0100)

----------------------------------------------------------------

----------------------------------------------------------------

Laurent Vivier (1):
  libqos: fix qvring_init()

Lin Ma (1):
  iothread: check iothread->ctx before aio_context_unref to avoid
    assertion

Stefan Hajnoczi (3):
  coroutine: add qemu_coroutine_entered() function
  test-coroutine: test qemu_coroutine_entered()
  linux-aio: fix re-entrant completion processing

Yaowei Bai (2):
  block: mirror: fix wrong comment of mirror_start
  aio-posix: avoid unnecessary aio_epoll_enabled() calls

 aio-posix.c               | 12 +++++++-----
 block/linux-aio.c         |  9 ++++++---
 include/block/block_int.h |  2 +-
 include/qemu/coroutine.h  | 13 +++++++++++++
 iothread.c                |  3 +++
 tests/libqos/virtio.c     |  2 +-
 tests/test-coroutine.c    | 42 ++++++++++++++++++++++++++++++++++++++++++
 util/qemu-coroutine.c     |  5 +++++
 8 files changed, 78 insertions(+), 10 deletions(-)

-- 
2.7.4

Re: [Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

On Mon, Nov 09, 2015 at 08:35:56AM +0100, Markus Armbruster wrote:
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Fri, 11/06 18:07, Peter Maydell wrote:
> >> On 6 November 2015 at 17:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >> > The following changes since commit 4b59f39bc9a03afcc74b2fa28da7c3189fca507c:
> >> >
> >> >   Merge remote-tracking branch
> >> > 'remotes/mjt/tags/pull-trivial-patches-2015-11-06' into staging
> >> > (2015-11-06 12:50:24 +0000)
> >> >
> >> > are available in the git repository at:
> >> >
> >> >   git://github.com/stefanha/qemu.git tags/block-pull-request
> >> >
> >> > for you to fetch changes up to 6f707181b1bd6ccf2d2fd9397039c7ef6fa4a9fd:
> >> >
> >> >   blockdev: acquire AioContext in hmp_commit() (2015-11-06 15:41:00 +0000)
> >> >
> >> > ----------------------------------------------------------------
> >> 
> >> Build failure on OSX :-(
> >> 
> >> /Users/pm215/src/qemu-for-merges/aio-posix.c  CC    block/qcow.o
> >> :442:37: error: no member named 'epollfd' in 'struct AioContext'
> >>         epoll_handler.pfd.fd = ctx->epollfd;
> >>                                ~~~  ^
> >> 
> >
> > :(
> >
> > I think it is harmless to always include this member in AioContext. Stefan,
> > could you squash this into patch 5?  Thanks!
> >
> > Fam
> >
> > ---
> >
> > diff --git a/include/block/aio.h b/include/block/aio.h
> > index 91737d5..735f1f8 100644
> > --- a/include/block/aio.h
> > +++ b/include/block/aio.h
> > @@ -124,11 +124,9 @@ struct AioContext {
> >      QEMUTimerListGroup tlg;
> >  
> >      int external_disable_cnt;
> > -#ifdef CONFIG_EPOLL
> >      int epollfd;
> >      bool epoll_enabled;
> >      bool epoll_available;
> > -#endif
> >  };
> 
> Replace by the ifdeffery by a comment pointing to CONFIG_EPOLL, perhaps?

Done.  Will send a v2 pull request.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PULL 0/7] Block patches

[Markus Armbruster]

Fam Zheng <famz@redhat.com> writes:

> On Fri, 11/06 18:07, Peter Maydell wrote:
>> On 6 November 2015 at 17:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> > The following changes since commit 4b59f39bc9a03afcc74b2fa28da7c3189fca507c:
>> >
>> >   Merge remote-tracking branch
>> > 'remotes/mjt/tags/pull-trivial-patches-2015-11-06' into staging
>> > (2015-11-06 12:50:24 +0000)
>> >
>> > are available in the git repository at:
>> >
>> >   git://github.com/stefanha/qemu.git tags/block-pull-request
>> >
>> > for you to fetch changes up to 6f707181b1bd6ccf2d2fd9397039c7ef6fa4a9fd:
>> >
>> >   blockdev: acquire AioContext in hmp_commit() (2015-11-06 15:41:00 +0000)
>> >
>> > ----------------------------------------------------------------
>> 
>> Build failure on OSX :-(
>> 
>> /Users/pm215/src/qemu-for-merges/aio-posix.c  CC    block/qcow.o
>> :442:37: error: no member named 'epollfd' in 'struct AioContext'
>>         epoll_handler.pfd.fd = ctx->epollfd;
>>                                ~~~  ^
>> 
>
> :(
>
> I think it is harmless to always include this member in AioContext. Stefan,
> could you squash this into patch 5?  Thanks!
>
> Fam
>
> ---
>
> diff --git a/include/block/aio.h b/include/block/aio.h
> index 91737d5..735f1f8 100644
> --- a/include/block/aio.h
> +++ b/include/block/aio.h
> @@ -124,11 +124,9 @@ struct AioContext {
>      QEMUTimerListGroup tlg;
>  
>      int external_disable_cnt;
> -#ifdef CONFIG_EPOLL
>      int epollfd;
>      bool epoll_enabled;
>      bool epoll_available;
> -#endif
>  };

Replace by the ifdeffery by a comment pointing to CONFIG_EPOLL, perhaps?

Re: [Qemu-devel] [PULL 0/7] Block patches

[Fam Zheng]

On Fri, 11/06 18:07, Peter Maydell wrote:
> On 6 November 2015 at 17:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > The following changes since commit 4b59f39bc9a03afcc74b2fa28da7c3189fca507c:
> >
> >   Merge remote-tracking branch 'remotes/mjt/tags/pull-trivial-patches-2015-11-06' into staging (2015-11-06 12:50:24 +0000)
> >
> > are available in the git repository at:
> >
> >   git://github.com/stefanha/qemu.git tags/block-pull-request
> >
> > for you to fetch changes up to 6f707181b1bd6ccf2d2fd9397039c7ef6fa4a9fd:
> >
> >   blockdev: acquire AioContext in hmp_commit() (2015-11-06 15:41:00 +0000)
> >
> > ----------------------------------------------------------------
> 
> Build failure on OSX :-(
> 
> /Users/pm215/src/qemu-for-merges/aio-posix.c  CC    block/qcow.o
> :442:37: error: no member named 'epollfd' in 'struct AioContext'
>         epoll_handler.pfd.fd = ctx->epollfd;
>                                ~~~  ^
> 

:(

I think it is harmless to always include this member in AioContext. Stefan,
could you squash this into patch 5?  Thanks!

Fam

---

diff --git a/include/block/aio.h b/include/block/aio.h
index 91737d5..735f1f8 100644
--- a/include/block/aio.h
+++ b/include/block/aio.h
@@ -124,11 +124,9 @@ struct AioContext {
     QEMUTimerListGroup tlg;
 
     int external_disable_cnt;
-#ifdef CONFIG_EPOLL
     int epollfd;
     bool epoll_enabled;
     bool epoll_available;
-#endif
 };
                                                                               
 /**                                                                           

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 6 November 2015 at 17:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The following changes since commit 4b59f39bc9a03afcc74b2fa28da7c3189fca507c:
>
>   Merge remote-tracking branch 'remotes/mjt/tags/pull-trivial-patches-2015-11-06' into staging (2015-11-06 12:50:24 +0000)
>
> are available in the git repository at:
>
>   git://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 6f707181b1bd6ccf2d2fd9397039c7ef6fa4a9fd:
>
>   blockdev: acquire AioContext in hmp_commit() (2015-11-06 15:41:00 +0000)
>
> ----------------------------------------------------------------

Build failure on OSX :-(

/Users/pm215/src/qemu-for-merges/aio-posix.c  CC    block/qcow.o
:442:37: error: no member named 'epollfd' in 'struct AioContext'
        epoll_handler.pfd.fd = ctx->epollfd;
                               ~~~  ^

thanks
-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 4b59f39bc9a03afcc74b2fa28da7c3189fca507c:

  Merge remote-tracking branch 'remotes/mjt/tags/pull-trivial-patches-2015-11-06' into staging (2015-11-06 12:50:24 +0000)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 6f707181b1bd6ccf2d2fd9397039c7ef6fa4a9fd:

  blockdev: acquire AioContext in hmp_commit() (2015-11-06 15:41:00 +0000)

----------------------------------------------------------------

----------------------------------------------------------------

Denis V. Lunev (1):
  monitor: add missed aio_context_acquire into vm_completion call

Fam Zheng (3):
  aio: Introduce aio_external_disabled
  aio: Introduce aio_context_setup
  aio: Introduce aio-epoll.c

Michael S. Tsirkin (2):
  dataplane: simplify indirect descriptor read
  dataplane: support non-contigious s/g

Stefan Hajnoczi (1):
  blockdev: acquire AioContext in hmp_commit()

 aio-posix.c                 | 188 +++++++++++++++++++++++++++++++++++++++++++-
 aio-win32.c                 |   4 +
 async.c                     |  13 ++-
 blockdev.c                  |  12 ++-
 hw/virtio/dataplane/vring.c |  96 ++++++++++++++--------
 include/block/aio.h         |  24 ++++++
 monitor.c                   |  11 ++-
 7 files changed, 309 insertions(+), 39 deletions(-)

-- 
2.5.0

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 25 September 2015 at 08:50, Jeff Cody <jcody@redhat.com> wrote:
> The following changes since commit eb9d0ea063fc7bdfab76b84085602a9e48d13ec7:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20150924' into staging (2015-09-24 01:32:11 +0100)
>
> are available in the git repository at:
>
>
>   git@github.com:codyprime/qemu-kvm-jtc.git tags/block-pull-request
>
> for you to fetch changes up to e6fd57ea297ec3aad32b24090c5d3757a99df3fe:
>
>   sheepdog: refine discard support (2015-09-25 10:25:19 -0400)
>
> ----------------------------------------------------------------
> Block patches
> ----------------------------------------------------------------
>

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Jeff Cody]

The following changes since commit eb9d0ea063fc7bdfab76b84085602a9e48d13ec7:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20150924' into staging (2015-09-24 01:32:11 +0100)

are available in the git repository at:


  git@github.com:codyprime/qemu-kvm-jtc.git tags/block-pull-request

for you to fetch changes up to e6fd57ea297ec3aad32b24090c5d3757a99df3fe:

  sheepdog: refine discard support (2015-09-25 10:25:19 -0400)

----------------------------------------------------------------
Block patches
----------------------------------------------------------------

Hitoshi Mitake (2):
  sheepdog: use per AIOCB dirty indexes for non overlapping requests
  sheepdog: refine discard support

Liu Yuan (1):
  sheepdog: add reopen support

Peter Lieven (2):
  block/nfs: fix calculation of allocated file size
  block/nfs: cache allocated filesize for read-only files

Wen Congyang (2):
  block: Introduce a new API bdrv_co_no_copy_on_readv()
  Backup: don't do copy-on-read in before_write_notifier

 block/backup.c        |  20 ++++--
 block/io.c            |  12 +++-
 block/nfs.c           |  38 +++++++++++-
 block/sheepdog.c      | 168 +++++++++++++++++++++++++++++++++++++++-----------
 include/block/block.h |   9 ++-
 trace-events          |   1 +
 6 files changed, 200 insertions(+), 48 deletions(-)

-- 
1.9.3

Re: [Qemu-devel] [PULL 0/7] Block patches

[Peter Maydell]

On 30 April 2015 at 20:10, Kevin Wolf <kwolf@redhat.com> wrote:
> The following changes since commit 06feaacfb4cfef10cc0c93d97df7bfc8a71dbc7e:
>
>   Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2015-04-30 12:04:11 +0100)
>
> are available in the git repository at:
>
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to 4a4d614ff56b4cf15e83629946afe51dc116053f:
>
>   Enable NVMe start controller for Windows guest. (2015-04-30 15:35:26 +0200)
>
> ----------------------------------------------------------------
> Block patches
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 0/7] Block patches

[Kevin Wolf]

The following changes since commit 06feaacfb4cfef10cc0c93d97df7bfc8a71dbc7e:

  Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2015-04-30 12:04:11 +0100)

are available in the git repository at:


  git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to 4a4d614ff56b4cf15e83629946afe51dc116053f:

  Enable NVMe start controller for Windows guest. (2015-04-30 15:35:26 +0200)

----------------------------------------------------------------
Block patches

----------------------------------------------------------------
Daniel Stekloff (1):
      Enable NVMe start controller for Windows guest.

Kevin Wolf (1):
      MAINTAINERS: Add qemu-block list where missing

Stefan Hajnoczi (5):
      MAINTAINERS: make virtio-blk Stefan Hajnoczi's responsibility
      MAINTAINERS: split out image formats
      MAINTAINERS: make block I/O path Stefan Hajnoczi's responsibility
      MAINTAINERS: make image fuzzer Stefan Hajnoczi's responsibility
      MAINTAINERS: make block layer core Kevin Wolf's responsibility

 MAINTAINERS     | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 hw/block/nvme.c |   7 ++++
 2 files changed, 110 insertions(+), 7 deletions(-)

[Qemu-devel] [PULL 0/7] Block patches

[Stefan Hajnoczi]

The following changes since commit 1680d485777ecf436d724631ea8722cc0c66990e:

  Merge remote-tracking branch 'rth/tcg-ldst-6' into staging (2013-10-14 09:59:59 -0700)

are available in the git repository at:


  git://github.com/stefanha/qemu.git block

for you to fetch changes up to dbbcaa8d4358fdf3c42bf01e9e2d687300e84770:

  vmdk: fix VMFS extent parsing (2013-10-18 13:41:36 +0200)

----------------------------------------------------------------
Andreas Färber (1):
      sd: Avoid access to NULL BlockDriverState

Fam Zheng (3):
      blockdev: fix cdrom read_only flag
      vmdk: Only read cid from image file when opening
      vmdk: fix VMFS extent parsing

Max Reitz (1):
      block/raw-win32: Always use -errno in hdev_open

Mike Qiu (1):
      hmp: drop bogus "[not inserted]"

Stefan Weil (1):
      virtio: Remove unneeded memcpy

 block/raw-win32.c     |  5 ++---
 block/vmdk.c          | 10 +++++-----
 blockdev.c            |  7 ++++---
 hmp.c                 |  2 --
 hw/block/virtio-blk.c |  1 -
 hw/sd/sd.c            |  2 +-
 6 files changed, 12 insertions(+), 15 deletions(-)

-- 
1.8.3.1

Re: [Qemu-devel] [PULL 0/7] Block patches

[Anthony Liguori]

Pulled.  Thanks.

Regards,

Anthony Liguori

[Qemu-devel] [PULL 0/7] Block patches

[Kevin Wolf]

The following changes since commit 38aea177d93556aada7c4c7aa530f0050715e293:

  Merge remote-tracking branch 'pmaydell/configury.next' into staging (2013-06-17 08:57:57 -0500)

are available in the git repository at:


  git://repo.or.cz/qemu/kevin.git for-anthony

for you to fetch changes up to 0bed087df24c7b3fae366f239b9d150de3309416:

  vmdk: Allow reading variable size descriptor files (2013-06-17 17:47:59 +0200)

----------------------------------------------------------------
Evgeny Budilovsky (1):
      vmdk: Allow reading variable size descriptor files

Keith Busch (1):
      NVMe: Initial commit for new storage interface

Liu Yuan (2):
      sheepdog: fix snapshot tag initialization
      sheepdog: support 'qemu-img snapshot -a'

Richard W.M. Jones (2):
      block/curl.c: Refuse to open the handle for writes.
      curl: Don't set curl options on the handle just before it's going to be deleted.

Stefan Hajnoczi (1):
      vmdk: byteswap VMDK4Header.desc_offset field

 MAINTAINERS              |   5 +
 block/curl.c             |   8 +-
 block/sheepdog.c         |  14 +-
 block/vmdk.c             |  34 +-
 default-configs/pci.mak  |   1 +
 hw/block/Makefile.objs   |   1 +
 hw/block/nvme.c          | 885 +++++++++++++++++++++++++++++++++++++++++++++++
 hw/block/nvme.h          | 711 +++++++++++++++++++++++++++++++++++++
 include/hw/pci/pci_ids.h |   1 +
 9 files changed, 1641 insertions(+), 19 deletions(-)
 create mode 100644 hw/block/nvme.c
 create mode 100644 hw/block/nvme.h

[Qemu-devel] [PULL 0/7] Block patches

[Kevin Wolf]

Some more fixes for 0.13.

The following changes since commit cdcf9153e5e17dde340135fee5dcc7c299f2d4f5:

  etrax: Update ethernet mgm-ctrl reg on writes (2010-07-25 21:03:56 +0200)

are available in the git repository at:
  git://repo.or.cz/qemu/kevin.git for-anthony

Alex Williamson (1):
      virtio-blk: Create exit function to unregister savevm

Aurelien Jarno (1):
      ide/atapi: add support for GET EVENT STATUS NOTIFICATION

Blue Swirl (1):
      Fix -snapshot deleting images on disk change

Bruce Rogers (1):
      move 'unsafe' to end of caching modes in help

Christoph Hellwig (1):
      block: default to 0 minimal / optiomal I/O size

Stefan Weil (1):
      block: Use error codes from lower levels for error message

Yoshiaki Tamura (1):
      block migration: propagate return value when bdrv_write() returns < 0

 block-migration.c |    6 +++++-
 block.c           |   32 ++++++++++++++++++++++++--------
 block.h           |    1 +
 block_int.h       |    4 ++--
 blockdev.c        |    1 +
 hw/ide/core.c     |   15 +++++++++++++++
 hw/virtio-blk.c   |    8 ++++++++
 hw/virtio-pci.c   |    1 +
 hw/virtio.h       |    1 +
 qemu-options.hx   |    2 +-
 10 files changed, 59 insertions(+), 12 deletions(-)