[Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Pavel Dovgalyuk]

The set of patches include the block-related updates
of the record/replay icount feature:
 - application of 'snapshot' option on the file layer instead of
   the top one: command line and documentation fix
 - implementation of bdrv_snapshot_goto for blkreplay driver
 - start/stop fix in replay mode with attached block devices
 - record/replay of bh oneshot events, used in the block layer

These patches are rebased upon the following series:
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg05568.html

---

Pavel Dovgalyuk (6):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      replay: add BH oneshot event for block layer


 block/blkreplay.c        |    8 ++++++++
 block/block-backend.c    |    8 +++++---
 block/io.c               |   32 ++++++++++++++++++++++++++++++--
 block/iscsi.c            |    5 +++--
 block/nfs.c              |    5 +++--
 block/null.c             |    4 +++-
 block/nvme.c             |    6 ++++--
 block/rbd.c              |    5 +++--
 block/vxhs.c             |    5 +++--
 cpus.c                   |    2 --
 docs/replay.txt          |   12 +++++++++---
 include/sysemu/replay.h  |    3 +++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.c          |    4 +++-
 stubs/Makefile.objs      |    1 +
 stubs/replay-user.c      |    9 +++++++++
 vl.c                     |   11 +++++++++--
 18 files changed, 113 insertions(+), 24 deletions(-)
 create mode 100644 stubs/replay-user.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [for-4.2 PATCH 1/6] block: implement bdrv_snapshot_goto for blkreplay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch enables making snapshots with blkreplay used in
block devices.
This function is required to make bdrv_snapshot_goto without
calling .bdrv_open which is not implemented.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Acked-by: Kevin Wolf <kwolf@redhat.com>
---
 block/blkreplay.c |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/block/blkreplay.c b/block/blkreplay.c
index 2b7931b940..c96ac8f4bc 100644
--- a/block/blkreplay.c
+++ b/block/blkreplay.c
@@ -126,6 +126,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)
     return ret;
 }
 
+static int blkreplay_snapshot_goto(BlockDriverState *bs,
+                                   const char *snapshot_id)
+{
+    return bdrv_snapshot_goto(bs->file->bs, snapshot_id, NULL);
+}
+
 static BlockDriver bdrv_blkreplay = {
     .format_name            = "blkreplay",
     .instance_size          = 0,
@@ -140,6 +146,8 @@ static BlockDriver bdrv_blkreplay = {
     .bdrv_co_pwrite_zeroes  = blkreplay_co_pwrite_zeroes,
     .bdrv_co_pdiscard       = blkreplay_co_pdiscard,
     .bdrv_co_flush          = blkreplay_co_flush,
+
+    .bdrv_snapshot_goto     = blkreplay_snapshot_goto,
 };
 
 static void bdrv_blkreplay_init(void)

[Qemu-devel] [for-4.2 PATCH 2/6] replay: disable default snapshot for record/replay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch disables setting '-snapshot' option on by default
in record/replay mode. This is needed for creating vmstates in record
and replay modes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Acked-by: Kevin Wolf <kwolf@redhat.com>
---
 vl.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/vl.c b/vl.c
index b426b32134..b73fd078a8 100644
--- a/vl.c
+++ b/vl.c
@@ -1202,7 +1202,7 @@ static void configure_blockdev(BlockdevOptionsQueue *bdo_queue,
         qapi_free_BlockdevOptions(bdo->bdo);
         g_free(bdo);
     }
-    if (snapshot || replay_mode != REPLAY_MODE_NONE) {
+    if (snapshot) {
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot,
                           NULL, NULL);
     }
@@ -3051,7 +3051,13 @@ int main(int argc, char **argv, char **envp)
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
                 break;
             case QEMU_OPTION_snapshot:
-                snapshot = 1;
+                {
+                    Error *blocker = NULL;
+                    snapshot = 1;
+                    error_setg(&blocker, QERR_REPLAY_NOT_SUPPORTED,
+                               "-snapshot");
+                    replay_add_blocker(blocker);
+                }
                 break;
             case QEMU_OPTION_numa:
                 opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),

[Qemu-devel] [for-4.2 PATCH 3/6] replay: update docs for record/replay with block devices

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

This patch updates the description of the command lines for using
record/replay with attached block devices.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 docs/replay.txt |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/docs/replay.txt b/docs/replay.txt
index ee6aee9861..ce97c3f72f 100644
--- a/docs/replay.txt
+++ b/docs/replay.txt
@@ -27,7 +27,7 @@ Usage of the record/replay:
  * First, record the execution with the following command line:
     qemu-system-i386 \
      -icount shift=7,rr=record,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay:
  * After recording, you can replay it by using another command line:
     qemu-system-i386 \
      -icount shift=7,rr=replay,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of
 bdrv coroutine functions at the top of block drivers stack.
 To record and replay block operations the drive must be configured
 as following:
- -drive file=disk.qcow2,if=none,id=img-direct
+ -drive file=disk.qcow2,if=none,snapshot,id=img-direct
  -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
  -device ide-hd,drive=img-blkreplay
 
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start
 of replaying. It also can be loaded while replaying to roll back
 the execution.
 
+'snapshot' flag of the disk image must be removed to save the snapshots
+in the overlay (or original image) instead of using the temporary overlay.
+ -drive file=disk.ovl,if=none,id=img-direct
+ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+ -device ide-hd,drive=img-blkreplay
+
 Use QEMU monitor to create additional snapshots. 'savevm <name>' command
 created the snapshot and 'loadvm <name>' restores it. To prevent corruption
 of the original disk image, use overlay files linked to the original images.

[Qemu-devel] [for-4.2 PATCH 4/6] replay: don't drain/flush bdrv queue while RR is working

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

In record/replay mode bdrv queue is controlled by replay mechanism.
It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty
queue, but flushing the queue is still impossible there,
because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in
record/replay mode.

Stopping the machine when the IO requests are not finished is needed
for the debugging. E.g., breakpoint may be set at the specified step,
and forcing the IO requests to finish may break the determinism
of the execution.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Acked-by: Kevin Wolf <kwolf@redhat.com>
---
 block/io.c |   28 ++++++++++++++++++++++++++++
 cpus.c     |    2 --
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/block/io.c b/block/io.c
index 06305c6ea6..2e71bcb8d6 100644
--- a/block/io.c
+++ b/block/io.c
@@ -32,6 +32,7 @@
 #include "qemu/cutils.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
+#include "sysemu/replay.h"
 
 #define NOT_DONE 0x7fffffff /* used while emulated sync operation in progress */
 
@@ -598,6 +599,15 @@ void bdrv_drain_all_begin(void)
         return;
     }
 
+    /*
+     * bdrv queue is managed by record/replay,
+     * waiting for finishing the I/O requests may
+     * be infinite
+     */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     /* AIO_WAIT_WHILE() with a NULL context can only be called from the main
      * loop AioContext, so make sure we're in the main context. */
     assert(qemu_get_current_aio_context() == qemu_get_aio_context());
@@ -627,6 +637,15 @@ void bdrv_drain_all_end(void)
     BlockDriverState *bs = NULL;
     int drained_end_counter = 0;
 
+    /*
+     * bdrv queue is managed by record/replay,
+     * waiting for finishing the I/O requests may
+     * be endless
+     */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     while ((bs = bdrv_next_all_states(bs))) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
 
@@ -1997,6 +2016,15 @@ int bdrv_flush_all(void)
     BlockDriverState *bs = NULL;
     int result = 0;
 
+    /*
+     * bdrv queue is managed by record/replay,
+     * creating new flush request for stopping
+     * the VM may break the determinism
+     */
+    if (replay_events_enabled()) {
+        return result;
+    }
+
     for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
         int ret;
diff --git a/cpus.c b/cpus.c
index a7120ffbd5..f6b7252fdb 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1083,7 +1083,6 @@ static int do_vm_stop(RunState state, bool send_stop)
     }
 
     bdrv_drain_all();
-    replay_disable_events();
     ret = bdrv_flush_all();
 
     return ret;
@@ -2169,7 +2168,6 @@ int vm_prepare_start(void)
     /* We are sending this now, but the CPUs will be resumed shortly later */
     qapi_event_send_resume();
 
-    replay_enable_events();
     cpu_enable_ticks();
     runstate_set(RUN_STATE_RUNNING);
     vm_state_notify(1, RUN_STATE_RUNNING);

[Qemu-devel] [for-4.2 PATCH 5/6] replay: finish record/replay before closing the disks

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

After recent updates block devices cannot be closed on qemu exit.
This happens due to the block request polling when replay is not finished.
Therefore now we stop execution recording before closing the block devices.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay.c |    2 ++
 vl.c            |    1 +
 2 files changed, 3 insertions(+)

diff --git a/replay/replay.c b/replay/replay.c
index 6a62ec3811..71c4e6b777 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -385,6 +385,8 @@ void replay_finish(void)
     g_free(replay_snapshot);
     replay_snapshot = NULL;
 
+    replay_mode = REPLAY_MODE_NONE;
+
     replay_finish_events();
 }
 
diff --git a/vl.c b/vl.c
index b73fd078a8..2a341f5ad2 100644
--- a/vl.c
+++ b/vl.c
@@ -4499,6 +4499,7 @@ int main(int argc, char **argv, char **envp)
 
     /* No more vcpu or device emulation activity beyond this point */
     vm_shutdown();
+    replay_finish();
 
     job_cancel_sync_all();
     bdrv_close_all();

[Qemu-devel] [for-4.2 PATCH 6/6] replay: add BH oneshot event for block layer

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

Replay is capable of recording normal BH events, but sometimes
there are single use callbacks scheduled with aio_bh_schedule_oneshot
function. This patch enables recording and replaying such callbacks.
Block layer uses these events for calling the completion function.
Replaying these calls makes the execution deterministic.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Acked-by: Kevin Wolf <kwolf@redhat.com>
---
 block/block-backend.c    |    8 +++++---
 block/io.c               |    4 ++--
 block/iscsi.c            |    5 +++--
 block/nfs.c              |    5 +++--
 block/null.c             |    4 +++-
 block/nvme.c             |    6 ++++--
 block/rbd.c              |    5 +++--
 block/vxhs.c             |    5 +++--
 include/sysemu/replay.h  |    3 +++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.c          |    2 +-
 stubs/Makefile.objs      |    1 +
 stubs/replay-user.c      |    9 +++++++++
 14 files changed, 57 insertions(+), 17 deletions(-)
 create mode 100644 stubs/replay-user.c

diff --git a/block/block-backend.c b/block/block-backend.c
index 0056b526b8..b8b45e9e82 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -17,6 +17,7 @@
 #include "block/throttle-groups.h"
 #include "sysemu/blockdev.h"
 #include "sysemu/sysemu.h"
+#include "sysemu/replay.h"
 #include "qapi/error.h"
 #include "qapi/qapi-events-block.h"
 #include "qemu/id.h"
@@ -1296,7 +1297,8 @@ BlockAIOCB *blk_abort_aio_request(BlockBackend *blk,
     acb->blk = blk;
     acb->ret = ret;
 
-    aio_bh_schedule_oneshot(blk_get_aio_context(blk), error_callback_bh, acb);
+    replay_bh_schedule_oneshot_event(blk_get_aio_context(blk),
+                                     error_callback_bh, acb);
     return &acb->common;
 }
 
@@ -1352,8 +1354,8 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes,
 
     acb->has_returned = true;
     if (acb->rwco.ret != NOT_DONE) {
-        aio_bh_schedule_oneshot(blk_get_aio_context(blk),
-                                blk_aio_complete_bh, acb);
+        replay_bh_schedule_oneshot_event(blk_get_aio_context(blk),
+                                         blk_aio_complete_bh, acb);
     }
 
     return &acb->common;
diff --git a/block/io.c b/block/io.c
index 2e71bcb8d6..ebff47e6e1 100644
--- a/block/io.c
+++ b/block/io.c
@@ -367,8 +367,8 @@ static void coroutine_fn bdrv_co_yield_to_drain(BlockDriverState *bs,
     if (bs) {
         bdrv_inc_in_flight(bs);
     }
-    aio_bh_schedule_oneshot(bdrv_get_aio_context(bs),
-                            bdrv_co_drain_bh_cb, &data);
+    replay_bh_schedule_oneshot_event(bdrv_get_aio_context(bs),
+                                     bdrv_co_drain_bh_cb, &data);
 
     qemu_coroutine_yield();
     /* If we are resumed from some other event (such as an aio completion or a
diff --git a/block/iscsi.c b/block/iscsi.c
index 506bf5f875..2ced15066a 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -40,6 +40,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/uuid.h"
+#include "sysemu/replay.h"
 #include "qapi/error.h"
 #include "qapi/qapi-commands-misc.h"
 #include "qapi/qmp/qdict.h"
@@ -280,8 +281,8 @@ iscsi_co_generic_cb(struct iscsi_context *iscsi, int status,
     }
 
     if (iTask->co) {
-        aio_bh_schedule_oneshot(iTask->iscsilun->aio_context,
-                                 iscsi_co_generic_bh_cb, iTask);
+        replay_bh_schedule_oneshot_event(iTask->iscsilun->aio_context,
+                                         iscsi_co_generic_bh_cb, iTask);
     } else {
         iTask->complete = 1;
     }
diff --git a/block/nfs.c b/block/nfs.c
index d93241b3bb..cfd6c956b3 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -37,6 +37,7 @@
 #include "qemu/uri.h"
 #include "qemu/cutils.h"
 #include "sysemu/sysemu.h"
+#include "sysemu/replay.h"
 #include "qapi/qapi-visit-block-core.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
@@ -257,8 +258,8 @@ nfs_co_generic_cb(int ret, struct nfs_context *nfs, void *data,
     if (task->ret < 0) {
         error_report("NFS Error: %s", nfs_get_error(nfs));
     }
-    aio_bh_schedule_oneshot(task->client->aio_context,
-                            nfs_co_generic_bh_cb, task);
+    replay_bh_schedule_oneshot_event(task->client->aio_context,
+                                     nfs_co_generic_bh_cb, task);
 }
 
 static int coroutine_fn nfs_co_preadv(BlockDriverState *bs, uint64_t offset,
diff --git a/block/null.c b/block/null.c
index 699aa295cb..15e1d56746 100644
--- a/block/null.c
+++ b/block/null.c
@@ -17,6 +17,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "block/block_int.h"
+#include "sysemu/replay.h"
 
 #define NULL_OPT_LATENCY "latency-ns"
 #define NULL_OPT_ZEROES  "read-zeroes"
@@ -179,7 +180,8 @@ static inline BlockAIOCB *null_aio_common(BlockDriverState *bs,
         timer_mod_ns(&acb->timer,
                      qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + s->latency_ns);
     } else {
-        aio_bh_schedule_oneshot(bdrv_get_aio_context(bs), null_bh_cb, acb);
+        replay_bh_schedule_oneshot_event(bdrv_get_aio_context(bs),
+                                         null_bh_cb, acb);
     }
     return &acb->common;
 }
diff --git a/block/nvme.c b/block/nvme.c
index c28755cc31..ac652e878c 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -22,6 +22,7 @@
 #include "qemu/option.h"
 #include "qemu/vfio-helpers.h"
 #include "block/block_int.h"
+#include "sysemu/replay.h"
 #include "trace.h"
 
 #include "block/nvme.h"
@@ -350,7 +351,8 @@ static bool nvme_process_completion(BDRVNVMeState *s, NVMeQueuePair *q)
         smp_mb_release();
         *q->cq.doorbell = cpu_to_le32(q->cq.head);
         if (!qemu_co_queue_empty(&q->free_req_queue)) {
-            aio_bh_schedule_oneshot(s->aio_context, nvme_free_req_queue_cb, q);
+            replay_bh_schedule_oneshot_event(s->aio_context,
+                                             nvme_free_req_queue_cb, q);
         }
     }
     q->busy = false;
@@ -934,7 +936,7 @@ static void nvme_rw_cb(void *opaque, int ret)
         /* The rw coroutine hasn't yielded, don't try to enter. */
         return;
     }
-    aio_bh_schedule_oneshot(data->ctx, nvme_rw_cb_bh, data);
+    replay_bh_schedule_oneshot_event(data->ctx, nvme_rw_cb_bh, data);
 }
 
 static coroutine_fn int nvme_co_prw_aligned(BlockDriverState *bs,
diff --git a/block/rbd.c b/block/rbd.c
index 59757b3120..a1ce5b9292 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -22,6 +22,7 @@
 #include "block/qdict.h"
 #include "crypto/secret.h"
 #include "qemu/cutils.h"
+#include "sysemu/replay.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qjson.h"
@@ -884,8 +885,8 @@ static void rbd_finish_aiocb(rbd_completion_t c, RADOSCB *rcb)
     rcb->ret = rbd_aio_get_return_value(c);
     rbd_aio_release(c);
 
-    aio_bh_schedule_oneshot(bdrv_get_aio_context(acb->common.bs),
-                            rbd_finish_bh, rcb);
+    replay_bh_schedule_oneshot_event(bdrv_get_aio_context(acb->common.bs),
+                                     rbd_finish_bh, rcb);
 }
 
 static int rbd_aio_discard_wrapper(rbd_image_t image,
diff --git a/block/vxhs.c b/block/vxhs.c
index 77fd5eb20d..d79fc97df6 100644
--- a/block/vxhs.c
+++ b/block/vxhs.c
@@ -22,6 +22,7 @@
 #include "qapi/error.h"
 #include "qemu/uuid.h"
 #include "crypto/tlscredsx509.h"
+#include "sysemu/replay.h"
 
 #define VXHS_OPT_FILENAME           "filename"
 #define VXHS_OPT_VDISK_ID           "vdisk-id"
@@ -105,8 +106,8 @@ static void vxhs_iio_callback(void *ctx, uint32_t opcode, uint32_t error)
             trace_vxhs_iio_callback(error);
         }
 
-        aio_bh_schedule_oneshot(bdrv_get_aio_context(acb->common.bs),
-                                vxhs_complete_aio_bh, acb);
+        replay_bh_schedule_oneshot_event(bdrv_get_aio_context(acb->common.bs),
+                                         vxhs_complete_aio_bh, acb);
         break;
 
     default:
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 28ce19e919..1476ab533e 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -140,6 +140,9 @@ void replay_enable_events(void);
 bool replay_events_enabled(void);
 /*! Adds bottom half event to the queue */
 void replay_bh_schedule_event(QEMUBH *bh);
+/* Adds oneshot bottom half event to the queue */
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque);
 /*! Adds input event to the queue */
 void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 008e80f636..302b84043a 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -36,6 +36,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_BH_ONESHOT:
+        ((QEMUBHFunc *)event->opaque)(event->opaque2);
+        break;
     case REPLAY_ASYNC_EVENT_INPUT:
         qemu_input_event_send_impl(NULL, (InputEvent *)event->opaque);
         qapi_free_InputEvent((InputEvent *)event->opaque);
@@ -131,6 +134,17 @@ void replay_bh_schedule_event(QEMUBH *bh)
     }
 }
 
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque)
+{
+    if (events_enabled) {
+        uint64_t id = replay_get_current_icount();
+        replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
+    } else {
+        aio_bh_schedule_oneshot(ctx, cb, opaque);
+    }
+}
+
 void replay_add_input_event(struct InputEvent *event)
 {
     replay_add_event(REPLAY_ASYNC_EVENT_INPUT, event, NULL, 0);
@@ -161,6 +175,7 @@ static void replay_save_event(Event *event, int checkpoint)
         /* save event-specific data */
         switch (event->event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_BH_ONESHOT:
             replay_put_qword(event->id);
             break;
         case REPLAY_ASYNC_EVENT_INPUT:
@@ -216,6 +231,7 @@ static Event *replay_read_event(int checkpoint)
     /* Events that has not to be in the queue */
     switch (replay_state.read_event_kind) {
     case REPLAY_ASYNC_EVENT_BH:
+    case REPLAY_ASYNC_EVENT_BH_ONESHOT:
         if (replay_state.read_event_id == -1) {
             replay_state.read_event_id = replay_get_qword();
         }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index afba9a3e0c..55fca1ac6b 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -51,6 +51,7 @@ enum ReplayEvents {
 
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
+    REPLAY_ASYNC_EVENT_BH_ONESHOT,
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_EVENT_CHAR_READ,
diff --git a/replay/replay.c b/replay/replay.c
index 71c4e6b777..242d36a858 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -22,7 +22,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe02007
+#define REPLAY_VERSION              0xe02008
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 9c7393b08c..4a50e95ec3 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -20,6 +20,7 @@ stub-obj-y += monitor.o
 stub-obj-y += notify-event.o
 stub-obj-y += qtest.o
 stub-obj-y += replay.o
+stub-obj-y += replay-user.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
 stub-obj-y += sysbus.o
diff --git a/stubs/replay-user.c b/stubs/replay-user.c
new file mode 100644
index 0000000000..2ad9e27203
--- /dev/null
+++ b/stubs/replay-user.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+#include "sysemu/replay.h"
+#include "sysemu/sysemu.h"
+
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque)
+{
+    aio_bh_schedule_oneshot(ctx, cb, opaque);
+}

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[no-reply]

Patchew URL: https://patchew.org/QEMU/156438176555.22071.10523120047318890136.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
/var/tmp/patchew-tester-tmp-z2gfg7al/src/replay/replay-events.c: In function ‘replay_bh_schedule_oneshot_event’:
/var/tmp/patchew-tester-tmp-z2gfg7al/src/replay/replay-events.c:141:23: error: implicit declaration of function ‘replay_get_current_icount’; did you mean ‘replay_get_current_step’? [-Werror=implicit-function-declaration]
  141 |         uint64_t id = replay_get_current_icount();
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~
      |                       replay_get_current_step
/var/tmp/patchew-tester-tmp-z2gfg7al/src/replay/replay-events.c:141:23: error: nested extern declaration of ‘replay_get_current_icount’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make: *** [/var/tmp/patchew-tester-tmp-z2gfg7al/src/rules.mak:69: replay/replay-events.o] Error 1
make: *** Waiting for unfinished jobs....


The full log is available at
http://patchew.org/logs/156438176555.22071.10523120047318890136.stgit@pasha-Precision-3630-Tower/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[no-reply]

Patchew URL: https://patchew.org/QEMU/156438176555.22071.10523120047318890136.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      ui/x_keymap.o
  CC      ui/gtk.o
  CC      ui/curses.o
/tmp/qemu-test/src/replay/replay-events.c:141:23: error: implicit declaration of function 'replay_get_current_icount' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        uint64_t id = replay_get_current_icount();
                      ^
/tmp/qemu-test/src/replay/replay-events.c:141:23: note: did you mean 'replay_get_current_step'?
/tmp/qemu-test/src/include/sysemu/replay.h:78:10: note: 'replay_get_current_step' declared here
uint64_t replay_get_current_step(void);
         ^
/tmp/qemu-test/src/replay/replay-events.c:141:23: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
        uint64_t id = replay_get_current_icount();
                      ^
2 errors generated.


The full log is available at
http://patchew.org/logs/156438176555.22071.10523120047318890136.stgit@pasha-Precision-3630-Tower/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Pavel Dovgalyuk]

> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:
> 
> >> -----Original Message-----
> >> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> >> Sent: Tuesday, September 17, 2019 10:02 PM
> >> To: Pavel Dovgalyuk
> >> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
> >> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
> >> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
> >> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
> >> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
> >> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
> >> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
> >>
> >>
> >> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> >>
> >> > The set of patches include the block-related updates
> >> > of the record/replay icount feature:
> >> >  - application of 'snapshot' option on the file layer instead of
> >> >    the top one: command line and documentation fix
> >> >  - implementation of bdrv_snapshot_goto for blkreplay driver
> >> >  - start/stop fix in replay mode with attached block devices
> >> >  - record/replay of bh oneshot events, used in the block layer
> >>
> >> Can we come up with a reasonable smoke test to verify record/replay is
> >> functional? AIUI we don't need a full OS but we do need a block device
> >> to store the replay data. Could we use one of the simple system tests
> >> like "memory" and run that through a record and replay cycle?
> >
> > That's would be great.
> > I'm not familiar with testing framework and couldn't find "memory"
> > test that you refer to.
> 
> The memory test code is in tests/tcg/multiarch/system and gets combined
> with the boot code for whichever target can build it. For example on
> aarch64 it is run like:
> 
>   timeout 15  $BLD/aarch64-softmmu/qemu-system-aarch64 -monitor none -display none -chardev
> file,path=memory.out,id=output  -M virt -cpu max -display none -semihosting-config
> enable=on,target=native,chardev=output -kernel memory

Yes, rr testing could be that simple, but in full system emulation mode.
Simplest tests may be run even without the block devices:

qemu-system-aarch64 -monitor none -display none -chardev file,path=memory.out,id=output
-M virt -cpu max -kernel memory -net none -icount shift=5,rr=record,rrfile=record.bin

Better testing must include some block device like Linux boot image or something similar.

> The test binaries will be in $BLD/tests/tcg/aarch64-softmmu and are
> built when you run "make check-tcg" and have either the appropriate
> cross compilers installed on your system or docker enabled and setup
> (see docs/devel/testing.rst).
> 
> > Replay test must at least the following:
> > 1. record some execution
> > 2. replay that execution
> >
> > And there could be several endings:
> > 1. record stalled
> > 2. record crashed
> > 3. replay stalled
> > 4. replay crashed
> > 5. all executions finished successfully
> 
> If you can provide an appropriate set of invocations I can plumb them
> into the Makefiles for you.

That will be great, thank you.

Pavel Dovgalyuk

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Alex Bennée]

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

>> -----Original Message-----
>> From: Alex Bennée [mailto:alex.bennee@linaro.org]
>> Sent: Tuesday, September 17, 2019 10:02 PM
>> To: Pavel Dovgalyuk
>> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
>> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
>> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
>> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
>> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
>> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
>> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
>>
>>
>> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
>>
>> > The set of patches include the block-related updates
>> > of the record/replay icount feature:
>> >  - application of 'snapshot' option on the file layer instead of
>> >    the top one: command line and documentation fix
>> >  - implementation of bdrv_snapshot_goto for blkreplay driver
>> >  - start/stop fix in replay mode with attached block devices
>> >  - record/replay of bh oneshot events, used in the block layer
>>
>> Can we come up with a reasonable smoke test to verify record/replay is
>> functional? AIUI we don't need a full OS but we do need a block device
>> to store the replay data. Could we use one of the simple system tests
>> like "memory" and run that through a record and replay cycle?
>
> That's would be great.
> I'm not familiar with testing framework and couldn't find "memory"
> test that you refer to.

The memory test code is in tests/tcg/multiarch/system and gets combined
with the boot code for whichever target can build it. For example on
aarch64 it is run like:

  timeout 15  $BLD/aarch64-softmmu/qemu-system-aarch64 -monitor none -display none -chardev file,path=memory.out,id=output  -M virt -cpu max -display none -semihosting-config enable=on,target=native,chardev=output -kernel memory

The test binaries will be in $BLD/tests/tcg/aarch64-softmmu and are
built when you run "make check-tcg" and have either the appropriate
cross compilers installed on your system or docker enabled and setup
(see docs/devel/testing.rst).

> Replay test must at least the following:
> 1. record some execution
> 2. replay that execution
>
> And there could be several endings:
> 1. record stalled
> 2. record crashed
> 3. replay stalled
> 4. replay crashed
> 5. all executions finished successfully

If you can provide an appropriate set of invocations I can plumb them
into the Makefiles for you.

>
> Pavel Dovgalyuk
>
>>
>> >
>> > ---
>> >
>> > Pavel Dovgaluk (6):
>> >       block: implement bdrv_snapshot_goto for blkreplay
>> >       replay: disable default snapshot for record/replay
>> >       replay: update docs for record/replay with block devices
>> >       replay: don't drain/flush bdrv queue while RR is working
>> >       replay: finish record/replay before closing the disks
>> >       replay: add BH oneshot event for block layer
>> >
>> >
>> >  block/blkreplay.c        |    8 ++++++++
>> >  block/block-backend.c    |    9 ++++++---
>> >  block/io.c               |   32 ++++++++++++++++++++++++++++++--
>> >  block/iscsi.c            |    5 +++--
>> >  block/nfs.c              |    6 ++++--
>> >  block/null.c             |    4 +++-
>> >  block/nvme.c             |    6 ++++--
>> >  block/rbd.c              |    5 +++--
>> >  block/vxhs.c             |    5 +++--
>> >  cpus.c                   |    2 --
>> >  docs/replay.txt          |   12 +++++++++---
>> >  include/sysemu/replay.h  |    4 ++++
>> >  replay/replay-events.c   |   16 ++++++++++++++++
>> >  replay/replay-internal.h |    1 +
>> >  replay/replay.c          |    2 ++
>> >  stubs/Makefile.objs      |    1 +
>> >  stubs/replay-user.c      |    9 +++++++++
>> >  vl.c                     |   11 +++++++++--
>> >  18 files changed, 115 insertions(+), 23 deletions(-)
>> >  create mode 100644 stubs/replay-user.c
>>
>>
>> --
>> Alex Bennée


--
Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Pavel Dovgalyuk]

> -----Original Message-----
> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> Sent: Tuesday, September 17, 2019 10:02 PM
> To: Pavel Dovgalyuk
> Cc: qemu-devel@nongnu.org; kwolf@redhat.com; peter.maydell@linaro.org;
> crosthwaite.peter@gmail.com; boost.lists@gmail.com; artem.k.pisarenko@gmail.com;
> quintela@redhat.com; ciro.santilli@gmail.com; jasowang@redhat.com; mst@redhat.com;
> armbru@redhat.com; mreitz@redhat.com; maria.klimushenkova@ispras.ru; dovgaluk@ispras.ru;
> kraxel@redhat.com; pavel.dovgaluk@ispras.ru; thomas.dullien@googlemail.com;
> pbonzini@redhat.com; dgilbert@redhat.com; rth@twiddle.net
> Subject: Re: [for-4.2 PATCH 0/6] Block-related record/replay fixes
> 
> 
> Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:
> 
> > The set of patches include the block-related updates
> > of the record/replay icount feature:
> >  - application of 'snapshot' option on the file layer instead of
> >    the top one: command line and documentation fix
> >  - implementation of bdrv_snapshot_goto for blkreplay driver
> >  - start/stop fix in replay mode with attached block devices
> >  - record/replay of bh oneshot events, used in the block layer
> 
> Can we come up with a reasonable smoke test to verify record/replay is
> functional? AIUI we don't need a full OS but we do need a block device
> to store the replay data. Could we use one of the simple system tests
> like "memory" and run that through a record and replay cycle?

That's would be great.
I'm not familiar with testing framework and couldn't find "memory" test that you refer to.
Replay test must at least the following:
1. record some execution
2. replay that execution

And there could be several endings:
1. record stalled
2. record crashed
3. replay stalled
4. replay crashed
5. all executions finished successfully

Pavel Dovgalyuk

> 
> >
> > ---
> >
> > Pavel Dovgaluk (6):
> >       block: implement bdrv_snapshot_goto for blkreplay
> >       replay: disable default snapshot for record/replay
> >       replay: update docs for record/replay with block devices
> >       replay: don't drain/flush bdrv queue while RR is working
> >       replay: finish record/replay before closing the disks
> >       replay: add BH oneshot event for block layer
> >
> >
> >  block/blkreplay.c        |    8 ++++++++
> >  block/block-backend.c    |    9 ++++++---
> >  block/io.c               |   32 ++++++++++++++++++++++++++++++--
> >  block/iscsi.c            |    5 +++--
> >  block/nfs.c              |    6 ++++--
> >  block/null.c             |    4 +++-
> >  block/nvme.c             |    6 ++++--
> >  block/rbd.c              |    5 +++--
> >  block/vxhs.c             |    5 +++--
> >  cpus.c                   |    2 --
> >  docs/replay.txt          |   12 +++++++++---
> >  include/sysemu/replay.h  |    4 ++++
> >  replay/replay-events.c   |   16 ++++++++++++++++
> >  replay/replay-internal.h |    1 +
> >  replay/replay.c          |    2 ++
> >  stubs/Makefile.objs      |    1 +
> >  stubs/replay-user.c      |    9 +++++++++
> >  vl.c                     |   11 +++++++++--
> >  18 files changed, 115 insertions(+), 23 deletions(-)
> >  create mode 100644 stubs/replay-user.c
> 
> 
> --
> Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Alex Bennée]

Pavel Dovgalyuk <pavel.dovgaluk@gmail.com> writes:

> The set of patches include the block-related updates
> of the record/replay icount feature:
>  - application of 'snapshot' option on the file layer instead of
>    the top one: command line and documentation fix
>  - implementation of bdrv_snapshot_goto for blkreplay driver
>  - start/stop fix in replay mode with attached block devices
>  - record/replay of bh oneshot events, used in the block layer

Can we come up with a reasonable smoke test to verify record/replay is
functional? AIUI we don't need a full OS but we do need a block device
to store the replay data. Could we use one of the simple system tests
like "memory" and run that through a record and replay cycle?

If we can defend the basic functionally in "make check" then breakage is
less likely to slip through and you'll have less bisecting to do.

>
> ---
>
> Pavel Dovgaluk (6):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       replay: add BH oneshot event for block layer
>
>
>  block/blkreplay.c        |    8 ++++++++
>  block/block-backend.c    |    9 ++++++---
>  block/io.c               |   32 ++++++++++++++++++++++++++++++--
>  block/iscsi.c            |    5 +++--
>  block/nfs.c              |    6 ++++--
>  block/null.c             |    4 +++-
>  block/nvme.c             |    6 ++++--
>  block/rbd.c              |    5 +++--
>  block/vxhs.c             |    5 +++--
>  cpus.c                   |    2 --
>  docs/replay.txt          |   12 +++++++++---
>  include/sysemu/replay.h  |    4 ++++
>  replay/replay-events.c   |   16 ++++++++++++++++
>  replay/replay-internal.h |    1 +
>  replay/replay.c          |    2 ++
>  stubs/Makefile.objs      |    1 +
>  stubs/replay-user.c      |    9 +++++++++
>  vl.c                     |   11 +++++++++--
>  18 files changed, 115 insertions(+), 23 deletions(-)
>  create mode 100644 stubs/replay-user.c


--
Alex Bennée

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[no-reply]

Patchew URL: https://patchew.org/QEMU/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

./tests/docker/docker.py --engine auto build qemu:fedora tests/docker/dockerfiles/fedora.docker   --add-current-user  
Image is up to date.
  LD      docker-test-mingw@fedora.mo
cc: fatal error: no input files
compilation terminated.
make: *** [docker-test-mingw@fedora.mo] Error 4



The full log is available at
http://patchew.org/logs/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[no-reply]

Patchew URL: https://patchew.org/QEMU/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

./tests/docker/docker.py --engine auto build qemu:fedora tests/docker/dockerfiles/fedora.docker   --add-current-user  
Image is up to date.
  LD      docker-test-debug@fedora.mo
cc: fatal error: no input files
compilation terminated.
make: *** [docker-test-debug@fedora.mo] Error 4



The full log is available at
http://patchew.org/logs/156872146565.1757.3033215873677512474.stgit@pasha-Precision-3630-Tower/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

[Qemu-devel] [for-4.2 PATCH 0/6] Block-related record/replay fixes

[Pavel Dovgalyuk]

The set of patches include the block-related updates
of the record/replay icount feature:
 - application of 'snapshot' option on the file layer instead of
   the top one: command line and documentation fix
 - implementation of bdrv_snapshot_goto for blkreplay driver
 - start/stop fix in replay mode with attached block devices
 - record/replay of bh oneshot events, used in the block layer

---

Pavel Dovgaluk (6):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      replay: add BH oneshot event for block layer


 block/blkreplay.c        |    8 ++++++++
 block/block-backend.c    |    9 ++++++---
 block/io.c               |   32 ++++++++++++++++++++++++++++++--
 block/iscsi.c            |    5 +++--
 block/nfs.c              |    6 ++++--
 block/null.c             |    4 +++-
 block/nvme.c             |    6 ++++--
 block/rbd.c              |    5 +++--
 block/vxhs.c             |    5 +++--
 cpus.c                   |    2 --
 docs/replay.txt          |   12 +++++++++---
 include/sysemu/replay.h  |    4 ++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.c          |    2 ++
 stubs/Makefile.objs      |    1 +
 stubs/replay-user.c      |    9 +++++++++
 vl.c                     |   11 +++++++++--
 18 files changed, 115 insertions(+), 23 deletions(-)
 create mode 100644 stubs/replay-user.c

-- 
Pavel Dovgalyuk