[Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Pavel Dovgalyuk]

GDB remote protocol supports reverse debugging of the targets.
It includes 'reverse step' and 'reverse continue' operations.
The first one finds the previous step of the execution,
and the second one is intended to stop at the last breakpoint that
would happen when the program is executed normally.

Reverse debugging is possible in the replay mode, when at least
one snapshot was created at the record or replay phase.
QEMU can use these snapshots for travelling back in time with GDB.

Running the execution in replay mode allows using GDB reverse debugging
commands:
 - reverse-stepi (or rsi): Steps one instruction to the past.
   QEMU loads on of the prior snapshots and proceeds to the desired
   instruction forward. When that step is reaches, execution stops.
 - reverse-continue (or rc): Runs execution "backwards".
   QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
   and replaying the execution. Then QEMU loads snapshots again and
   replays to the latest breakpoint. When there are no breakpoints in
   the examined section of the execution, QEMU finds one more snapshot
   and tries again. After the first snapshot is processed, execution
   stops at this snapshot.

The set of patches include the following modifications:
 - gdbstub update for reverse debugging support
 - functions that automatically perform reverse step and reverse
   continue operations
 - hmp/qmp commands for manipulating the replay process
 - improvement of the snapshotting for saving the execution step
   in the snapshot parameters
 - other record/replay fixes

The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180207

---

Pavel Dovgalyuk (17):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      migration: introduce icount field for snapshots
      qcow2: introduce icount field for snapshots
      replay: introduce info hmp/qmp command
      replay: introduce breakpoint at the specified step
      replay: implement replay_seek command to proceed to the desired step
      replay: flush events when exitting
      timer: remove replay clock probe in deadline calculation
      replay: refine replay-time module
      translator: fix breakpoint processing
      replay: flush rr queue before loading the vmstate
      gdbstub: add reverse step support in replay mode
      gdbstub: add reverse continue support in replay mode


 accel/tcg/translator.c    |    8 +
 block/blkreplay.c         |    8 +
 block/io.c                |   22 +++
 block/qapi.c              |   11 +-
 block/qcow2-snapshot.c    |    9 +
 block/qcow2.h             |    2 
 blockdev.c                |    3 
 cpus.c                    |   19 ++-
 docs/replay.txt           |   12 +-
 exec.c                    |    6 +
 gdbstub.c                 |   50 +++++++-
 hmp-commands-info.hx      |   14 ++
 hmp-commands.hx           |   30 +++++
 hmp.h                     |    3 
 include/block/snapshot.h  |    1 
 include/sysemu/replay.h   |   18 +++
 migration/savevm.c        |   11 +-
 qapi/block-core.json      |    5 +
 qapi/block.json           |    3 
 qapi/misc.json            |   69 +++++++++++
 replay/Makefile.objs      |    3 
 replay/replay-debugging.c |  286 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c    |   14 --
 replay/replay-internal.h  |   10 +-
 replay/replay-time.c      |   27 ++--
 replay/replay.c           |   22 +++
 stubs/replay.c            |   10 ++
 util/qemu-timer.c         |   11 --
 vl.c                      |   11 +-
 29 files changed, 625 insertions(+), 73 deletions(-)
 create mode 100644 replay/replay-debugging.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH 01/17] block: implement bdrv_snapshot_goto for blkreplay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch enables making snapshots with blkreplay used in
block devices.
This function is required to make bdrv_snapshot_goto without
calling .bdrv_open which is not implemented.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/blkreplay.c |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/block/blkreplay.c b/block/blkreplay.c
index fe5a9b4..ec0aa82 100755
--- a/block/blkreplay.c
+++ b/block/blkreplay.c
@@ -127,6 +127,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)
     return ret;
 }
 
+static int blkreplay_snapshot_goto(BlockDriverState *bs,
+                                   const char *snapshot_id)
+{
+    return bdrv_snapshot_goto(bs->file->bs, snapshot_id, NULL);
+}
+
 static BlockDriver bdrv_blkreplay = {
     .format_name            = "blkreplay",
     .instance_size          = 0,
@@ -142,6 +148,8 @@ static BlockDriver bdrv_blkreplay = {
     .bdrv_co_pwrite_zeroes  = blkreplay_co_pwrite_zeroes,
     .bdrv_co_pdiscard       = blkreplay_co_pdiscard,
     .bdrv_co_flush          = blkreplay_co_flush,
+
+    .bdrv_snapshot_goto     = blkreplay_snapshot_goto,
 };
 
 static void bdrv_blkreplay_init(void)

[Qemu-devel] [RFC PATCH 02/17] replay: disable default snapshot for record/replay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch disables setting '-snapshot' option on by default
in record/replay mode. This is needed for creating vmstates in record
and replay modes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 vl.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/vl.c b/vl.c
index fce1fd1..ca4239c 100644
--- a/vl.c
+++ b/vl.c
@@ -3226,7 +3226,13 @@ int main(int argc, char **argv, char **envp)
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
                 break;
             case QEMU_OPTION_snapshot:
-                snapshot = 1;
+                {
+                    Error *blocker = NULL;
+                    snapshot = 1;
+                    error_setg(&blocker, QERR_REPLAY_NOT_SUPPORTED,
+                               "-snapshot");
+                    replay_add_blocker(blocker);
+                }
                 break;
             case QEMU_OPTION_numa:
                 opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),
@@ -4529,7 +4535,7 @@ int main(int argc, char **argv, char **envp)
         qapi_free_BlockdevOptions(bdo->bdo);
         g_free(bdo);
     }
-    if (snapshot || replay_mode != REPLAY_MODE_NONE) {
+    if (snapshot) {
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot,
                           NULL, NULL);
     }

[Qemu-devel] [RFC PATCH 03/17] replay: update docs for record/replay with block devices

[Pavel Dovgalyuk]

This patch updates the description of the command lines for using
record/replay with attached block devices.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 docs/replay.txt |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/docs/replay.txt b/docs/replay.txt
index 2e21e9c..f7def53 100644
--- a/docs/replay.txt
+++ b/docs/replay.txt
@@ -27,7 +27,7 @@ Usage of the record/replay:
  * First, record the execution with the following command line:
     qemu-system-i386 \
      -icount shift=7,rr=record,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay:
  * After recording, you can replay it by using another command line:
     qemu-system-i386 \
      -icount shift=7,rr=replay,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of
 bdrv coroutine functions at the top of block drivers stack.
 To record and replay block operations the drive must be configured
 as following:
- -drive file=disk.qcow2,if=none,id=img-direct
+ -drive file=disk.qcow2,if=none,snapshot,id=img-direct
  -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
  -device ide-hd,drive=img-blkreplay
 
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start
 of replaying. It also can be loaded while replaying to roll back
 the execution.
 
+'snapshot' flag of the disk image must be removed to save the snapshots
+in the overlay (or original image) instead of using the temporary overlay.
+ -drive file=disk.ovl,if=none,id=img-direct
+ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+ -device ide-hd,drive=img-blkreplay
+
 Use QEMU monitor to create additional snapshots. 'savevm <name>' command
 created the snapshot and 'loadvm <name>' restores it. To prevent corruption
 of the original disk image, use overlay files linked to the original images.

[Qemu-devel] [RFC PATCH 04/17] replay: don't drain/flush bdrv queue while RR is working

[Pavel Dovgalyuk]

In record/replay mode bdrv queue is controlled by replay mechanism.
It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty
queue, but flushing the queue is still impossible there,
because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in
record/replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/io.c |   22 ++++++++++++++++++++++
 cpus.c     |    2 --
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/block/io.c b/block/io.c
index bd9a19a..1ceefbc 100644
--- a/block/io.c
+++ b/block/io.c
@@ -32,6 +32,7 @@
 #include "qemu/cutils.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
+#include "sysemu/replay.h"
 
 #define NOT_DONE 0x7fffffff /* used while emulated sync operation in progress */
 
@@ -407,6 +408,13 @@ void bdrv_drain_all_begin(void)
     BdrvNextIterator it;
     GSList *aio_ctxs = NULL, *ctx;
 
+    /* bdrv queue is managed by record/replay,
+       waiting for finishing the I/O requests may
+       be infinite */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     /* BDRV_POLL_WHILE() for a node can only be called from its own I/O thread
      * or the main loop AioContext. We potentially use BDRV_POLL_WHILE() on
      * nodes in several different AioContexts, so make sure we're in the main
@@ -458,6 +466,13 @@ void bdrv_drain_all_end(void)
     BlockDriverState *bs;
     BdrvNextIterator it;
 
+    /* bdrv queue is managed by record/replay,
+       waiting for finishing the I/O requests may
+       be endless */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
 
@@ -1832,6 +1847,13 @@ int bdrv_flush_all(void)
     BlockDriverState *bs = NULL;
     int result = 0;
 
+    /* bdrv queue is managed by record/replay,
+       creating new flush request for stopping
+       the VM may break the determinism */
+    if (replay_events_enabled()) {
+        return result;
+    }
+
     for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
         int ret;
diff --git a/cpus.c b/cpus.c
index 38eba8b..140cc4f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1016,7 +1016,6 @@ static int do_vm_stop(RunState state, bool send_stop)
     }
 
     bdrv_drain_all();
-    replay_disable_events();
     ret = bdrv_flush_all();
 
     return ret;
@@ -2059,7 +2058,6 @@ int vm_prepare_start(void)
         qapi_event_send_stop(&error_abort);
         res = -1;
     } else {
-        replay_enable_events();
         cpu_enable_ticks();
         runstate_set(RUN_STATE_RUNNING);
         vm_state_notify(1, RUN_STATE_RUNNING);

[Qemu-devel] [RFC PATCH 05/17] replay: finish record/replay before closing the disks

[Pavel Dovgalyuk]

After recent updates block devices cannot be closed on qemu exit.
This happens due to the block request polling when replay is not finished.
Therefore now we stop execution recording before closing the block devices.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay.c |    2 ++
 vl.c            |    1 +
 2 files changed, 3 insertions(+)

diff --git a/replay/replay.c b/replay/replay.c
index 8228261..58a986f 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -366,6 +366,8 @@ void replay_finish(void)
     g_free(replay_snapshot);
     replay_snapshot = NULL;
 
+    replay_mode = REPLAY_MODE_NONE;
+
     replay_finish_events();
 }
 
diff --git a/vl.c b/vl.c
index ca4239c..98157d8 100644
--- a/vl.c
+++ b/vl.c
@@ -4743,6 +4743,7 @@ int main(int argc, char **argv, char **envp)
 
     /* No more vcpu or device emulation activity beyond this point */
     vm_shutdown();
+    replay_finish();
 
     bdrv_close_all();
 

[Qemu-devel] [RFC PATCH 06/17] migration: introduce icount field for snapshots

[Pavel Dovgalyuk]

Saving icount as a parameters of the snapshot allows navigation between
them in the execution replay scenario.
This information can be used for finding a specific snapshot for rewinding
the recorded execution to the specific moment of the time.
E.g., 'reverse step' action needs to load the nearest snapshot which is
prior to the current moment of time .

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 block/qapi.c             |   11 +++++++----
 blockdev.c               |    3 +++
 include/block/snapshot.h |    1 +
 migration/savevm.c       |    1 +
 qapi/block-core.json     |    5 ++++-
 qapi/block.json          |    3 ++-
 6 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/block/qapi.c b/block/qapi.c
index 04c6fc6..b51a23a 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -210,6 +210,7 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs,
         info->date_nsec     = sn_tab[i].date_nsec;
         info->vm_clock_sec  = sn_tab[i].vm_clock_nsec / 1000000000;
         info->vm_clock_nsec = sn_tab[i].vm_clock_nsec % 1000000000;
+        info->icount        = sn_tab[i].icount;
 
         info_list = g_new0(SnapshotInfoList, 1);
         info_list->value = info;
@@ -654,8 +655,8 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f,
 
     if (!sn) {
         func_fprintf(f,
-                     "%-10s%-20s%7s%20s%15s",
-                     "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK");
+                     "%-10s%-18s%7s%20s%13s%11s",
+                     "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK", "ICOUNT");
     } else {
         ti = sn->date_sec;
         localtime_r(&ti, &tm);
@@ -669,12 +670,13 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f,
                  (int)(secs % 60),
                  (int)((sn->vm_clock_nsec / 1000000) % 1000));
         func_fprintf(f,
-                     "%-10s%-20s%7s%20s%15s",
+                     "%-10s%-18s%7s%20s%13s%11"PRId64,
                      sn->id_str, sn->name,
                      get_human_readable_size(buf1, sizeof(buf1),
                                              sn->vm_state_size),
                      date_buf,
-                     clock_buf);
+                     clock_buf,
+                     sn->icount);
     }
 }
 
@@ -842,6 +844,7 @@ void bdrv_image_info_dump(fprintf_function func_fprintf, void *f,
                 .date_nsec = elem->value->date_nsec,
                 .vm_clock_nsec = elem->value->vm_clock_sec * 1000000000ULL +
                                  elem->value->vm_clock_nsec,
+                .icount = elem->value->icount,
             };
 
             pstrcpy(sn.id_str, sizeof(sn.id_str), elem->value->id);
diff --git a/blockdev.c b/blockdev.c
index c31bf3d..6771b79 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -56,6 +56,7 @@
 #include "block/trace.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/qtest.h"
+#include "sysemu/replay.h"
 #include "qemu/cutils.h"
 #include "qemu/help_option.h"
 #include "qemu/throttle-options.h"
@@ -1348,6 +1349,7 @@ SnapshotInfo *qmp_blockdev_snapshot_delete_internal_sync(const char *device,
     info->vm_state_size = sn.vm_state_size;
     info->vm_clock_nsec = sn.vm_clock_nsec % 1000000000;
     info->vm_clock_sec = sn.vm_clock_nsec / 1000000000;
+    info->icount = sn.icount;
 
     return info;
 
@@ -1556,6 +1558,7 @@ static void internal_snapshot_prepare(BlkActionState *common,
     sn->date_sec = tv.tv_sec;
     sn->date_nsec = tv.tv_usec * 1000;
     sn->vm_clock_nsec = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    sn->icount = replay_get_current_step();
 
     ret1 = bdrv_snapshot_create(bs, sn);
     if (ret1 < 0) {
diff --git a/include/block/snapshot.h b/include/block/snapshot.h
index f73d109..c9c8975 100644
--- a/include/block/snapshot.h
+++ b/include/block/snapshot.h
@@ -42,6 +42,7 @@ typedef struct QEMUSnapshotInfo {
     uint32_t date_sec; /* UTC date of the snapshot */
     uint32_t date_nsec;
     uint64_t vm_clock_nsec; /* VM clock relative to boot */
+    uint64_t icount; /* record/replay step */
 } QEMUSnapshotInfo;
 
 int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
diff --git a/migration/savevm.c b/migration/savevm.c
index e2be02a..6ca67fb 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2270,6 +2270,7 @@ int save_snapshot(const char *name, Error **errp)
     sn->date_sec = tv.tv_sec;
     sn->date_nsec = tv.tv_usec * 1000;
     sn->vm_clock_nsec = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    sn->icount = replay_get_current_step();
 
     if (name) {
         ret = bdrv_snapshot_find(bs, old_sn, name);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index c50517b..675b24e 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -25,13 +25,16 @@
 #
 # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec
 #
+# @icount: current instruction count for execution record/replay
+#
 # Since: 1.3
 #
 ##
 { 'struct': 'SnapshotInfo',
   'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int',
             'date-sec': 'int', 'date-nsec': 'int',
-            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } }
+            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int',
+            'icount': 'int' } }
 
 ##
 # @ImageInfoSpecificQCow2EncryptionBase:
diff --git a/qapi/block.json b/qapi/block.json
index c694524..2f364f7 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -148,7 +148,8 @@
 #                    "date-sec": 1000012,
 #                    "date-nsec": 10,
 #                    "vm-clock-sec": 100,
-#                    "vm-clock-nsec": 20
+#                    "vm-clock-nsec": 20,
+#                    "icount": 220414
 #      }
 #    }
 #

[Qemu-devel] [RFC PATCH 07/17] qcow2: introduce icount field for snapshots

[Pavel Dovgalyuk]

This patch introduces the icount field for saving within the snapshot.
It is required for navigation between the snapshots in record/replay mode.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 block/qcow2-snapshot.c |    9 +++++++++
 block/qcow2.h          |    2 ++
 2 files changed, 11 insertions(+)

diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 74293be..4f2357e 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs)
             sn->disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
         }
 
+        if (extra_data_size >= 24) {
+            sn->icount = be64_to_cpu(extra.icount);
+        } else {
+            sn->icount = 0;
+        }
+
         /* Read snapshot ID */
         sn->id_str = g_malloc(id_str_size + 1);
         ret = bdrv_pread(bs->file, offset, sn->id_str, id_str_size);
@@ -209,6 +215,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)
         memset(&extra, 0, sizeof(extra));
         extra.vm_state_size_large = cpu_to_be64(sn->vm_state_size);
         extra.disk_size = cpu_to_be64(sn->disk_size);
+        extra.icount = cpu_to_be64(sn->icount);
 
         id_str_size = strlen(sn->id_str);
         name_size = strlen(sn->name);
@@ -372,6 +379,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
     sn->date_sec = sn_info->date_sec;
     sn->date_nsec = sn_info->date_nsec;
     sn->vm_clock_nsec = sn_info->vm_clock_nsec;
+    sn->icount = sn_info->icount;
 
     /* Allocate the L1 table of the snapshot and copy the current one there. */
     l1_table_offset = qcow2_alloc_clusters(bs, s->l1_size * sizeof(uint64_t));
@@ -691,6 +699,7 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab)
         sn_info->date_sec = sn->date_sec;
         sn_info->date_nsec = sn->date_nsec;
         sn_info->vm_clock_nsec = sn->vm_clock_nsec;
+        sn_info->icount = sn->icount;
     }
     *psn_tab = sn_tab;
     return s->nb_snapshots;
diff --git a/block/qcow2.h b/block/qcow2.h
index adf5c39..8880937 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -151,6 +151,7 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {
 typedef struct QEMU_PACKED QCowSnapshotExtraData {
     uint64_t vm_state_size_large;
     uint64_t disk_size;
+    uint64_t icount;
 } QCowSnapshotExtraData;
 
 
@@ -164,6 +165,7 @@ typedef struct QCowSnapshot {
     uint32_t date_sec;
     uint32_t date_nsec;
     uint64_t vm_clock_nsec;
+    uint64_t icount;
 } QCowSnapshot;
 
 struct Qcow2Cache;

[Qemu-devel] [RFC PATCH 08/17] replay: introduce info hmp/qmp command

[Pavel Dovgalyuk]

This patch introduces 'info replay' monitor command and
corresponding qmp request.
These commands request the current record/replay mode, replay log file name,
and the execution step (number or recorded/replayed instructions).

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 hmp-commands-info.hx      |   14 ++++++++++++++
 hmp.h                     |    1 +
 qapi/misc.json            |   36 ++++++++++++++++++++++++++++++++++++
 replay/Makefile.objs      |    3 ++-
 replay/replay-debugging.c |   41 +++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h  |    2 ++
 replay/replay.c           |    3 +--
 7 files changed, 97 insertions(+), 3 deletions(-)
 create mode 100644 replay/replay-debugging.c

diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
index ddfcd5a..f5631be 100644
--- a/hmp-commands-info.hx
+++ b/hmp-commands-info.hx
@@ -883,6 +883,20 @@ STEXI
 Show SEV information.
 ETEXI
 
+    {
+        .name       = "replay",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show parameters of the record/replay",
+        .cmd        = hmp_info_replay,
+    },
+
+STEXI
+@item info replay
+@findex info replay
+Display the current record/replay mode and the currently executing step.
+ETEXI
+
 STEXI
 @end table
 ETEXI
diff --git a/hmp.h b/hmp.h
index 4e2ec37..084fb62 100644
--- a/hmp.h
+++ b/hmp.h
@@ -144,5 +144,6 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict);
 void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);
 void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
+void hmp_info_replay(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/qapi/misc.json b/qapi/misc.json
index 5636f4a..2ded924 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3449,3 +3449,39 @@
 ##
 { 'command': 'x-oob-test', 'data' : { 'lock': 'bool' },
   'allow-oob': true }
+
+##
+# @ReplayInfo:
+#
+# Status of the record/replay mode.
+#
+# @mode: current mode.
+#
+# @filename: name of the record/replay log file.
+#
+# @step: current step number.
+#
+# Since: 2.13
+#
+##
+{ 'struct': 'ReplayInfo',
+  'data': { 'mode': 'ReplayMode', '*filename': 'str', 'step': 'int' } }
+
+
+##
+# @info_replay:
+#
+# Retrieves the status of the execution record/replay.
+#
+# Returns: structure with the properties of the record/replay.
+#
+# Since: 2.13
+#
+# Example:
+#
+# -> { "execute": "info_replay" }
+# <- { "return": { "mode": "play", "filename": "log.rr", "step": 220414 } }
+#
+##
+{ 'command': 'info_replay',
+  'returns': 'ReplayInfo' }
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index cee6539..6694e3e 100644
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -6,4 +6,5 @@ common-obj-y += replay-input.o
 common-obj-y += replay-char.o
 common-obj-y += replay-snapshot.o
 common-obj-y += replay-net.o
-common-obj-y += replay-audio.o
\ No newline at end of file
+common-obj-y += replay-audio.o
+common-obj-y += replay-debugging.o
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
new file mode 100644
index 0000000..2777e57
--- /dev/null
+++ b/replay/replay-debugging.c
@@ -0,0 +1,41 @@
+/*
+ * replay-debugging.c
+ *
+ * Copyright (c) 2010-2018 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "sysemu/replay.h"
+#include "replay-internal.h"
+#include "hmp.h"
+#include "monitor/monitor.h"
+#include "qapi/qapi-commands-misc.h"
+
+void hmp_info_replay(Monitor *mon, const QDict *qdict)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        monitor_printf(mon, "No record/replay\n");
+    } else {
+        monitor_printf(mon, "%s execution '%s': current step = %"PRId64"\n",
+            replay_mode == REPLAY_MODE_RECORD ? "Recording" : "Replaying",
+            replay_filename, replay_get_current_step());
+    }
+}
+
+ReplayInfo *qmp_info_replay(Error **errp)
+{
+    ReplayInfo *retval = g_new0(ReplayInfo, 1);
+    retval->mode = replay_mode;
+    if (replay_filename) {
+        retval->filename = g_strdup(replay_filename);
+        retval->has_filename = true;
+    }
+    retval->step = replay_get_current_step();
+    return retval;
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index ac4b27b..ef82b5e 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -91,6 +91,8 @@ extern ReplayState replay_state;
 
 /* File for replay writing */
 extern FILE *replay_file;
+/*! Name of replay file  */
+extern char *replay_filename;
 
 void replay_put_byte(uint8_t byte);
 void replay_put_event(uint8_t event);
diff --git a/replay/replay.c b/replay/replay.c
index 58a986f..8b70d7d 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -29,8 +29,7 @@
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 char *replay_snapshot;
 
-/* Name of replay file  */
-static char *replay_filename;
+char *replay_filename;
 ReplayState replay_state;
 static GSList *replay_blockers;
 

[Qemu-devel] [RFC PATCH 09/17] replay: introduce breakpoint at the specified step

[Pavel Dovgalyuk]

This patch introduces replay_break qmp and hmp commands.
These commands allow stopping at the specified instruction.
It may be useful for debugging when there are some known
events that should be investigated.
The commands have one argument - number of instructions
executed since the start of the replay.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 hmp-commands.hx           |   15 ++++++++++++
 hmp.h                     |    1 +
 include/sysemu/replay.h   |    3 ++
 qapi/misc.json            |   17 ++++++++++++++
 replay/replay-debugging.c |   55 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h  |    4 +++
 replay/replay.c           |   17 ++++++++++++++
 7 files changed, 112 insertions(+)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index 35d862a..f162f5e 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1820,6 +1820,21 @@ Set QOM property @var{property} of object at location @var{path} to value @var{v
 ETEXI
 
     {
+        .name       = "replay_break",
+        .args_type  = "step:i",
+        .params     = "step",
+        .help       = "sets breakpoint on the specified step of the replay",
+        .cmd        = hmp_replay_break,
+    },
+
+STEXI
+@item replay_break @var{step}
+@findex replay_break
+Set breakpoint on the specified step of the replay.
+Execution stops when the specified step is reached.
+ETEXI
+
+    {
         .name       = "info",
         .args_type  = "item:s?",
         .params     = "[subcommand]",
diff --git a/hmp.h b/hmp.h
index 084fb62..5ef8f56 100644
--- a/hmp.h
+++ b/hmp.h
@@ -145,5 +145,6 @@ void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);
 void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
 void hmp_info_replay(Monitor *mon, const QDict *qdict);
+void hmp_replay_break(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 3ced6bc..98d709c 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -71,6 +71,9 @@ void replay_start(void);
 void replay_finish(void);
 /*! Adds replay blocker with the specified error description */
 void replay_add_blocker(Error *reason);
+/*! Sets breakpoint at the specified step.
+    If step = -1LL the existing breakpoint is removed. */
+void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
 
 /* Processing the instructions */
 
diff --git a/qapi/misc.json b/qapi/misc.json
index 2ded924..920099c 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3485,3 +3485,20 @@
 ##
 { 'command': 'info_replay',
   'returns': 'ReplayInfo' }
+
+##
+# @replay_break:
+#
+# Set breakpoint on the specified step of the replay.
+# Execution stops when the specified step is reached.
+#
+# @step: execution step to stop at
+#
+# Since: 2.13
+#
+# Example:
+#
+# -> { "execute": "replay_break", "data": { "step": 220414 } }
+#
+##
+{ 'command': 'replay_break', 'data': { 'step': 'int' } }
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index 2777e57..330bd91 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -16,6 +16,8 @@
 #include "hmp.h"
 #include "monitor/monitor.h"
 #include "qapi/qapi-commands-misc.h"
+#include "qapi/qmp/qdict.h"
+#include "qemu/timer.h"
 
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
@@ -39,3 +41,56 @@ ReplayInfo *qmp_info_replay(Error **errp)
     retval->step = replay_get_current_step();
     return retval;
 }
+
+void replay_break(int64_t step, QEMUTimerCB callback, void *opaque)
+{
+    assert(replay_mode == REPLAY_MODE_PLAY);
+    assert(replay_mutex_locked());
+
+    replay_break_step = step;
+    if (replay_break_timer) {
+        timer_del(replay_break_timer);
+        timer_free(replay_break_timer);
+        replay_break_timer = NULL;
+    }
+
+    if (replay_break_step == -1LL) {
+        return;
+    }
+    assert(replay_break_step >= replay_get_current_step());
+    assert(callback);
+
+    replay_break_timer = timer_new_ns(QEMU_CLOCK_REALTIME, callback, opaque);
+}
+
+static void replay_stop_vm(void *opaque)
+{
+    vm_stop(RUN_STATE_PAUSED);
+    replay_break(-1LL, NULL, NULL);
+}
+
+void qmp_replay_break(int64_t step, Error **errp)
+{
+    if (replay_mode ==  REPLAY_MODE_PLAY) {
+        if (step >= replay_get_current_step()) {
+            replay_break(step, replay_stop_vm, NULL);
+        } else {
+            error_setg(errp, "cannot set break at the step in the past");
+        }
+    } else {
+        error_setg(errp, "setting the break is allowed only in play mode");
+    }
+}
+
+void hmp_replay_break(Monitor *mon, const QDict *qdict)
+{
+    int64_t step = qdict_get_try_int(qdict, "step", -1LL);
+    Error *err = NULL;
+
+    qmp_replay_break(step, &err);
+    if (err) {
+        monitor_printf(mon, "replay_break error: %s\n", error_get_pretty(err));
+        error_free(err);
+        return;
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index ef82b5e..34d19eb 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -93,6 +93,10 @@ extern ReplayState replay_state;
 extern FILE *replay_file;
 /*! Name of replay file  */
 extern char *replay_filename;
+/*! Step of the replay breakpoint */
+extern int64_t replay_break_step;
+/*! Timer for the replay breakpoint callback */
+extern QEMUTimer *replay_break_timer;
 
 void replay_put_byte(uint8_t byte);
 void replay_put_event(uint8_t event);
diff --git a/replay/replay.c b/replay/replay.c
index 8b70d7d..dcce902 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -33,6 +33,10 @@ char *replay_filename;
 ReplayState replay_state;
 static GSList *replay_blockers;
 
+/* Replay breakpoints */
+int64_t replay_break_step = -1LL;
+QEMUTimer *replay_break_timer;
+
 bool replay_next_event_is(int event)
 {
     bool res = false;
@@ -72,6 +76,13 @@ int replay_get_instructions(void)
     replay_mutex_lock();
     if (replay_next_event_is(EVENT_INSTRUCTION)) {
         res = replay_state.instructions_count;
+        if (replay_break_step != -1LL) {
+            uint64_t current = replay_get_current_step();
+            assert(replay_break_step >= current);
+            if (current + res > replay_break_step) {
+                res = replay_break_step - current;
+            }
+        }
     }
     replay_mutex_unlock();
     return res;
@@ -98,6 +109,12 @@ void replay_account_executed_instructions(void)
                    will be read from the log. */
                 qemu_notify_event();
             }
+            /* Execution reached the break step */
+            if (replay_break_step == replay_state.current_step) {
+                /* Cannot make callback directly from the vCPU thread */
+                timer_mod_ns(replay_break_timer,
+                    qemu_clock_get_ns(QEMU_CLOCK_REALTIME));
+            }
         }
     }
 }

[Qemu-devel] [RFC PATCH 10/17] replay: implement replay_seek command to proceed to the desired step

[Pavel Dovgalyuk]

This patch adds hmp/qmp command replay_seek which proceeds the execution
to the specified step.
It automatically loads nearest snapshot and replays the execution to find
the desired step.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 hmp-commands.hx           |   15 ++++++++
 hmp.h                     |    1 +
 qapi/misc.json            |   16 ++++++++
 replay/replay-debugging.c |   88 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 120 insertions(+)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index f162f5e..18b287e 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1835,6 +1835,21 @@ Execution stops when the specified step is reached.
 ETEXI
 
     {
+        .name       = "replay_seek",
+        .args_type  = "step:i",
+        .params     = "step",
+        .help       = "rewinds replay to the specified step",
+        .cmd        = hmp_replay_seek,
+    },
+
+STEXI
+@item replay_seek @var{step}
+@findex replay_seek
+Automatically proceeds to the specified step, when replaying
+the execution.
+ETEXI
+
+    {
         .name       = "info",
         .args_type  = "item:s?",
         .params     = "[subcommand]",
diff --git a/hmp.h b/hmp.h
index 5ef8f56..31f830c 100644
--- a/hmp.h
+++ b/hmp.h
@@ -146,5 +146,6 @@ void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
 void hmp_info_replay(Monitor *mon, const QDict *qdict);
 void hmp_replay_break(Monitor *mon, const QDict *qdict);
+void hmp_replay_seek(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/qapi/misc.json b/qapi/misc.json
index 920099c..98d7be7 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3502,3 +3502,19 @@
 #
 ##
 { 'command': 'replay_break', 'data': { 'step': 'int' } }
+
+##
+# @replay_seek:
+#
+# Automatically proceeds to the specified step, when replaying
+# the execution.
+#
+# @step: destination execution step
+#
+# Since: 2.13
+#
+# Example:
+#
+# -> { "execute": "replay_seek", "data": { "step": 220414 } }
+##
+{ 'command': 'replay_seek', 'data': { 'step': 'int' } }
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index 330bd91..677714e 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -18,6 +18,8 @@
 #include "qapi/qapi-commands-misc.h"
 #include "qapi/qmp/qdict.h"
 #include "qemu/timer.h"
+#include "block/snapshot.h"
+#include "migration/snapshot.h"
 
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
@@ -94,3 +96,89 @@ void hmp_replay_break(Monitor *mon, const QDict *qdict)
         return;
     }
 }
+
+static char *replay_find_nearest_snapshot(int64_t step, int64_t* snapshot_step)
+{
+    BlockDriverState *bs;
+    QEMUSnapshotInfo *sn_tab;
+    QEMUSnapshotInfo *nearest = NULL;
+    char *ret = NULL;
+    int nb_sns, i;
+    AioContext *aio_context;
+
+    *snapshot_step = -1;
+
+    bs = bdrv_all_find_vmstate_bs();
+    if (!bs) {
+        goto fail;
+    }
+    aio_context = bdrv_get_aio_context(bs);
+
+    aio_context_acquire(aio_context);
+    nb_sns = bdrv_snapshot_list(bs, &sn_tab);
+    aio_context_release(aio_context);
+
+    for (i = 0; i < nb_sns; i++) {
+        if (bdrv_all_find_snapshot(sn_tab[i].name, &bs) == 0) {
+            if (sn_tab[i].icount <= step
+                && (!nearest || nearest->icount < sn_tab[i].icount)) {
+                nearest = &sn_tab[i];
+            }
+        }
+    }
+    if (nearest) {
+        ret = g_strdup(nearest->name);
+        *snapshot_step = nearest->icount;
+    }
+    g_free(sn_tab);
+
+fail:
+    return ret;
+}
+
+static void replay_seek(int64_t step, Error **errp, QEMUTimerCB callback)
+{
+    char *snapshot = NULL;
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        error_setg(errp, "replay must be enabled to seek");
+        return;
+    }
+    if (!replay_snapshot) {
+        error_setg(errp, "snapshotting is disabled");
+        return;
+    }
+    int64_t snapshot_step = -1;
+    snapshot = replay_find_nearest_snapshot(step, &snapshot_step);
+    if (snapshot) {
+        if (step < replay_get_current_step()
+            || replay_get_current_step() < snapshot_step) {
+            vm_stop(RUN_STATE_RESTORE_VM);
+            load_snapshot(snapshot, errp);
+        }
+        g_free(snapshot);
+    }
+    if (replay_get_current_step() <= step) {
+        replay_break(step, callback, NULL);
+        vm_start();
+    } else {
+        error_setg(errp, "cannot seek to the specified step");
+    }
+}
+
+void qmp_replay_seek(int64_t step, Error **errp)
+{
+    replay_seek(step, errp, replay_stop_vm);
+}
+
+void hmp_replay_seek(Monitor *mon, const QDict *qdict)
+{
+    int64_t step = qdict_get_try_int(qdict, "step", -1LL);
+    Error *err = NULL;
+
+    qmp_replay_seek(step, &err);
+    if (err) {
+        monitor_printf(mon, "replay_seek error: %s\n", error_get_pretty(err));
+        error_free(err);
+        return;
+    }
+}

[Qemu-devel] [RFC PATCH 11/17] replay: flush events when exitting

[Pavel Dovgalyuk]

This patch adds events processing when emulation finishes instead
of just cleaning the queue. Now the bdrv coroutines will be in consistent
state when emulator closes. It allows correct polling of the block layer
at exit.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 replay/replay-events.c   |   14 +-------------
 replay/replay-internal.h |    2 --
 2 files changed, 1 insertion(+), 15 deletions(-)

diff --git a/replay/replay-events.c b/replay/replay-events.c
index 707de38..0964a82 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -94,18 +94,6 @@ void replay_disable_events(void)
     }
 }
 
-void replay_clear_events(void)
-{
-    g_assert(replay_mutex_locked());
-
-    while (!QTAILQ_EMPTY(&events_list)) {
-        Event *event = QTAILQ_FIRST(&events_list);
-        QTAILQ_REMOVE(&events_list, event, events);
-
-        g_free(event);
-    }
-}
-
 /*! Adds specified async event to the queue */
 void replay_add_event(ReplayAsyncEventKind event_kind,
                       void *opaque,
@@ -308,7 +296,7 @@ void replay_init_events(void)
 void replay_finish_events(void)
 {
     events_enabled = false;
-    replay_clear_events();
+    replay_flush_events();
 }
 
 bool replay_events_enabled(void)
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 34d19eb..a2221e5 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -148,8 +148,6 @@ void replay_init_events(void);
 void replay_finish_events(void);
 /*! Flushes events queue */
 void replay_flush_events(void);
-/*! Clears events list before loading new VM state */
-void replay_clear_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */

[Qemu-devel] [RFC PATCH 12/17] timer: remove replay clock probe in deadline calculation

[Pavel Dovgalyuk]

Ciro Santilli reported that commit a5ed352596a8b7eb2f9acce34371b944ac3056c4
breaks the execution replay. It happens due to the probing the clock
for the new instances of iothread.
However, this probing was made in replay mode for the timer lists that
are empty.
This patch removes clock probing in replay mode.
It is an artifact of the old version with another thread model.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 util/qemu-timer.c |   11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)

diff --git a/util/qemu-timer.c b/util/qemu-timer.c
index 2ed1bf2..86bfe84 100644
--- a/util/qemu-timer.c
+++ b/util/qemu-timer.c
@@ -578,17 +578,10 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg)
 {
     int64_t deadline = -1;
     QEMUClockType type;
-    bool play = replay_mode == REPLAY_MODE_PLAY;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
         if (qemu_clock_use_for_deadline(type)) {
-            if (!play || type == QEMU_CLOCK_REALTIME) {
-                deadline = qemu_soonest_timeout(deadline,
-                                                timerlist_deadline_ns(tlg->tl[type]));
-            } else {
-                /* Read clock from the replay file and
-                   do not calculate the deadline, based on virtual clock. */
-                qemu_clock_get_ns(type);
-            }
+            deadline = qemu_soonest_timeout(deadline,
+                                            timerlist_deadline_ns(tlg->tl[type]));
         }
     }
     return deadline;

[Qemu-devel] [RFC PATCH 13/17] replay: refine replay-time module

[Pavel Dovgalyuk]

This patch removes refactoring artifacts from the replay/replay-time.c

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 replay/replay-time.c |   27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)

diff --git a/replay/replay-time.c b/replay/replay-time.c
index 6a7565e..40030b8 100644
--- a/replay/replay-time.c
+++ b/replay/replay-time.c
@@ -17,14 +17,12 @@
 
 int64_t replay_save_clock(ReplayClockKind kind, int64_t clock)
 {
+    g_assert(replay_file);
+    g_assert(replay_mutex_locked());
 
-    if (replay_file) {
-        g_assert(replay_mutex_locked());
-
-        replay_save_instructions();
-        replay_put_event(EVENT_CLOCK + kind);
-        replay_put_qword(clock);
-    }
+    replay_save_instructions();
+    replay_put_event(EVENT_CLOCK + kind);
+    replay_put_qword(clock);
 
     return clock;
 }
@@ -46,20 +44,15 @@ void replay_read_next_clock(ReplayClockKind kind)
 /*! Reads next clock event from the input. */
 int64_t replay_read_clock(ReplayClockKind kind)
 {
+    int64_t ret;
     g_assert(replay_file && replay_mutex_locked());
 
     replay_account_executed_instructions();
 
-    if (replay_file) {
-        int64_t ret;
-        if (replay_next_event_is(EVENT_CLOCK + kind)) {
-            replay_read_next_clock(kind);
-        }
-        ret = replay_state.cached_clock[kind];
-
-        return ret;
+    if (replay_next_event_is(EVENT_CLOCK + kind)) {
+        replay_read_next_clock(kind);
     }
+    ret = replay_state.cached_clock[kind];
 
-    error_report("REPLAY INTERNAL ERROR %d", __LINE__);
-    exit(1);
+    return ret;
 }

[Qemu-devel] [RFC PATCH 14/17] translator: fix breakpoint processing

[Pavel Dovgalyuk]

QEMU cannot pass through the breakpoints when 'si' command is used
in remote gdb. This patch disables inserting the breakpoints
when we are already single stepping though the gdb remote protocol.
This patch also fixes icount calculation for the blocks that include
breakpoints - instruction with breakpoint is not executed and shouldn't
be used in icount calculation.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 accel/tcg/translator.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c
index 23c6602..3c7a035 100644
--- a/accel/tcg/translator.c
+++ b/accel/tcg/translator.c
@@ -35,6 +35,7 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
                      CPUState *cpu, TranslationBlock *tb)
 {
     int max_insns;
+    int bp_insn = 0;
 
     /* Initialize DisasContext */
     db->tb = tb;
@@ -73,11 +74,13 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
         tcg_debug_assert(db->is_jmp == DISAS_NEXT);  /* no early exit */
 
         /* Pass breakpoint hits to target for further processing */
-        if (unlikely(!QTAILQ_EMPTY(&cpu->breakpoints))) {
+        if (!db->singlestep_enabled
+            && unlikely(!QTAILQ_EMPTY(&cpu->breakpoints))) {
             CPUBreakpoint *bp;
             QTAILQ_FOREACH(bp, &cpu->breakpoints, entry) {
                 if (bp->pc == db->pc_next) {
                     if (ops->breakpoint_check(db, cpu, bp)) {
+                        bp_insn = 1;
                         break;
                     }
                 }
@@ -119,7 +122,7 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
 
     /* Emit code to exit the TB, as indicated by db->is_jmp.  */
     ops->tb_stop(db, cpu);
-    gen_tb_end(db->tb, db->num_insns);
+    gen_tb_end(db->tb, db->num_insns - bp_insn);
 
     /* The disas_log hook may use these values rather than recompute.  */
     db->tb->size = db->pc_next - db->pc_first;

[Qemu-devel] [RFC PATCH 15/17] replay: flush rr queue before loading the vmstate

[Pavel Dovgalyuk]

Non-empty record/replay queue prevents saving and loading the VM state,
because it includes pending bottom halves and block coroutines.
But when the new VM state is loaded, we don't have to preserve the consistency
of the current state anymore. Therefore this patch just flushes the queue
allowing the coroutines to finish.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 include/sysemu/replay.h  |    2 ++
 migration/savevm.c       |   10 ++++------
 replay/replay-internal.h |    2 --
 3 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 98d709c..84a1ec5 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -132,6 +132,8 @@ void replay_disable_events(void);
 void replay_enable_events(void);
 /*! Returns true when saving events is enabled */
 bool replay_events_enabled(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
 /*! Adds bottom half event to the queue */
 void replay_bh_schedule_event(QEMUBH *bh);
 /*! Adds input event to the queue */
diff --git a/migration/savevm.c b/migration/savevm.c
index 6ca67fb..c69b187 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2419,12 +2419,6 @@ int load_snapshot(const char *name, Error **errp)
     AioContext *aio_context;
     MigrationIncomingState *mis = migration_incoming_get_current();
 
-    if (!replay_can_snapshot()) {
-        error_report("Record/replay does not allow loading snapshot "
-                     "right now. Try once more later.");
-        return -EINVAL;
-    }
-
     if (!bdrv_all_can_snapshot(&bs)) {
         error_setg(errp,
                    "Device '%s' is writable but does not support snapshots",
@@ -2458,6 +2452,10 @@ int load_snapshot(const char *name, Error **errp)
         return -EINVAL;
     }
 
+    /* Flush the record/replay queue. Now the VM state is going
+       to change. Therefore we don't need to preserve its consistency */
+    replay_flush_events();
+
     /* Flush all IO requests so they don't interfere with the new state.  */
     bdrv_drain_all_begin();
 
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index a2221e5..08ef2ec 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -146,8 +146,6 @@ void replay_read_next_clock(unsigned int kind);
 void replay_init_events(void);
 /*! Clears internal data structures for events handling */
 void replay_finish_events(void);
-/*! Flushes events queue */
-void replay_flush_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */

[Qemu-devel] [RFC PATCH 16/17] gdbstub: add reverse step support in replay mode

[Pavel Dovgalyuk]

GDB remote protocol supports two reverse debugging commands:
reverse step and reverse continue.
This patch adds support of the first one to the gdbstub.
Reverse step is intended to step one instruction in the backwards
direction. This is not possible in regular execution.
But replayed execution is deterministic, therefore we can load one of
the prior snapshots and proceed to the desired step. It is equivalent
to stepping one instruction back.
There should be at least one snapshot preceding the debugged part of
the replay log.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 accel/tcg/translator.c    |    1 +
 cpus.c                    |   14 +++++++++++---
 exec.c                    |    5 +++++
 gdbstub.c                 |   42 +++++++++++++++++++++++++++++++++++++++---
 include/sysemu/replay.h   |    7 +++++++
 replay/replay-debugging.c |   33 +++++++++++++++++++++++++++++++++
 stubs/replay.c            |    5 +++++
 7 files changed, 101 insertions(+), 6 deletions(-)

diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c
index 3c7a035..4adb37c 100644
--- a/accel/tcg/translator.c
+++ b/accel/tcg/translator.c
@@ -17,6 +17,7 @@
 #include "exec/gen-icount.h"
 #include "exec/log.h"
 #include "exec/translator.h"
+#include "sysemu/replay.h"
 
 /* Pairs with tcg_clear_temp_count.
    To be called by #TranslatorOps.{translate_insn,tb_stop} if
diff --git a/cpus.c b/cpus.c
index 140cc4f..3fb9321 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1042,9 +1042,17 @@ static bool cpu_can_run(CPUState *cpu)
 
 static void cpu_handle_guest_debug(CPUState *cpu)
 {
-    gdb_set_stop_cpu(cpu);
-    qemu_system_debug_request();
-    cpu->stopped = true;
+    if (!replay_running_debug()) {
+        gdb_set_stop_cpu(cpu);
+        qemu_system_debug_request();
+        cpu->stopped = true;
+    } else {
+        if (!cpu->singlestep_enabled) {
+            cpu_single_step(cpu, SSTEP_ENABLE);
+        } else {
+            cpu_single_step(cpu, 0);
+        }
+    }
 }
 
 #ifdef CONFIG_LINUX
diff --git a/exec.c b/exec.c
index 02b1efe..37f171f 100644
--- a/exec.c
+++ b/exec.c
@@ -2556,6 +2556,11 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)
     QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {
         if (cpu_watchpoint_address_matches(wp, vaddr, len)
             && (wp->flags & flags)) {
+            if (replay_running_debug()) {
+                /* Don't process the watchpoints when we are
+                   in a reverse debugging operation. */
+                return;
+            }
             if (flags == BP_MEM_READ) {
                 wp->flags |= BP_WATCHPOINT_HIT_READ;
             } else {
diff --git a/gdbstub.c b/gdbstub.c
index 3c38073..86dec56 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -39,6 +39,7 @@
 #include "sysemu/kvm.h"
 #include "exec/semihost.h"
 #include "exec/exec-all.h"
+#include "sysemu/replay.h"
 
 #ifdef CONFIG_USER_ONLY
 #define GDB_ATTACHED "0"
@@ -334,6 +335,19 @@ typedef struct GDBState {
  */
 static int sstep_flags = SSTEP_ENABLE|SSTEP_NOIRQ|SSTEP_NOTIMER;
 
+/*! Retrieves flags for single step mode. */
+static int get_sstep_flags(void)
+{
+    /* In replay mode all events written into the log should be replayed.
+     * That is why NOIRQ flag is removed in this mode.
+     */
+    if (replay_mode != REPLAY_MODE_NONE) {
+        return SSTEP_ENABLE;
+    } else {
+        return sstep_flags;
+    }
+}
+
 static GDBState *gdbserver_state;
 
 bool gdb_has_xml;
@@ -424,7 +438,7 @@ static int gdb_continue_partial(GDBState *s, char *newstates)
     CPU_FOREACH(cpu) {
         if (newstates[cpu->cpu_index] == 's') {
             trace_gdbstub_op_stepping(cpu->cpu_index);
-            cpu_single_step(cpu, sstep_flags);
+            cpu_single_step(cpu, get_sstep_flags());
         }
     }
     s->running_state = 1;
@@ -443,7 +457,7 @@ static int gdb_continue_partial(GDBState *s, char *newstates)
                 break; /* nothing to do here */
             case 's':
                 trace_gdbstub_op_stepping(cpu->cpu_index);
-                cpu_single_step(cpu, sstep_flags);
+                cpu_single_step(cpu, get_sstep_flags());
                 cpu_resume(cpu);
                 flag = 1;
                 break;
@@ -1072,9 +1086,28 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
             addr = strtoull(p, (char **)&p, 16);
             gdb_set_cpu_pc(s, addr);
         }
-        cpu_single_step(s->c_cpu, sstep_flags);
+        cpu_single_step(s->c_cpu, get_sstep_flags());
         gdb_continue(s);
         return RS_IDLE;
+    case 'b':
+        /* Backward debugging commands */
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            switch (*p) {
+            case 's':
+                if (replay_reverse_step()) {
+                    gdb_continue(s);
+                    return RS_IDLE;
+                } else {
+                    put_packet(s, "E14");
+                    break;
+                }
+            default:
+                goto unknown_command;
+            }
+        } else {
+            put_packet(s, "E22");
+        }
+        goto unknown_command;
     case 'F':
         {
             target_ulong ret;
@@ -1337,6 +1370,9 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
             if (cc->gdb_core_xml_file != NULL) {
                 pstrcat(buf, sizeof(buf), ";qXfer:features:read+");
             }
+            if (replay_mode == REPLAY_MODE_PLAY) {
+                pstrcat(buf, sizeof(buf), ";ReverseStep+");
+            }
             put_packet(s, buf);
             break;
         }
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 84a1ec5..611eabb 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -74,6 +74,13 @@ void replay_add_blocker(Error *reason);
 /*! Sets breakpoint at the specified step.
     If step = -1LL the existing breakpoint is removed. */
 void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
+/*! Start making one step in backward direction.
+    Used by gdbstub for backwards debugging.
+    Returns true on success. */
+bool replay_reverse_step(void);
+/*! Returns true if replay module is processing
+    reverse_continue or reverse_step request */
+bool replay_running_debug(void);
 
 /* Processing the instructions */
 
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index 677714e..c07bc7b 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -21,6 +21,13 @@
 #include "block/snapshot.h"
 #include "migration/snapshot.h"
 
+static bool replay_is_debugging;
+
+bool replay_running_debug(void)
+{
+    return replay_is_debugging;
+}
+
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
     if (replay_mode == REPLAY_MODE_NONE) {
@@ -182,3 +189,29 @@ void hmp_replay_seek(Monitor *mon, const QDict *qdict)
         return;
     }
 }
+
+static void replay_stop_vm_debug(void *opaque)
+{
+    replay_is_debugging = false;
+    vm_stop(RUN_STATE_DEBUG);
+    replay_break(-1LL, NULL, NULL);
+}
+
+bool replay_reverse_step(void)
+{
+    Error *err = NULL;
+
+    assert(replay_mode == REPLAY_MODE_PLAY);
+
+    if (replay_get_current_step() != 0) {
+        replay_seek(replay_get_current_step() - 1, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            return false;
+        }
+        replay_is_debugging = true;
+        return true;
+    }
+
+    return false;
+}
diff --git a/stubs/replay.c b/stubs/replay.c
index 04279ab..b0fba0e 100644
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -80,3 +80,8 @@ void replay_mutex_lock(void)
 void replay_mutex_unlock(void)
 {
 }
+
+bool replay_reverse_step(void)
+{
+    return false;
+}

[Qemu-devel] [RFC PATCH 17/17] gdbstub: add reverse continue support in replay mode

[Pavel Dovgalyuk]

This patch adds support of the reverse continue operation for gdbstub.
Reverse continue finds the last breakpoint that would happen in normal
execution from the beginning to the current moment.
Implementation of the reverse continue replays the execution twice:
to find the breakpoints that were hit and to seek to the last breakpoint.
Reverse continue loads the previous snapshot and tries to find the breakpoint
since that moment. If there are no such breakpoints, it proceeds to
the earlier snapshot, and so on. When no breakpoints or watchpoints were
hit at all, execution stops at the beginning of the replay log.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 cpus.c                    |    3 ++
 exec.c                    |    1 +
 gdbstub.c                 |   10 ++++++-
 include/sysemu/replay.h   |    6 ++++
 replay/replay-debugging.c |   69 +++++++++++++++++++++++++++++++++++++++++++++
 stubs/replay.c            |    5 +++
 6 files changed, 93 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index 3fb9321..9e81e9a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1048,6 +1048,9 @@ static void cpu_handle_guest_debug(CPUState *cpu)
         cpu->stopped = true;
     } else {
         if (!cpu->singlestep_enabled) {
+            /* Report about the breakpoint and
+               make a single step to skip it */
+            replay_breakpoint();
             cpu_single_step(cpu, SSTEP_ENABLE);
         } else {
             cpu_single_step(cpu, 0);
diff --git a/exec.c b/exec.c
index 37f171f..c31e23d 100644
--- a/exec.c
+++ b/exec.c
@@ -2559,6 +2559,7 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)
             if (replay_running_debug()) {
                 /* Don't process the watchpoints when we are
                    in a reverse debugging operation. */
+                replay_breakpoint();
                 return;
             }
             if (flags == BP_MEM_READ) {
diff --git a/gdbstub.c b/gdbstub.c
index 86dec56..1645b7f 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -1101,6 +1101,14 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
                     put_packet(s, "E14");
                     break;
                 }
+            case 'c':
+                if (replay_reverse_continue()) {
+                    gdb_continue(s);
+                    return RS_IDLE;
+                } else {
+                    put_packet(s, "E14");
+                    break;
+                }
             default:
                 goto unknown_command;
             }
@@ -1371,7 +1379,7 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
                 pstrcat(buf, sizeof(buf), ";qXfer:features:read+");
             }
             if (replay_mode == REPLAY_MODE_PLAY) {
-                pstrcat(buf, sizeof(buf), ";ReverseStep+");
+                pstrcat(buf, sizeof(buf), ";ReverseStep+;ReverseContinue+");
             }
             put_packet(s, buf);
             break;
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 611eabb..a3113c1 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -78,9 +78,15 @@ void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
     Used by gdbstub for backwards debugging.
     Returns true on success. */
 bool replay_reverse_step(void);
+/*! Start searching the last breakpoint/watchpoint.
+    Used by gdbstub for backwards debugging.
+    Returns true if the process successfully started. */
+bool replay_reverse_continue(void);
 /*! Returns true if replay module is processing
     reverse_continue or reverse_step request */
 bool replay_running_debug(void);
+/*! Called in reverse debugging mode to collect breakpoint information */
+void replay_breakpoint(void);
 
 /* Processing the instructions */
 
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index c07bc7b..c3fb40d 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -22,6 +22,8 @@
 #include "migration/snapshot.h"
 
 static bool replay_is_debugging;
+static int64_t replay_last_breakpoint;
+static int64_t replay_last_snapshot;
 
 bool replay_running_debug(void)
 {
@@ -215,3 +217,70 @@ bool replay_reverse_step(void)
 
     return false;
 }
+
+static void replay_continue_end(void)
+{
+    replay_is_debugging = false;
+    vm_stop(RUN_STATE_DEBUG);
+    replay_break(-1LL, NULL, NULL);
+}
+
+static void replay_continue_stop(void *opaque)
+{
+    Error *err = NULL;
+    if (replay_last_breakpoint != -1LL) {
+        replay_seek(replay_last_breakpoint, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        return;
+    }
+    /* No breakpoints since the last snapshot.
+       Find previous snapshot and try again. */
+    if (replay_last_snapshot != 0) {
+        replay_seek(replay_last_snapshot - 1, &err, replay_continue_stop);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        replay_last_snapshot = replay_get_current_step();
+        return;
+    } else {
+        /* Seek to the very first step */
+        replay_seek(0, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        return;
+    }
+    replay_continue_end();
+}
+
+bool replay_reverse_continue(void)
+{
+    Error *err = NULL;
+
+    assert(replay_mode == REPLAY_MODE_PLAY);
+
+    if (replay_get_current_step() != 0) {
+        replay_seek(replay_get_current_step() - 1, &err, replay_continue_stop);
+        if (err) {
+            error_free(err);
+            return false;
+        }
+        replay_last_breakpoint = -1LL;
+        replay_is_debugging = true;
+        replay_last_snapshot = replay_get_current_step();
+        return true;
+    }
+
+    return false;
+}
+
+void replay_breakpoint(void)
+{
+    assert(replay_mode == REPLAY_MODE_PLAY);
+    replay_last_breakpoint = replay_get_current_step();
+}
diff --git a/stubs/replay.c b/stubs/replay.c
index b0fba0e..781974e 100644
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -85,3 +85,8 @@ bool replay_reverse_step(void)
 {
     return false;
 }
+
+bool replay_reverse_continue(void)
+{
+    return false;
+}

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Pavel Dovgalyuk]

> From: Pavel Dovgalyuk [mailto:Pavel.Dovgaluk@ispras.ru]
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-180207

This should be https://github.com/ispras/qemu/tree/rr-180425


Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH 06/17] migration: introduce icount field for snapshots

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> Saving icount as a parameters of the snapshot allows navigation between
> them in the execution replay scenario.
> This information can be used for finding a specific snapshot for rewinding
> the recorded execution to the specific moment of the time.
> E.g., 'reverse step' action needs to load the nearest snapshot which is
> prior to the current moment of time .
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  block/qapi.c             |   11 +++++++----
>  blockdev.c               |    3 +++
>  include/block/snapshot.h |    1 +
>  migration/savevm.c       |    1 +
>  qapi/block-core.json     |    5 ++++-
>  qapi/block.json          |    3 ++-
>  6 files changed, 18 insertions(+), 6 deletions(-)

Using scripts/git.orderfile may make your patches easier to review, by
hoisting interfaces above implementation.

> +++ b/qapi/block-core.json
> @@ -25,13 +25,16 @@
>  #
>  # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec
>  #
> +# @icount: current instruction count for execution record/replay

Missing a '(since 2.13)' tag.

> +#
>  # Since: 1.3
>  #
>  ##
>  { 'struct': 'SnapshotInfo',
>    'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int',
>              'date-sec': 'int', 'date-nsec': 'int',
> -            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } }
> +            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int',
> +            'icount': 'int' } }

'icount' should be optional, as older qcow2 images will not have that
information available.  Also, putting patch 7 before this patch may make
more sense.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC PATCH 07/17] qcow2: introduce icount field for snapshots

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> This patch introduces the icount field for saving within the snapshot.
> It is required for navigation between the snapshots in record/replay mode.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---
>  block/qcow2-snapshot.c |    9 +++++++++
>  block/qcow2.h          |    2 ++
>  2 files changed, 11 insertions(+)

Missing a change to docs/interop/qcow2.txt.  You cannot add new features
into the qcow2 internal snapshot metadata without first documenting them
as part of the qcow2 spec.

> 
> diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
> index 74293be..4f2357e 100644
> --- a/block/qcow2-snapshot.c
> +++ b/block/qcow2-snapshot.c
> @@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs)
>              sn->disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
>          }
>  
> +        if (extra_data_size >= 24) {
> +            sn->icount = be64_to_cpu(extra.icount);

That's VERY dangerous without a spec change.  If someone else adds extra
data in some other format, then you will misinterpret their extra
information as your icount information.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC PATCH 08/17] replay: introduce info hmp/qmp command

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> This patch introduces 'info replay' monitor command and
> corresponding qmp request.
> These commands request the current record/replay mode, replay log file name,
> and the execution step (number or recorded/replayed instructions).
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---

Focusing on the QMP portion:


> +++ b/qapi/misc.json
> @@ -3449,3 +3449,39 @@
>  ##
>  { 'command': 'x-oob-test', 'data' : { 'lock': 'bool' },
>    'allow-oob': true }
> +
> +##
> +# @ReplayInfo:
> +#
> +# Status of the record/replay mode.
> +#
> +# @mode: current mode.
> +#
> +# @filename: name of the record/replay log file.
> +#
> +# @step: current step number.
> +#
> +# Since: 2.13
> +#
> +##
> +{ 'struct': 'ReplayInfo',
> +  'data': { 'mode': 'ReplayMode', '*filename': 'str', 'step': 'int' } }
> +
> +
> +##
> +# @info_replay:

Please call this 'query-replay', for consistency with other QMP commands
(your version looks too much like an HMP command).

> +#
> +# Retrieves the status of the execution record/replay.
> +#
> +# Returns: structure with the properties of the record/replay.
> +#
> +# Since: 2.13
> +#
> +# Example:
> +#
> +# -> { "execute": "info_replay" }
> +# <- { "return": { "mode": "play", "filename": "log.rr", "step": 220414 } }
> +#
> +##
> +{ 'command': 'info_replay',
> +  'returns': 'ReplayInfo' }

Please put these additions to misc.json be grouped closer to the
existing ReplayMode definition

> diff --git a/replay/Makefile.objs b/replay/Makefile.objs
> index cee6539..6694e3e 100644
> --- a/replay/Makefile.objs
> +++ b/replay/Makefile.objs
> @@ -6,4 +6,5 @@ common-obj-y += replay-input.o
>  common-obj-y += replay-char.o
>  common-obj-y += replay-snapshot.o
>  common-obj-y += replay-net.o
> -common-obj-y += replay-audio.o
> \ No newline at end of file

Eww, that's an existing trivial bug that we should fix (POSIX says that
Makefiles should be text files, and that text files should end in a
newline).  Just because GNU make handles a missing trailing newline on
makefile include fragments doesn't mean we should rely on it.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC PATCH 09/17] replay: introduce breakpoint at the specified step

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> This patch introduces replay_break qmp and hmp commands.
> These commands allow stopping at the specified instruction.
> It may be useful for debugging when there are some known
> events that should be investigated.
> The commands have one argument - number of instructions
> executed since the start of the replay.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---

> +++ b/qapi/misc.json
> @@ -3485,3 +3485,20 @@
>  ##
>  { 'command': 'info_replay',
>    'returns': 'ReplayInfo' }
> +
> +##
> +# @replay_break:

This should be 'replay-break' (QMP commands should favor '-' over '_').

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC PATCH 10/17] replay: implement replay_seek command to proceed to the desired step

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> This patch adds hmp/qmp command replay_seek which proceeds the execution
> to the specified step.
> It automatically loads nearest snapshot and replays the execution to find
> the desired step.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> ---

> +++ b/qapi/misc.json
> @@ -3502,3 +3502,19 @@
>  #
>  ##
>  { 'command': 'replay_break', 'data': { 'step': 'int' } }
> +
> +##
> +# @replay_seek:

This should be 'replay-seek'.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC PATCH 06/17] migration: introduce icount field for snapshots

[Pavel Dovgalyuk]

> From: Eric Blake [mailto:eblake@redhat.com]
> On 04/25/2018 07:46 AM, Pavel Dovgalyuk wrote:
> > Saving icount as a parameters of the snapshot allows navigation between
> > them in the execution replay scenario.
> > This information can be used for finding a specific snapshot for rewinding
> > the recorded execution to the specific moment of the time.
> > E.g., 'reverse step' action needs to load the nearest snapshot which is
> > prior to the current moment of time .
> >
> > Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> > ---
> >  block/qapi.c             |   11 +++++++----
> >  blockdev.c               |    3 +++
> >  include/block/snapshot.h |    1 +
> >  migration/savevm.c       |    1 +
> >  qapi/block-core.json     |    5 ++++-
> >  qapi/block.json          |    3 ++-
> >  6 files changed, 18 insertions(+), 6 deletions(-)
> 
> Using scripts/git.orderfile may make your patches easier to review, by
> hoisting interfaces above implementation.

Thank you. This script makes the patches look much better.


Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
<Pavel.Dovgaluk@ispras.ru> wrote:
> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
>
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
>

Hi Pavel,

1)

Can you provide more details on how to run the reverse debugging? In
particular how to take the checkpoint?

My test setup is described in detail at:
https://github.com/cirosantilli/qemu-test/tree/8127452e5685ed233dc7357a1fe34b7a2d173480
command "x86_64/reverse-debug".

Here are the actual commands:

#!/usr/bin/env bash
set -eu
dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
cmd="\
time \
./x86_64-softmmu/qemu-system-x86_64 \
-M pc \
-append 'root=/dev/sda console=ttyS0 nokaslr printk.time=y -
lkmc_eval=\"/rand_check.out;/sbin/ifup -a;wget -S
google.com;/poweroff.out;\"' \
-kernel '${dir}/out/x86_64/buildroot/images/bzImage' \
-nographic \
-serial mon:stdio \
-monitor telnet::45454,server,nowait \
\
-drive file='${dir}/out/x86_64/buildroot/images/rootfs.ext2.qcow2,if=none,id=img-direct,format=qcow2,snapshot'
\
-drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
-device ide-hd,drive=img-blkreplay \
\
-netdev user,id=net1 \
-device rtl8139,netdev=net1 \
-object filter-replay,id=replay,netdev=net1 \
"
cmd="${cmd} $@"
echo "$cmd"
eval "$cmd -icount 'shift=7,rr=record,rrfile=replay.bin'"
eval "$cmd -icount 'shift=7,rr=replay,rrfile=replay.bin' -S -s"

Then I take a snapshot right at the beginning of the execution:

telnet 45454
savevm a

And on another shell:

/data/git/linux-kernel-module-cheat/out/x86_64/buildroot/host/usr/bin/x86_64-linux-gdb
\
-q \
-ex 'file vmlinux' \
-ex 'target remote localhost:1234' \
-ex 'break start_kernel' \
-ex 'continue' \

But now if I try on GDB:

next
next
next
reverse-continue

hoping to go back to start_kernel, but nothing happens.

Same behavior if I take the snapshot after reaching start_kernel instead.

2)

I wonder if it would be possible to expose checkpoint taking through
GDB example via:
https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html

Or some other more convenient checkpoint generation method, e.g.
automatically take checkpoints every N instructions.

> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
>
> The set of patches include the following modifications:
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
>  - other record/replay fixes
>
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-180207
>
> ---
>
> Pavel Dovgalyuk (17):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       migration: introduce icount field for snapshots
>       qcow2: introduce icount field for snapshots
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay_seek command to proceed to the desired step
>       replay: flush events when exitting
>       timer: remove replay clock probe in deadline calculation
>       replay: refine replay-time module
>       translator: fix breakpoint processing
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>
>
>  accel/tcg/translator.c    |    8 +
>  block/blkreplay.c         |    8 +
>  block/io.c                |   22 +++
>  block/qapi.c              |   11 +-
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    2
>  blockdev.c                |    3
>  cpus.c                    |   19 ++-
>  docs/replay.txt           |   12 +-
>  exec.c                    |    6 +
>  gdbstub.c                 |   50 +++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   30 +++++
>  hmp.h                     |    3
>  include/block/snapshot.h  |    1
>  include/sysemu/replay.h   |   18 +++
>  migration/savevm.c        |   11 +-
>  qapi/block-core.json      |    5 +
>  qapi/block.json           |    3
>  qapi/misc.json            |   69 +++++++++++
>  replay/Makefile.objs      |    3
>  replay/replay-debugging.c |  286 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-events.c    |   14 --
>  replay/replay-internal.h  |   10 +-
>  replay/replay-time.c      |   27 ++--
>  replay/replay.c           |   22 +++
>  stubs/replay.c            |   10 ++
>  util/qemu-timer.c         |   11 --
>  vl.c                      |   11 +-
>  29 files changed, 625 insertions(+), 73 deletions(-)
>  create mode 100644 replay/replay-debugging.c
>
> --
> Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Pavel Dovgalyuk]

> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
> On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
> <Pavel.Dovgaluk@ispras.ru> wrote:
> > GDB remote protocol supports reverse debugging of the targets.
> > It includes 'reverse step' and 'reverse continue' operations.
> > The first one finds the previous step of the execution,
> > and the second one is intended to stop at the last breakpoint that
> > would happen when the program is executed normally.
> >
> > Reverse debugging is possible in the replay mode, when at least
> > one snapshot was created at the record or replay phase.
> > QEMU can use these snapshots for travelling back in time with GDB.
> >
> 
> Hi Pavel,
> 
> 1)
> 
> Can you provide more details on how to run the reverse debugging? In
> particular how to take the checkpoint?

There is some information in docs/replay.txt, but I guess, that I can give some more.

> 
> My test setup is described in detail at:
> https://github.com/cirosantilli/qemu-test/tree/8127452e5685ed233dc7357a1fe34b7a2d173480
> command "x86_64/reverse-debug".
> 
> Here are the actual commands:
> 
> #!/usr/bin/env bash
> set -eu
> dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
> cmd="\
> time \
> ./x86_64-softmmu/qemu-system-x86_64 \
> -M pc \
> -append 'root=/dev/sda console=ttyS0 nokaslr printk.time=y -
> lkmc_eval=\"/rand_check.out;/sbin/ifup -a;wget -S
> google.com;/poweroff.out;\"' \
> -kernel '${dir}/out/x86_64/buildroot/images/bzImage' \
> -nographic \
> -serial mon:stdio \
> -monitor telnet::45454,server,nowait \
> \
> -drive file='${dir}/out/x86_64/buildroot/images/rootfs.ext2.qcow2,if=none,id=img-
> direct,format=qcow2,snapshot'

The main thing for reverse debugging is snapshotting.
Therefore you should have an image that does not use temporary overlay file (snapshot option).
I'm using the following command line for record:

rm ./images/xp.ovl
# create overlay to avoid modifying the original image
./bin/qemu-img create -f qcow2 -b xp.qcow2 ./images/xp.ovl
./bin/qemu-system-i386 \
# This is workaround for XP. I wonder is it needed for the current version or not.
 -global apic-common.vapic=off \
# using newly created overlay instead of the original image
# rrsnapshot creates the snapshot at the start
 -icount shift=7,rr=record,rrfile=xp.replay,rrsnapshot=init -drive file=./images/xp.ovl,if=none,id=img-direct \
 -drive driver=blkreplay,if=none,image=img-direct,id=img-replay -device ide-hd,drive=img-replay -net none -m 256M -monitor stdio

While recording I can create some snapshots with savevm.
Command line for replaying differs only in "rr" option. rrsnapshot there loads the initial snapshot.
Any of the previously created snapshots may be specified.
You can also create new snapshots while replaying.


> \
> -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
> -device ide-hd,drive=img-blkreplay \
> \
> -netdev user,id=net1 \
> -device rtl8139,netdev=net1 \
> -object filter-replay,id=replay,netdev=net1 \
> "
> cmd="${cmd} $@"
> echo "$cmd"
> eval "$cmd -icount 'shift=7,rr=record,rrfile=replay.bin'"
> eval "$cmd -icount 'shift=7,rr=replay,rrfile=replay.bin' -S -s"
> 
> Then I take a snapshot right at the beginning of the execution:
> 
> telnet 45454
> savevm a
> 
> And on another shell:
> 
> /data/git/linux-kernel-module-cheat/out/x86_64/buildroot/host/usr/bin/x86_64-linux-gdb
> \
> -q \
> -ex 'file vmlinux' \
> -ex 'target remote localhost:1234' \
> -ex 'break start_kernel' \
> -ex 'continue' \
> 
> But now if I try on GDB:
> 
> next
> next
> next
> reverse-continue
> 
> hoping to go back to start_kernel, but nothing happens.

Yes, because you are missing your snapshot, that was actually created in the temporary overlay.

> Same behavior if I take the snapshot after reaching start_kernel instead.
> 
> 2)
> 
> I wonder if it would be possible to expose checkpoint taking through
> GDB example via:
> https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html

We'll check this out.

> Or some other more convenient checkpoint generation method, e.g.
> automatically take checkpoints every N instructions.

We implemented 'taking snapshots every N seconds', but I'll prefer to submit
it later, after approving the main idea.

> > Running the execution in replay mode allows using GDB reverse debugging
> > commands:
> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >    instruction forward. When that step is reaches, execution stops.
> >  - reverse-continue (or rc): Runs execution "backwards".
> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >    and replaying the execution. Then QEMU loads snapshots again and
> >    replays to the latest breakpoint. When there are no breakpoints in
> >    the examined section of the execution, QEMU finds one more snapshot
> >    and tries again. After the first snapshot is processed, execution
> >    stops at this snapshot.
> >
> > The set of patches include the following modifications:
> >  - gdbstub update for reverse debugging support
> >  - functions that automatically perform reverse step and reverse
> >    continue operations
> >  - hmp/qmp commands for manipulating the replay process
> >  - improvement of the snapshotting for saving the execution step
> >    in the snapshot parameters
> >  - other record/replay fixes
> >
> > The patches are available in the repository:
> > https://github.com/ispras/qemu/tree/rr-180207

Pavel Dovgalyuk

[Qemu-devel] Fwd: [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

Forgetting about debugging, I belive there is a deadlock in the replay
at 63d426dfa4fbfac3d50cda3f553cd975de2b85ea , but it is rare.

I have only reproduced it on ARM so far, and I haven't checked pre-patch.

The setup is https://github.com/cirosantilli/qemu-test/tree/6a3497f0d84e7c86ef80f7322e24e8a149b93214
with images-ab21ef58deed8536bc159c2afd680a4fabd68510.zip

Then try to run it several times with:

i=0; while true; do date; echo $i; ../qemu-test/arm/rr; i=$(($i+1)); done

I think the deadlock can happen in a few different places, but the
most common is when the kernel is doing disk related stuff, the last
messages before getting stuck are:

[   11.530325] ALSA device list:
[   11.531451]   No soundcards found.

and what would follow on a normal replay would be:

[   11.551904] EXT4-fs (vda): couldn't mount as ext3 due to feature
incompatibilities
[   11.619238] EXT4-fs (vda): mounted filesystem without journal. Opts: (null)

I then attach GDB with:

gdb -q ./arm-softmmu/qemu-system-arm `pgrep qemu`

and then:

>>> thread apply all bt

Thread 5 (Thread 0x7f59c6efb700 (LWP 22096)):
#0  0x00007f59e7aa9072 in futex_wait_cancelable (private=<optimized
out>, expected=0, futex_word=0x55a8e99801d8) at
../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  0x00007f59e7aa9072 in __pthread_cond_wait_common (abstime=0x0,
mutex=0x55a8e89cbf40 <qemu_global_mutex>, cond=0x55a8e99801b0) at
pthread_cond_wait.c:502
#2  0x00007f59e7aa9072 in __pthread_cond_wait (cond=0x55a8e99801b0,
mutex=0x55a8e89cbf40 <qemu_global_mutex>) at pthread_cond_wait.c:655
#3  0x000055a8e7f4f178 in qemu_cond_wait_impl (cond=0x55a8e99801b0,
mutex=0x55a8e89cbf40 <qemu_global_mutex>, file=0x55a8e80b10a8
"/home/ciro/git/qemu/cpus.c", line=1175) at
util/qemu-thread-posix.c:164
#4  0x000055a8e7999965 in qemu_tcg_rr_wait_io_event
(cpu=0x55a8e986b330) at /home/ciro/git/qemu/cpus.c:1175
#5  0x000055a8e799a1f5 in qemu_tcg_rr_cpu_thread_fn
(arg=0x55a8e986b330) at /home/ciro/git/qemu/cpus.c:1502
#6  0x00007f59e7aa27fc in start_thread (arg=0x7f59c6efb700) at
pthread_create.c:465
#7  0x00007f59e77cfb5f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f59c76fc700 (LWP 22095)):
#0  0x00007f59e77c3a4b in __GI_ppoll (fds=0x7f59b8000b10, nfds=1,
timeout=<optimized out>, sigmask=0x0) at
../sysdeps/unix/sysv/linux/ppoll.c:39
#1  0x000055a8e7f4a02e in qemu_poll_ns (fds=0x7f59b8000b10, nfds=1,
timeout=-1) at util/qemu-timer.c:322
#2  0x000055a8e7f4cb5e in aio_poll (ctx=0x55a8e978eab0, blocking=true)
at util/aio-posix.c:629
#3  0x000055a8e7b5f084 in iothread_run (opaque=0x55a8e970c710) at
iothread.c:64
#4  0x00007f59e7aa27fc in start_thread (arg=0x7f59c76fc700) at
pthread_create.c:465
#5  0x00007f59e77cfb5f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f59ced65700 (LWP 22093)):
#0  0x00007f59e77c9a49 in syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f59e88456ef in g_cond_wait () at
/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x000055a8e7f43157 in wait_for_trace_records_available () at
trace/simple.c:150
#3  0x000055a8e7f431b8 in writeout_thread (opaque=0x0) at
trace/simple.c:169
#4  0x00007f59e8827645 in  () at
/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f59e7aa27fc in start_thread (arg=0x7f59ced65700) at
pthread_create.c:465
#6  0x00007f59e77cfb5f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f59cf566700 (LWP 22092)):
#0  0x00007f59e77c9a49 in syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055a8e7f4f5d8 in qemu_futex_wait (f=0x55a8e8e48418
<rcu_call_ready_event>, val=4294967295) at
/home/ciro/git/qemu/include/qemu/futex.h:29
#2  0x000055a8e7f4f79f in qemu_event_wait (ev=0x55a8e8e48418
<rcu_call_ready_event>) at util/qemu-thread-posix.c:445
#3  0x000055a8e7f67d2d in call_rcu_thread (opaque=0x0) at
util/rcu.c:261
#4  0x00007f59e7aa27fc in start_thread (arg=0x7f59cf566700) at
pthread_create.c:465
#5  0x00007f59e77cfb5f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f59ecf03280 (LWP 22091)):
#0  0x00007f59e77c3a4b in __GI_ppoll (fds=0x55a8e9860aa0, nfds=5,
timeout=<optimized out>, sigmask=0x0) at
../sysdeps/unix/sysv/linux/ppoll.c:39
#1  0x000055a8e7f4a0c4 in qemu_poll_ns (fds=0x55a8e9860aa0, nfds=5,
timeout=1000000000) at util/qemu-timer.c:334
#2  0x000055a8e7f4b176 in os_host_main_loop_wait (timeout=1000000000)
at util/main-loop.c:258
#3  0x000055a8e7f4b241 in main_loop_wait (nonblocking=0) at
util/main-loop.c:522
#4  0x000055a8e7b66fed in main_loop () at vl.c:1943
#5  0x000055a8e7b6ead4 in main (argc=24, argv=0x7fff6fe0f328,
envp=0x7fff6fe0f3f0) at vl.c:4740

On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
<Pavel.Dovgaluk@ispras.ru> wrote:
> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
>
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
>
> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
>
> The set of patches include the following modifications:
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
>  - other record/replay fixes
>
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-180207
>
> ---
>
> Pavel Dovgalyuk (17):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       migration: introduce icount field for snapshots
>       qcow2: introduce icount field for snapshots
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay_seek command to proceed to the desired step
>       replay: flush events when exitting
>       timer: remove replay clock probe in deadline calculation
>       replay: refine replay-time module
>       translator: fix breakpoint processing
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>
>
>  accel/tcg/translator.c    |    8 +
>  block/blkreplay.c         |    8 +
>  block/io.c                |   22 +++
>  block/qapi.c              |   11 +-
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    2
>  blockdev.c                |    3
>  cpus.c                    |   19 ++-
>  docs/replay.txt           |   12 +-
>  exec.c                    |    6 +
>  gdbstub.c                 |   50 +++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   30 +++++
>  hmp.h                     |    3
>  include/block/snapshot.h  |    1
>  include/sysemu/replay.h   |   18 +++
>  migration/savevm.c        |   11 +-
>  qapi/block-core.json      |    5 +
>  qapi/block.json           |    3
>  qapi/misc.json            |   69 +++++++++++
>  replay/Makefile.objs      |    3
>  replay/replay-debugging.c |  286 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-events.c    |   14 --
>  replay/replay-internal.h  |   10 +-
>  replay/replay-time.c      |   27 ++--
>  replay/replay.c           |   22 +++
>  stubs/replay.c            |   10 ++
>  util/qemu-timer.c         |   11 --
>  vl.c                      |   11 +-
>  29 files changed, 625 insertions(+), 73 deletions(-)
>  create mode 100644 replay/replay-debugging.c
>
> --
> Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

On Sat, Apr 28, 2018 at 9:12 AM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:
>> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
>> On Thu, Apr 26, 2018 at 1:34 PM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:
>> >> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
>> >> On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
>> >> <Pavel.Dovgaluk@ispras.ru> wrote:
>> >> > GDB remote protocol supports reverse debugging of the targets.
>> >> > It includes 'reverse step' and 'reverse continue' operations.
>> >> > The first one finds the previous step of the execution,
>> >> > and the second one is intended to stop at the last breakpoint that
>> >> > would happen when the program is executed normally.
>> >> >
>> >> > Reverse debugging is possible in the replay mode, when at least
>> >> > one snapshot was created at the record or replay phase.
>> >> > QEMU can use these snapshots for travelling back in time with GDB.
>> >> >
>> >>
>> >> Hi Pavel,
>> >>
>> >> 1)
>> >>
>> >> Can you provide more details on how to run the reverse debugging? In
>> >> particular how to take the checkpoint?
>> >
>> > There is some information in docs/replay.txt, but I guess, that I can give some more.
>> >
>> >>
>> >> My test setup is described in detail at:
>> >> https://github.com/cirosantilli/qemu-test/tree/8127452e5685ed233dc7357a1fe34b7a2d173480
>> >> command "x86_64/reverse-debug".
>> >>
>> >> Here are the actual commands:
>> >>
>> >> #!/usr/bin/env bash
>> >> set -eu
>> >> dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
>> >> cmd="\
>> >> time \
>> >> ./x86_64-softmmu/qemu-system-x86_64 \
>> >> -M pc \
>> >> -append 'root=/dev/sda console=ttyS0 nokaslr printk.time=y -
>> >> lkmc_eval=\"/rand_check.out;/sbin/ifup -a;wget -S
>> >> google.com;/poweroff.out;\"' \
>> >> -kernel '${dir}/out/x86_64/buildroot/images/bzImage' \
>> >> -nographic \
>> >> -serial mon:stdio \
>> >> -monitor telnet::45454,server,nowait \
>> >> \
>> >> -drive file='${dir}/out/x86_64/buildroot/images/rootfs.ext2.qcow2,if=none,id=img-
>> >> direct,format=qcow2,snapshot'
>> >
>> > The main thing for reverse debugging is snapshotting.
>> > Therefore you should have an image that does not use temporary overlay file (snapshot
>> option).
>> > I'm using the following command line for record:
>> >
>> > rm ./images/xp.ovl
>> > # create overlay to avoid modifying the original image
>> > ./bin/qemu-img create -f qcow2 -b xp.qcow2 ./images/xp.ovl
>> > ./bin/qemu-system-i386 \
>> > # This is workaround for XP. I wonder is it needed for the current version or not.
>> >  -global apic-common.vapic=off \
>> > # using newly created overlay instead of the original image
>> > # rrsnapshot creates the snapshot at the start
>> >  -icount shift=7,rr=record,rrfile=xp.replay,rrsnapshot=init -drive
>> file=./images/xp.ovl,if=none,id=img-direct \
>> >  -drive driver=blkreplay,if=none,image=img-direct,id=img-replay -device ide-hd,drive=img-
>> replay -net none -m 256M -monitor stdio
>> >
>> > While recording I can create some snapshots with savevm.
>> > Command line for replaying differs only in "rr" option. rrsnapshot there loads the initial
>> snapshot.
>> > Any of the previously created snapshots may be specified.
>> > You can also create new snapshots while replaying.
>> >
>>
>> How is the snapshot to be used chosen? Does this patch make it try to
>> smartly use the snapshot that is closest to the target break?
>
> Yes, it selects the closest snapshot.
>
>> Does rrsnapshot select which snapshot will be used, or just creates a
>> snapshot at the start or record?
>
> rrsnapshot creates a snapshot at record and loads it at start.
> It is required, because the disk image is modified by the execution,
> when 'snapshot' option is omitted.
>
>> I have modified my commands to remove snapshot from -drive, and add
>> rrsnapshot=init to -icount and the following works:
>>
>> b start_kernel
>> n
>> n
>> n
>> b
>> n
>> n
>> rc
>
> Great!
>
>> However, if after the "b start_kernel" I make a new snapshot on telnet
>> with "savevm a" to try and make the restore faster, then
>> reverse-continue fails.
>
> That's strange. What did it say?
>

Nothing, it just stayed on the same line.

>> Also, if I do "loadvm a" after "savevm a" while the debugger is
>> attached at start_kernel, the monitor just hangs. Is there a way to
>> restore snapshots while debugging of replay is going on?
>
> Never tried to do this.
> I'll check this out.
>
>
> Pavel Dovgalyuk
>

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

On Sat, Apr 28, 2018 at 10:27 AM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:
>
>
>> -----Original Message-----
>> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
>> Sent: Saturday, April 28, 2018 11:13 AM
>> To: Pavel Dovgalyuk
>> Subject: Re: [RFC PATCH 00/17] reverse debugging
>>
>> Forgetting about debugging, I belive there is a deadlock in the replay
>> at 63d426dfa4fbfac3d50cda3f553cd975de2b85ea , but it is rare.
>>
>> I have only reproduced it on ARM so far, and I haven't checked pre-patch.
>>
>> The setup is https://github.com/cirosantilli/qemu-
>> test/tree/6a3497f0d84e7c86ef80f7322e24e8a149b93214
>> with images-ab21ef58deed8536bc159c2afd680a4fabd68510.zip
>>
>> Then try to run it several times with:
>>
>> i=0; while true; do date; echo $i; ../qemu-test/arm/rr; i=$(($i+1)); done
>>
>> I think the deadlock can happen in a few different places, but the
>> most common is when the kernel is doing disk related stuff, the last
>> messages before getting stuck are:
>
> It usually happens when there is some bugs in the implementation of the virtual devices.
> Our customers mostly emulates x86-based systems, therefore most of
> the ARM hardware is untested.
>

Hi Pete, do you know anything about this? Traces at:
http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg05218.html
command at: https://github.com/cirosantilli/qemu-test/blob/6a3497f0d84e7c86ef80f7322e24e8a149b93214/arm/rr

@Pavel: I recommend always replying to both me and to qemu-devel to
preserve a better history of our talk on the tracker.

>> [   11.530325] ALSA device list:
>> [   11.531451]   No soundcards found.
>>
>> and what would follow on a normal replay would be:
>>
>> [   11.551904] EXT4-fs (vda): couldn't mount as ext3 due to feature
>> incompatibilities
>> [   11.619238] EXT4-fs (vda): mounted filesystem without journal. Opts: (null)
>>
>> I then attach GDB with:
>>
>> gdb -q ./arm-softmmu/qemu-system-arm `pgrep qemu`
>
>
>
>
> Pavel Dovgalyuk
>

Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

On Thu, Apr 26, 2018 at 1:34 PM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:

> > From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
> > On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
> > <Pavel.Dovgaluk@ispras.ru> wrote:
> > > GDB remote protocol supports reverse debugging of the targets.
> > > It includes 'reverse step' and 'reverse continue' operations.
> > > The first one finds the previous step of the execution,
> > > and the second one is intended to stop at the last breakpoint that
> > > would happen when the program is executed normally.
> > >
> > > Reverse debugging is possible in the replay mode, when at least
> > > one snapshot was created at the record or replay phase.
> > > QEMU can use these snapshots for travelling back in time with GDB.
> > >
> >
> > Hi Pavel,
> >
> > 1)
> >
> > Can you provide more details on how to run the reverse debugging? In
> > particular how to take the checkpoint?
>
> There is some information in docs/replay.txt, but I guess, that I can give
> some more.
>
> >
> > My test setup is described in detail at:
> > https://github.com/cirosantilli/qemu-test/tree/8127452e5685e
> d233dc7357a1fe34b7a2d173480
> > command "x86_64/reverse-debug".
> >
> > Here are the actual commands:
> >
> > #!/usr/bin/env bash
> > set -eu
> > dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
> > cmd="\
> > time \
> > ./x86_64-softmmu/qemu-system-x86_64 \
> > -M pc \
> > -append 'root=/dev/sda console=ttyS0 nokaslr printk.time=y -
> > lkmc_eval=\"/rand_check.out;/sbin/ifup -a;wget -S
> > google.com;/poweroff.out;\"' \
> > -kernel '${dir}/out/x86_64/buildroot/images/bzImage' \
> > -nographic \
> > -serial mon:stdio \
> > -monitor telnet::45454,server,nowait \
> > \
> > -drive file='${dir}/out/x86_64/buildroot/images/rootfs.ext2.qcow2,
> if=none,id=img-
> > direct,format=qcow2,snapshot'
>
> The main thing for reverse debugging is snapshotting.
> Therefore you should have an image that does not use temporary overlay
> file (snapshot option).
> I'm using the following command line for record:
>
> rm ./images/xp.ovl
> # create overlay to avoid modifying the original image
> ./bin/qemu-img create -f qcow2 -b xp.qcow2 ./images/xp.ovl
> ./bin/qemu-system-i386 \
> # This is workaround for XP. I wonder is it needed for the current version
> or not.
>  -global apic-common.vapic=off \
> # using newly created overlay instead of the original image
> # rrsnapshot creates the snapshot at the start
>  -icount shift=7,rr=record,rrfile=xp.replay,rrsnapshot=init -drive
> file=./images/xp.ovl,if=none,id=img-direct \
>  -drive driver=blkreplay,if=none,image=img-direct,id=img-replay -device
> ide-hd,drive=img-replay -net none -m 256M -monitor stdio
>
> While recording I can create some snapshots with savevm.
> Command line for replaying differs only in "rr" option. rrsnapshot there
> loads the initial snapshot.
> Any of the previously created snapshots may be specified.
> You can also create new snapshots while replaying.
>
>
> > \
> > -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
> > -device ide-hd,drive=img-blkreplay \
> > \
> > -netdev user,id=net1 \
> > -device rtl8139,netdev=net1 \
> > -object filter-replay,id=replay,netdev=net1 \
> > "
> > cmd="${cmd} $@"
> > echo "$cmd"
> > eval "$cmd -icount 'shift=7,rr=record,rrfile=replay.bin'"
> > eval "$cmd -icount 'shift=7,rr=replay,rrfile=replay.bin' -S -s"
> >
> > Then I take a snapshot right at the beginning of the execution:
> >
> > telnet 45454
> > savevm a
> >
> > And on another shell:
> >
> > /data/git/linux-kernel-module-cheat/out/x86_64/buildroot/hos
> t/usr/bin/x86_64-linux-gdb
> > \
> > -q \
> > -ex 'file vmlinux' \
> > -ex 'target remote localhost:1234' \
> > -ex 'break start_kernel' \
> > -ex 'continue' \
> >
> > But now if I try on GDB:
> >
> > next
> > next
> > next
> > reverse-continue
> >
> > hoping to go back to start_kernel, but nothing happens.
>
> Yes, because you are missing your snapshot, that was actually created in
> the temporary overlay.
>
> > Same behavior if I take the snapshot after reaching start_kernel instead.
> >
> > 2)
> >
> > I wonder if it would be possible to expose checkpoint taking through
> > GDB example via:
> > https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html
>
> We'll check this out.
>

Actually, this is not needed, I have learnt now that you can send QEMU
monitor commands with `monitor <qemu-monitor-command>`, so e.g. `monitor
savevm a`.


>
> > Or some other more convenient checkpoint generation method, e.g.
> > automatically take checkpoints every N instructions.
>
> We implemented 'taking snapshots every N seconds', but I'll prefer to
> submit
> it later, after approving the main idea.
>
> > > Running the execution in replay mode allows using GDB reverse debugging
> > > commands:
> > >  - reverse-stepi (or rsi): Steps one instruction to the past.
> > >    QEMU loads on of the prior snapshots and proceeds to the desired
> > >    instruction forward. When that step is reaches, execution stops.
> > >  - reverse-continue (or rc): Runs execution "backwards".
> > >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> > >    and replaying the execution. Then QEMU loads snapshots again and
> > >    replays to the latest breakpoint. When there are no breakpoints in
> > >    the examined section of the execution, QEMU finds one more snapshot
> > >    and tries again. After the first snapshot is processed, execution
> > >    stops at this snapshot.
> > >
> > > The set of patches include the following modifications:
> > >  - gdbstub update for reverse debugging support
> > >  - functions that automatically perform reverse step and reverse
> > >    continue operations
> > >  - hmp/qmp commands for manipulating the replay process
> > >  - improvement of the snapshotting for saving the execution step
> > >    in the snapshot parameters
> > >  - other record/replay fixes
> > >
> > > The patches are available in the repository:
> > > https://github.com/ispras/qemu/tree/rr-180207
>
> Pavel Dovgalyuk
>
>