[Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Pavel Dovgalyuk]

GDB remote protocol supports reverse debugging of the targets.
It includes 'reverse step' and 'reverse continue' operations.
The first one finds the previous step of the execution,
and the second one is intended to stop at the last breakpoint that
would happen when the program is executed normally.

Reverse debugging is possible in the replay mode, when at least
one snapshot was created at the record or replay phase.
QEMU can use these snapshots for travelling back in time with GDB.

Running the execution in replay mode allows using GDB reverse debugging
commands:
 - reverse-stepi (or rsi): Steps one instruction to the past.
   QEMU loads on of the prior snapshots and proceeds to the desired
   instruction forward. When that step is reaches, execution stops.
 - reverse-continue (or rc): Runs execution "backwards".
   QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
   and replaying the execution. Then QEMU loads snapshots again and
   replays to the latest breakpoint. When there are no breakpoints in
   the examined section of the execution, QEMU finds one more snapshot
   and tries again. After the first snapshot is processed, execution
   stops at this snapshot.

The set of patches include the following modifications:
 - fixes of record/replay caused by the QEMU core changes
 - gdbstub update for reverse debugging support
 - functions that automatically perform reverse step and reverse
   continue operations
 - hmp/qmp commands for manipulating the replay process
 - improvement of the snapshotting for saving the execution step
   in the snapshot parameters
 - adding new clock for correct timer events from vnc and slirp
 - other record/replay fixes

The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180911

v6 changes:
 - rebased to the new version of master
 - fixed build of linux-user configurations
 - added new clock for slirp and vnc timers

v5 changes:
 - multiple fixes of record/replay bugs appeared after QEMU core update
 - changed reverse debugging to 'since 3.1'

v4 changes:
 - changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)

v3 changes:
 - Fixed PS/2 bug with save/load vm, which caused failures of the replay.
 - Rebased to the new code base.
 - Minor fixes.

v2 changes:
 - documented reverse debugging
 - fixed start vmstate loading in record mode
 - documented qcow2 changes (as suggested by Eric Blake)
 - made icount SnapshotInfo field optional (as suggested by Eric Blake)
 - renamed qmp commands (as suggested by Eric Blake)
 - minor changes

---

Pavel Dovgalyuk (25):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      qcow2: introduce icount field for snapshots
      migration: introduce icount field for snapshots
      replay: provide and accessor for rr filename
      replay: introduce info hmp/qmp command
      replay: introduce breakpoint at the specified step
      replay: implement replay-seek command to proceed to the desired step
      replay: flush events when exiting
      replay: refine replay-time module
      translator: fix breakpoint processing
      replay: flush rr queue before loading the vmstate
      gdbstub: add reverse step support in replay mode
      gdbstub: add reverse continue support in replay mode
      replay: describe reverse debugging in docs/replay.txt
      replay: allow loading any snapshots before recording
      replay: wake up vCPU when replaying
      replay: replay BH for IDE trim operation
      replay: add BH oneshot event for block layer
      timer: introduce new virtual clock
      slirp: fix ipv6 timers
      ui: fix virtual timers


 accel/tcg/translator.c    |    9 +
 block/blkreplay.c         |    8 +
 block/block-backend.c     |    3 
 block/io.c                |   22 +++
 block/qapi.c              |   17 ++-
 block/qcow2-snapshot.c    |    9 +
 block/qcow2.h             |    2 
 blockdev.c                |   10 ++
 cpus.c                    |   50 +++++---
 docs/interop/qcow2.txt    |    4 +
 docs/replay.txt           |   45 +++++++
 exec.c                    |    6 +
 gdbstub.c                 |   50 +++++++-
 hmp-commands-info.hx      |   14 ++
 hmp-commands.hx           |   30 +++++
 hmp.h                     |    3 
 hw/ide/core.c             |    3 
 include/block/snapshot.h  |    1 
 include/qemu/timer.h      |    9 +
 include/sysemu/replay.h   |   26 ++++
 migration/savevm.c        |   15 +-
 qapi/block-core.json      |    5 +
 qapi/block.json           |    3 
 qapi/misc.json            |   68 +++++++++++
 replay/Makefile.objs      |    3 
 replay/replay-debugging.c |  287 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c    |   30 +++--
 replay/replay-internal.h  |    9 +
 replay/replay-snapshot.c  |   17 ++-
 replay/replay-time.c      |   32 ++---
 replay/replay.c           |   38 ++++++
 slirp/ip6_icmp.c          |    7 +
 stubs/Makefile.objs       |    1 
 stubs/replay-user.c       |    9 +
 stubs/replay.c            |   10 ++
 ui/input.c                |    8 +
 util/qemu-timer.c         |    2 
 vl.c                      |   18 ++-
 38 files changed, 791 insertions(+), 92 deletions(-)
 create mode 100644 replay/replay-debugging.c
 create mode 100644 stubs/replay-user.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [PATCH v6 01/25] block: implement bdrv_snapshot_goto for blkreplay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch enables making snapshots with blkreplay used in
block devices.
This function is required to make bdrv_snapshot_goto without
calling .bdrv_open which is not implemented.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/blkreplay.c |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/block/blkreplay.c b/block/blkreplay.c
index b5d9efd..142dfe3 100755
--- a/block/blkreplay.c
+++ b/block/blkreplay.c
@@ -126,6 +126,12 @@ static int coroutine_fn blkreplay_co_flush(BlockDriverState *bs)
     return ret;
 }
 
+static int blkreplay_snapshot_goto(BlockDriverState *bs,
+                                   const char *snapshot_id)
+{
+    return bdrv_snapshot_goto(bs->file->bs, snapshot_id, NULL);
+}
+
 static BlockDriver bdrv_blkreplay = {
     .format_name            = "blkreplay",
     .instance_size          = 0,
@@ -140,6 +146,8 @@ static BlockDriver bdrv_blkreplay = {
     .bdrv_co_pwrite_zeroes  = blkreplay_co_pwrite_zeroes,
     .bdrv_co_pdiscard       = blkreplay_co_pdiscard,
     .bdrv_co_flush          = blkreplay_co_flush,
+
+    .bdrv_snapshot_goto     = blkreplay_snapshot_goto,
 };
 
 static void bdrv_blkreplay_init(void)

[Qemu-devel] [PATCH v6 02/25] replay: disable default snapshot for record/replay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

This patch disables setting '-snapshot' option on by default
in record/replay mode. This is needed for creating vmstates in record
and replay modes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 vl.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/vl.c b/vl.c
index 5ba06ad..c32de28 100644
--- a/vl.c
+++ b/vl.c
@@ -3112,7 +3112,13 @@ int main(int argc, char **argv, char **envp)
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
                 break;
             case QEMU_OPTION_snapshot:
-                snapshot = 1;
+                {
+                    Error *blocker = NULL;
+                    snapshot = 1;
+                    error_setg(&blocker, QERR_REPLAY_NOT_SUPPORTED,
+                               "-snapshot");
+                    replay_add_blocker(blocker);
+                }
                 break;
             case QEMU_OPTION_numa:
                 opts = qemu_opts_parse_noisily(qemu_find_opts("numa"),
@@ -4447,7 +4453,7 @@ int main(int argc, char **argv, char **envp)
         qapi_free_BlockdevOptions(bdo->bdo);
         g_free(bdo);
     }
-    if (snapshot || replay_mode != REPLAY_MODE_NONE) {
+    if (snapshot) {
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot,
                           NULL, NULL);
     }

[Qemu-devel] [PATCH v6 03/25] replay: update docs for record/replay with block devices

[Pavel Dovgalyuk]

This patch updates the description of the command lines for using
record/replay with attached block devices.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 docs/replay.txt |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/docs/replay.txt b/docs/replay.txt
index 2e21e9c..f7def53 100644
--- a/docs/replay.txt
+++ b/docs/replay.txt
@@ -27,7 +27,7 @@ Usage of the record/replay:
  * First, record the execution with the following command line:
     qemu-system-i386 \
      -icount shift=7,rr=record,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -35,7 +35,7 @@ Usage of the record/replay:
  * After recording, you can replay it by using another command line:
     qemu-system-i386 \
      -icount shift=7,rr=replay,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,id=img-direct \
+     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
      -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
      -device ide-hd,drive=img-blkreplay \
      -netdev user,id=net1 -device rtl8139,netdev=net1 \
@@ -223,7 +223,7 @@ Block devices record/replay module intercepts calls of
 bdrv coroutine functions at the top of block drivers stack.
 To record and replay block operations the drive must be configured
 as following:
- -drive file=disk.qcow2,if=none,id=img-direct
+ -drive file=disk.qcow2,if=none,snapshot,id=img-direct
  -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
  -device ide-hd,drive=img-blkreplay
 
@@ -252,6 +252,12 @@ This snapshot is created at start of recording and restored at start
 of replaying. It also can be loaded while replaying to roll back
 the execution.
 
+'snapshot' flag of the disk image must be removed to save the snapshots
+in the overlay (or original image) instead of using the temporary overlay.
+ -drive file=disk.ovl,if=none,id=img-direct
+ -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+ -device ide-hd,drive=img-blkreplay
+
 Use QEMU monitor to create additional snapshots. 'savevm <name>' command
 created the snapshot and 'loadvm <name>' restores it. To prevent corruption
 of the original disk image, use overlay files linked to the original images.

[Qemu-devel] [PATCH v6 04/25] replay: don't drain/flush bdrv queue while RR is working

[Pavel Dovgalyuk]

In record/replay mode bdrv queue is controlled by replay mechanism.
It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty
queue, but flushing the queue is still impossible there,
because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in
record/replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/io.c |   22 ++++++++++++++++++++++
 cpus.c     |    2 --
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/block/io.c b/block/io.c
index 7100344..4d7a6e4 100644
--- a/block/io.c
+++ b/block/io.c
@@ -32,6 +32,7 @@
 #include "qemu/cutils.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
+#include "sysemu/replay.h"
 
 #define NOT_DONE 0x7fffffff /* used while emulated sync operation in progress */
 
@@ -533,6 +534,13 @@ void bdrv_drain_all_begin(void)
         return;
     }
 
+    /* bdrv queue is managed by record/replay,
+       waiting for finishing the I/O requests may
+       be infinite */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     /* AIO_WAIT_WHILE() with a NULL context can only be called from the main
      * loop AioContext, so make sure we're in the main context. */
     assert(qemu_get_current_aio_context() == qemu_get_aio_context());
@@ -561,6 +569,13 @@ void bdrv_drain_all_end(void)
 {
     BlockDriverState *bs = NULL;
 
+    /* bdrv queue is managed by record/replay,
+       waiting for finishing the I/O requests may
+       be endless */
+    if (replay_events_enabled()) {
+        return;
+    }
+
     while ((bs = bdrv_next_all_states(bs))) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
 
@@ -1993,6 +2008,13 @@ int bdrv_flush_all(void)
     BlockDriverState *bs = NULL;
     int result = 0;
 
+    /* bdrv queue is managed by record/replay,
+       creating new flush request for stopping
+       the VM may break the determinism */
+    if (replay_events_enabled()) {
+        return result;
+    }
+
     for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
         AioContext *aio_context = bdrv_get_aio_context(bs);
         int ret;
diff --git a/cpus.c b/cpus.c
index f257a6e..0ce5640 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1058,7 +1058,6 @@ static int do_vm_stop(RunState state, bool send_stop)
     }
 
     bdrv_drain_all();
-    replay_disable_events();
     ret = bdrv_flush_all();
 
     return ret;
@@ -2115,7 +2114,6 @@ int vm_prepare_start(void)
     /* We are sending this now, but the CPUs will be resumed shortly later */
     qapi_event_send_resume(&error_abort);
 
-    replay_enable_events();
     cpu_enable_ticks();
     runstate_set(RUN_STATE_RUNNING);
     vm_state_notify(1, RUN_STATE_RUNNING);

[Qemu-devel] [PATCH v6 05/25] replay: finish record/replay before closing the disks

[Pavel Dovgalyuk]

After recent updates block devices cannot be closed on qemu exit.
This happens due to the block request polling when replay is not finished.
Therefore now we stop execution recording before closing the block devices.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay.c |    2 ++
 vl.c            |    1 +
 2 files changed, 3 insertions(+)

diff --git a/replay/replay.c b/replay/replay.c
index 8228261..58a986f 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -366,6 +366,8 @@ void replay_finish(void)
     g_free(replay_snapshot);
     replay_snapshot = NULL;
 
+    replay_mode = REPLAY_MODE_NONE;
+
     replay_finish_events();
 }
 
diff --git a/vl.c b/vl.c
index c32de28..3ac4d23 100644
--- a/vl.c
+++ b/vl.c
@@ -4656,6 +4656,7 @@ int main(int argc, char **argv, char **envp)
 
     /* No more vcpu or device emulation activity beyond this point */
     vm_shutdown();
+    replay_finish();
 
     job_cancel_sync_all();
     bdrv_close_all();

[Qemu-devel] [PATCH v6 06/25] qcow2: introduce icount field for snapshots

[Pavel Dovgalyuk]

This patch introduces the icount field for saving within the snapshot.
It is required for navigation between the snapshots in record/replay mode.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - documented format changes in docs/interop/qcow2.txt
   (suggested by Eric Blake)
---
 block/qcow2-snapshot.c |    7 +++++++
 block/qcow2.h          |    2 ++
 docs/interop/qcow2.txt |    4 ++++
 3 files changed, 13 insertions(+)

diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index bb6a5b7..d682946 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -103,6 +103,12 @@ int qcow2_read_snapshots(BlockDriverState *bs)
             sn->disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
         }
 
+        if (extra_data_size >= 24) {
+            sn->icount = be64_to_cpu(extra.icount);
+        } else {
+            sn->icount = -1ULL;
+        }
+
         /* Read snapshot ID */
         sn->id_str = g_malloc(id_str_size + 1);
         ret = bdrv_pread(bs->file, offset, sn->id_str, id_str_size);
@@ -209,6 +215,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)
         memset(&extra, 0, sizeof(extra));
         extra.vm_state_size_large = cpu_to_be64(sn->vm_state_size);
         extra.disk_size = cpu_to_be64(sn->disk_size);
+        extra.icount = cpu_to_be64(sn->icount);
 
         id_str_size = strlen(sn->id_str);
         name_size = strlen(sn->name);
diff --git a/block/qcow2.h b/block/qcow2.h
index 81b844e..b24ec93 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -148,6 +148,7 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {
 typedef struct QEMU_PACKED QCowSnapshotExtraData {
     uint64_t vm_state_size_large;
     uint64_t disk_size;
+    uint64_t icount;
 } QCowSnapshotExtraData;
 
 
@@ -161,6 +162,7 @@ typedef struct QCowSnapshot {
     uint32_t date_sec;
     uint32_t date_nsec;
     uint64_t vm_clock_nsec;
+    uint64_t icount;
 } QCowSnapshot;
 
 struct Qcow2Cache;
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 845d40a..aa1101d 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -506,6 +506,10 @@ Snapshot table entry:
 
                     Byte 48 - 55:   Virtual disk size of the snapshot in bytes
 
+                    Byte 56 - 63:   icount value which corresponds to
+                                    the record/replay step when the snapshot
+                                    was taken
+
                     Version 3 images must include extra data at least up to
                     byte 55.
 

[Qemu-devel] [PATCH v6 07/25] migration: introduce icount field for snapshots

[Pavel Dovgalyuk]

Saving icount as a parameters of the snapshot allows navigation between
them in the execution replay scenario.
This information can be used for finding a specific snapshot for rewinding
the recorded execution to the specific moment of the time.
E.g., 'reverse step' action needs to load the nearest snapshot which is
prior to the current moment of time .

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - made icount in SnapshotInfo optional (suggested by Eric Blake)
---
 block/qapi.c             |   17 +++++++++++++----
 block/qcow2-snapshot.c   |    2 ++
 blockdev.c               |   10 ++++++++++
 include/block/snapshot.h |    1 +
 migration/savevm.c       |    5 +++++
 qapi/block-core.json     |    5 ++++-
 qapi/block.json          |    3 ++-
 7 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/block/qapi.c b/block/qapi.c
index c66f949..9f2ccfa 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -210,6 +210,7 @@ int bdrv_query_snapshot_info_list(BlockDriverState *bs,
         info->date_nsec     = sn_tab[i].date_nsec;
         info->vm_clock_sec  = sn_tab[i].vm_clock_nsec / 1000000000;
         info->vm_clock_nsec = sn_tab[i].vm_clock_nsec % 1000000000;
+        info->icount        = sn_tab[i].icount;
 
         info_list = g_new0(SnapshotInfoList, 1);
         info_list->value = info;
@@ -663,14 +664,15 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f,
                         QEMUSnapshotInfo *sn)
 {
     char buf1[128], date_buf[128], clock_buf[128];
+    char icount_buf[128] = {0};
     struct tm tm;
     time_t ti;
     int64_t secs;
 
     if (!sn) {
         func_fprintf(f,
-                     "%-10s%-20s%7s%20s%15s",
-                     "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK");
+                     "%-10s%-18s%7s%20s%13s%11s",
+                     "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK", "ICOUNT");
     } else {
         ti = sn->date_sec;
         localtime_r(&ti, &tm);
@@ -683,13 +685,18 @@ void bdrv_snapshot_dump(fprintf_function func_fprintf, void *f,
                  (int)((secs / 60) % 60),
                  (int)(secs % 60),
                  (int)((sn->vm_clock_nsec / 1000000) % 1000));
+        if (sn->icount != -1ULL) {
+            snprintf(icount_buf, sizeof(icount_buf),
+                "%"PRId64, sn->icount);
+        }
         func_fprintf(f,
-                     "%-10s%-20s%7s%20s%15s",
+                     "%-10s%-18s%7s%20s%13s%11s",
                      sn->id_str, sn->name,
                      get_human_readable_size(buf1, sizeof(buf1),
                                              sn->vm_state_size),
                      date_buf,
-                     clock_buf);
+                     clock_buf,
+                     icount_buf);
     }
 }
 
@@ -857,6 +864,8 @@ void bdrv_image_info_dump(fprintf_function func_fprintf, void *f,
                 .date_nsec = elem->value->date_nsec,
                 .vm_clock_nsec = elem->value->vm_clock_sec * 1000000000ULL +
                                  elem->value->vm_clock_nsec,
+                .icount = elem->value->has_icount ?
+                          elem->value->icount : -1ULL,
             };
 
             pstrcpy(sn.id_str, sizeof(sn.id_str), elem->value->id);
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index d682946..96b57f4 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -379,6 +379,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
     sn->date_sec = sn_info->date_sec;
     sn->date_nsec = sn_info->date_nsec;
     sn->vm_clock_nsec = sn_info->vm_clock_nsec;
+    sn->icount = sn_info->icount;
 
     /* Allocate the L1 table of the snapshot and copy the current one there. */
     l1_table_offset = qcow2_alloc_clusters(bs, s->l1_size * sizeof(uint64_t));
@@ -698,6 +699,7 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab)
         sn_info->date_sec = sn->date_sec;
         sn_info->date_nsec = sn->date_nsec;
         sn_info->vm_clock_nsec = sn->vm_clock_nsec;
+        sn_info->icount = sn->icount;
     }
     *psn_tab = sn_tab;
     return s->nb_snapshots;
diff --git a/blockdev.c b/blockdev.c
index 72f5347..54b4b9d 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -57,6 +57,7 @@
 #include "block/trace.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/qtest.h"
+#include "sysemu/replay.h"
 #include "qemu/cutils.h"
 #include "qemu/help_option.h"
 #include "qemu/throttle-options.h"
@@ -1239,6 +1240,10 @@ SnapshotInfo *qmp_blockdev_snapshot_delete_internal_sync(const char *device,
     info->vm_state_size = sn.vm_state_size;
     info->vm_clock_nsec = sn.vm_clock_nsec % 1000000000;
     info->vm_clock_sec = sn.vm_clock_nsec / 1000000000;
+    if (sn.icount != -1ULL) {
+        info->icount = sn.icount;
+        info->has_icount = true;
+    }
 
     return info;
 
@@ -1447,6 +1452,11 @@ static void internal_snapshot_prepare(BlkActionState *common,
     sn->date_sec = tv.tv_sec;
     sn->date_nsec = tv.tv_usec * 1000;
     sn->vm_clock_nsec = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    if (replay_mode != REPLAY_MODE_NONE) {
+        sn->icount = replay_get_current_step();
+    } else {
+        sn->icount = -1ULL;
+    }
 
     ret1 = bdrv_snapshot_create(bs, sn);
     if (ret1 < 0) {
diff --git a/include/block/snapshot.h b/include/block/snapshot.h
index f73d109..c9c8975 100644
--- a/include/block/snapshot.h
+++ b/include/block/snapshot.h
@@ -42,6 +42,7 @@ typedef struct QEMUSnapshotInfo {
     uint32_t date_sec; /* UTC date of the snapshot */
     uint32_t date_nsec;
     uint64_t vm_clock_nsec; /* VM clock relative to boot */
+    uint64_t icount; /* record/replay step */
 } QEMUSnapshotInfo;
 
 int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
diff --git a/migration/savevm.c b/migration/savevm.c
index 13e51f0..0c21d67 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2457,6 +2457,11 @@ int save_snapshot(const char *name, Error **errp)
     sn->date_sec = tv.tv_sec;
     sn->date_nsec = tv.tv_usec * 1000;
     sn->vm_clock_nsec = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    if (replay_mode != REPLAY_MODE_NONE) {
+        sn->icount = replay_get_current_step();
+    } else {
+        sn->icount = -1ULL;
+    }
 
     if (name) {
         ret = bdrv_snapshot_find(bs, old_sn, name);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 4c7a37a..456af87 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -26,13 +26,16 @@
 #
 # @vm-clock-nsec: fractional part in nano seconds to be used with vm-clock-sec
 #
+# @icount: current instruction count for execution record/replay (since 3.1)
+#
 # Since: 1.3
 #
 ##
 { 'struct': 'SnapshotInfo',
   'data': { 'id': 'str', 'name': 'str', 'vm-state-size': 'int',
             'date-sec': 'int', 'date-nsec': 'int',
-            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int' } }
+            'vm-clock-sec': 'int', 'vm-clock-nsec': 'int',
+            '*icount': 'int' } }
 
 ##
 # @ImageInfoSpecificQCow2EncryptionBase:
diff --git a/qapi/block.json b/qapi/block.json
index 11f01f2..a6396a9 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -176,7 +176,8 @@
 #                    "date-sec": 1000012,
 #                    "date-nsec": 10,
 #                    "vm-clock-sec": 100,
-#                    "vm-clock-nsec": 20
+#                    "vm-clock-nsec": 20,
+#                    "icount": 220414
 #      }
 #    }
 #

[Qemu-devel] [PATCH v6 08/25] replay: provide and accessor for rr filename

[Pavel Dovgalyuk]

This patch adds an accessor function for the name of the record/replay
log file. Adding an accessor instead of making variable global,
prevents accidental modification of this variable by other modules.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 include/sysemu/replay.h |    2 ++
 replay/replay.c         |    5 +++++
 2 files changed, 7 insertions(+)

diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index bb8660b..a5510f2 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -71,6 +71,8 @@ void replay_start(void);
 void replay_finish(void);
 /*! Adds replay blocker with the specified error description */
 void replay_add_blocker(Error *reason);
+/*! Returns name of the replay log file */
+const char *replay_get_filename(void);
 
 /* Processing the instructions */
 
diff --git a/replay/replay.c b/replay/replay.c
index 58a986f..80afbd5 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -375,3 +375,8 @@ void replay_add_blocker(Error *reason)
 {
     replay_blockers = g_slist_prepend(replay_blockers, reason);
 }
+
+const char *replay_get_filename(void)
+{
+    return replay_filename;
+}

[Qemu-devel] [PATCH v6 09/25] replay: introduce info hmp/qmp command

[Pavel Dovgalyuk]

This patch introduces 'info replay' monitor command and
corresponding qmp request.
These commands request the current record/replay mode, replay log file name,
and the execution step (number or recorded/replayed instructions).

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Acked-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

--

v2:
 - renamed info_replay qmp into query-replay (suggested by Eric Blake)
---
 hmp-commands-info.hx      |   14 ++++++++++++++
 hmp.h                     |    1 +
 qapi/misc.json            |   35 +++++++++++++++++++++++++++++++++++
 replay/Makefile.objs      |    3 ++-
 replay/replay-debugging.c |   41 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 93 insertions(+), 1 deletion(-)
 create mode 100644 replay/replay-debugging.c

diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
index cbee8b9..9f2f35e 100644
--- a/hmp-commands-info.hx
+++ b/hmp-commands-info.hx
@@ -918,6 +918,20 @@ STEXI
 Show SEV information.
 ETEXI
 
+    {
+        .name       = "replay",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show parameters of the record/replay",
+        .cmd        = hmp_info_replay,
+    },
+
+STEXI
+@item info replay
+@findex info replay
+Display the current record/replay mode and the currently executing step.
+ETEXI
+
 STEXI
 @end table
 ETEXI
diff --git a/hmp.h b/hmp.h
index 5f1addc..d792149 100644
--- a/hmp.h
+++ b/hmp.h
@@ -148,5 +148,6 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict);
 void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);
 void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
+void hmp_info_replay(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/qapi/misc.json b/qapi/misc.json
index d450cfe..e246ce3 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3100,6 +3100,41 @@
   'data': [ 'none', 'record', 'play' ] }
 
 ##
+# @ReplayInfo:
+#
+# Status of the record/replay mode.
+#
+# @mode: current mode.
+#
+# @filename: name of the record/replay log file.
+#
+# @step: current step number.
+#
+# Since: 3.1
+#
+##
+{ 'struct': 'ReplayInfo',
+  'data': { 'mode': 'ReplayMode', '*filename': 'str', 'step': 'int' } }
+
+##
+# @query-replay:
+#
+# Retrieves the status of the execution record/replay.
+#
+# Returns: structure with the properties of the record/replay.
+#
+# Since: 3.1
+#
+# Example:
+#
+# -> { "execute": "query-replay" }
+# <- { "return": { "mode": "play", "filename": "log.rr", "step": 220414 } }
+#
+##
+{ 'command': 'query-replay',
+  'returns': 'ReplayInfo' }
+
+##
 # @xen-load-devices-state:
 #
 # Load the state of all devices from file. The RAM and the block devices
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index cee6539..6694e3e 100644
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -6,4 +6,5 @@ common-obj-y += replay-input.o
 common-obj-y += replay-char.o
 common-obj-y += replay-snapshot.o
 common-obj-y += replay-net.o
-common-obj-y += replay-audio.o
\ No newline at end of file
+common-obj-y += replay-audio.o
+common-obj-y += replay-debugging.o
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
new file mode 100644
index 0000000..2d364fa
--- /dev/null
+++ b/replay/replay-debugging.c
@@ -0,0 +1,41 @@
+/*
+ * replay-debugging.c
+ *
+ * Copyright (c) 2010-2018 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "sysemu/replay.h"
+#include "replay-internal.h"
+#include "hmp.h"
+#include "monitor/monitor.h"
+#include "qapi/qapi-commands-misc.h"
+
+void hmp_info_replay(Monitor *mon, const QDict *qdict)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        monitor_printf(mon, "No record/replay\n");
+    } else {
+        monitor_printf(mon, "%s execution '%s': current step = %"PRId64"\n",
+            replay_mode == REPLAY_MODE_RECORD ? "Recording" : "Replaying",
+            replay_get_filename(), replay_get_current_step());
+    }
+}
+
+ReplayInfo *qmp_query_replay(Error **errp)
+{
+    ReplayInfo *retval = g_new0(ReplayInfo, 1);
+    retval->mode = replay_mode;
+    if (replay_get_filename()) {
+        retval->filename = g_strdup(replay_get_filename());
+        retval->has_filename = true;
+    }
+    retval->step = replay_get_current_step();
+    return retval;
+}

[Qemu-devel] [PATCH v6 10/25] replay: introduce breakpoint at the specified step

[Pavel Dovgalyuk]

This patch introduces replay_break qmp and hmp commands.
These commands allow stopping at the specified instruction.
It may be useful for debugging when there are some known
events that should be investigated.
The commands have one argument - number of instructions
executed since the start of the replay.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - renamed replay_break qmp command into replay-break
   (suggested by Eric Blake)
---
 hmp-commands.hx           |   15 ++++++++++++
 hmp.h                     |    1 +
 include/sysemu/replay.h   |    3 ++
 qapi/misc.json            |   17 ++++++++++++++
 replay/replay-debugging.c |   55 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h  |    4 +++
 replay/replay.c           |   17 ++++++++++++++
 7 files changed, 112 insertions(+)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index db0c681..aa0794e 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1888,6 +1888,21 @@ Set QOM property @var{property} of object at location @var{path} to value @var{v
 ETEXI
 
     {
+        .name       = "replay_break",
+        .args_type  = "step:i",
+        .params     = "step",
+        .help       = "sets breakpoint on the specified step of the replay",
+        .cmd        = hmp_replay_break,
+    },
+
+STEXI
+@item replay_break @var{step}
+@findex replay_break
+Set breakpoint on the specified step of the replay.
+Execution stops when the specified step is reached.
+ETEXI
+
+    {
         .name       = "info",
         .args_type  = "item:s?",
         .params     = "[subcommand]",
diff --git a/hmp.h b/hmp.h
index d792149..ad94abe 100644
--- a/hmp.h
+++ b/hmp.h
@@ -149,5 +149,6 @@ void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict);
 void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
 void hmp_info_replay(Monitor *mon, const QDict *qdict);
+void hmp_replay_break(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index a5510f2..c9ba75a 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -73,6 +73,9 @@ void replay_finish(void);
 void replay_add_blocker(Error *reason);
 /*! Returns name of the replay log file */
 const char *replay_get_filename(void);
+/*! Sets breakpoint at the specified step.
+    If step = -1LL the existing breakpoint is removed. */
+void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
 
 /* Processing the instructions */
 
diff --git a/qapi/misc.json b/qapi/misc.json
index e246ce3..4fcd211 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3135,6 +3135,23 @@
   'returns': 'ReplayInfo' }
 
 ##
+# @replay-break:
+#
+# Set breakpoint on the specified step of the replay.
+# Execution stops when the specified step is reached.
+#
+# @step: execution step to stop at
+#
+# Since: 3.1
+#
+# Example:
+#
+# -> { "execute": "replay-break", "data": { "step": 220414 } }
+#
+##
+{ 'command': 'replay-break', 'data': { 'step': 'int' } }
+
+##
 # @xen-load-devices-state:
 #
 # Load the state of all devices from file. The RAM and the block devices
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index 2d364fa..c6b80c0 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -16,6 +16,8 @@
 #include "hmp.h"
 #include "monitor/monitor.h"
 #include "qapi/qapi-commands-misc.h"
+#include "qapi/qmp/qdict.h"
+#include "qemu/timer.h"
 
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
@@ -39,3 +41,56 @@ ReplayInfo *qmp_query_replay(Error **errp)
     retval->step = replay_get_current_step();
     return retval;
 }
+
+void replay_break(int64_t step, QEMUTimerCB callback, void *opaque)
+{
+    assert(replay_mode == REPLAY_MODE_PLAY);
+    assert(replay_mutex_locked());
+
+    replay_break_step = step;
+    if (replay_break_timer) {
+        timer_del(replay_break_timer);
+        timer_free(replay_break_timer);
+        replay_break_timer = NULL;
+    }
+
+    if (replay_break_step == -1LL) {
+        return;
+    }
+    assert(replay_break_step >= replay_get_current_step());
+    assert(callback);
+
+    replay_break_timer = timer_new_ns(QEMU_CLOCK_REALTIME, callback, opaque);
+}
+
+static void replay_stop_vm(void *opaque)
+{
+    vm_stop(RUN_STATE_PAUSED);
+    replay_break(-1LL, NULL, NULL);
+}
+
+void qmp_replay_break(int64_t step, Error **errp)
+{
+    if (replay_mode ==  REPLAY_MODE_PLAY) {
+        if (step >= replay_get_current_step()) {
+            replay_break(step, replay_stop_vm, NULL);
+        } else {
+            error_setg(errp, "cannot set break at the step in the past");
+        }
+    } else {
+        error_setg(errp, "setting the break is allowed only in play mode");
+    }
+}
+
+void hmp_replay_break(Monitor *mon, const QDict *qdict)
+{
+    int64_t step = qdict_get_try_int(qdict, "step", -1LL);
+    Error *err = NULL;
+
+    qmp_replay_break(step, &err);
+    if (err) {
+        monitor_printf(mon, "replay_break error: %s\n", error_get_pretty(err));
+        error_free(err);
+        return;
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 4f82676..f9cad9d 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -91,6 +91,10 @@ extern ReplayState replay_state;
 
 /* File for replay writing */
 extern FILE *replay_file;
+/*! Step of the replay breakpoint */
+extern int64_t replay_break_step;
+/*! Timer for the replay breakpoint callback */
+extern QEMUTimer *replay_break_timer;
 
 void replay_put_byte(uint8_t byte);
 void replay_put_event(uint8_t event);
diff --git a/replay/replay.c b/replay/replay.c
index 80afbd5..a106211 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -34,6 +34,10 @@ static char *replay_filename;
 ReplayState replay_state;
 static GSList *replay_blockers;
 
+/* Replay breakpoints */
+int64_t replay_break_step = -1LL;
+QEMUTimer *replay_break_timer;
+
 bool replay_next_event_is(int event)
 {
     bool res = false;
@@ -73,6 +77,13 @@ int replay_get_instructions(void)
     replay_mutex_lock();
     if (replay_next_event_is(EVENT_INSTRUCTION)) {
         res = replay_state.instructions_count;
+        if (replay_break_step != -1LL) {
+            uint64_t current = replay_get_current_step();
+            assert(replay_break_step >= current);
+            if (current + res > replay_break_step) {
+                res = replay_break_step - current;
+            }
+        }
     }
     replay_mutex_unlock();
     return res;
@@ -99,6 +110,12 @@ void replay_account_executed_instructions(void)
                    will be read from the log. */
                 qemu_notify_event();
             }
+            /* Execution reached the break step */
+            if (replay_break_step == replay_state.current_step) {
+                /* Cannot make callback directly from the vCPU thread */
+                timer_mod_ns(replay_break_timer,
+                    qemu_clock_get_ns(QEMU_CLOCK_REALTIME));
+            }
         }
     }
 }

[Qemu-devel] [PATCH v6 11/25] replay: implement replay-seek command to proceed to the desired step

[Pavel Dovgalyuk]

This patch adds hmp/qmp commands replay_seek/replay-seek that proceed
the execution to the specified step.
The commands automatically loads nearest snapshot and replay the execution
to find the desired step.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - renamed replay_seek qmp command into replay-seek
   (suggested by Eric Blake)
---
 hmp-commands.hx           |   15 ++++++++
 hmp.h                     |    1 +
 qapi/misc.json            |   16 ++++++++
 replay/replay-debugging.c |   89 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 121 insertions(+)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index aa0794e..8d4065b 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1903,6 +1903,21 @@ Execution stops when the specified step is reached.
 ETEXI
 
     {
+        .name       = "replay_seek",
+        .args_type  = "step:i",
+        .params     = "step",
+        .help       = "rewinds replay to the specified step",
+        .cmd        = hmp_replay_seek,
+    },
+
+STEXI
+@item replay_seek @var{step}
+@findex replay_seek
+Automatically proceeds to the specified step, when replaying
+the execution.
+ETEXI
+
+    {
         .name       = "info",
         .args_type  = "item:s?",
         .params     = "[subcommand]",
diff --git a/hmp.h b/hmp.h
index ad94abe..a7d3bce 100644
--- a/hmp.h
+++ b/hmp.h
@@ -150,5 +150,6 @@ void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict);
 void hmp_info_sev(Monitor *mon, const QDict *qdict);
 void hmp_info_replay(Monitor *mon, const QDict *qdict);
 void hmp_replay_break(Monitor *mon, const QDict *qdict);
+void hmp_replay_seek(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/qapi/misc.json b/qapi/misc.json
index 4fcd211..7389cc6 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3152,6 +3152,22 @@
 { 'command': 'replay-break', 'data': { 'step': 'int' } }
 
 ##
+# @replay-seek:
+#
+# Automatically proceeds to the specified step, when replaying
+# the execution.
+#
+# @step: destination execution step
+#
+# Since: 3.1
+#
+# Example:
+#
+# -> { "execute": "replay-seek", "data": { "step": 220414 } }
+##
+{ 'command': 'replay-seek', 'data': { 'step': 'int' } }
+
+##
 # @xen-load-devices-state:
 #
 # Load the state of all devices from file. The RAM and the block devices
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index c6b80c0..e605ee0 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -18,6 +18,8 @@
 #include "qapi/qapi-commands-misc.h"
 #include "qapi/qmp/qdict.h"
 #include "qemu/timer.h"
+#include "block/snapshot.h"
+#include "migration/snapshot.h"
 
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
@@ -94,3 +96,90 @@ void hmp_replay_break(Monitor *mon, const QDict *qdict)
         return;
     }
 }
+
+static char *replay_find_nearest_snapshot(int64_t step, int64_t* snapshot_step)
+{
+    BlockDriverState *bs;
+    QEMUSnapshotInfo *sn_tab;
+    QEMUSnapshotInfo *nearest = NULL;
+    char *ret = NULL;
+    int nb_sns, i;
+    AioContext *aio_context;
+
+    *snapshot_step = -1;
+
+    bs = bdrv_all_find_vmstate_bs();
+    if (!bs) {
+        goto fail;
+    }
+    aio_context = bdrv_get_aio_context(bs);
+
+    aio_context_acquire(aio_context);
+    nb_sns = bdrv_snapshot_list(bs, &sn_tab);
+    aio_context_release(aio_context);
+
+    for (i = 0; i < nb_sns; i++) {
+        if (bdrv_all_find_snapshot(sn_tab[i].name, &bs) == 0) {
+            if (sn_tab[i].icount != -1ULL
+                && sn_tab[i].icount <= step
+                && (!nearest || nearest->icount < sn_tab[i].icount)) {
+                nearest = &sn_tab[i];
+            }
+        }
+    }
+    if (nearest) {
+        ret = g_strdup(nearest->name);
+        *snapshot_step = nearest->icount;
+    }
+    g_free(sn_tab);
+
+fail:
+    return ret;
+}
+
+static void replay_seek(int64_t step, Error **errp, QEMUTimerCB callback)
+{
+    char *snapshot = NULL;
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        error_setg(errp, "replay must be enabled to seek");
+        return;
+    }
+    if (!replay_snapshot) {
+        error_setg(errp, "snapshotting is disabled");
+        return;
+    }
+    int64_t snapshot_step = -1;
+    snapshot = replay_find_nearest_snapshot(step, &snapshot_step);
+    if (snapshot) {
+        if (step < replay_get_current_step()
+            || replay_get_current_step() < snapshot_step) {
+            vm_stop(RUN_STATE_RESTORE_VM);
+            load_snapshot(snapshot, errp);
+        }
+        g_free(snapshot);
+    }
+    if (replay_get_current_step() <= step) {
+        replay_break(step, callback, NULL);
+        vm_start();
+    } else {
+        error_setg(errp, "cannot seek to the specified step");
+    }
+}
+
+void qmp_replay_seek(int64_t step, Error **errp)
+{
+    replay_seek(step, errp, replay_stop_vm);
+}
+
+void hmp_replay_seek(Monitor *mon, const QDict *qdict)
+{
+    int64_t step = qdict_get_try_int(qdict, "step", -1LL);
+    Error *err = NULL;
+
+    qmp_replay_seek(step, &err);
+    if (err) {
+        monitor_printf(mon, "replay_seek error: %s\n", error_get_pretty(err));
+        error_free(err);
+        return;
+    }
+}

[Qemu-devel] [PATCH v6 12/25] replay: flush events when exiting

[Pavel Dovgalyuk]

This patch adds events processing when emulation finishes instead
of just cleaning the queue. Now the bdrv coroutines will be in consistent
state when emulator closes. It allows correct polling of the block layer
at exit.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 replay/replay-events.c   |   14 +-------------
 replay/replay-internal.h |    2 --
 2 files changed, 1 insertion(+), 15 deletions(-)

diff --git a/replay/replay-events.c b/replay/replay-events.c
index 707de38..0964a82 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -94,18 +94,6 @@ void replay_disable_events(void)
     }
 }
 
-void replay_clear_events(void)
-{
-    g_assert(replay_mutex_locked());
-
-    while (!QTAILQ_EMPTY(&events_list)) {
-        Event *event = QTAILQ_FIRST(&events_list);
-        QTAILQ_REMOVE(&events_list, event, events);
-
-        g_free(event);
-    }
-}
-
 /*! Adds specified async event to the queue */
 void replay_add_event(ReplayAsyncEventKind event_kind,
                       void *opaque,
@@ -308,7 +296,7 @@ void replay_init_events(void)
 void replay_finish_events(void)
 {
     events_enabled = false;
-    replay_clear_events();
+    replay_flush_events();
 }
 
 bool replay_events_enabled(void)
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index f9cad9d..b824b7a 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -148,8 +148,6 @@ void replay_init_events(void);
 void replay_finish_events(void);
 /*! Flushes events queue */
 void replay_flush_events(void);
-/*! Clears events list before loading new VM state */
-void replay_clear_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */

[Qemu-devel] [PATCH v6 13/25] replay: refine replay-time module

[Pavel Dovgalyuk]

This patch removes refactoring artifacts from the replay/replay-time.c

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 replay/replay-time.c |   32 +++++++++++++-------------------
 1 file changed, 13 insertions(+), 19 deletions(-)

diff --git a/replay/replay-time.c b/replay/replay-time.c
index 17caf35..8986054 100644
--- a/replay/replay-time.c
+++ b/replay/replay-time.c
@@ -17,16 +17,15 @@
 
 int64_t replay_save_clock(ReplayClockKind kind, int64_t clock, int64_t raw_icount)
 {
-    if (replay_file) {
-        g_assert(replay_mutex_locked());
+    g_assert(replay_file);
+    g_assert(replay_mutex_locked());
 
-        /* Due to the caller's locking requirements we get the icount from it instead
-         * of using replay_save_instructions().
-         */
-        replay_advance_current_step(raw_icount);
-        replay_put_event(EVENT_CLOCK + kind);
-        replay_put_qword(clock);
-    }
+    /* Due to the caller's locking requirements we get the icount from it instead
+     * of using replay_save_instructions().
+     */
+    replay_advance_current_step(raw_icount);
+    replay_put_event(EVENT_CLOCK + kind);
+    replay_put_qword(clock);
 
     return clock;
 }
@@ -48,20 +47,15 @@ void replay_read_next_clock(ReplayClockKind kind)
 /*! Reads next clock event from the input. */
 int64_t replay_read_clock(ReplayClockKind kind)
 {
+    int64_t ret;
     g_assert(replay_file && replay_mutex_locked());
 
     replay_account_executed_instructions();
 
-    if (replay_file) {
-        int64_t ret;
-        if (replay_next_event_is(EVENT_CLOCK + kind)) {
-            replay_read_next_clock(kind);
-        }
-        ret = replay_state.cached_clock[kind];
-
-        return ret;
+    if (replay_next_event_is(EVENT_CLOCK + kind)) {
+        replay_read_next_clock(kind);
     }
+    ret = replay_state.cached_clock[kind];
 
-    error_report("REPLAY INTERNAL ERROR %d", __LINE__);
-    exit(1);
+    return ret;
 }

[Qemu-devel] [PATCH v6 14/25] translator: fix breakpoint processing

[Pavel Dovgalyuk]

QEMU cannot pass through the breakpoints when 'si' command is used
in remote gdb. This patch disables inserting the breakpoints
when we are already single stepping though the gdb remote protocol.
This patch also fixes icount calculation for the blocks that include
breakpoints - instruction with breakpoint is not executed and shouldn't
be used in icount calculation.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 accel/tcg/translator.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c
index 0f9dca9..afd0a49 100644
--- a/accel/tcg/translator.c
+++ b/accel/tcg/translator.c
@@ -34,6 +34,8 @@ void translator_loop_temp_check(DisasContextBase *db)
 void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
                      CPUState *cpu, TranslationBlock *tb)
 {
+    int bp_insn = 0;
+
     /* Initialize DisasContext */
     db->tb = tb;
     db->pc_first = tb->pc;
@@ -71,11 +73,13 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
         tcg_debug_assert(db->is_jmp == DISAS_NEXT);  /* no early exit */
 
         /* Pass breakpoint hits to target for further processing */
-        if (unlikely(!QTAILQ_EMPTY(&cpu->breakpoints))) {
+        if (!db->singlestep_enabled
+            && unlikely(!QTAILQ_EMPTY(&cpu->breakpoints))) {
             CPUBreakpoint *bp;
             QTAILQ_FOREACH(bp, &cpu->breakpoints, entry) {
                 if (bp->pc == db->pc_next) {
                     if (ops->breakpoint_check(db, cpu, bp)) {
+                        bp_insn = 1;
                         break;
                     }
                 }
@@ -118,7 +122,7 @@ void translator_loop(const TranslatorOps *ops, DisasContextBase *db,
 
     /* Emit code to exit the TB, as indicated by db->is_jmp.  */
     ops->tb_stop(db, cpu);
-    gen_tb_end(db->tb, db->num_insns);
+    gen_tb_end(db->tb, db->num_insns - bp_insn);
 
     /* The disas_log hook may use these values rather than recompute.  */
     db->tb->size = db->pc_next - db->pc_first;

[Qemu-devel] [PATCH v6 15/25] replay: flush rr queue before loading the vmstate

[Pavel Dovgalyuk]

Non-empty record/replay queue prevents saving and loading the VM state,
because it includes pending bottom halves and block coroutines.
But when the new VM state is loaded, we don't have to preserve the consistency
of the current state anymore. Therefore this patch just flushes the queue
allowing the coroutines to finish.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 include/sysemu/replay.h  |    2 ++
 migration/savevm.c       |   10 ++++------
 replay/replay-internal.h |    2 --
 3 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index c9ba75a..7a4890d 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -140,6 +140,8 @@ void replay_disable_events(void);
 void replay_enable_events(void);
 /*! Returns true when saving events is enabled */
 bool replay_events_enabled(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
 /*! Adds bottom half event to the queue */
 void replay_bh_schedule_event(QEMUBH *bh);
 /*! Adds input event to the queue */
diff --git a/migration/savevm.c b/migration/savevm.c
index 0c21d67..1894f45 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2610,12 +2610,6 @@ int load_snapshot(const char *name, Error **errp)
     AioContext *aio_context;
     MigrationIncomingState *mis = migration_incoming_get_current();
 
-    if (!replay_can_snapshot()) {
-        error_report("Record/replay does not allow loading snapshot "
-                     "right now. Try once more later.");
-        return -EINVAL;
-    }
-
     if (!bdrv_all_can_snapshot(&bs)) {
         error_setg(errp,
                    "Device '%s' is writable but does not support snapshots",
@@ -2649,6 +2643,10 @@ int load_snapshot(const char *name, Error **errp)
         return -EINVAL;
     }
 
+    /* Flush the record/replay queue. Now the VM state is going
+       to change. Therefore we don't need to preserve its consistency */
+    replay_flush_events();
+
     /* Flush all IO requests so they don't interfere with the new state.  */
     bdrv_drain_all_begin();
 
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index b824b7a..a857ca4 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -146,8 +146,6 @@ void replay_read_next_clock(unsigned int kind);
 void replay_init_events(void);
 /*! Clears internal data structures for events handling */
 void replay_finish_events(void);
-/*! Flushes events queue */
-void replay_flush_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */

[Qemu-devel] [PATCH v6 16/25] gdbstub: add reverse step support in replay mode

[Pavel Dovgalyuk]

GDB remote protocol supports two reverse debugging commands:
reverse step and reverse continue.
This patch adds support of the first one to the gdbstub.
Reverse step is intended to step one instruction in the backwards
direction. This is not possible in regular execution.
But replayed execution is deterministic, therefore we can load one of
the prior snapshots and proceed to the desired step. It is equivalent
to stepping one instruction back.
There should be at least one snapshot preceding the debugged part of
the replay log.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 accel/tcg/translator.c    |    1 +
 cpus.c                    |   14 +++++++++++---
 exec.c                    |    5 +++++
 gdbstub.c                 |   42 +++++++++++++++++++++++++++++++++++++++---
 include/sysemu/replay.h   |    7 +++++++
 replay/replay-debugging.c |   33 +++++++++++++++++++++++++++++++++
 stubs/replay.c            |    5 +++++
 7 files changed, 101 insertions(+), 6 deletions(-)

diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c
index afd0a49..33a543e 100644
--- a/accel/tcg/translator.c
+++ b/accel/tcg/translator.c
@@ -17,6 +17,7 @@
 #include "exec/gen-icount.h"
 #include "exec/log.h"
 #include "exec/translator.h"
+#include "sysemu/replay.h"
 
 /* Pairs with tcg_clear_temp_count.
    To be called by #TranslatorOps.{translate_insn,tb_stop} if
diff --git a/cpus.c b/cpus.c
index 0ce5640..09248dd 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1084,9 +1084,17 @@ static bool cpu_can_run(CPUState *cpu)
 
 static void cpu_handle_guest_debug(CPUState *cpu)
 {
-    gdb_set_stop_cpu(cpu);
-    qemu_system_debug_request();
-    cpu->stopped = true;
+    if (!replay_running_debug()) {
+        gdb_set_stop_cpu(cpu);
+        qemu_system_debug_request();
+        cpu->stopped = true;
+    } else {
+        if (!cpu->singlestep_enabled) {
+            cpu_single_step(cpu, SSTEP_ENABLE);
+        } else {
+            cpu_single_step(cpu, 0);
+        }
+    }
 }
 
 #ifdef CONFIG_LINUX
diff --git a/exec.c b/exec.c
index 6826c83..eedddba 100644
--- a/exec.c
+++ b/exec.c
@@ -2738,6 +2738,11 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)
     QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) {
         if (cpu_watchpoint_address_matches(wp, vaddr, len)
             && (wp->flags & flags)) {
+            if (replay_running_debug()) {
+                /* Don't process the watchpoints when we are
+                   in a reverse debugging operation. */
+                return;
+            }
             if (flags == BP_MEM_READ) {
                 wp->flags |= BP_WATCHPOINT_HIT_READ;
             } else {
diff --git a/gdbstub.c b/gdbstub.c
index d6ab950..fa108f4 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -39,6 +39,7 @@
 #include "sysemu/kvm.h"
 #include "exec/semihost.h"
 #include "exec/exec-all.h"
+#include "sysemu/replay.h"
 
 #ifdef CONFIG_USER_ONLY
 #define GDB_ATTACHED "0"
@@ -334,6 +335,19 @@ typedef struct GDBState {
  */
 static int sstep_flags = SSTEP_ENABLE|SSTEP_NOIRQ|SSTEP_NOTIMER;
 
+/*! Retrieves flags for single step mode. */
+static int get_sstep_flags(void)
+{
+    /* In replay mode all events written into the log should be replayed.
+     * That is why NOIRQ flag is removed in this mode.
+     */
+    if (replay_mode != REPLAY_MODE_NONE) {
+        return SSTEP_ENABLE;
+    } else {
+        return sstep_flags;
+    }
+}
+
 static GDBState *gdbserver_state;
 
 bool gdb_has_xml;
@@ -424,7 +438,7 @@ static int gdb_continue_partial(GDBState *s, char *newstates)
     CPU_FOREACH(cpu) {
         if (newstates[cpu->cpu_index] == 's') {
             trace_gdbstub_op_stepping(cpu->cpu_index);
-            cpu_single_step(cpu, sstep_flags);
+            cpu_single_step(cpu, get_sstep_flags());
         }
     }
     s->running_state = 1;
@@ -443,7 +457,7 @@ static int gdb_continue_partial(GDBState *s, char *newstates)
                 break; /* nothing to do here */
             case 's':
                 trace_gdbstub_op_stepping(cpu->cpu_index);
-                cpu_single_step(cpu, sstep_flags);
+                cpu_single_step(cpu, get_sstep_flags());
                 cpu_resume(cpu);
                 flag = 1;
                 break;
@@ -1082,9 +1096,28 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
             addr = strtoull(p, (char **)&p, 16);
             gdb_set_cpu_pc(s, addr);
         }
-        cpu_single_step(s->c_cpu, sstep_flags);
+        cpu_single_step(s->c_cpu, get_sstep_flags());
         gdb_continue(s);
         return RS_IDLE;
+    case 'b':
+        /* Backward debugging commands */
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            switch (*p) {
+            case 's':
+                if (replay_reverse_step()) {
+                    gdb_continue(s);
+                    return RS_IDLE;
+                } else {
+                    put_packet(s, "E14");
+                    break;
+                }
+            default:
+                goto unknown_command;
+            }
+        } else {
+            put_packet(s, "E22");
+        }
+        goto unknown_command;
     case 'F':
         {
             target_ulong ret;
@@ -1347,6 +1380,9 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
             if (cc->gdb_core_xml_file != NULL) {
                 pstrcat(buf, sizeof(buf), ";qXfer:features:read+");
             }
+            if (replay_mode == REPLAY_MODE_PLAY) {
+                pstrcat(buf, sizeof(buf), ";ReverseStep+");
+            }
             put_packet(s, buf);
             break;
         }
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 7a4890d..55a272a 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -76,6 +76,13 @@ const char *replay_get_filename(void);
 /*! Sets breakpoint at the specified step.
     If step = -1LL the existing breakpoint is removed. */
 void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
+/*! Start making one step in backward direction.
+    Used by gdbstub for backwards debugging.
+    Returns true on success. */
+bool replay_reverse_step(void);
+/*! Returns true if replay module is processing
+    reverse_continue or reverse_step request */
+bool replay_running_debug(void);
 
 /* Processing the instructions */
 
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index e605ee0..a49d91e 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -21,6 +21,13 @@
 #include "block/snapshot.h"
 #include "migration/snapshot.h"
 
+static bool replay_is_debugging;
+
+bool replay_running_debug(void)
+{
+    return replay_is_debugging;
+}
+
 void hmp_info_replay(Monitor *mon, const QDict *qdict)
 {
     if (replay_mode == REPLAY_MODE_NONE) {
@@ -183,3 +190,29 @@ void hmp_replay_seek(Monitor *mon, const QDict *qdict)
         return;
     }
 }
+
+static void replay_stop_vm_debug(void *opaque)
+{
+    replay_is_debugging = false;
+    vm_stop(RUN_STATE_DEBUG);
+    replay_break(-1LL, NULL, NULL);
+}
+
+bool replay_reverse_step(void)
+{
+    Error *err = NULL;
+
+    assert(replay_mode == REPLAY_MODE_PLAY);
+
+    if (replay_get_current_step() != 0) {
+        replay_seek(replay_get_current_step() - 1, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            return false;
+        }
+        replay_is_debugging = true;
+        return true;
+    }
+
+    return false;
+}
diff --git a/stubs/replay.c b/stubs/replay.c
index 4ac6078..521552f 100644
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -80,3 +80,8 @@ void replay_mutex_lock(void)
 void replay_mutex_unlock(void)
 {
 }
+
+bool replay_reverse_step(void)
+{
+    return false;
+}

[Qemu-devel] [PATCH v6 17/25] gdbstub: add reverse continue support in replay mode

[Pavel Dovgalyuk]

This patch adds support of the reverse continue operation for gdbstub.
Reverse continue finds the last breakpoint that would happen in normal
execution from the beginning to the current moment.
Implementation of the reverse continue replays the execution twice:
to find the breakpoints that were hit and to seek to the last breakpoint.
Reverse continue loads the previous snapshot and tries to find the breakpoint
since that moment. If there are no such breakpoints, it proceeds to
the earlier snapshot, and so on. When no breakpoints or watchpoints were
hit at all, execution stops at the beginning of the replay log.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 cpus.c                    |    3 ++
 exec.c                    |    1 +
 gdbstub.c                 |   10 ++++++-
 include/sysemu/replay.h   |    6 ++++
 replay/replay-debugging.c |   69 +++++++++++++++++++++++++++++++++++++++++++++
 stubs/replay.c            |    5 +++
 6 files changed, 93 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index 09248dd..f31a70a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1090,6 +1090,9 @@ static void cpu_handle_guest_debug(CPUState *cpu)
         cpu->stopped = true;
     } else {
         if (!cpu->singlestep_enabled) {
+            /* Report about the breakpoint and
+               make a single step to skip it */
+            replay_breakpoint();
             cpu_single_step(cpu, SSTEP_ENABLE);
         } else {
             cpu_single_step(cpu, 0);
diff --git a/exec.c b/exec.c
index eedddba..02d71b8 100644
--- a/exec.c
+++ b/exec.c
@@ -2741,6 +2741,7 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags)
             if (replay_running_debug()) {
                 /* Don't process the watchpoints when we are
                    in a reverse debugging operation. */
+                replay_breakpoint();
                 return;
             }
             if (flags == BP_MEM_READ) {
diff --git a/gdbstub.c b/gdbstub.c
index fa108f4..ecdc0b2 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -1111,6 +1111,14 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
                     put_packet(s, "E14");
                     break;
                 }
+            case 'c':
+                if (replay_reverse_continue()) {
+                    gdb_continue(s);
+                    return RS_IDLE;
+                } else {
+                    put_packet(s, "E14");
+                    break;
+                }
             default:
                 goto unknown_command;
             }
@@ -1381,7 +1389,7 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
                 pstrcat(buf, sizeof(buf), ";qXfer:features:read+");
             }
             if (replay_mode == REPLAY_MODE_PLAY) {
-                pstrcat(buf, sizeof(buf), ";ReverseStep+");
+                pstrcat(buf, sizeof(buf), ";ReverseStep+;ReverseContinue+");
             }
             put_packet(s, buf);
             break;
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 55a272a..c1646b0 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -80,9 +80,15 @@ void replay_break(int64_t step, QEMUTimerCB callback, void *opaque);
     Used by gdbstub for backwards debugging.
     Returns true on success. */
 bool replay_reverse_step(void);
+/*! Start searching the last breakpoint/watchpoint.
+    Used by gdbstub for backwards debugging.
+    Returns true if the process successfully started. */
+bool replay_reverse_continue(void);
 /*! Returns true if replay module is processing
     reverse_continue or reverse_step request */
 bool replay_running_debug(void);
+/*! Called in reverse debugging mode to collect breakpoint information */
+void replay_breakpoint(void);
 
 /* Processing the instructions */
 
diff --git a/replay/replay-debugging.c b/replay/replay-debugging.c
index a49d91e..81434d8 100644
--- a/replay/replay-debugging.c
+++ b/replay/replay-debugging.c
@@ -22,6 +22,8 @@
 #include "migration/snapshot.h"
 
 static bool replay_is_debugging;
+static int64_t replay_last_breakpoint;
+static int64_t replay_last_snapshot;
 
 bool replay_running_debug(void)
 {
@@ -216,3 +218,70 @@ bool replay_reverse_step(void)
 
     return false;
 }
+
+static void replay_continue_end(void)
+{
+    replay_is_debugging = false;
+    vm_stop(RUN_STATE_DEBUG);
+    replay_break(-1LL, NULL, NULL);
+}
+
+static void replay_continue_stop(void *opaque)
+{
+    Error *err = NULL;
+    if (replay_last_breakpoint != -1LL) {
+        replay_seek(replay_last_breakpoint, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        return;
+    }
+    /* No breakpoints since the last snapshot.
+       Find previous snapshot and try again. */
+    if (replay_last_snapshot != 0) {
+        replay_seek(replay_last_snapshot - 1, &err, replay_continue_stop);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        replay_last_snapshot = replay_get_current_step();
+        return;
+    } else {
+        /* Seek to the very first step */
+        replay_seek(0, &err, replay_stop_vm_debug);
+        if (err) {
+            error_free(err);
+            replay_continue_end();
+        }
+        return;
+    }
+    replay_continue_end();
+}
+
+bool replay_reverse_continue(void)
+{
+    Error *err = NULL;
+
+    assert(replay_mode == REPLAY_MODE_PLAY);
+
+    if (replay_get_current_step() != 0) {
+        replay_seek(replay_get_current_step() - 1, &err, replay_continue_stop);
+        if (err) {
+            error_free(err);
+            return false;
+        }
+        replay_last_breakpoint = -1LL;
+        replay_is_debugging = true;
+        replay_last_snapshot = replay_get_current_step();
+        return true;
+    }
+
+    return false;
+}
+
+void replay_breakpoint(void)
+{
+    assert(replay_mode == REPLAY_MODE_PLAY);
+    replay_last_breakpoint = replay_get_current_step();
+}
diff --git a/stubs/replay.c b/stubs/replay.c
index 521552f..ee64972 100644
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -85,3 +85,8 @@ bool replay_reverse_step(void)
 {
     return false;
 }
+
+bool replay_reverse_continue(void)
+{
+    return false;
+}

[Qemu-devel] [PATCH v6 18/25] replay: describe reverse debugging in docs/replay.txt

[Pavel Dovgalyuk]

This patch updates the documentation and describes usage of the reverse
debugging in QEMU+GDB.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 docs/replay.txt |   33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/docs/replay.txt b/docs/replay.txt
index f7def53..086d3f8 100644
--- a/docs/replay.txt
+++ b/docs/replay.txt
@@ -293,6 +293,39 @@ for recording and replaying must contain identical number of ports in record
 and replay modes, but their backends may differ.
 E.g., '-serial stdio' in record mode, and '-serial null' in replay mode.
 
+Reverse debugging
+-----------------
+
+Reverse debugging allows "executing" the program in reverse direction.
+GDB remote protocol supports "reverse step" and "reverse continue"
+commands. The first one steps single instruction backwards in time,
+and the second one finds the last breakpoint in the past.
+
+Recorded executions may be used to enable reverse debugging. QEMU can't
+execute the code in backwards direction, but can load a snapshot and
+replay forward to find the desired position or breakpoint.
+
+The following GDB commands are supported:
+ - reverse-stepi (or rsi) - step one instruction backwards
+ - reverse-continue (or rc) - find last breakpoint in the past
+
+Reverse step loads the nearest snapshot and replays the execution until
+the required instruction is met.
+
+Reverse continue may include several passes of examining the execution
+between the snapshots. Each of the passes include the following steps:
+ 1. loading the snapshot
+ 2. replaying to examine the breakpoints
+ 3. if breakpoint or watchpoint was met
+    - loading the snaphot again
+    - replaying to the required breakpoint
+ 4. else
+    - proceeding to the p.1 with the earlier snapshot
+
+Therefore usage of the reverse debugging requires at least one snapshot
+created in advance. See the "Snapshotting" section to learn about running
+record/replay and creating the snapshot in these modes.
+
 Replay log format
 -----------------
 

[Qemu-devel] [PATCH v6 19/25] replay: allow loading any snapshots before recording

[Pavel Dovgalyuk]

This patch enables using -loadvm in recording mode to allow starting
the execution recording from any of the available snapshots.
It also fixes loading of the record/replay state, therefore snapshots
created in replay mode may also be used for starting the new recording.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 replay/replay-snapshot.c |   17 ++++++++++++-----
 vl.c                     |    7 ++++---
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c
index 2ab85cf..16bacc9 100644
--- a/replay/replay-snapshot.c
+++ b/replay/replay-snapshot.c
@@ -33,11 +33,18 @@ static int replay_pre_save(void *opaque)
 static int replay_post_load(void *opaque, int version_id)
 {
     ReplayState *state = opaque;
-    fseek(replay_file, state->file_offset, SEEK_SET);
-    qemu_clock_set_last(QEMU_CLOCK_HOST, state->host_clock_last);
-    /* If this was a vmstate, saved in recording mode,
-       we need to initialize replay data fields. */
-    replay_fetch_data_kind();
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        fseek(replay_file, state->file_offset, SEEK_SET);
+        qemu_clock_set_last(QEMU_CLOCK_HOST, state->host_clock_last);
+        /* If this was a vmstate, saved in recording mode,
+           we need to initialize replay data fields. */
+        replay_fetch_data_kind();
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        /* This is only useful for loading the initial state.
+           Therefore reset all the counters. */
+        state->instructions_count = 0;
+        state->block_request_id = 0;
+    }
 
     return 0;
 }
diff --git a/vl.c b/vl.c
index 3ac4d23..bac495f 100644
--- a/vl.c
+++ b/vl.c
@@ -4619,15 +4619,16 @@ int main(int argc, char **argv, char **envp)
     replay_checkpoint(CHECKPOINT_RESET);
     qemu_system_reset(SHUTDOWN_CAUSE_NONE);
     register_global_state();
-    if (replay_mode != REPLAY_MODE_NONE) {
-        replay_vmstate_init();
-    } else if (loadvm) {
+    if (loadvm) {
         Error *local_err = NULL;
         if (load_snapshot(loadvm, &local_err) < 0) {
             error_report_err(local_err);
             autostart = 0;
         }
     }
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_vmstate_init();
+    }
 
     qdev_prop_check_globals();
     if (vmstate_dump_file) {

[Qemu-devel] [PATCH v6 20/25] replay: wake up vCPU when replaying

[Pavel Dovgalyuk]

In record/replay icount mode vCPU thread and iothread synchronize
the execution using the checkpoints.
vCPU thread processes the virtual timers and iothread processes all others.
When iothread wants to wake up sleeping vCPU thread, it sends dummy queued
work. Therefore it could be the following sequence of the events in
record mode:
 - IO: sending dummy work
 - IO: processing timers
 - CPU: wakeup
 - CPU: clearing dummy work
 - CPU: processing virtual timers

But due to the races in replay mode the sequence may change:
 - IO: sending dummy work
 - CPU: wakeup
 - CPU: clearing dummy work
 - CPU: sleeping again because nothing to do
 - IO: Processing timers
 - CPU: zzzz

In this case vCPU will not wake up, because dummy work is not to be set up
again.

This patch tries to wake up the vCPU when it sleeps and the icount warp
checkpoint isn't met. It means that vCPU has something to do, because
there are no other reasons of non-matching warp checkpoint.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v5: improve checking that vCPU is still sleeping
---
 cpus.c                  |   31 +++++++++++++++++++++----------
 include/sysemu/replay.h |    3 +++
 replay/replay.c         |   12 ++++++++++++
 3 files changed, 36 insertions(+), 10 deletions(-)

diff --git a/cpus.c b/cpus.c
index f31a70a..b12a02f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -576,18 +576,29 @@ void qemu_start_warp_timer(void)
         return;
     }
 
-    /* warp clock deterministically in record/replay mode */
-    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
-        return;
-    }
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        if (!all_cpu_threads_idle()) {
+            return;
+        }
 
-    if (!all_cpu_threads_idle()) {
-        return;
-    }
+        if (qtest_enabled()) {
+            /* When testing, qtest commands advance icount.  */
+            return;
+        }
 
-    if (qtest_enabled()) {
-        /* When testing, qtest commands advance icount.  */
-        return;
+        replay_checkpoint(CHECKPOINT_CLOCK_WARP_START);
+    } else {
+        /* warp clock deterministically in record/replay mode */
+        if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) {
+            /* vCPU is sleeping and warp can't be started.
+               It is probably a race condition: notification sent
+               to vCPU was processed in advance and vCPU went to sleep.
+               Therefore we have to wake it up for doing someting. */
+            if (replay_has_checkpoint()) {
+                qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
+            }
+            return;
+        }
     }
 
     /* We want to use the earliest deadline from ALL vm_clocks */
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index c1646b0..3130e45 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -144,6 +144,9 @@ void replay_shutdown_request(ShutdownCause cause);
     Returns 0 in PLAY mode if checkpoint was not found.
     Returns 1 in all other cases. */
 bool replay_checkpoint(ReplayCheckpoint checkpoint);
+/*! Used to determine that checkpoint is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_checkpoint(void);
 
 /* Asynchronous events queue */
 
diff --git a/replay/replay.c b/replay/replay.c
index a106211..2145686 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -241,6 +241,18 @@ out:
     return res;
 }
 
+bool replay_has_checkpoint(void)
+{
+    bool res = false;
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        g_assert(replay_mutex_locked());
+        replay_account_executed_instructions();
+        res = EVENT_CHECKPOINT <= replay_state.data_kind
+              && replay_state.data_kind <= EVENT_CHECKPOINT_LAST;
+    }
+    return res;
+}
+
 static void replay_enable(const char *fname, int mode)
 {
     const char *fmode = NULL;

[Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[Pavel Dovgalyuk]

This patch makes IDE trim BH deterministic, because it affects
the device state. Therefore its invocation should be replayed
instead of running at the random moment.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 hw/ide/core.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/ide/core.c b/hw/ide/core.c
index 2c62efc..04e22e7 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -35,6 +35,7 @@
 #include "sysemu/block-backend.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
+#include "sysemu/replay.h"
 
 #include "hw/ide/internal.h"
 #include "trace.h"
@@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
 done:
     iocb->aiocb = NULL;
     if (iocb->bh) {
-        qemu_bh_schedule(iocb->bh);
+        replay_bh_schedule_event(iocb->bh);
     }
 }
 

[Qemu-devel] [PATCH v6 22/25] replay: add BH oneshot event for block layer

[Pavel Dovgalyuk]

Replay is capable of recording normal BH events, but sometimes
there are single use callbacks scheduled with aio_bh_schedule_oneshot
function. This patch enables recording and replaying such callbacks.
Block layer uses these events for calling the completion function.
Replaying these calls makes the execution deterministic.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v6:
 - moved stub function to the separate file for fixing linux-user build
---
 block/block-backend.c    |    3 ++-
 include/sysemu/replay.h  |    3 +++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.c          |    2 +-
 stubs/Makefile.objs      |    1 +
 stubs/replay-user.c      |    9 +++++++++
 7 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 stubs/replay-user.c

diff --git a/block/block-backend.c b/block/block-backend.c
index fa12063..712bafd 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -17,6 +17,7 @@
 #include "block/throttle-groups.h"
 #include "sysemu/blockdev.h"
 #include "sysemu/sysemu.h"
+#include "sysemu/replay.h"
 #include "qapi/error.h"
 #include "qapi/qapi-events-block.h"
 #include "qemu/id.h"
@@ -1369,7 +1370,7 @@ static BlockAIOCB *blk_aio_prwv(BlockBackend *blk, int64_t offset, int bytes,
 
     acb->has_returned = true;
     if (acb->rwco.ret != NOT_DONE) {
-        aio_bh_schedule_oneshot(blk_get_aio_context(blk),
+        replay_bh_schedule_oneshot_event(blk_get_aio_context(blk),
                                 blk_aio_complete_bh, acb);
     }
 
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 3130e45..9da90e1 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -160,6 +160,9 @@ bool replay_events_enabled(void);
 void replay_flush_events(void);
 /*! Adds bottom half event to the queue */
 void replay_bh_schedule_event(QEMUBH *bh);
+/*! Adds oneshot bottom half event to the queue */
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque);
 /*! Adds input event to the queue */
 void replay_input_event(QemuConsole *src, InputEvent *evt);
 /*! Adds input sync event to the queue */
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 0964a82..83a7d81 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -37,6 +37,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_BH_ONESHOT:
+        ((QEMUBHFunc*)event->opaque)(event->opaque2);
+        break;
     case REPLAY_ASYNC_EVENT_INPUT:
         qemu_input_event_send_impl(NULL, (InputEvent *)event->opaque);
         qapi_free_InputEvent((InputEvent *)event->opaque);
@@ -132,6 +135,17 @@ void replay_bh_schedule_event(QEMUBH *bh)
     }
 }
 
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque)
+{
+    if (events_enabled) {
+        uint64_t id = replay_get_current_step();
+        replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
+    } else {
+        aio_bh_schedule_oneshot(ctx, cb, opaque);
+    }
+}
+
 void replay_add_input_event(struct InputEvent *event)
 {
     replay_add_event(REPLAY_ASYNC_EVENT_INPUT, event, NULL, 0);
@@ -162,6 +176,7 @@ static void replay_save_event(Event *event, int checkpoint)
         /* save event-specific data */
         switch (event->event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_BH_ONESHOT:
             replay_put_qword(event->id);
             break;
         case REPLAY_ASYNC_EVENT_INPUT:
@@ -216,6 +231,7 @@ static Event *replay_read_event(int checkpoint)
     /* Events that has not to be in the queue */
     switch (replay_state.read_event_kind) {
     case REPLAY_ASYNC_EVENT_BH:
+    case REPLAY_ASYNC_EVENT_BH_ONESHOT:
         if (replay_state.read_event_id == -1) {
             replay_state.read_event_id = replay_get_qword();
         }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index a857ca4..548d4d8 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -51,6 +51,7 @@ enum ReplayEvents {
 
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
+    REPLAY_ASYNC_EVENT_BH_ONESHOT,
     REPLAY_ASYNC_EVENT_INPUT,
     REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_EVENT_CHAR_READ,
diff --git a/replay/replay.c b/replay/replay.c
index 2145686..355e5ae 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -22,7 +22,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe02007
+#define REPLAY_VERSION              0xe02008
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 53d3f32..2acbac3 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -24,6 +24,7 @@ stub-obj-y += monitor.o
 stub-obj-y += notify-event.o
 stub-obj-y += qtest.o
 stub-obj-y += replay.o
+stub-obj-y += replay-user.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
 stub-obj-y += slirp.o
diff --git a/stubs/replay-user.c b/stubs/replay-user.c
new file mode 100644
index 0000000..2ad9e27
--- /dev/null
+++ b/stubs/replay-user.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+#include "sysemu/replay.h"
+#include "sysemu/sysemu.h"
+
+void replay_bh_schedule_oneshot_event(AioContext *ctx,
+    QEMUBHFunc *cb, void *opaque)
+{
+    aio_bh_schedule_oneshot(ctx, cb, opaque);
+}

[Qemu-devel] [PATCH v6 23/25] timer: introduce new virtual clock

[Pavel Dovgalyuk]

Slirp and VNC modules use virtual clock for processing some events that
are related to the guest execution speed.
But virtual clock-related events are consideres to be deterministic and
are recorded/replayed by icount mechanism. But slirp and VNC lie outside
the recorded guest core (which includes CPU and peripherals).
Therefore slirp and VNC are external for the guest, but should work at
guest speed.
This patch introduces new virtual clock which can be used for external
subsystems for running timers that are synchronized with the guest.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 include/qemu/timer.h |    9 +++++++++
 util/qemu-timer.c    |    2 ++
 2 files changed, 11 insertions(+)

diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 39ea907..a005ed2 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -42,6 +42,14 @@
  * In icount mode, this clock counts nanoseconds while the virtual
  * machine is running.  It is used to increase @QEMU_CLOCK_VIRTUAL
  * while the CPUs are sleeping and thus not executing instructions.
+ *
+ * @QEMU_CLOCK_VIRTUAL_EXT: virtual clock for external subsystems
+ *
+ * The virtual clock only runs during the emulation. It stops
+ * when the virtual machine is stopped. The timers for this clock
+ * do not recorded in rr mode, therefore this clock could be used
+ * for the subsystems that operate outside the guest core.
+ *
  */
 
 typedef enum {
@@ -49,6 +57,7 @@ typedef enum {
     QEMU_CLOCK_VIRTUAL = 1,
     QEMU_CLOCK_HOST = 2,
     QEMU_CLOCK_VIRTUAL_RT = 3,
+    QEMU_CLOCK_VIRTUAL_EXT = 4,
     QEMU_CLOCK_MAX
 } QEMUClockType;
 
diff --git a/util/qemu-timer.c b/util/qemu-timer.c
index 86bfe84..eb60d8f 100644
--- a/util/qemu-timer.c
+++ b/util/qemu-timer.c
@@ -496,6 +496,7 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)
 
     switch (timer_list->clock->type) {
     case QEMU_CLOCK_REALTIME:
+    case QEMU_CLOCK_VIRTUAL_EXT:
         break;
     default:
     case QEMU_CLOCK_VIRTUAL:
@@ -597,6 +598,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
         return get_clock();
     default:
     case QEMU_CLOCK_VIRTUAL:
+    case QEMU_CLOCK_VIRTUAL_EXT:
         if (use_icount) {
             return cpu_get_icount();
         } else {

[Qemu-devel] [PATCH v6 24/25] slirp: fix ipv6 timers

[Pavel Dovgalyuk]

ICMP implementation for IPv6 uses timers based on virtual clock.
This is incorrect because this service is not related to the guest state,
and its events should not be recorded and replayed.
This patch changes using virtual clock to the new virtual_ext clock.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 slirp/ip6_icmp.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/slirp/ip6_icmp.c b/slirp/ip6_icmp.c
index ee333d0..3f41187 100644
--- a/slirp/ip6_icmp.c
+++ b/slirp/ip6_icmp.c
@@ -17,7 +17,7 @@ static void ra_timer_handler(void *opaque)
 {
     Slirp *slirp = opaque;
     timer_mod(slirp->ra_timer,
-              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + NDP_Interval);
+              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT) + NDP_Interval);
     ndp_send_ra(slirp);
 }
 
@@ -27,9 +27,10 @@ void icmp6_init(Slirp *slirp)
         return;
     }
 
-    slirp->ra_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, ra_timer_handler, slirp);
+    slirp->ra_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_EXT,
+                                   ra_timer_handler, slirp);
     timer_mod(slirp->ra_timer,
-              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + NDP_Interval);
+              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT) + NDP_Interval);
 }
 
 void icmp6_cleanup(Slirp *slirp)

[Qemu-devel] [PATCH v6 25/25] ui: fix virtual timers

[Pavel Dovgalyuk]

UI uses timers based on virtual clock for managing key queue.
This is incorrect because this service is not related to the guest state,
and its events should not be recorded and replayed. But these timers should
stop when the guest is not executing.
This patch changes using virtual clock to the new virtual_ext clock,
which runs as virtual clock, but its timers are not saved to the log.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
---
 ui/input.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/ui/input.c b/ui/input.c
index 51b1019..dd7f6d7 100644
--- a/ui/input.c
+++ b/ui/input.c
@@ -271,7 +271,7 @@ static void qemu_input_queue_process(void *opaque)
         item = QTAILQ_FIRST(queue);
         switch (item->type) {
         case QEMU_INPUT_QUEUE_DELAY:
-            timer_mod(item->timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL)
+            timer_mod(item->timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT)
                       + item->delay_ms);
             return;
         case QEMU_INPUT_QUEUE_EVENT:
@@ -301,7 +301,7 @@ static void qemu_input_queue_delay(struct QemuInputEventQueueHead *queue,
     queue_count++;
 
     if (start_timer) {
-        timer_mod(item->timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL)
+        timer_mod(item->timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT)
                   + item->delay_ms);
     }
 }
@@ -448,8 +448,8 @@ void qemu_input_event_send_key_delay(uint32_t delay_ms)
     }
 
     if (!kbd_timer) {
-        kbd_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, qemu_input_queue_process,
-                                 &kbd_queue);
+        kbd_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_EXT,
+                                 qemu_input_queue_process, &kbd_queue);
     }
     if (queue_count < queue_limit) {
         qemu_input_queue_delay(&kbd_queue, kbd_timer,

Re: [Qemu-devel] [PATCH v6 24/25] slirp: fix ipv6 timers

[Samuel Thibault]

Pavel Dovgalyuk, le mer. 12 sept. 2018 11:20:07 +0300, a ecrit:
> ICMP implementation for IPv6 uses timers based on virtual clock.
> This is incorrect because this service is not related to the guest state,
> and its events should not be recorded and replayed.
> This patch changes using virtual clock to the new virtual_ext clock.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

As discussed previously

Reviewed-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

> ---
>  slirp/ip6_icmp.c |    7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/slirp/ip6_icmp.c b/slirp/ip6_icmp.c
> index ee333d0..3f41187 100644
> --- a/slirp/ip6_icmp.c
> +++ b/slirp/ip6_icmp.c
> @@ -17,7 +17,7 @@ static void ra_timer_handler(void *opaque)
>  {
>      Slirp *slirp = opaque;
>      timer_mod(slirp->ra_timer,
> -              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + NDP_Interval);
> +              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT) + NDP_Interval);
>      ndp_send_ra(slirp);
>  }
>  
> @@ -27,9 +27,10 @@ void icmp6_init(Slirp *slirp)
>          return;
>      }
>  
> -    slirp->ra_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, ra_timer_handler, slirp);
> +    slirp->ra_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_EXT,
> +                                   ra_timer_handler, slirp);
>      timer_mod(slirp->ra_timer,
> -              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + NDP_Interval);
> +              qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_EXT) + NDP_Interval);
>  }
>  
>  void icmp6_cleanup(Slirp *slirp)
> 
> 

-- 
Samuel
<T> csp.tar.gz:     ascii text
 -+- #ens-mim - vive les browsers qui prennent des initiatives à la con -+-

Re: [Qemu-devel] [PATCH v6 20/25] replay: wake up vCPU when replaying

[Paolo Bonzini]

On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
> This patch tries to wake up the vCPU when it sleeps and the icount warp
> checkpoint isn't met. It means that vCPU has something to do, because
> there are no other reasons of non-matching warp checkpoint.

What happens if !replay_has_checkpoint()?  Should that be an assertion?

Thanks,

Paolo

Re: [Qemu-devel] [PATCH v6 22/25] replay: add BH oneshot event for block layer

[Paolo Bonzini]

On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
> + uint64_t id = replay_get_current_step();
> + replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);

Why does it need an id, while REPLAY_ASYNC_EVENT_BH does not?

Paolo

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Paolo Bonzini]

On 12/09/2018 10:17, Pavel Dovgalyuk wrote:
> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
> 
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
> 
> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
> 
> The set of patches include the following modifications:
>  - fixes of record/replay caused by the QEMU core changes
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
>  - adding new clock for correct timer events from vnc and slirp
>  - other record/replay fixes
> 
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-180911
> 
> v6 changes:
>  - rebased to the new version of master
>  - fixed build of linux-user configurations
>  - added new clock for slirp and vnc timers
> 
> v5 changes:
>  - multiple fixes of record/replay bugs appeared after QEMU core update
>  - changed reverse debugging to 'since 3.1'
> 
> v4 changes:
>  - changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
> 
> v3 changes:
>  - Fixed PS/2 bug with save/load vm, which caused failures of the replay.
>  - Rebased to the new code base.
>  - Minor fixes.
> 
> v2 changes:
>  - documented reverse debugging
>  - fixed start vmstate loading in record mode
>  - documented qcow2 changes (as suggested by Eric Blake)
>  - made icount SnapshotInfo field optional (as suggested by Eric Blake)
>  - renamed qmp commands (as suggested by Eric Blake)
>  - minor changes
> 
> ---
> 
> Pavel Dovgalyuk (25):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       qcow2: introduce icount field for snapshots
>       migration: introduce icount field for snapshots
>       replay: provide and accessor for rr filename
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay-seek command to proceed to the desired step
>       replay: flush events when exiting
>       replay: refine replay-time module
>       translator: fix breakpoint processing
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>       replay: describe reverse debugging in docs/replay.txt
>       replay: allow loading any snapshots before recording
>       replay: wake up vCPU when replaying
>       replay: replay BH for IDE trim operation
>       replay: add BH oneshot event for block layer
>       timer: introduce new virtual clock
>       slirp: fix ipv6 timers
>       ui: fix virtual timers
> 
> 
>  accel/tcg/translator.c    |    9 +
>  block/blkreplay.c         |    8 +
>  block/block-backend.c     |    3 
>  block/io.c                |   22 +++
>  block/qapi.c              |   17 ++-
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    2 
>  blockdev.c                |   10 ++
>  cpus.c                    |   50 +++++---
>  docs/interop/qcow2.txt    |    4 +
>  docs/replay.txt           |   45 +++++++
>  exec.c                    |    6 +
>  gdbstub.c                 |   50 +++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   30 +++++
>  hmp.h                     |    3 
>  hw/ide/core.c             |    3 
>  include/block/snapshot.h  |    1 
>  include/qemu/timer.h      |    9 +
>  include/sysemu/replay.h   |   26 ++++
>  migration/savevm.c        |   15 +-
>  qapi/block-core.json      |    5 +
>  qapi/block.json           |    3 
>  qapi/misc.json            |   68 +++++++++++
>  replay/Makefile.objs      |    3 
>  replay/replay-debugging.c |  287 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-events.c    |   30 +++--
>  replay/replay-internal.h  |    9 +
>  replay/replay-snapshot.c  |   17 ++-
>  replay/replay-time.c      |   32 ++---
>  replay/replay.c           |   38 ++++++
>  slirp/ip6_icmp.c          |    7 +
>  stubs/Makefile.objs       |    1 
>  stubs/replay-user.c       |    9 +
>  stubs/replay.c            |   10 ++
>  ui/input.c                |    8 +
>  util/qemu-timer.c         |    2 
>  vl.c                      |   18 ++-
>  38 files changed, 791 insertions(+), 92 deletions(-)
>  create mode 100644 replay/replay-debugging.c
>  create mode 100644 stubs/replay-user.c
> 

For now I'm queuing 12, 14, 19, 20 (pending question to you) and 23-25.

Kevin, can you take a look at patches 1-5?  I cannot quite evaluate if 4
has any scary ramifications.

Paolo

Re: [Qemu-devel] [PATCH v6 20/25] replay: wake up vCPU when replaying

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
> > This patch tries to wake up the vCPU when it sleeps and the icount warp
> > checkpoint isn't met. It means that vCPU has something to do, because
> > there are no other reasons of non-matching warp checkpoint.
> 
> What happens if !replay_has_checkpoint()?  Should that be an assertion?

The condition may be true, only when vCPU thread is sleeping.
In all other cases (e.g., running) the condition is false and we
have nothing to do.

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 22/25] replay: add BH oneshot event for block layer

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
> > + uint64_t id = replay_get_current_step();
> > + replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
> 
> Why does it need an id, while REPLAY_ASYNC_EVENT_BH does not?

Because _oneshot() function takes only context and callback,
and BH object is created inside that function.
Therefore to use the normal bh scheduling events we have to refactor
block code - add bh creation, scheduling, and freeing.

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 12/09/2018 10:17, Pavel Dovgalyuk wrote:
> > GDB remote protocol supports reverse debugging of the targets.
> > It includes 'reverse step' and 'reverse continue' operations.
> > The first one finds the previous step of the execution,
> > and the second one is intended to stop at the last breakpoint that
> > would happen when the program is executed normally.
> >
> > Reverse debugging is possible in the replay mode, when at least
> > one snapshot was created at the record or replay phase.
> > QEMU can use these snapshots for travelling back in time with GDB.
> >
> > Running the execution in replay mode allows using GDB reverse debugging
> > commands:
> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >    instruction forward. When that step is reaches, execution stops.
> >  - reverse-continue (or rc): Runs execution "backwards".
> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >    and replaying the execution. Then QEMU loads snapshots again and
> >    replays to the latest breakpoint. When there are no breakpoints in
> >    the examined section of the execution, QEMU finds one more snapshot
> >    and tries again. After the first snapshot is processed, execution
> >    stops at this snapshot.
> >
> > The set of patches include the following modifications:
> >  - fixes of record/replay caused by the QEMU core changes
> >  - gdbstub update for reverse debugging support
> >  - functions that automatically perform reverse step and reverse
> >    continue operations
> >  - hmp/qmp commands for manipulating the replay process
> >  - improvement of the snapshotting for saving the execution step
> >    in the snapshot parameters
> >  - adding new clock for correct timer events from vnc and slirp
> >  - other record/replay fixes
> >
> > The patches are available in the repository:
> > https://github.com/ispras/qemu/tree/rr-180911
> >
> > v6 changes:
> >  - rebased to the new version of master
> >  - fixed build of linux-user configurations
> >  - added new clock for slirp and vnc timers
> >
> > v5 changes:
> >  - multiple fixes of record/replay bugs appeared after QEMU core update
> >  - changed reverse debugging to 'since 3.1'
> >
> > v4 changes:
> >  - changed 'since 2.13' to 'since 3.0' in json (as suggested by Eric Blake)
> >
> > v3 changes:
> >  - Fixed PS/2 bug with save/load vm, which caused failures of the replay.
> >  - Rebased to the new code base.
> >  - Minor fixes.
> >
> > v2 changes:
> >  - documented reverse debugging
> >  - fixed start vmstate loading in record mode
> >  - documented qcow2 changes (as suggested by Eric Blake)
> >  - made icount SnapshotInfo field optional (as suggested by Eric Blake)
> >  - renamed qmp commands (as suggested by Eric Blake)
> >  - minor changes
> >
> > ---
> >
> > Pavel Dovgalyuk (25):
> >       block: implement bdrv_snapshot_goto for blkreplay
> >       replay: disable default snapshot for record/replay
> >       replay: update docs for record/replay with block devices
> >       replay: don't drain/flush bdrv queue while RR is working
> >       replay: finish record/replay before closing the disks
> >       qcow2: introduce icount field for snapshots
> >       migration: introduce icount field for snapshots
> >       replay: provide and accessor for rr filename
> >       replay: introduce info hmp/qmp command
> >       replay: introduce breakpoint at the specified step
> >       replay: implement replay-seek command to proceed to the desired step
> >       replay: flush events when exiting
> >       replay: refine replay-time module
> >       translator: fix breakpoint processing
> >       replay: flush rr queue before loading the vmstate
> >       gdbstub: add reverse step support in replay mode
> >       gdbstub: add reverse continue support in replay mode
> >       replay: describe reverse debugging in docs/replay.txt
> >       replay: allow loading any snapshots before recording
> >       replay: wake up vCPU when replaying
> >       replay: replay BH for IDE trim operation
> >       replay: add BH oneshot event for block layer
> >       timer: introduce new virtual clock
> >       slirp: fix ipv6 timers
> >       ui: fix virtual timers
> >
> >
> >  accel/tcg/translator.c    |    9 +
> >  block/blkreplay.c         |    8 +
> >  block/block-backend.c     |    3
> >  block/io.c                |   22 +++
> >  block/qapi.c              |   17 ++-
> >  block/qcow2-snapshot.c    |    9 +
> >  block/qcow2.h             |    2
> >  blockdev.c                |   10 ++
> >  cpus.c                    |   50 +++++---
> >  docs/interop/qcow2.txt    |    4 +
> >  docs/replay.txt           |   45 +++++++
> >  exec.c                    |    6 +
> >  gdbstub.c                 |   50 +++++++-
> >  hmp-commands-info.hx      |   14 ++
> >  hmp-commands.hx           |   30 +++++
> >  hmp.h                     |    3
> >  hw/ide/core.c             |    3
> >  include/block/snapshot.h  |    1
> >  include/qemu/timer.h      |    9 +
> >  include/sysemu/replay.h   |   26 ++++
> >  migration/savevm.c        |   15 +-
> >  qapi/block-core.json      |    5 +
> >  qapi/block.json           |    3
> >  qapi/misc.json            |   68 +++++++++++
> >  replay/Makefile.objs      |    3
> >  replay/replay-debugging.c |  287 +++++++++++++++++++++++++++++++++++++++++++++
> >  replay/replay-events.c    |   30 +++--
> >  replay/replay-internal.h  |    9 +
> >  replay/replay-snapshot.c  |   17 ++-
> >  replay/replay-time.c      |   32 ++---
> >  replay/replay.c           |   38 ++++++
> >  slirp/ip6_icmp.c          |    7 +
> >  stubs/Makefile.objs       |    1
> >  stubs/replay-user.c       |    9 +
> >  stubs/replay.c            |   10 ++
> >  ui/input.c                |    8 +
> >  util/qemu-timer.c         |    2
> >  vl.c                      |   18 ++-
> >  38 files changed, 791 insertions(+), 92 deletions(-)
> >  create mode 100644 replay/replay-debugging.c
> >  create mode 100644 stubs/replay-user.c
> >
> 
> For now I'm queuing 12, 14, 19, 20 (pending question to you) and 23-25.

What about patch 21?

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 22/25] replay: add BH oneshot event for block layer

[Paolo Bonzini]

On 13/09/2018 13:13, Pavel Dovgalyuk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
>>> + uint64_t id = replay_get_current_step();
>>> + replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
>>
>> Why does it need an id, while REPLAY_ASYNC_EVENT_BH does not?
> 
> Because _oneshot() function takes only context and callback,
> and BH object is created inside that function.
> Therefore to use the normal bh scheduling events we have to refactor
> block code - add bh creation, scheduling, and freeing.

No, I'm asking only why the id is 0 for REPLAY_ASYNC_EVENT_BH and
replay_get_current_step() for REPLAY_ASYNC_EVENT_BH_ONESHOT.

Thanks,

Paolo

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Paolo Bonzini]

On 13/09/2018 15:40, Pavel Dovgalyuk wrote:
>> For now I'm queuing 12, 14, 19, 20 (pending question to you) and 23-25.
> What about patch 21?

I'd want an ACK from the IDE maintainer.  Let's add him to Cc.

Paolo

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[John Snow]

On 09/12/2018 04:19 AM, Pavel Dovgalyuk wrote:
> This patch makes IDE trim BH deterministic, because it affects
> the device state. Therefore its invocation should be replayed
> instead of running at the random moment.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  hw/ide/core.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/ide/core.c b/hw/ide/core.c
> index 2c62efc..04e22e7 100644
> --- a/hw/ide/core.c
> +++ b/hw/ide/core.c
> @@ -35,6 +35,7 @@
>  #include "sysemu/block-backend.h"
>  #include "qapi/error.h"
>  #include "qemu/cutils.h"
> +#include "sysemu/replay.h"
>  
>  #include "hw/ide/internal.h"
>  #include "trace.h"
> @@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
>  done:
>      iocb->aiocb = NULL;
>      if (iocb->bh) {
> -        qemu_bh_schedule(iocb->bh);
> +        replay_bh_schedule_event(iocb->bh);
>      }
>  }
>  
> 
> 

Just passing by: Why do we need to change this call, but nothing else in
IDE?

I don't mind conceptually, but it's odd to me that of all the calls I
make in this emulator that change state somewhere that this is the only
one you need to hijack for the replay feature.

Is this a necessarily complete change?

--js

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[Pavel Dovgalyuk]

> From: John Snow [mailto:jsnow@redhat.com]
> On 09/12/2018 04:19 AM, Pavel Dovgalyuk wrote:
> > This patch makes IDE trim BH deterministic, because it affects
> > the device state. Therefore its invocation should be replayed
> > instead of running at the random moment.
> >
> > Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > ---
> >  hw/ide/core.c |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/ide/core.c b/hw/ide/core.c
> > index 2c62efc..04e22e7 100644
> > --- a/hw/ide/core.c
> > +++ b/hw/ide/core.c
> > @@ -35,6 +35,7 @@
> >  #include "sysemu/block-backend.h"
> >  #include "qapi/error.h"
> >  #include "qemu/cutils.h"
> > +#include "sysemu/replay.h"
> >
> >  #include "hw/ide/internal.h"
> >  #include "trace.h"
> > @@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
> >  done:
> >      iocb->aiocb = NULL;
> >      if (iocb->bh) {
> > -        qemu_bh_schedule(iocb->bh);
> > +        replay_bh_schedule_event(iocb->bh);
> >      }
> >  }
> >
> Just passing by: Why do we need to change this call, but nothing else in
> IDE?

This call is responsible for a bug that was reproducible.

> I don't mind conceptually, but it's odd to me that of all the calls I
> make in this emulator that change state somewhere that this is the only
> one you need to hijack for the replay feature.
> 
> Is this a necessarily complete change?

Maybe not. We can hardly analyze all peripheral devices code and fix all the calls.
But I think we can improve that patch and at least look through ide core to fix other calls.

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 22/25] replay: add BH oneshot event for block layer

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 13/09/2018 13:13, Pavel Dovgalyuk wrote:
> >> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> >> On 12/09/2018 10:19, Pavel Dovgalyuk wrote:
> >>> + uint64_t id = replay_get_current_step();
> >>> + replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
> >>
> >> Why does it need an id, while REPLAY_ASYNC_EVENT_BH does not?
> >
> > Because _oneshot() function takes only context and callback,
> > and BH object is created inside that function.
> > Therefore to use the normal bh scheduling events we have to refactor
> > block code - add bh creation, scheduling, and freeing.
> 
> No, I'm asking only why the id is 0 for REPLAY_ASYNC_EVENT_BH and
> replay_get_current_step() for REPLAY_ASYNC_EVENT_BH_ONESHOT.
> 
> Thanks,
> 
> Paolo

Sorry, I missed your point.
Maybe you looked at the wrong function, because both of these events have an id:

void replay_bh_schedule_event(QEMUBH *bh)
{
    if (events_enabled) {
        uint64_t id = replay_get_current_step();
        replay_add_event(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
    } else {
        qemu_bh_schedule(bh);
    }
}

void replay_bh_schedule_oneshot_event(AioContext *ctx,
    QEMUBHFunc *cb, void *opaque)
{
    if (events_enabled) {
        uint64_t id = replay_get_current_step();
        replay_add_event(REPLAY_ASYNC_EVENT_BH_ONESHOT, cb, opaque, id);
    } else {
        aio_bh_schedule_oneshot(ctx, cb, opaque);
    }
}

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[Pavel Dovgalyuk]

> From: Pavel Dovgalyuk [mailto:dovgaluk@ispras.ru]
> > From: John Snow [mailto:jsnow@redhat.com]
> > On 09/12/2018 04:19 AM, Pavel Dovgalyuk wrote:
> > > This patch makes IDE trim BH deterministic, because it affects
> > > the device state. Therefore its invocation should be replayed
> > > instead of running at the random moment.
> > >
> > > Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> > > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > > ---
> > >  hw/ide/core.c |    3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/hw/ide/core.c b/hw/ide/core.c
> > > index 2c62efc..04e22e7 100644
> > > --- a/hw/ide/core.c
> > > +++ b/hw/ide/core.c
> > > @@ -35,6 +35,7 @@
> > >  #include "sysemu/block-backend.h"
> > >  #include "qapi/error.h"
> > >  #include "qemu/cutils.h"
> > > +#include "sysemu/replay.h"
> > >
> > >  #include "hw/ide/internal.h"
> > >  #include "trace.h"
> > > @@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
> > >  done:
> > >      iocb->aiocb = NULL;
> > >      if (iocb->bh) {
> > > -        qemu_bh_schedule(iocb->bh);
> > > +        replay_bh_schedule_event(iocb->bh);
> > >      }
> > >  }
> > >
> > Just passing by: Why do we need to change this call, but nothing else in
> > IDE?
> 
> This call is responsible for a bug that was reproducible.
> 
> > I don't mind conceptually, but it's odd to me that of all the calls I
> > make in this emulator that change state somewhere that this is the only
> > one you need to hijack for the replay feature.
> >
> > Is this a necessarily complete change?


I found one more BH in ide/core:

static void ide_restart_cb(void *opaque, int running, RunState state)
{
    IDEBus *bus = opaque;

    if (!running)
        return;

    if (!bus->bh) {
        bus->bh = qemu_bh_new(ide_restart_bh, bus);
        qemu_bh_schedule(bus->bh);
    }
}

void ide_register_restart_cb(IDEBus *bus)
{
    if (bus->dma->ops->restart_dma) {
        bus->vmstate = qemu_add_vm_change_state_handler(ide_restart_cb, bus);
    }
}

As I understand, it is called when VM start/stop event happen.
These events are not related to the guest state.

Does this BH change the guest state somehow?

Pavel Dovgalyuk

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[John Snow]

On 09/14/2018 01:48 AM, Pavel Dovgalyuk wrote:
>> From: John Snow [mailto:jsnow@redhat.com]
>> On 09/12/2018 04:19 AM, Pavel Dovgalyuk wrote:
>>> This patch makes IDE trim BH deterministic, because it affects
>>> the device state. Therefore its invocation should be replayed
>>> instead of running at the random moment.
>>>
>>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>>> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
>>> ---
>>>  hw/ide/core.c |    3 ++-
>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/hw/ide/core.c b/hw/ide/core.c
>>> index 2c62efc..04e22e7 100644
>>> --- a/hw/ide/core.c
>>> +++ b/hw/ide/core.c
>>> @@ -35,6 +35,7 @@
>>>  #include "sysemu/block-backend.h"
>>>  #include "qapi/error.h"
>>>  #include "qemu/cutils.h"
>>> +#include "sysemu/replay.h"
>>>
>>>  #include "hw/ide/internal.h"
>>>  #include "trace.h"
>>> @@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
>>>  done:
>>>      iocb->aiocb = NULL;
>>>      if (iocb->bh) {
>>> -        qemu_bh_schedule(iocb->bh);
>>> +        replay_bh_schedule_event(iocb->bh);
>>>      }
>>>  }
>>>
>> Just passing by: Why do we need to change this call, but nothing else in
>> IDE?
> 
> This call is responsible for a bug that was reproducible.
> 
>> I don't mind conceptually, but it's odd to me that of all the calls I
>> make in this emulator that change state somewhere that this is the only
>> one you need to hijack for the replay feature.
>>
>> Is this a necessarily complete change?
> 
> Maybe not. We can hardly analyze all peripheral devices code and fix all the calls.
> But I think we can improve that patch and at least look through ide core to fix other calls.
> 
> Pavel Dovgalyuk
> 

It just seems odd that if you're working on a replay mechanism that
requires you to intercept my QEMU API calls that you're only changing a
trim callback.

I'd kind of expect that you don't need to intercept any, unless these
are legacy calls that I shouldn't be making at all and you have a more
generic intercept somewhere deeper in the codebase.

In that case, I really ought to hustle off of my use of legacy calls.

What are the criteria for things you need to intercept/wrap?

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[John Snow]

On 09/14/2018 03:27 AM, Pavel Dovgalyuk wrote:
>> From: Pavel Dovgalyuk [mailto:dovgaluk@ispras.ru]
>>> From: John Snow [mailto:jsnow@redhat.com]
>>> On 09/12/2018 04:19 AM, Pavel Dovgalyuk wrote:
>>>> This patch makes IDE trim BH deterministic, because it affects
>>>> the device state. Therefore its invocation should be replayed
>>>> instead of running at the random moment.
>>>>
>>>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>>>> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
>>>> ---
>>>>  hw/ide/core.c |    3 ++-
>>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/hw/ide/core.c b/hw/ide/core.c
>>>> index 2c62efc..04e22e7 100644
>>>> --- a/hw/ide/core.c
>>>> +++ b/hw/ide/core.c
>>>> @@ -35,6 +35,7 @@
>>>>  #include "sysemu/block-backend.h"
>>>>  #include "qapi/error.h"
>>>>  #include "qemu/cutils.h"
>>>> +#include "sysemu/replay.h"
>>>>
>>>>  #include "hw/ide/internal.h"
>>>>  #include "trace.h"
>>>> @@ -479,7 +480,7 @@ static void ide_issue_trim_cb(void *opaque, int ret)
>>>>  done:
>>>>      iocb->aiocb = NULL;
>>>>      if (iocb->bh) {
>>>> -        qemu_bh_schedule(iocb->bh);
>>>> +        replay_bh_schedule_event(iocb->bh);
>>>>      }
>>>>  }
>>>>
>>> Just passing by: Why do we need to change this call, but nothing else in
>>> IDE?
>>
>> This call is responsible for a bug that was reproducible.
>>
>>> I don't mind conceptually, but it's odd to me that of all the calls I
>>> make in this emulator that change state somewhere that this is the only
>>> one you need to hijack for the replay feature.
>>>
>>> Is this a necessarily complete change?
> 
> 
> I found one more BH in ide/core:
> 
> static void ide_restart_cb(void *opaque, int running, RunState state)
> {
>     IDEBus *bus = opaque;
> 
>     if (!running)
>         return;
> 
>     if (!bus->bh) {
>         bus->bh = qemu_bh_new(ide_restart_bh, bus);
>         qemu_bh_schedule(bus->bh);
>     }
> }
> 
> void ide_register_restart_cb(IDEBus *bus)
> {
>     if (bus->dma->ops->restart_dma) {
>         bus->vmstate = qemu_add_vm_change_state_handler(ide_restart_cb, bus);
>     }
> }
> 
> As I understand, it is called when VM start/stop event happen.
> These events are not related to the guest state.
> 
> Does this BH change the guest state somehow?
> 
> Pavel Dovgalyuk
> 
> 

Shouldn't change guest state all by itself.

ide_restart_bh does, though. (Changes device registers, can cause block
IO to occur, etc.)

--js

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[Paolo Bonzini]

On 14/09/2018 16:00, John Snow wrote:
>> Maybe not. We can hardly analyze all peripheral devices code and fix all the calls.
>> But I think we can improve that patch and at least look through ide core to fix other calls.
>>
>> Pavel Dovgalyuk
>>
> It just seems odd that if you're working on a replay mechanism that
> requires you to intercept my QEMU API calls that you're only changing a
> trim callback.

You need it only here because the block layer is already calling
replay_bh_schedule_event (actually replay_bh_schedule_oneshot_event
after patch 22 of this series) on reads and writes.

Paolo

> I'd kind of expect that you don't need to intercept any, unless these
> are legacy calls that I shouldn't be making at all and you have a more
> generic intercept somewhere deeper in the codebase.
> 
> In that case, I really ought to hustle off of my use of legacy calls.
> 
> What are the criteria for things you need to intercept/wrap?

Re: [Qemu-devel] [PATCH v6 21/25] replay: replay BH for IDE trim operation

[Peter Maydell]

On 14 September 2018 at 16:19, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 14/09/2018 16:00, John Snow wrote:
>>> Maybe not. We can hardly analyze all peripheral devices code and fix all the calls.
>>> But I think we can improve that patch and at least look through ide core to fix other calls.
>>>
>>> Pavel Dovgalyuk
>>>
>> It just seems odd that if you're working on a replay mechanism that
>> requires you to intercept my QEMU API calls that you're only changing a
>> trim callback.
>
> You need it only here because the block layer is already calling
> replay_bh_schedule_event (actually replay_bh_schedule_oneshot_event
> after patch 22 of this series) on reads and writes.

Do we have documentation describing when a device model needs to care
about record/replay ?

thanks
-- PMM

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Artem Pisarenko]

It wasn't so easy to apply this patch due to problems in compilation of
version you pointed to, and due to content distortions introduced by mail
archive, but I got it worked finally :)

Applying this patch finally made all my tests succeed... almost :)

Now qemu may hang in random moment of emulation, but not hard. Symptoms
looks like I've already reported here:
https://bugs.launchpad.net/qemu/+bug/1790460 . So, this isn't
record/replay-specific. Although, without rr= option I wasn't able cause
this issue to reveal itself, but it doesn't make much sense due to
instability of issue's nature and its hard reproducibility.

Commit I tested against (with patches
applied): 53a19a9a5f9811a911e9b69ef36afb0d66b5d85c .


вт, 9 окт. 2018 г. в 17:26, Pavel Dovgalyuk <dovgaluk@ispras.ru>:

> Maybe this will help?
>
>
>
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg560780.html
>
>
>
> Pavel Dovgalyuk
>
>
>
> *From:* Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com]
> *Sent:* Tuesday, October 09, 2018 2:24 PM
> *To:* Pavel Dovgalyuk
>
>
> *Cc:* Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
> *Subject:* Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and
> adding reverse debugging
>
>
>
> (Since all previous patches are already merged to master, I'm running
> tests against latest (almost) version from master branch. Following results
> are based on master commit dafd95053611aa14dda40266857608d12ddce658 .)
>
>
>
> Applying this patch made Tests 1 and 2 succeed (at least I wasn't able to
> acheive failures with several attempts).
>
> Also I've tried few tests without sleep=off and/or rtc base options. All
> of them succeed too, except one case - removing sleep=off (regardless of
> -rtc option values or its presence at all) causes qemu to hang hard in
> recording mode at very startup. Process needs to be killed.
>
>
>
> Some info from debugger:
>
>     qemu-system-x86_64 [13231] [cores: 2,4,5,7]
>
>           Thread #1 [qemu-system-x86] 13231 [core: 2] (Suspended :
> Container)
>
>                       __lll_lock_wait() at lowlevellock.S:135
> 0x7f00b116626d
>
>                       __GI___pthread_mutex_lock() at
> pthread_mutex_lock.c:80 0x7f00b115fdbd
>
>                       qemu_mutex_lock_impl() at qemu-thread-posix.c:66
> 0x947ac4
>
>                       replay_mutex_lock() at replay-internal.c:206
> 0x7f3dea
>
>                       os_host_main_loop_wait() at main-loop.c:235
> 0x94335e
>
>                       main_loop_wait() at main-loop.c:497 0x943429
>
>                       main_loop() at vl.c:1,853 0x5be70f
>
>                       main() at vl.c:4,575 0x5c56e0
>
>           Thread #2 [qemu-system-x86] 13282 [core: 4] (Suspended :
> Container)
>
>           Thread #3 [qemu-system-x86] 13283 [core: 5] (Suspended :
> Container)
>
>           Thread #4 [qemu-system-x86] 13284 [core: 7] (Suspended : Step)
>
>                       cpu_get_icount_raw() at cpus.c:301 0x45a0a0
>
>                       replay_get_current_step() at replay.c:67 0x7f2f14
>
>                       replay_save_instructions() at replay-internal.c:225
> 0x7f3ea0
>
>                       replay_save_clock() at replay-time.c:24 0x7f483d
>
>                       icount_warp_rt() at cpus.c:512 0x45a745
>
>                       qemu_account_warp_timer() at cpus.c:690
> 0x45ad55
>
>                       qemu_tcg_rr_cpu_thread_fn() at cpus.c:1,498
> 0x45c554
>
>                       qemu_thread_start() at qemu-thread-posix.c:504
> 0x9485cf
>
>                       start_thread() at pthread_create.c:333
> 0x7f00b115d6ba
>
>                       clone() at clone.S:109 0x7f00b0e9341d
>
>     gdb (7.11.1)
>
>
>
> Threads #2,3 are just waiting in poll or similar. Nothing extraordinary.
>
>
>
> Thread #4 cycles inside do {} while() loop of cpu_get_icount_raw()
> function:
>
>     do {
>
>         start = seqlock_read_begin(&timers_state.vm_clock_seqlock);
>
>         icount = cpu_get_icount_raw_locked();
>
>     } while (seqlock_read_retry(&timers_state.vm_clock_seqlock, start));
>
>
>
> Value of timers_state.vm_clock_seqlock.sequence is always 3.
>
>
>
> вт, 9 окт. 2018 г. в 15:04, Pavel Dovgalyuk <dovgaluk@ispras.ru>:
>
> Please try the following patch.
>
> There was a problem with rtc option in record/replay mode.
>
>
>
> diff --git a/vl.c b/vl.c
>
> index 40d5d0f..afe1c20 100644
>
> --- a/vl.c
>
> +++ b/vl.c
>
> @@ -2885,6 +2885,7 @@ int main(int argc, char **argv, char **envp)
>
>      DisplayState *ds;
>
>      QemuOpts *opts, *machine_opts;
>
>      QemuOpts *icount_opts = NULL, *accel_opts = NULL;
>
> +    QemuOpts *rtc_opts = NULL;
>
>      QemuOptsList *olist;
>
>      int optind;
>
>      const char *optarg;
>
> @@ -3691,12 +3692,11 @@ int main(int argc, char **argv, char **envp)
>
>                  warn_report("This option is ignored and will be removed
> soon");
>
>                  break;
>
>              case QEMU_OPTION_rtc:
>
> -                opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),
> optarg,
>
> -                                               false);
>
> -                if (!opts) {
>
> +                rtc_opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),
>
> +                                                   optarg, false);
>
> +                if (!rtc_opts) {
>
>                      exit(1);
>
>                  }
>
> -                configure_rtc(opts);
>
>                  break;
>
>              case QEMU_OPTION_tb_size:
>
> #ifndef CONFIG_TCG
>
> @@ -3907,6 +3907,9 @@ int main(int argc, char **argv, char **envp)
>
>      loc_set_none();
>
>      replay_configure(icount_opts);
>
> +    if (rtc_opts) {
>
> +        configure_rtc(rtc_opts);
>
> +    }
>
>      if (incoming && !preconfig_exit_requested) {
>
>          error_report("'preconfig' and 'incoming' options are "
>
>
>
> Pavel Dovgalyuk
>
>
>
> *From:* Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com]
> *Sent:* Thursday, October 04, 2018 4:16 PM
> *To:* dovgaluk
> *Cc:* Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
> *Subject:* Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and
> adding reverse debugging
>
>
>
> No, it didn't changed test results, at least for
> https://github.com/ispras/qemu/tree/rr-180911 . Even step values it
> stucks on are same for most runs.
>
> Playing with master and my own branch gives different results for tests
> without sleep=off and -rtc base. It seems that patch you mentioned didn't
> changed them very much.
>
> The only thing can be said for sure, is that this patch does not fix
> issues completely. But MAY fix them partially or in some other specific
> cases...
>
>
>
> ср, 3 окт. 2018 г. в 12:47, dovgaluk <dovgaluk@ispras.ru>:
>
> Can you try applying this patch?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html
>
> I also encountered the problems with x86_64 replaying and found the
> misprint in
> the code which was fixed later, than sending the series to the mailing
> list.
>
> Pavel Dovgalyuk
>
>
> Artem Pisarenko писал 2018-10-02 10:02:
> > I've added "-monitor stdio" option to command line of Test 1 and
> > repeated entering command during execution:
> >
> >   QEMU 3.0.50 monitor - type 'help' for more information
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 311736195
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 318198367
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 324737211
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 329890795
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   ...
> >
> > Some notes on value of step it stucks on:
> > - mostly it's same (even across different record-replay pairs);
> > - stressing host during replay may cause it to change even for same
> > record-replay pair (i.e. different replay executions for same file
> > recorded).
> >
> > This specific case seems to be stable to reproduce.
> >
> > вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> > <artem.k.pisarenko@gmail.com>:
> >
> >> I've posted bug report with extended tests (incl. case without
> >> sleep=off). You may find guest image (kernel) in bug description.
> >> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
> >>
> >> The most annoying thing is that some issues are almost not
> >> reproducible. There are definitely race conditions somewhere in qemu
> >> code. Running 'stress-ng' utility with CPU and I/O stressors in
> >> parallel with qemu execution greatly minimizes amount of attempts
> >> when I'm trying to trigger some of issues I encounter.
> >>
> >> I'll try 'info monitor' command tomorrow, but no guarantees that
> >> I'll be able to reproduce issue again.
> >>
> >> Speaking about '-nographic' and SDL... I've noted that UI greatly
> >> minimizes possibility of hanging (but not avoids it completely) when
> >> using icount in general, so this effect isn't rr-specific. I've
> >> already reported this bug too.
> >>
> >> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
> >>
> >>> Artem Pisarenko писал 2018-09-30 14:01:
> >>>> Feature still broken :(
> >>>
> >>> Thanks for testing.
> >>>
> >>>>
> >>>> Brief description of my tests.
> >>>>
> >>>> Guest image is Linux, which just powers off after kernel boots
> >>>> (instead of proceeding to user-space /init or /sbin/init).
> >>>> Base cmdline:
> >>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
> >>> -cpu
> >>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
> >>> -initrd
> >>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
> >>>> -nographic -serial SERIAL_VALUE -icount
> >>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
> >>>
> >>> I've never tried it with sleep=off. Can you remove it and try
> >>> again?
> >>>
> >>> We also seen a problem with '-nographic'. When we remove this
> >>> option and
> >>> QEMU runs with SDL
> >>> window, everything is ok. There is some problem with main loop
> >>> which may
> >>> sleep when there
> >>> is no GUI to update, or something like that. We couldn't fix it
> >>> yet.
> >>>
> >>>>
> >>>> Test 1. When SERIAL_VALUE=none
> >>>> Running with RR_VALUE=record completes successfully.
> >>>> Running with RR_VALUE=replay doesn't completes. qemu process
> >>> just
> >>>> eating ~100% cpu and memory usage doesn't grow after some
> >>> moment. I
> >>>> don't see what happens because of problem no.2 (see below).
> >>>
> >>> Try 'info replay' monitor command. Does instruction counter
> >>> increases?
> >>>
> >>>>
> >>>> Test 2. When SERIAL_VALUE=stdio
> >>>> Running with RR_VALUE=record completes successfully.
> >>>>
> >>>> Running with RR_VALUE=replay caues exit with error:
> >>>>
> >>>> "qemu-system-x86_64: Missing character write event in the replay
> >>> log"
> >>>>
> >>>> These problems are same with qemu 2.12 (both vanilla and with
> >>> previous
> >>>> versions of these patches applied). Furthemore, I consider whole
> >>>> icount mode broken and determinism isn't achievable.
> >>>> The irony is that I actually don't need record/replay feature.
> >>> I've
> >>>> tried to use it only as instrument to debug failing determinism
> >>> in
> >>>> qemu code. But since replay/record feature itself relies on
> >>>> determinism, which is broken, it's no wonder why it fails also
> >>> (I just
> >>>> hoped to bypass it).
> >>>>
> >>>> Contact me if you need more details. I just tired a lot trying
> >>> to get
> >>>> all these things working... Hope is leaving me...
> >>>
> >>> Can you share the kernel in case the icount still broken?
> >>>
> >>> Pavel Dovgalyuk
> >> --
> >>
> >> С уважением,
> >> Артем Писаренко
> >  --
> >
> > С уважением,
> >   Артем Писаренко
> >
> > Links:
> > ------
> > [1] https://bugs.launchpad.net/qemu/+bug/1795369
>
> --
>
> С уважением,
>   Артем Писаренко
>
> --
>
> С уважением,
>   Артем Писаренко
>
-- 

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Pavel Dovgalyuk]

Maybe this will help?

 

https://www.mail-archive.com/qemu-devel@nongnu.org/msg560780.html

 

Pavel Dovgalyuk

 

From: Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com] 
Sent: Tuesday, October 09, 2018 2:24 PM
To: Pavel Dovgalyuk
Cc: Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

 

(Since all previous patches are already merged to master, I'm running tests against latest (almost) version from master branch. Following results are based on master commit dafd95053611aa14dda40266857608d12ddce658 .)

 

Applying this patch made Tests 1 and 2 succeed (at least I wasn't able to acheive failures with several attempts).

Also I've tried few tests without sleep=off and/or rtc base options. All of them succeed too, except one case - removing sleep=off (regardless of -rtc option values or its presence at all) causes qemu to hang hard in recording mode at very startup. Process needs to be killed.

 

Some info from debugger:

    qemu-system-x86_64 [13231] [cores: 2,4,5,7]    

          Thread #1 [qemu-system-x86] 13231 [core: 2] (Suspended : Container)    

                      __lll_lock_wait() at lowlevellock.S:135 0x7f00b116626d   

                      __GI___pthread_mutex_lock() at pthread_mutex_lock.c:80 0x7f00b115fdbd     

                      qemu_mutex_lock_impl() at qemu-thread-posix.c:66 0x947ac4      

                      replay_mutex_lock() at replay-internal.c:206 0x7f3dea       

                      os_host_main_loop_wait() at main-loop.c:235 0x94335e    

                      main_loop_wait() at main-loop.c:497 0x943429      

                      main_loop() at vl.c:1,853 0x5be70f   

                      main() at vl.c:4,575 0x5c56e0           

          Thread #2 [qemu-system-x86] 13282 [core: 4] (Suspended : Container)    

          Thread #3 [qemu-system-x86] 13283 [core: 5] (Suspended : Container)    

          Thread #4 [qemu-system-x86] 13284 [core: 7] (Suspended : Step) 

                      cpu_get_icount_raw() at cpus.c:301 0x45a0a0         

                      replay_get_current_step() at replay.c:67 0x7f2f14   

                      replay_save_instructions() at replay-internal.c:225 0x7f3ea0          

                      replay_save_clock() at replay-time.c:24 0x7f483d   

                      icount_warp_rt() at cpus.c:512 0x45a745     

                      qemu_account_warp_timer() at cpus.c:690 0x45ad55         

                      qemu_tcg_rr_cpu_thread_fn() at cpus.c:1,498 0x45c554    

                      qemu_thread_start() at qemu-thread-posix.c:504 0x9485cf 

                      start_thread() at pthread_create.c:333 0x7f00b115d6ba     

                      clone() at clone.S:109 0x7f00b0e9341d       

    gdb (7.11.1)          

 

Threads #2,3 are just waiting in poll or similar. Nothing extraordinary.

 

Thread #4 cycles inside do {} while() loop of cpu_get_icount_raw() function:

    do {

        start = seqlock_read_begin(&timers_state.vm_clock_seqlock);

        icount = cpu_get_icount_raw_locked();

    } while (seqlock_read_retry(&timers_state.vm_clock_seqlock, start));

 

Value of timers_state.vm_clock_seqlock.sequence is always 3.

 

вт, 9 окт. 2018 г. в 15:04, Pavel Dovgalyuk <dovgaluk@ispras.ru>:

Please try the following patch.

There was a problem with rtc option in record/replay mode.

 

diff --git a/vl.c b/vl.c

index 40d5d0f..afe1c20 100644

--- a/vl.c

+++ b/vl.c

@@ -2885,6 +2885,7 @@ int main(int argc, char **argv, char **envp)

     DisplayState *ds;

     QemuOpts *opts, *machine_opts;

     QemuOpts *icount_opts = NULL, *accel_opts = NULL;

+    QemuOpts *rtc_opts = NULL;

     QemuOptsList *olist;

     int optind;

     const char *optarg;

@@ -3691,12 +3692,11 @@ int main(int argc, char **argv, char **envp)

                 warn_report("This option is ignored and will be removed soon");

                 break;

             case QEMU_OPTION_rtc:

-                opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"), optarg,

-                                               false);

-                if (!opts) {

+                rtc_opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),

+                                                   optarg, false);

+                if (!rtc_opts) {

                     exit(1);

                 }

-                configure_rtc(opts);

                 break;

             case QEMU_OPTION_tb_size:

#ifndef CONFIG_TCG

@@ -3907,6 +3907,9 @@ int main(int argc, char **argv, char **envp)

     loc_set_none();

     replay_configure(icount_opts);

+    if (rtc_opts) {

+        configure_rtc(rtc_opts);

+    }

     if (incoming && !preconfig_exit_requested) {

         error_report("'preconfig' and 'incoming' options are "

 

Pavel Dovgalyuk

 

From: Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com] 
Sent: Thursday, October 04, 2018 4:16 PM
To: dovgaluk
Cc: Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

 

No, it didn't changed test results, at least for https://github.com/ispras/qemu/tree/rr-180911 . Even step values it stucks on are same for most runs.

Playing with master and my own branch gives different results for tests without sleep=off and -rtc base. It seems that patch you mentioned didn't changed them very much.

The only thing can be said for sure, is that this patch does not fix issues completely. But MAY fix them partially or in some other specific cases...

 

ср, 3 окт. 2018 г. в 12:47, dovgaluk <dovgaluk@ispras.ru>:

Can you try applying this patch?
https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html

I also encountered the problems with x86_64 replaying and found the 
misprint in
the code which was fixed later, than sending the series to the mailing 
list.

Pavel Dovgalyuk


Artem Pisarenko писал 2018-10-02 10:02:
> I've added "-monitor stdio" option to command line of Test 1 and
> repeated entering command during execution:
> 
>   QEMU 3.0.50 monitor - type 'help' for more information
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 311736195
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 318198367
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 324737211
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 329890795
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   ...
> 
> Some notes on value of step it stucks on:
> - mostly it's same (even across different record-replay pairs);
> - stressing host during replay may cause it to change even for same
> record-replay pair (i.e. different replay executions for same file
> recorded).
> 
> This specific case seems to be stable to reproduce.
> 
> вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> <artem.k.pisarenko@gmail.com>:
> 
>> I've posted bug report with extended tests (incl. case without
>> sleep=off). You may find guest image (kernel) in bug description.
>> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
>> 
>> The most annoying thing is that some issues are almost not
>> reproducible. There are definitely race conditions somewhere in qemu
>> code. Running 'stress-ng' utility with CPU and I/O stressors in
>> parallel with qemu execution greatly minimizes amount of attempts
>> when I'm trying to trigger some of issues I encounter.
>> 
>> I'll try 'info monitor' command tomorrow, but no guarantees that
>> I'll be able to reproduce issue again.
>> 
>> Speaking about '-nographic' and SDL... I've noted that UI greatly
>> minimizes possibility of hanging (but not avoids it completely) when
>> using icount in general, so this effect isn't rr-specific. I've
>> already reported this bug too.
>> 
>> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
>> 
>>> Artem Pisarenko писал 2018-09-30 14:01:
>>>> Feature still broken :(
>>> 
>>> Thanks for testing.
>>> 
>>>> 
>>>> Brief description of my tests.
>>>> 
>>>> Guest image is Linux, which just powers off after kernel boots
>>>> (instead of proceeding to user-space /init or /sbin/init).
>>>> Base cmdline:
>>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
>>> -cpu
>>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
>>> -initrd
>>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
>>>> -nographic -serial SERIAL_VALUE -icount
>>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
>>> 
>>> I've never tried it with sleep=off. Can you remove it and try
>>> again?
>>> 
>>> We also seen a problem with '-nographic'. When we remove this
>>> option and
>>> QEMU runs with SDL
>>> window, everything is ok. There is some problem with main loop
>>> which may
>>> sleep when there
>>> is no GUI to update, or something like that. We couldn't fix it
>>> yet.
>>> 
>>>> 
>>>> Test 1. When SERIAL_VALUE=none
>>>> Running with RR_VALUE=record completes successfully.
>>>> Running with RR_VALUE=replay doesn't completes. qemu process
>>> just
>>>> eating ~100% cpu and memory usage doesn't grow after some
>>> moment. I
>>>> don't see what happens because of problem no.2 (see below).
>>> 
>>> Try 'info replay' monitor command. Does instruction counter
>>> increases?
>>> 
>>>> 
>>>> Test 2. When SERIAL_VALUE=stdio
>>>> Running with RR_VALUE=record completes successfully.
>>>> 
>>>> Running with RR_VALUE=replay caues exit with error:
>>>> 
>>>> "qemu-system-x86_64: Missing character write event in the replay
>>> log"
>>>> 
>>>> These problems are same with qemu 2.12 (both vanilla and with
>>> previous
>>>> versions of these patches applied). Furthemore, I consider whole
>>>> icount mode broken and determinism isn't achievable.
>>>> The irony is that I actually don't need record/replay feature.
>>> I've
>>>> tried to use it only as instrument to debug failing determinism
>>> in
>>>> qemu code. But since replay/record feature itself relies on
>>>> determinism, which is broken, it's no wonder why it fails also
>>> (I just
>>>> hoped to bypass it).
>>>> 
>>>> Contact me if you need more details. I just tired a lot trying
>>> to get
>>>> all these things working... Hope is leaving me...
>>> 
>>> Can you share the kernel in case the icount still broken?
>>> 
>>> Pavel Dovgalyuk
>> --
>> 
>> С уважением,
>> Артем Писаренко
>  --
> 
> С уважением,
>   Артем Писаренко
> 
> Links:
> ------
> [1] https://bugs.launchpad.net/qemu/+bug/1795369

-- 

С уважением,
  Артем Писаренко

-- 

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Artem Pisarenko]

(Since all previous patches are already merged to master, I'm running tests
against latest (almost) version from master branch. Following results are
based on master commit dafd95053611aa14dda40266857608d12ddce658 .)

Applying this patch made Tests 1 and 2 succeed (at least I wasn't able to
acheive failures with several attempts).
Also I've tried few tests without sleep=off and/or rtc base options. All of
them succeed too, except one case - removing sleep=off (regardless of -rtc
option values or its presence at all) causes qemu to hang hard in recording
mode at very startup. Process needs to be killed.

Some info from debugger:
    qemu-system-x86_64 [13231] [cores: 2,4,5,7]
Thread #1 [qemu-system-x86] 13231 [core: 2] (Suspended : Container)
__lll_lock_wait() at lowlevellock.S:135 0x7f00b116626d
__GI___pthread_mutex_lock() at pthread_mutex_lock.c:80 0x7f00b115fdbd
qemu_mutex_lock_impl() at qemu-thread-posix.c:66 0x947ac4
replay_mutex_lock() at replay-internal.c:206 0x7f3dea
os_host_main_loop_wait() at main-loop.c:235 0x94335e
main_loop_wait() at main-loop.c:497 0x943429
main_loop() at vl.c:1,853 0x5be70f
main() at vl.c:4,575 0x5c56e0
Thread #2 [qemu-system-x86] 13282 [core: 4] (Suspended : Container)
Thread #3 [qemu-system-x86] 13283 [core: 5] (Suspended : Container)
Thread #4 [qemu-system-x86] 13284 [core: 7] (Suspended : Step)
cpu_get_icount_raw() at cpus.c:301 0x45a0a0
replay_get_current_step() at replay.c:67 0x7f2f14
replay_save_instructions() at replay-internal.c:225 0x7f3ea0
replay_save_clock() at replay-time.c:24 0x7f483d
icount_warp_rt() at cpus.c:512 0x45a745
qemu_account_warp_timer() at cpus.c:690 0x45ad55
qemu_tcg_rr_cpu_thread_fn() at cpus.c:1,498 0x45c554
qemu_thread_start() at qemu-thread-posix.c:504 0x9485cf
start_thread() at pthread_create.c:333 0x7f00b115d6ba
clone() at clone.S:109 0x7f00b0e9341d
    gdb (7.11.1)

Threads #2,3 are just waiting in poll or similar. Nothing extraordinary.

Thread #4 cycles inside do {} while() loop of cpu_get_icount_raw() function:
    do {
        start = seqlock_read_begin(&timers_state.vm_clock_seqlock);
        icount = cpu_get_icount_raw_locked();
    } while (seqlock_read_retry(&timers_state.vm_clock_seqlock, start));

Value of timers_state.vm_clock_seqlock.sequence is always 3.

вт, 9 окт. 2018 г. в 15:04, Pavel Dovgalyuk <dovgaluk@ispras.ru>:

> Please try the following patch.
>
> There was a problem with rtc option in record/replay mode.
>
>
>
> diff --git a/vl.c b/vl.c
>
> index 40d5d0f..afe1c20 100644
>
> --- a/vl.c
>
> +++ b/vl.c
>
> @@ -2885,6 +2885,7 @@ int main(int argc, char **argv, char **envp)
>
>      DisplayState *ds;
>
>      QemuOpts *opts, *machine_opts;
>
>      QemuOpts *icount_opts = NULL, *accel_opts = NULL;
>
> +    QemuOpts *rtc_opts = NULL;
>
>      QemuOptsList *olist;
>
>      int optind;
>
>      const char *optarg;
>
> @@ -3691,12 +3692,11 @@ int main(int argc, char **argv, char **envp)
>
>                  warn_report("This option is ignored and will be removed
> soon");
>
>                  break;
>
>              case QEMU_OPTION_rtc:
>
> -                opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),
> optarg,
>
> -                                               false);
>
> -                if (!opts) {
>
> +                rtc_opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),
>
> +                                                   optarg, false);
>
> +                if (!rtc_opts) {
>
>                      exit(1);
>
>                  }
>
> -                configure_rtc(opts);
>
>                  break;
>
>              case QEMU_OPTION_tb_size:
>
> #ifndef CONFIG_TCG
>
> @@ -3907,6 +3907,9 @@ int main(int argc, char **argv, char **envp)
>
>      loc_set_none();
>
>      replay_configure(icount_opts);
>
> +    if (rtc_opts) {
>
> +        configure_rtc(rtc_opts);
>
> +    }
>
>      if (incoming && !preconfig_exit_requested) {
>
>          error_report("'preconfig' and 'incoming' options are "
>
>
>
> Pavel Dovgalyuk
>
>
>
> *From:* Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com]
> *Sent:* Thursday, October 04, 2018 4:16 PM
> *To:* dovgaluk
> *Cc:* Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
> *Subject:* Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and
> adding reverse debugging
>
>
>
> No, it didn't changed test results, at least for
> https://github.com/ispras/qemu/tree/rr-180911 . Even step values it
> stucks on are same for most runs.
>
> Playing with master and my own branch gives different results for tests
> without sleep=off and -rtc base. It seems that patch you mentioned didn't
> changed them very much.
>
> The only thing can be said for sure, is that this patch does not fix
> issues completely. But MAY fix them partially or in some other specific
> cases...
>
>
>
> ср, 3 окт. 2018 г. в 12:47, dovgaluk <dovgaluk@ispras.ru>:
>
> Can you try applying this patch?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html
>
> I also encountered the problems with x86_64 replaying and found the
> misprint in
> the code which was fixed later, than sending the series to the mailing
> list.
>
> Pavel Dovgalyuk
>
>
> Artem Pisarenko писал 2018-10-02 10:02:
> > I've added "-monitor stdio" option to command line of Test 1 and
> > repeated entering command during execution:
> >
> >   QEMU 3.0.50 monitor - type 'help' for more information
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 311736195
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 318198367
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 324737211
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 329890795
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   ...
> >
> > Some notes on value of step it stucks on:
> > - mostly it's same (even across different record-replay pairs);
> > - stressing host during replay may cause it to change even for same
> > record-replay pair (i.e. different replay executions for same file
> > recorded).
> >
> > This specific case seems to be stable to reproduce.
> >
> > вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> > <artem.k.pisarenko@gmail.com>:
> >
> >> I've posted bug report with extended tests (incl. case without
> >> sleep=off). You may find guest image (kernel) in bug description.
> >> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
> >>
> >> The most annoying thing is that some issues are almost not
> >> reproducible. There are definitely race conditions somewhere in qemu
> >> code. Running 'stress-ng' utility with CPU and I/O stressors in
> >> parallel with qemu execution greatly minimizes amount of attempts
> >> when I'm trying to trigger some of issues I encounter.
> >>
> >> I'll try 'info monitor' command tomorrow, but no guarantees that
> >> I'll be able to reproduce issue again.
> >>
> >> Speaking about '-nographic' and SDL... I've noted that UI greatly
> >> minimizes possibility of hanging (but not avoids it completely) when
> >> using icount in general, so this effect isn't rr-specific. I've
> >> already reported this bug too.
> >>
> >> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
> >>
> >>> Artem Pisarenko писал 2018-09-30 14:01:
> >>>> Feature still broken :(
> >>>
> >>> Thanks for testing.
> >>>
> >>>>
> >>>> Brief description of my tests.
> >>>>
> >>>> Guest image is Linux, which just powers off after kernel boots
> >>>> (instead of proceeding to user-space /init or /sbin/init).
> >>>> Base cmdline:
> >>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
> >>> -cpu
> >>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
> >>> -initrd
> >>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
> >>>> -nographic -serial SERIAL_VALUE -icount
> >>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
> >>>
> >>> I've never tried it with sleep=off. Can you remove it and try
> >>> again?
> >>>
> >>> We also seen a problem with '-nographic'. When we remove this
> >>> option and
> >>> QEMU runs with SDL
> >>> window, everything is ok. There is some problem with main loop
> >>> which may
> >>> sleep when there
> >>> is no GUI to update, or something like that. We couldn't fix it
> >>> yet.
> >>>
> >>>>
> >>>> Test 1. When SERIAL_VALUE=none
> >>>> Running with RR_VALUE=record completes successfully.
> >>>> Running with RR_VALUE=replay doesn't completes. qemu process
> >>> just
> >>>> eating ~100% cpu and memory usage doesn't grow after some
> >>> moment. I
> >>>> don't see what happens because of problem no.2 (see below).
> >>>
> >>> Try 'info replay' monitor command. Does instruction counter
> >>> increases?
> >>>
> >>>>
> >>>> Test 2. When SERIAL_VALUE=stdio
> >>>> Running with RR_VALUE=record completes successfully.
> >>>>
> >>>> Running with RR_VALUE=replay caues exit with error:
> >>>>
> >>>> "qemu-system-x86_64: Missing character write event in the replay
> >>> log"
> >>>>
> >>>> These problems are same with qemu 2.12 (both vanilla and with
> >>> previous
> >>>> versions of these patches applied). Furthemore, I consider whole
> >>>> icount mode broken and determinism isn't achievable.
> >>>> The irony is that I actually don't need record/replay feature.
> >>> I've
> >>>> tried to use it only as instrument to debug failing determinism
> >>> in
> >>>> qemu code. But since replay/record feature itself relies on
> >>>> determinism, which is broken, it's no wonder why it fails also
> >>> (I just
> >>>> hoped to bypass it).
> >>>>
> >>>> Contact me if you need more details. I just tired a lot trying
> >>> to get
> >>>> all these things working... Hope is leaving me...
> >>>
> >>> Can you share the kernel in case the icount still broken?
> >>>
> >>> Pavel Dovgalyuk
> >> --
> >>
> >> С уважением,
> >> Артем Писаренко
> >  --
> >
> > С уважением,
> >   Артем Писаренко
> >
> > Links:
> > ------
> > [1] https://bugs.launchpad.net/qemu/+bug/1795369
>
> --
>
> С уважением,
>   Артем Писаренко
>
-- 

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Pavel Dovgalyuk]

Please try the following patch.

There was a problem with rtc option in record/replay mode.

 

diff --git a/vl.c b/vl.c

index 40d5d0f..afe1c20 100644

--- a/vl.c

+++ b/vl.c

@@ -2885,6 +2885,7 @@ int main(int argc, char **argv, char **envp)

     DisplayState *ds;

     QemuOpts *opts, *machine_opts;

     QemuOpts *icount_opts = NULL, *accel_opts = NULL;

+    QemuOpts *rtc_opts = NULL;

     QemuOptsList *olist;

     int optind;

     const char *optarg;

@@ -3691,12 +3692,11 @@ int main(int argc, char **argv, char **envp)

                 warn_report("This option is ignored and will be removed soon");

                 break;

             case QEMU_OPTION_rtc:

-                opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"), optarg,

-                                               false);

-                if (!opts) {

+                rtc_opts = qemu_opts_parse_noisily(qemu_find_opts("rtc"),

+                                                   optarg, false);

+                if (!rtc_opts) {

                     exit(1);

                 }

-                configure_rtc(opts);

                 break;

             case QEMU_OPTION_tb_size:

#ifndef CONFIG_TCG

@@ -3907,6 +3907,9 @@ int main(int argc, char **argv, char **envp)

     loc_set_none();

     replay_configure(icount_opts);

+    if (rtc_opts) {

+        configure_rtc(rtc_opts);

+    }

     if (incoming && !preconfig_exit_requested) {

         error_report("'preconfig' and 'incoming' options are "

 

Pavel Dovgalyuk

 

From: Artem Pisarenko [mailto:artem.k.pisarenko@gmail.com] 
Sent: Thursday, October 04, 2018 4:16 PM
To: dovgaluk
Cc: Pavel.Dovgaluk@ispras.ru; qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

 

No, it didn't changed test results, at least for https://github.com/ispras/qemu/tree/rr-180911 . Even step values it stucks on are same for most runs.

Playing with master and my own branch gives different results for tests without sleep=off and -rtc base. It seems that patch you mentioned didn't changed them very much.

The only thing can be said for sure, is that this patch does not fix issues completely. But MAY fix them partially or in some other specific cases...

 

ср, 3 окт. 2018 г. в 12:47, dovgaluk <dovgaluk@ispras.ru>:

Can you try applying this patch?
https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html

I also encountered the problems with x86_64 replaying and found the 
misprint in
the code which was fixed later, than sending the series to the mailing 
list.

Pavel Dovgalyuk


Artem Pisarenko писал 2018-10-02 10:02:
> I've added "-monitor stdio" option to command line of Test 1 and
> repeated entering command during execution:
> 
>   QEMU 3.0.50 monitor - type 'help' for more information
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 311736195
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 318198367
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 324737211
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 329890795
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   ...
> 
> Some notes on value of step it stucks on:
> - mostly it's same (even across different record-replay pairs);
> - stressing host during replay may cause it to change even for same
> record-replay pair (i.e. different replay executions for same file
> recorded).
> 
> This specific case seems to be stable to reproduce.
> 
> вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> <artem.k.pisarenko@gmail.com>:
> 
>> I've posted bug report with extended tests (incl. case without
>> sleep=off). You may find guest image (kernel) in bug description.
>> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
>> 
>> The most annoying thing is that some issues are almost not
>> reproducible. There are definitely race conditions somewhere in qemu
>> code. Running 'stress-ng' utility with CPU and I/O stressors in
>> parallel with qemu execution greatly minimizes amount of attempts
>> when I'm trying to trigger some of issues I encounter.
>> 
>> I'll try 'info monitor' command tomorrow, but no guarantees that
>> I'll be able to reproduce issue again.
>> 
>> Speaking about '-nographic' and SDL... I've noted that UI greatly
>> minimizes possibility of hanging (but not avoids it completely) when
>> using icount in general, so this effect isn't rr-specific. I've
>> already reported this bug too.
>> 
>> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
>> 
>>> Artem Pisarenko писал 2018-09-30 14:01:
>>>> Feature still broken :(
>>> 
>>> Thanks for testing.
>>> 
>>>> 
>>>> Brief description of my tests.
>>>> 
>>>> Guest image is Linux, which just powers off after kernel boots
>>>> (instead of proceeding to user-space /init or /sbin/init).
>>>> Base cmdline:
>>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
>>> -cpu
>>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
>>> -initrd
>>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
>>>> -nographic -serial SERIAL_VALUE -icount
>>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
>>> 
>>> I've never tried it with sleep=off. Can you remove it and try
>>> again?
>>> 
>>> We also seen a problem with '-nographic'. When we remove this
>>> option and
>>> QEMU runs with SDL
>>> window, everything is ok. There is some problem with main loop
>>> which may
>>> sleep when there
>>> is no GUI to update, or something like that. We couldn't fix it
>>> yet.
>>> 
>>>> 
>>>> Test 1. When SERIAL_VALUE=none
>>>> Running with RR_VALUE=record completes successfully.
>>>> Running with RR_VALUE=replay doesn't completes. qemu process
>>> just
>>>> eating ~100% cpu and memory usage doesn't grow after some
>>> moment. I
>>>> don't see what happens because of problem no.2 (see below).
>>> 
>>> Try 'info replay' monitor command. Does instruction counter
>>> increases?
>>> 
>>>> 
>>>> Test 2. When SERIAL_VALUE=stdio
>>>> Running with RR_VALUE=record completes successfully.
>>>> 
>>>> Running with RR_VALUE=replay caues exit with error:
>>>> 
>>>> "qemu-system-x86_64: Missing character write event in the replay
>>> log"
>>>> 
>>>> These problems are same with qemu 2.12 (both vanilla and with
>>> previous
>>>> versions of these patches applied). Furthemore, I consider whole
>>>> icount mode broken and determinism isn't achievable.
>>>> The irony is that I actually don't need record/replay feature.
>>> I've
>>>> tried to use it only as instrument to debug failing determinism
>>> in
>>>> qemu code. But since replay/record feature itself relies on
>>>> determinism, which is broken, it's no wonder why it fails also
>>> (I just
>>>> hoped to bypass it).
>>>> 
>>>> Contact me if you need more details. I just tired a lot trying
>>> to get
>>>> all these things working... Hope is leaving me...
>>> 
>>> Can you share the kernel in case the icount still broken?
>>> 
>>> Pavel Dovgalyuk
>> --
>> 
>> С уважением,
>> Артем Писаренко
>  --
> 
> С уважением,
>   Артем Писаренко
> 
> Links:
> ------
> [1] https://bugs.launchpad.net/qemu/+bug/1795369

-- 

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Artem Pisarenko]

No, it didn't changed test results, at least for
https://github.com/ispras/qemu/tree/rr-180911 . Even step values it stucks
on are same for most runs.
Playing with master and my own branch gives different results for tests
without sleep=off and -rtc base. It seems that patch you mentioned didn't
changed them very much.
The only thing can be said for sure, is that this patch does not fix issues
completely. But MAY fix them partially or in some other specific cases...

ср, 3 окт. 2018 г. в 12:47, dovgaluk <dovgaluk@ispras.ru>:

> Can you try applying this patch?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html
>
> I also encountered the problems with x86_64 replaying and found the
> misprint in
> the code which was fixed later, than sending the series to the mailing
> list.
>
> Pavel Dovgalyuk
>
>
> Artem Pisarenko писал 2018-10-02 10:02:
> > I've added "-monitor stdio" option to command line of Test 1 and
> > repeated entering command during execution:
> >
> >   QEMU 3.0.50 monitor - type 'help' for more information
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 311736195
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 318198367
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 324737211
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 329890795
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   (qemu) info replay
> >   Replaying execution 'icount_rr_capture.bin': current step =
> > 607069789
> >   ...
> >
> > Some notes on value of step it stucks on:
> > - mostly it's same (even across different record-replay pairs);
> > - stressing host during replay may cause it to change even for same
> > record-replay pair (i.e. different replay executions for same file
> > recorded).
> >
> > This specific case seems to be stable to reproduce.
> >
> > вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> > <artem.k.pisarenko@gmail.com>:
> >
> >> I've posted bug report with extended tests (incl. case without
> >> sleep=off). You may find guest image (kernel) in bug description.
> >> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
> >>
> >> The most annoying thing is that some issues are almost not
> >> reproducible. There are definitely race conditions somewhere in qemu
> >> code. Running 'stress-ng' utility with CPU and I/O stressors in
> >> parallel with qemu execution greatly minimizes amount of attempts
> >> when I'm trying to trigger some of issues I encounter.
> >>
> >> I'll try 'info monitor' command tomorrow, but no guarantees that
> >> I'll be able to reproduce issue again.
> >>
> >> Speaking about '-nographic' and SDL... I've noted that UI greatly
> >> minimizes possibility of hanging (but not avoids it completely) when
> >> using icount in general, so this effect isn't rr-specific. I've
> >> already reported this bug too.
> >>
> >> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
> >>
> >>> Artem Pisarenko писал 2018-09-30 14:01:
> >>>> Feature still broken :(
> >>>
> >>> Thanks for testing.
> >>>
> >>>>
> >>>> Brief description of my tests.
> >>>>
> >>>> Guest image is Linux, which just powers off after kernel boots
> >>>> (instead of proceeding to user-space /init or /sbin/init).
> >>>> Base cmdline:
> >>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
> >>> -cpu
> >>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
> >>> -initrd
> >>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
> >>>> -nographic -serial SERIAL_VALUE -icount
> >>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
> >>>
> >>> I've never tried it with sleep=off. Can you remove it and try
> >>> again?
> >>>
> >>> We also seen a problem with '-nographic'. When we remove this
> >>> option and
> >>> QEMU runs with SDL
> >>> window, everything is ok. There is some problem with main loop
> >>> which may
> >>> sleep when there
> >>> is no GUI to update, or something like that. We couldn't fix it
> >>> yet.
> >>>
> >>>>
> >>>> Test 1. When SERIAL_VALUE=none
> >>>> Running with RR_VALUE=record completes successfully.
> >>>> Running with RR_VALUE=replay doesn't completes. qemu process
> >>> just
> >>>> eating ~100% cpu and memory usage doesn't grow after some
> >>> moment. I
> >>>> don't see what happens because of problem no.2 (see below).
> >>>
> >>> Try 'info replay' monitor command. Does instruction counter
> >>> increases?
> >>>
> >>>>
> >>>> Test 2. When SERIAL_VALUE=stdio
> >>>> Running with RR_VALUE=record completes successfully.
> >>>>
> >>>> Running with RR_VALUE=replay caues exit with error:
> >>>>
> >>>> "qemu-system-x86_64: Missing character write event in the replay
> >>> log"
> >>>>
> >>>> These problems are same with qemu 2.12 (both vanilla and with
> >>> previous
> >>>> versions of these patches applied). Furthemore, I consider whole
> >>>> icount mode broken and determinism isn't achievable.
> >>>> The irony is that I actually don't need record/replay feature.
> >>> I've
> >>>> tried to use it only as instrument to debug failing determinism
> >>> in
> >>>> qemu code. But since replay/record feature itself relies on
> >>>> determinism, which is broken, it's no wonder why it fails also
> >>> (I just
> >>>> hoped to bypass it).
> >>>>
> >>>> Contact me if you need more details. I just tired a lot trying
> >>> to get
> >>>> all these things working... Hope is leaving me...
> >>>
> >>> Can you share the kernel in case the icount still broken?
> >>>
> >>> Pavel Dovgalyuk
> >> --
> >>
> >> С уважением,
> >> Артем Писаренко
> >  --
> >
> > С уважением,
> >   Артем Писаренко
> >
> > Links:
> > ------
> > [1] https://bugs.launchpad.net/qemu/+bug/1795369
>
> --

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[dovgaluk]

Can you try applying this patch?
https://www.mail-archive.com/qemu-devel@nongnu.org/msg563798.html

I also encountered the problems with x86_64 replaying and found the 
misprint in
the code which was fixed later, than sending the series to the mailing 
list.

Pavel Dovgalyuk


Artem Pisarenko писал 2018-10-02 10:02:
> I've added "-monitor stdio" option to command line of Test 1 and
> repeated entering command during execution:
> 
>   QEMU 3.0.50 monitor - type 'help' for more information
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 311736195
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 318198367
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 324737211
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 329890795
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   (qemu) info replay
>   Replaying execution 'icount_rr_capture.bin': current step =
> 607069789
>   ...
> 
> Some notes on value of step it stucks on:
> - mostly it's same (even across different record-replay pairs);
> - stressing host during replay may cause it to change even for same
> record-replay pair (i.e. different replay executions for same file
> recorded).
> 
> This specific case seems to be stable to reproduce.
> 
> вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko
> <artem.k.pisarenko@gmail.com>:
> 
>> I've posted bug report with extended tests (incl. case without
>> sleep=off). You may find guest image (kernel) in bug description.
>> https://bugs.launchpad.net/qemu/+bug/1795369 [1]
>> 
>> The most annoying thing is that some issues are almost not
>> reproducible. There are definitely race conditions somewhere in qemu
>> code. Running 'stress-ng' utility with CPU and I/O stressors in
>> parallel with qemu execution greatly minimizes amount of attempts
>> when I'm trying to trigger some of issues I encounter.
>> 
>> I'll try 'info monitor' command tomorrow, but no guarantees that
>> I'll be able to reproduce issue again.
>> 
>> Speaking about '-nographic' and SDL... I've noted that UI greatly
>> minimizes possibility of hanging (but not avoids it completely) when
>> using icount in general, so this effect isn't rr-specific. I've
>> already reported this bug too.
>> 
>> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
>> 
>>> Artem Pisarenko писал 2018-09-30 14:01:
>>>> Feature still broken :(
>>> 
>>> Thanks for testing.
>>> 
>>>> 
>>>> Brief description of my tests.
>>>> 
>>>> Guest image is Linux, which just powers off after kernel boots
>>>> (instead of proceeding to user-space /init or /sbin/init).
>>>> Base cmdline:
>>>> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048
>>> -cpu
>>>> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage
>>> -initrd
>>>> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
>>>> -nographic -serial SERIAL_VALUE -icount
>>>> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
>>> 
>>> I've never tried it with sleep=off. Can you remove it and try
>>> again?
>>> 
>>> We also seen a problem with '-nographic'. When we remove this
>>> option and
>>> QEMU runs with SDL
>>> window, everything is ok. There is some problem with main loop
>>> which may
>>> sleep when there
>>> is no GUI to update, or something like that. We couldn't fix it
>>> yet.
>>> 
>>>> 
>>>> Test 1. When SERIAL_VALUE=none
>>>> Running with RR_VALUE=record completes successfully.
>>>> Running with RR_VALUE=replay doesn't completes. qemu process
>>> just
>>>> eating ~100% cpu and memory usage doesn't grow after some
>>> moment. I
>>>> don't see what happens because of problem no.2 (see below).
>>> 
>>> Try 'info replay' monitor command. Does instruction counter
>>> increases?
>>> 
>>>> 
>>>> Test 2. When SERIAL_VALUE=stdio
>>>> Running with RR_VALUE=record completes successfully.
>>>> 
>>>> Running with RR_VALUE=replay caues exit with error:
>>>> 
>>>> "qemu-system-x86_64: Missing character write event in the replay
>>> log"
>>>> 
>>>> These problems are same with qemu 2.12 (both vanilla and with
>>> previous
>>>> versions of these patches applied). Furthemore, I consider whole
>>>> icount mode broken and determinism isn't achievable.
>>>> The irony is that I actually don't need record/replay feature.
>>> I've
>>>> tried to use it only as instrument to debug failing determinism
>>> in
>>>> qemu code. But since replay/record feature itself relies on
>>>> determinism, which is broken, it's no wonder why it fails also
>>> (I just
>>>> hoped to bypass it).
>>>> 
>>>> Contact me if you need more details. I just tired a lot trying
>>> to get
>>>> all these things working... Hope is leaving me...
>>> 
>>> Can you share the kernel in case the icount still broken?
>>> 
>>> Pavel Dovgalyuk
>> --
>> 
>> С уважением,
>> Артем Писаренко
>  --
> 
> С уважением,
>   Артем Писаренко
> 
> Links:
> ------
> [1] https://bugs.launchpad.net/qemu/+bug/1795369

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Artem Pisarenko]

I've added "-monitor stdio" option to command line of Test 1 and repeated
entering command during execution:

  QEMU 3.0.50 monitor - type 'help' for more information
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 311736195
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 318198367
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 324737211
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 329890795
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 607069789
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 607069789
  (qemu) info replay
  Replaying execution 'icount_rr_capture.bin': current step = 607069789
  ...

Some notes on value of step it stucks on:
- mostly it's same (even across different record-replay pairs);
- stressing host during replay may cause it to change even for same
record-replay pair (i.e. different replay executions for same file
recorded).

This specific case seems to be stable to reproduce.

вт, 2 окт. 2018 г. в 0:22, Artem Pisarenko <artem.k.pisarenko@gmail.com>:

> I've posted bug report with extended tests (incl. case without sleep=off).
> You may find guest image (kernel) in bug description.
> https://bugs.launchpad.net/qemu/+bug/1795369
>
> The most annoying thing is that some issues are almost not reproducible.
> There are definitely race conditions somewhere in qemu code. Running
> 'stress-ng' utility with CPU and I/O stressors in parallel with qemu
> execution greatly minimizes amount of attempts when I'm trying to trigger
> some of issues I encounter.
>
> I'll try 'info monitor' command tomorrow, but no guarantees that I'll be
> able to reproduce issue again.
>
> Speaking about '-nographic' and SDL... I've noted that UI greatly
> minimizes possibility of hanging (but not avoids it completely) when using
> icount in general, so this effect isn't rr-specific. I've already reported
> this bug too.
>
>
> пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:
>
>> Artem Pisarenko писал 2018-09-30 14:01:
>> > Feature still broken :(
>>
>> Thanks for testing.
>>
>> >
>> > Brief description of my tests.
>> >
>> > Guest image is Linux, which just powers off after kernel boots
>> > (instead of proceeding to user-space /init or /sbin/init).
>> > Base cmdline:
>> > qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048 -cpu
>> > qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage -initrd
>> > rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
>> > -nographic -serial SERIAL_VALUE -icount
>> > 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
>>
>> I've never tried it with sleep=off. Can you remove it and try again?
>>
>> We also seen a problem with '-nographic'. When we remove this option and
>> QEMU runs with SDL
>> window, everything is ok. There is some problem with main loop which may
>> sleep when there
>> is no GUI to update, or something like that. We couldn't fix it yet.
>>
>> >
>> > Test 1. When SERIAL_VALUE=none
>> > Running with RR_VALUE=record completes successfully.
>> > Running with RR_VALUE=replay doesn't completes. qemu process just
>> > eating ~100% cpu and memory usage doesn't grow after some moment. I
>> > don't see what happens because of problem no.2 (see below).
>>
>> Try 'info replay' monitor command. Does instruction counter increases?
>>
>> >
>> > Test 2. When SERIAL_VALUE=stdio
>> > Running with RR_VALUE=record completes successfully.
>> >
>> > Running with RR_VALUE=replay caues exit with error:
>> >
>> > "qemu-system-x86_64: Missing character write event in the replay log"
>> >
>> > These problems are same with qemu 2.12 (both vanilla and with previous
>> > versions of these patches applied). Furthemore, I consider whole
>> > icount mode broken and determinism isn't achievable.
>> > The irony is that I actually don't need record/replay feature. I've
>> > tried to use it only as instrument to debug failing determinism in
>> > qemu code. But since replay/record feature itself relies on
>> > determinism, which is broken, it's no wonder why it fails also (I just
>> > hoped to bypass it).
>> >
>> > Contact me if you need more details. I just tired a lot trying to get
>> > all these things working... Hope is leaving me...
>>
>> Can you share the kernel in case the icount still broken?
>>
>>
>> Pavel Dovgalyuk
>>
>> --
>
> С уважением,
>   Артем Писаренко
>
-- 

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[Artem Pisarenko]

I've posted bug report with extended tests (incl. case without sleep=off).
You may find guest image (kernel) in bug description.
https://bugs.launchpad.net/qemu/+bug/1795369

The most annoying thing is that some issues are almost not reproducible.
There are definitely race conditions somewhere in qemu code. Running
'stress-ng' utility with CPU and I/O stressors in parallel with qemu
execution greatly minimizes amount of attempts when I'm trying to trigger
some of issues I encounter.

I'll try 'info monitor' command tomorrow, but no guarantees that I'll be
able to reproduce issue again.

Speaking about '-nographic' and SDL... I've noted that UI greatly minimizes
possibility of hanging (but not avoids it completely) when using icount in
general, so this effect isn't rr-specific. I've already reported this bug
too.


пн, 1 окт. 2018 г., 20:14 dovgaluk <dovgaluk@ispras.ru>:

> Artem Pisarenko писал 2018-09-30 14:01:
> > Feature still broken :(
>
> Thanks for testing.
>
> >
> > Brief description of my tests.
> >
> > Guest image is Linux, which just powers off after kernel boots
> > (instead of proceeding to user-space /init or /sbin/init).
> > Base cmdline:
> > qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048 -cpu
> > qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage -initrd
> > rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
> > -nographic -serial SERIAL_VALUE -icount
> > 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin
>
> I've never tried it with sleep=off. Can you remove it and try again?
>
> We also seen a problem with '-nographic'. When we remove this option and
> QEMU runs with SDL
> window, everything is ok. There is some problem with main loop which may
> sleep when there
> is no GUI to update, or something like that. We couldn't fix it yet.
>
> >
> > Test 1. When SERIAL_VALUE=none
> > Running with RR_VALUE=record completes successfully.
> > Running with RR_VALUE=replay doesn't completes. qemu process just
> > eating ~100% cpu and memory usage doesn't grow after some moment. I
> > don't see what happens because of problem no.2 (see below).
>
> Try 'info replay' monitor command. Does instruction counter increases?
>
> >
> > Test 2. When SERIAL_VALUE=stdio
> > Running with RR_VALUE=record completes successfully.
> >
> > Running with RR_VALUE=replay caues exit with error:
> >
> > "qemu-system-x86_64: Missing character write event in the replay log"
> >
> > These problems are same with qemu 2.12 (both vanilla and with previous
> > versions of these patches applied). Furthemore, I consider whole
> > icount mode broken and determinism isn't achievable.
> > The irony is that I actually don't need record/replay feature. I've
> > tried to use it only as instrument to debug failing determinism in
> > qemu code. But since replay/record feature itself relies on
> > determinism, which is broken, it's no wonder why it fails also (I just
> > hoped to bypass it).
> >
> > Contact me if you need more details. I just tired a lot trying to get
> > all these things working... Hope is leaving me...
>
> Can you share the kernel in case the icount still broken?
>
>
> Pavel Dovgalyuk
>
> --

С уважением,
  Артем Писаренко

Re: [Qemu-devel] [PATCH v6 00/25] Fixing record/replay and adding reverse debugging

[dovgaluk]

Artem Pisarenko писал 2018-09-30 14:01:
> Feature still broken :(

Thanks for testing.

> 
> Brief description of my tests.
> 
> Guest image is Linux, which just powers off after kernel boots
> (instead of proceeding to user-space /init or /sbin/init).
> Base cmdline:
> qemu-system-x86_64 -nodefaults -machine pc,accel=tcg -m 2048 -cpu
> qemu64 -rtc clock=vm,base=2000-01-01T00:00:00 -kernel bzImage -initrd
> rootfs -append 'nokaslr console=ttyS0 rdinit=/init_poweroff'
> -nographic -serial SERIAL_VALUE -icount
> 1,sleep=off,rr=RR_VALUE,rrfile=icount_rr_capture.bin

I've never tried it with sleep=off. Can you remove it and try again?

We also seen a problem with '-nographic'. When we remove this option and 
QEMU runs with SDL
window, everything is ok. There is some problem with main loop which may 
sleep when there
is no GUI to update, or something like that. We couldn't fix it yet.

> 
> Test 1. When SERIAL_VALUE=none
> Running with RR_VALUE=record completes successfully.
> Running with RR_VALUE=replay doesn't completes. qemu process just
> eating ~100% cpu and memory usage doesn't grow after some moment. I
> don't see what happens because of problem no.2 (see below).

Try 'info replay' monitor command. Does instruction counter increases?

> 
> Test 2. When SERIAL_VALUE=stdio
> Running with RR_VALUE=record completes successfully.
> 
> Running with RR_VALUE=replay caues exit with error:
> 
> "qemu-system-x86_64: Missing character write event in the replay log"
> 
> These problems are same with qemu 2.12 (both vanilla and with previous
> versions of these patches applied). Furthemore, I consider whole
> icount mode broken and determinism isn't achievable.
> The irony is that I actually don't need record/replay feature. I've
> tried to use it only as instrument to debug failing determinism in
> qemu code. But since replay/record feature itself relies on
> determinism, which is broken, it's no wonder why it fails also (I just
> hoped to bypass it).
> 
> Contact me if you need more details. I just tired a lot trying to get
> all these things working... Hope is leaving me...

Can you share the kernel in case the icount still broken?


Pavel Dovgalyuk