[PATCH v3 0/9] Record/replay refactoring and stuff

[PATCH v3 0/9] Record/replay refactoring and stuff

[Pavel Dovgalyuk]

The following series includes the following record/replay-related changes:
- simplified async event processing
- updated record/replay documentation, which was also converted to rst
- avocado tests for record/replay of Linux for x86_64 and Aarch64
- some bugfixes

v3 changes:
 - rebased to master

v2 changes:
 - rebased to master
 - fixed some issues found by Richard Henderson

---

Pavel Dovgalyuk (9):
      replay: fix event queue flush for qemu shutdown
      replay: notify vCPU when BH is scheduled
      replay: rewrite async event handling
      replay: simplify async event processing
      docs: convert docs/devel/replay page to rst
      docs: move replay docs to docs/system/replay.rst
      tests/avocado: update replay_linux test
      tests/avocado: add replay Linux tests for virtio machine
      tests/avocado: add replay Linux test for Aarch64 machines


 accel/tcg/tcg-accel-ops-icount.c |   5 +-
 docs/devel/index-tcg.rst         |   1 +
 docs/devel/replay.rst            | 306 +++++++++++++++++++++++
 docs/devel/replay.txt            |  46 ----
 docs/replay.txt                  | 410 -------------------------------
 docs/system/index.rst            |   1 +
 docs/system/replay.rst           | 237 ++++++++++++++++++
 include/sysemu/cpu-timers.h      |   1 +
 include/sysemu/replay.h          |   9 +-
 replay/replay-events.c           |  56 ++---
 replay/replay-internal.h         |  37 ++-
 replay/replay-snapshot.c         |   2 -
 replay/replay.c                  |  75 +++---
 softmmu/icount.c                 |  12 +-
 stubs/icount.c                   |   4 +
 tests/avocado/replay_linux.py    |  86 ++++++-
 util/async.c                     |   8 +
 17 files changed, 726 insertions(+), 570 deletions(-)
 create mode 100644 docs/devel/replay.rst
 delete mode 100644 docs/devel/replay.txt
 delete mode 100644 docs/replay.txt
 create mode 100644 docs/system/replay.rst

--
Pavel Dovgalyuk

[PATCH v3 1/9] replay: fix event queue flush for qemu shutdown

[Pavel Dovgalyuk]

This patch fixes event queue flush in the case of emulator
shutdown. replay_finish_events should be called when replay_mode
is not cleared.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
---
 replay/replay.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/replay/replay.c b/replay/replay.c
index 6df2abc18c..2d3607998a 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -387,9 +387,8 @@ void replay_finish(void)
     g_free(replay_snapshot);
     replay_snapshot = NULL;
 
-    replay_mode = REPLAY_MODE_NONE;
-
     replay_finish_events();
+    replay_mode = REPLAY_MODE_NONE;
 }
 
 void replay_add_blocker(Error *reason)

[PATCH v3 2/9] replay: notify vCPU when BH is scheduled

[Pavel Dovgalyuk]

vCPU execution should be suspended when new BH is scheduled.
This is needed to avoid guest timeouts caused by the long cycles
of the execution. In replay mode execution may hang when
vCPU sleeps and block event comes to the queue.
This patch adds notification which wakes up vCPU or interrupts
execution of guest code.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

--

v2: changed first_cpu to current_cpu (suggested by Richard Henderson)
---
 include/sysemu/cpu-timers.h |    1 +
 softmmu/icount.c            |    8 ++++++++
 stubs/icount.c              |    4 ++++
 util/async.c                |    8 ++++++++
 4 files changed, 21 insertions(+)

diff --git a/include/sysemu/cpu-timers.h b/include/sysemu/cpu-timers.h
index ed6ee5c46c..2e786fe7fb 100644
--- a/include/sysemu/cpu-timers.h
+++ b/include/sysemu/cpu-timers.h
@@ -59,6 +59,7 @@ int64_t icount_round(int64_t count);
 /* if the CPUs are idle, start accounting real time to virtual clock. */
 void icount_start_warp_timer(void);
 void icount_account_warp_timer(void);
+void icount_notify_exit(void);
 
 /*
  * CPU Ticks and Clock
diff --git a/softmmu/icount.c b/softmmu/icount.c
index 5ca271620d..1cafec5014 100644
--- a/softmmu/icount.c
+++ b/softmmu/icount.c
@@ -486,3 +486,11 @@ void icount_configure(QemuOpts *opts, Error **errp)
                    qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) +
                    NANOSECONDS_PER_SECOND / 10);
 }
+
+void icount_notify_exit(void)
+{
+    if (icount_enabled() && current_cpu) {
+        qemu_cpu_kick(current_cpu);
+        qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
+    }
+}
diff --git a/stubs/icount.c b/stubs/icount.c
index f13c43568b..6df8c2bf7d 100644
--- a/stubs/icount.c
+++ b/stubs/icount.c
@@ -43,3 +43,7 @@ void icount_account_warp_timer(void)
 {
     abort();
 }
+
+void icount_notify_exit(void)
+{
+}
diff --git a/util/async.c b/util/async.c
index 554ba70cca..75f50f47c4 100644
--- a/util/async.c
+++ b/util/async.c
@@ -33,6 +33,7 @@
 #include "block/raw-aio.h"
 #include "qemu/coroutine_int.h"
 #include "qemu/coroutine-tls.h"
+#include "sysemu/cpu-timers.h"
 #include "trace.h"
 
 /***********************************************************/
@@ -185,6 +186,13 @@ void qemu_bh_schedule_idle(QEMUBH *bh)
 void qemu_bh_schedule(QEMUBH *bh)
 {
     aio_bh_enqueue(bh, BH_SCHEDULED);
+    /*
+     * Workaround for record/replay.
+     * vCPU execution should be suspended when new BH is set.
+     * This is needed to avoid guest timeouts caused
+     * by the long cycles of the execution.
+     */
+    icount_notify_exit();
 }
 
 /* This func is async.

[PATCH v3 3/9] replay: rewrite async event handling

[Pavel Dovgalyuk]

This patch decouples checkpoints and async events.
It was a tricky part of replay implementation. Now it becomes
much simpler and easier to maintain.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
---
 accel/tcg/tcg-accel-ops-icount.c |    5 +--
 docs/replay.txt                  |   11 ++----
 include/sysemu/replay.h          |    9 ++++-
 replay/replay-events.c           |   20 +++-------
 replay/replay-internal.h         |    6 +--
 replay/replay-snapshot.c         |    1 -
 replay/replay.c                  |   74 +++++++++++++++-----------------------
 softmmu/icount.c                 |    4 ++
 8 files changed, 54 insertions(+), 76 deletions(-)

diff --git a/accel/tcg/tcg-accel-ops-icount.c b/accel/tcg/tcg-accel-ops-icount.c
index 24520ea112..8f1dda4344 100644
--- a/accel/tcg/tcg-accel-ops-icount.c
+++ b/accel/tcg/tcg-accel-ops-icount.c
@@ -84,8 +84,7 @@ void icount_handle_deadline(void)
      * Don't interrupt cpu thread, when these events are waiting
      * (i.e., there is no checkpoint)
      */
-    if (deadline == 0
-        && (replay_mode != REPLAY_MODE_PLAY || replay_has_checkpoint())) {
+    if (deadline == 0) {
         icount_notify_aio_contexts();
     }
 }
@@ -109,7 +108,7 @@ void icount_prepare_for_run(CPUState *cpu)
 
     replay_mutex_lock();
 
-    if (cpu->icount_budget == 0 && replay_has_checkpoint()) {
+    if (cpu->icount_budget == 0) {
         icount_notify_aio_contexts();
     }
 }
diff --git a/docs/replay.txt b/docs/replay.txt
index 5b008ca491..6c9fdff09d 100644
--- a/docs/replay.txt
+++ b/docs/replay.txt
@@ -366,11 +366,9 @@ Here is the list of events that are written into the log:
    Argument: 4-byte number of executed instructions.
  - EVENT_INTERRUPT. Used to synchronize interrupt processing.
  - EVENT_EXCEPTION. Used to synchronize exception handling.
- - EVENT_ASYNC. This is a group of events. They are always processed
-   together with checkpoints. When such an event is generated, it is
-   stored in the queue and processed only when checkpoint occurs.
-   Every such event is followed by 1-byte checkpoint id and 1-byte
-   async event id from the following list:
+ - EVENT_ASYNC. This is a group of events. When such an event is generated,
+   it is stored in the queue and processed in icount_account_warp_timer().
+   Every such event has it's own id from the following list:
      - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes
        callbacks that affect virtual machine state, but normally called
        asynchronously.
@@ -405,6 +403,5 @@ Here is the list of events that are written into the log:
  - EVENT_CLOCK + clock_id. Group of events for host clock read operations.
    Argument: 8-byte clock value.
  - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of
-   CPU, internal threads, and asynchronous input events. May be followed
-   by one or more EVENT_ASYNC events.
+   CPU, internal threads, and asynchronous input events.
  - EVENT_END. Last event in the log.
diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h
index 032256533b..9af0ac32f0 100644
--- a/include/sysemu/replay.h
+++ b/include/sysemu/replay.h
@@ -161,9 +161,14 @@ void replay_shutdown_request(ShutdownCause cause);
     Returns 0 in PLAY mode if checkpoint was not found.
     Returns 1 in all other cases. */
 bool replay_checkpoint(ReplayCheckpoint checkpoint);
-/*! Used to determine that checkpoint is pending.
+/*! Used to determine that checkpoint or async event is pending.
     Does not proceed to the next event in the log. */
-bool replay_has_checkpoint(void);
+bool replay_has_event(void);
+/*
+ * Processes the async events added to the queue (while recording)
+ * or reads the events from the file (while replaying).
+ */
+void replay_async_events(void);
 
 /* Asynchronous events queue */
 
diff --git a/replay/replay-events.c b/replay/replay-events.c
index ac47c89834..db1decf9dd 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -170,12 +170,11 @@ void replay_block_event(QEMUBH *bh, uint64_t id)
     }
 }
 
-static void replay_save_event(Event *event, int checkpoint)
+static void replay_save_event(Event *event)
 {
     if (replay_mode != REPLAY_MODE_PLAY) {
         /* put the event into the file */
         replay_put_event(EVENT_ASYNC);
-        replay_put_byte(checkpoint);
         replay_put_byte(event->event_kind);
 
         /* save event-specific data */
@@ -206,34 +205,27 @@ static void replay_save_event(Event *event, int checkpoint)
 }
 
 /* Called with replay mutex locked */
-void replay_save_events(int checkpoint)
+void replay_save_events(void)
 {
     g_assert(replay_mutex_locked());
-    g_assert(checkpoint != CHECKPOINT_CLOCK_WARP_START);
-    g_assert(checkpoint != CHECKPOINT_CLOCK_VIRTUAL);
     while (!QTAILQ_EMPTY(&events_list)) {
         Event *event = QTAILQ_FIRST(&events_list);
-        replay_save_event(event, checkpoint);
+        replay_save_event(event);
         replay_run_event(event);
         QTAILQ_REMOVE(&events_list, event, events);
         g_free(event);
     }
 }
 
-static Event *replay_read_event(int checkpoint)
+static Event *replay_read_event(void)
 {
     Event *event;
     if (replay_state.read_event_kind == -1) {
-        replay_state.read_event_checkpoint = replay_get_byte();
         replay_state.read_event_kind = replay_get_byte();
         replay_state.read_event_id = -1;
         replay_check_error();
     }
 
-    if (checkpoint != replay_state.read_event_checkpoint) {
-        return NULL;
-    }
-
     /* Events that has not to be in the queue */
     switch (replay_state.read_event_kind) {
     case REPLAY_ASYNC_EVENT_BH:
@@ -294,11 +286,11 @@ static Event *replay_read_event(int checkpoint)
 }
 
 /* Called with replay mutex locked */
-void replay_read_events(int checkpoint)
+void replay_read_events(void)
 {
     g_assert(replay_mutex_locked());
     while (replay_state.data_kind == EVENT_ASYNC) {
-        Event *event = replay_read_event(checkpoint);
+        Event *event = replay_read_event();
         if (!event) {
             break;
         }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index dada623527..59797c86cf 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -87,8 +87,6 @@ typedef struct ReplayState {
     int32_t read_event_kind;
     /*! Asynchronous event id read from the log */
     uint64_t read_event_id;
-    /*! Asynchronous event checkpoint id read from the log */
-    int32_t read_event_checkpoint;
 } ReplayState;
 extern ReplayState replay_state;
 
@@ -151,9 +149,9 @@ void replay_finish_events(void);
 /*! Returns true if there are any unsaved events in the queue */
 bool replay_has_events(void);
 /*! Saves events from queue into the file */
-void replay_save_events(int checkpoint);
+void replay_save_events(void);
 /*! Read events from the file into the input queue */
-void replay_read_events(int checkpoint);
+void replay_read_events(void);
 /*! Adds specified async event to the queue */
 void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque,
                       void *opaque2, uint64_t id);
diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c
index e8767a1937..7e935deb15 100644
--- a/replay/replay-snapshot.c
+++ b/replay/replay-snapshot.c
@@ -61,7 +61,6 @@ static const VMStateDescription vmstate_replay = {
         VMSTATE_UINT64(block_request_id, ReplayState),
         VMSTATE_INT32(read_event_kind, ReplayState),
         VMSTATE_UINT64(read_event_id, ReplayState),
-        VMSTATE_INT32(read_event_checkpoint, ReplayState),
         VMSTATE_END_OF_LIST()
     },
 };
diff --git a/replay/replay.c b/replay/replay.c
index 2d3607998a..ccd7edec76 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -22,7 +22,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe0200a
+#define REPLAY_VERSION              0xe0200b
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
@@ -171,64 +171,49 @@ void replay_shutdown_request(ShutdownCause cause)
 
 bool replay_checkpoint(ReplayCheckpoint checkpoint)
 {
-    bool res = false;
-    static bool in_checkpoint;
     assert(EVENT_CHECKPOINT + checkpoint <= EVENT_CHECKPOINT_LAST);
 
-    if (!replay_file) {
-        return true;
-    }
-
-    if (in_checkpoint) {
-        /*
-           Recursion occurs when HW event modifies timers.
-           Prevent performing icount warp in this case and
-           wait for another invocation of the checkpoint.
-        */
-        g_assert(replay_mode == REPLAY_MODE_PLAY);
-        return false;
-    }
-    in_checkpoint = true;
-
     replay_save_instructions();
 
     if (replay_mode == REPLAY_MODE_PLAY) {
         g_assert(replay_mutex_locked());
         if (replay_next_event_is(EVENT_CHECKPOINT + checkpoint)) {
             replay_finish_event();
-        } else if (replay_state.data_kind != EVENT_ASYNC) {
-            res = false;
-            goto out;
+        } else {
+            return false;
         }
-        replay_read_events(checkpoint);
-        /* replay_read_events may leave some unread events.
-           Return false if not all of the events associated with
-           checkpoint were processed */
-        res = replay_state.data_kind != EVENT_ASYNC;
     } else if (replay_mode == REPLAY_MODE_RECORD) {
         g_assert(replay_mutex_locked());
         replay_put_event(EVENT_CHECKPOINT + checkpoint);
-        /* This checkpoint belongs to several threads.
-           Processing events from different threads is
-           non-deterministic */
-        if (checkpoint != CHECKPOINT_CLOCK_WARP_START
-            /* FIXME: this is temporary fix, other checkpoints
-                      may also be invoked from the different threads someday.
-                      Asynchronous event processing should be refactored
-                      to create additional replay event kind which is
-                      nailed to the one of the threads and which processes
-                      the event queue. */
-            && checkpoint != CHECKPOINT_CLOCK_VIRTUAL) {
-            replay_save_events(checkpoint);
-        }
-        res = true;
     }
-out:
-    in_checkpoint = false;
-    return res;
+    return true;
+}
+
+void replay_async_events(void)
+{
+    static bool processing = false;
+    /*
+     * If we are already processing the events, recursion may occur
+     * in case of incorrect implementation when HW event modifies timers.
+     * Timer modification may invoke the icount warp, event processing,
+     * and cause the recursion.
+     */
+    g_assert(!processing);
+    processing = true;
+
+    replay_save_instructions();
+
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        g_assert(replay_mutex_locked());
+        replay_read_events();
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        g_assert(replay_mutex_locked());
+        replay_save_events();
+    }
+    processing = false;
 }
 
-bool replay_has_checkpoint(void)
+bool replay_has_event(void)
 {
     bool res = false;
     if (replay_mode == REPLAY_MODE_PLAY) {
@@ -236,6 +221,7 @@ bool replay_has_checkpoint(void)
         replay_account_executed_instructions();
         res = EVENT_CHECKPOINT <= replay_state.data_kind
               && replay_state.data_kind <= EVENT_CHECKPOINT_LAST;
+        res = res || replay_state.data_kind == EVENT_ASYNC;
     }
     return res;
 }
diff --git a/softmmu/icount.c b/softmmu/icount.c
index 1cafec5014..4504433e16 100644
--- a/softmmu/icount.c
+++ b/softmmu/icount.c
@@ -322,7 +322,7 @@ void icount_start_warp_timer(void)
              * to vCPU was processed in advance and vCPU went to sleep.
              * Therefore we have to wake it up for doing someting.
              */
-            if (replay_has_checkpoint()) {
+            if (replay_has_event()) {
                 qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
             }
             return;
@@ -404,6 +404,8 @@ void icount_account_warp_timer(void)
         return;
     }
 
+    replay_async_events();
+
     /* warp clock deterministically in record/replay mode */
     if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_ACCOUNT)) {
         return;

[PATCH v3 4/9] replay: simplify async event processing

[Pavel Dovgalyuk]

This patch joins replay event id and async event id into single byte in the log.
It makes processing a bit faster and log a bit smaller.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

--

v2: minor enum fixes (suggested by Richard Henderson)
---
 replay/replay-events.c   |   36 ++++++++++++++----------------------
 replay/replay-internal.h |   31 ++++++++++++++-----------------
 replay/replay-snapshot.c |    1 -
 replay/replay.c          |    5 +++--
 4 files changed, 31 insertions(+), 42 deletions(-)

diff --git a/replay/replay-events.c b/replay/replay-events.c
index db1decf9dd..af0721cc1a 100644
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -174,8 +174,8 @@ static void replay_save_event(Event *event)
 {
     if (replay_mode != REPLAY_MODE_PLAY) {
         /* put the event into the file */
-        replay_put_event(EVENT_ASYNC);
-        replay_put_byte(event->event_kind);
+        g_assert(event->event_kind < REPLAY_ASYNC_COUNT);
+        replay_put_event(EVENT_ASYNC + event->event_kind);
 
         /* save event-specific data */
         switch (event->event_kind) {
@@ -220,14 +220,10 @@ void replay_save_events(void)
 static Event *replay_read_event(void)
 {
     Event *event;
-    if (replay_state.read_event_kind == -1) {
-        replay_state.read_event_kind = replay_get_byte();
-        replay_state.read_event_id = -1;
-        replay_check_error();
-    }
+    ReplayAsyncEventKind event_kind = replay_state.data_kind - EVENT_ASYNC;
 
     /* Events that has not to be in the queue */
-    switch (replay_state.read_event_kind) {
+    switch (event_kind) {
     case REPLAY_ASYNC_EVENT_BH:
     case REPLAY_ASYNC_EVENT_BH_ONESHOT:
         if (replay_state.read_event_id == -1) {
@@ -236,17 +232,17 @@ static Event *replay_read_event(void)
         break;
     case REPLAY_ASYNC_EVENT_INPUT:
         event = g_new0(Event, 1);
-        event->event_kind = replay_state.read_event_kind;
+        event->event_kind = event_kind;
         event->opaque = replay_read_input_event();
         return event;
     case REPLAY_ASYNC_EVENT_INPUT_SYNC:
         event = g_new0(Event, 1);
-        event->event_kind = replay_state.read_event_kind;
+        event->event_kind = event_kind;
         event->opaque = 0;
         return event;
     case REPLAY_ASYNC_EVENT_CHAR_READ:
         event = g_new0(Event, 1);
-        event->event_kind = replay_state.read_event_kind;
+        event->event_kind = event_kind;
         event->opaque = replay_event_char_read_load();
         return event;
     case REPLAY_ASYNC_EVENT_BLOCK:
@@ -256,18 +252,17 @@ static Event *replay_read_event(void)
         break;
     case REPLAY_ASYNC_EVENT_NET:
         event = g_new0(Event, 1);
-        event->event_kind = replay_state.read_event_kind;
+        event->event_kind = event_kind;
         event->opaque = replay_event_net_load();
         return event;
     default:
-        error_report("Unknown ID %d of replay event",
-            replay_state.read_event_kind);
+        error_report("Unknown ID %d of replay event", event_kind);
         exit(1);
         break;
     }
 
     QTAILQ_FOREACH(event, &events_list, events) {
-        if (event->event_kind == replay_state.read_event_kind
+        if (event->event_kind == event_kind
             && (replay_state.read_event_id == -1
                 || replay_state.read_event_id == event->id)) {
             break;
@@ -276,12 +271,8 @@ static Event *replay_read_event(void)
 
     if (event) {
         QTAILQ_REMOVE(&events_list, event, events);
-    } else {
-        return NULL;
     }
 
-    /* Read event-specific data */
-
     return event;
 }
 
@@ -289,13 +280,14 @@ static Event *replay_read_event(void)
 void replay_read_events(void)
 {
     g_assert(replay_mutex_locked());
-    while (replay_state.data_kind == EVENT_ASYNC) {
+    while (replay_state.data_kind >= EVENT_ASYNC
+        && replay_state.data_kind <= EVENT_ASYNC_LAST) {
         Event *event = replay_read_event();
         if (!event) {
             break;
         }
         replay_finish_event();
-        replay_state.read_event_kind = -1;
+        replay_state.read_event_id = -1;
         replay_run_event(event);
 
         g_free(event);
@@ -304,7 +296,7 @@ void replay_read_events(void)
 
 void replay_init_events(void)
 {
-    replay_state.read_event_kind = -1;
+    replay_state.read_event_id = -1;
 }
 
 void replay_finish_events(void)
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 59797c86cf..301131c1e6 100644
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -12,6 +12,19 @@
  *
  */
 
+/* Asynchronous events IDs */
+
+typedef enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_EVENT_BH,
+    REPLAY_ASYNC_EVENT_BH_ONESHOT,
+    REPLAY_ASYNC_EVENT_INPUT,
+    REPLAY_ASYNC_EVENT_INPUT_SYNC,
+    REPLAY_ASYNC_EVENT_CHAR_READ,
+    REPLAY_ASYNC_EVENT_BLOCK,
+    REPLAY_ASYNC_EVENT_NET,
+    REPLAY_ASYNC_COUNT
+} ReplayAsyncEventKind;
+
 /* Any changes to order/number of events will need to bump REPLAY_VERSION */
 enum ReplayEvents {
     /* for instruction event */
@@ -22,6 +35,7 @@ enum ReplayEvents {
     EVENT_EXCEPTION,
     /* for async events */
     EVENT_ASYNC,
+    EVENT_ASYNC_LAST = EVENT_ASYNC + REPLAY_ASYNC_COUNT - 1,
     /* for shutdown requests, range allows recovery of ShutdownCause */
     EVENT_SHUTDOWN,
     EVENT_SHUTDOWN_LAST = EVENT_SHUTDOWN + SHUTDOWN_CAUSE__MAX,
@@ -49,21 +63,6 @@ enum ReplayEvents {
     EVENT_COUNT
 };
 
-/* Asynchronous events IDs */
-
-enum ReplayAsyncEventKind {
-    REPLAY_ASYNC_EVENT_BH,
-    REPLAY_ASYNC_EVENT_BH_ONESHOT,
-    REPLAY_ASYNC_EVENT_INPUT,
-    REPLAY_ASYNC_EVENT_INPUT_SYNC,
-    REPLAY_ASYNC_EVENT_CHAR_READ,
-    REPLAY_ASYNC_EVENT_BLOCK,
-    REPLAY_ASYNC_EVENT_NET,
-    REPLAY_ASYNC_COUNT
-};
-
-typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
-
 typedef struct ReplayState {
     /*! Cached clock values. */
     int64_t cached_clock[REPLAY_CLOCK_COUNT];
@@ -83,8 +82,6 @@ typedef struct ReplayState {
     uint64_t block_request_id;
     /*! Prior value of the host clock */
     uint64_t host_clock_last;
-    /*! Asynchronous event type read from the log */
-    int32_t read_event_kind;
     /*! Asynchronous event id read from the log */
     uint64_t read_event_id;
 } ReplayState;
diff --git a/replay/replay-snapshot.c b/replay/replay-snapshot.c
index 7e935deb15..10a7cf7992 100644
--- a/replay/replay-snapshot.c
+++ b/replay/replay-snapshot.c
@@ -59,7 +59,6 @@ static const VMStateDescription vmstate_replay = {
         VMSTATE_UINT32(has_unread_data, ReplayState),
         VMSTATE_UINT64(file_offset, ReplayState),
         VMSTATE_UINT64(block_request_id, ReplayState),
-        VMSTATE_INT32(read_event_kind, ReplayState),
         VMSTATE_UINT64(read_event_id, ReplayState),
         VMSTATE_END_OF_LIST()
     },
diff --git a/replay/replay.c b/replay/replay.c
index ccd7edec76..4c396bb376 100644
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -22,7 +22,7 @@
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
-#define REPLAY_VERSION              0xe0200b
+#define REPLAY_VERSION              0xe0200c
 /* Size of replay log header */
 #define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
 
@@ -221,7 +221,8 @@ bool replay_has_event(void)
         replay_account_executed_instructions();
         res = EVENT_CHECKPOINT <= replay_state.data_kind
               && replay_state.data_kind <= EVENT_CHECKPOINT_LAST;
-        res = res || replay_state.data_kind == EVENT_ASYNC;
+        res = res || (EVENT_ASYNC <= replay_state.data_kind
+                     && replay_state.data_kind <= EVENT_ASYNC_LAST);
     }
     return res;
 }

[PATCH v3 5/9] docs: convert docs/devel/replay page to rst

[Pavel Dovgalyuk]

This patch converts prior .txt replay devel documentation to .rst.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
---
 docs/devel/index-tcg.rst |    1 +
 docs/devel/replay.rst    |   54 ++++++++++++++++++++++++++++++++++++++++++++++
 docs/devel/replay.txt    |   46 ---------------------------------------
 3 files changed, 55 insertions(+), 46 deletions(-)
 create mode 100644 docs/devel/replay.rst
 delete mode 100644 docs/devel/replay.txt

diff --git a/docs/devel/index-tcg.rst b/docs/devel/index-tcg.rst
index 0b0ad12c22..7b9760b26f 100644
--- a/docs/devel/index-tcg.rst
+++ b/docs/devel/index-tcg.rst
@@ -13,3 +13,4 @@ are only implementing things for HW accelerated hypervisors.
    multi-thread-tcg
    tcg-icount
    tcg-plugins
+   replay
diff --git a/docs/devel/replay.rst b/docs/devel/replay.rst
new file mode 100644
index 0000000000..dd8bf3b195
--- /dev/null
+++ b/docs/devel/replay.rst
@@ -0,0 +1,54 @@
+..
+   Copyright (c) 2022, ISP RAS
+   Written by Pavel Dovgalyuk
+
+=======================
+Execution Record/Replay
+=======================
+
+Record/replay mechanism, that could be enabled through icount mode, expects
+the virtual devices to satisfy the following requirements.
+
+The main idea behind this document is that everything that affects
+the guest state during execution in icount mode should be deterministic.
+
+Timers
+------
+
+All virtual devices should use virtual clock for timers that change the guest
+state. Virtual clock is deterministic, therefore such timers are deterministic
+too.
+
+Virtual devices can also use realtime clock for the events that do not change
+the guest state directly. When the clock ticking should depend on VM execution
+speed, use virtual clock with EXTERNAL attribute. It is not deterministic,
+but its speed depends on the guest execution. This clock is used by
+the virtual devices (e.g., slirp routing device) that lie outside the
+replayed guest.
+
+Bottom halves
+-------------
+
+Bottom half callbacks, that affect the guest state, should be invoked through
+replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions.
+Their invocations are saved in record mode and synchronized with the existing
+log in replay mode.
+
+Saving/restoring the VM state
+-----------------------------
+
+All fields in the device state structure (including virtual timers)
+should be restored by loadvm to the same values they had before savevm.
+
+Avoid accessing other devices' state, because the order of saving/restoring
+is not defined. It means that you should not call functions like
+'update_irq' in post_load callback. Save everything explicitly to avoid
+the dependencies that may make restoring the VM state non-deterministic.
+
+Stopping the VM
+---------------
+
+Stopping the guest should not interfere with its state (with the exception
+of the network connections, that could be broken by the remote timeouts).
+VM can be stopped at any moment of replay by the user. Restarting the VM
+after that stop should not break the replay by the unneeded guest state change.
diff --git a/docs/devel/replay.txt b/docs/devel/replay.txt
deleted file mode 100644
index e641c35add..0000000000
--- a/docs/devel/replay.txt
+++ /dev/null
@@ -1,46 +0,0 @@
-Record/replay mechanism, that could be enabled through icount mode, expects
-the virtual devices to satisfy the following requirements.
-
-The main idea behind this document is that everything that affects
-the guest state during execution in icount mode should be deterministic.
-
-Timers
-======
-
-All virtual devices should use virtual clock for timers that change the guest
-state. Virtual clock is deterministic, therefore such timers are deterministic
-too.
-
-Virtual devices can also use realtime clock for the events that do not change
-the guest state directly. When the clock ticking should depend on VM execution
-speed, use virtual clock with EXTERNAL attribute. It is not deterministic,
-but its speed depends on the guest execution. This clock is used by
-the virtual devices (e.g., slirp routing device) that lie outside the
-replayed guest.
-
-Bottom halves
-=============
-
-Bottom half callbacks, that affect the guest state, should be invoked through
-replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions.
-Their invocations are saved in record mode and synchronized with the existing
-log in replay mode.
-
-Saving/restoring the VM state
-=============================
-
-All fields in the device state structure (including virtual timers)
-should be restored by loadvm to the same values they had before savevm.
-
-Avoid accessing other devices' state, because the order of saving/restoring
-is not defined. It means that you should not call functions like
-'update_irq' in post_load callback. Save everything explicitly to avoid
-the dependencies that may make restoring the VM state non-deterministic.
-
-Stopping the VM
-===============
-
-Stopping the guest should not interfere with its state (with the exception
-of the network connections, that could be broken by the remote timeouts).
-VM can be stopped at any moment of replay by the user. Restarting the VM
-after that stop should not break the replay by the unneeded guest state change.

[PATCH v3 6/9] docs: move replay docs to docs/system/replay.rst

[Pavel Dovgalyuk]

This patch adds replay description page, converting prior
text from docs/replay.txt.
The text was also updated and some sections were moved
to devel part of the docs.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
---
 docs/devel/replay.rst  |  264 ++++++++++++++++++++++++++++++-
 docs/replay.txt        |  407 ------------------------------------------------
 docs/system/index.rst  |    1 
 docs/system/replay.rst |  237 ++++++++++++++++++++++++++++
 4 files changed, 496 insertions(+), 413 deletions(-)
 delete mode 100644 docs/replay.txt
 create mode 100644 docs/system/replay.rst

diff --git a/docs/devel/replay.rst b/docs/devel/replay.rst
index dd8bf3b195..0244be8b9c 100644
--- a/docs/devel/replay.rst
+++ b/docs/devel/replay.rst
@@ -1,20 +1,149 @@
 ..
    Copyright (c) 2022, ISP RAS
-   Written by Pavel Dovgalyuk
+   Written by Pavel Dovgalyuk and Alex Bennée
 
 =======================
 Execution Record/Replay
 =======================
 
-Record/replay mechanism, that could be enabled through icount mode, expects
-the virtual devices to satisfy the following requirements.
+Core concepts
+=============
+
+Record/replay functions are used for the deterministic replay of qemu
+execution. Execution recording writes a non-deterministic events log, which
+can be later used for replaying the execution anywhere and for unlimited
+number of times. Execution replaying reads the log and replays all
+non-deterministic events including external input, hardware clocks,
+and interrupts.
+
+Several parts of QEMU include function calls to make event log recording
+and replaying.
+Devices' models that have non-deterministic input from external devices were
+changed to write every external event into the execution log immediately.
+E.g. network packets are written into the log when they arrive into the virtual
+network adapter.
+
+All non-deterministic events are coming from these devices. But to
+replay them we need to know at which moments they occur. We specify
+these moments by counting the number of instructions executed between
+every pair of consecutive events.
+
+Academic papers with description of deterministic replay implementation:
+
+* `Deterministic Replay of System's Execution with Multi-target QEMU Simulator for Dynamic Analysis and Reverse Debugging <https://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html>`_
+* `Don't panic: reverse debugging of kernel drivers <https://dl.acm.org/citation.cfm?id=2786805.2803179>`_
+
+Modifications of qemu include:
+
+ * wrappers for clock and time functions to save their return values in the log
+ * saving different asynchronous events (e.g. system shutdown) into the log
+ * synchronization of the bottom halves execution
+ * synchronization of the threads from thread pool
+ * recording/replaying user input (mouse, keyboard, and microphone)
+ * adding internal checkpoints for cpu and io synchronization
+ * network filter for recording and replaying the packets
+ * block driver for making block layer deterministic
+ * serial port input record and replay
+ * recording of random numbers obtained from the external sources
+
+Instruction counting
+--------------------
+
+QEMU should work in icount mode to use record/replay feature. icount was
+designed to allow deterministic execution in absence of external inputs
+of the virtual machine. We also use icount to control the occurrence of the
+non-deterministic events. The number of instructions elapsed from the last event
+is written to the log while recording the execution. In replay mode we
+can predict when to inject that event using the instruction counter.
+
+Locking and thread synchronisation
+----------------------------------
+
+Previously the synchronisation of the main thread and the vCPU thread
+was ensured by the holding of the BQL. However the trend has been to
+reduce the time the BQL was held across the system including under TCG
+system emulation. As it is important that batches of events are kept
+in sequence (e.g. expiring timers and checkpoints in the main thread
+while instruction checkpoints are written by the vCPU thread) we need
+another lock to keep things in lock-step. This role is now handled by
+the replay_mutex_lock. It used to be held only for each event being
+written but now it is held for a whole execution period. This results
+in a deterministic ping-pong between the two main threads.
+
+As the BQL is now a finer grained lock than the replay_lock it is almost
+certainly a bug, and a source of deadlocks, to take the
+replay_mutex_lock while the BQL is held. This is enforced by an assert.
+While the unlocks are usually in the reverse order, this is not
+necessary; you can drop the replay_lock while holding the BQL, without
+doing a more complicated unlock_iothread/replay_unlock/lock_iothread
+sequence.
+
+Checkpoints
+-----------
+
+Replaying the execution of virtual machine is bound by sources of
+non-determinism. These are inputs from clock and peripheral devices,
+and QEMU thread scheduling. Thread scheduling affect on processing events
+from timers, asynchronous input-output, and bottom halves.
 
-The main idea behind this document is that everything that affects
+Invocations of timers are coupled with clock reads and changing the state
+of the virtual machine. Reads produce non-deterministic data taken from
+host clock. And VM state changes should preserve their order. Their relative
+order in replay mode must replicate the order of callbacks in record mode.
+To preserve this order we use checkpoints. When a specific clock is processed
+in record mode we save to the log special "checkpoint" event.
+Checkpoints here do not refer to virtual machine snapshots. They are just
+record/replay events used for synchronization.
+
+QEMU in replay mode will try to invoke timers processing in random moment
+of time. That's why we do not process a group of timers until the checkpoint
+event will be read from the log. Such an event allows synchronizing CPU
+execution and timer events.
+
+Two other checkpoints govern the "warping" of the virtual clock.
+While the virtual machine is idle, the virtual clock increments at
+1 ns per *real time* nanosecond.  This is done by setting up a timer
+(called the warp timer) on the virtual real time clock, so that the
+timer fires at the next deadline of the virtual clock; the virtual clock
+is then incremented (which is called "warping" the virtual clock) as
+soon as the timer fires or the CPUs need to go out of the idle state.
+Two functions are used for this purpose; because these actions change
+virtual machine state and must be deterministic, each of them creates a
+checkpoint. ``icount_start_warp_timer`` checks if the CPUs are idle and if so
+starts accounting real time to virtual clock. ``icount_account_warp_timer``
+is called when the CPUs get an interrupt or when the warp timer fires,
+and it warps the virtual clock by the amount of real time that has passed
+since ``icount_start_warp_timer``.
+
+Virtual devices
+===============
+
+Record/replay mechanism, that could be enabled through icount mode, expects
+the virtual devices to satisfy the following requirement:
+everything that affects
 the guest state during execution in icount mode should be deterministic.
 
 Timers
 ------
 
+Timers are used to execute callbacks from different subsystems of QEMU
+at the specified moments of time. There are several kinds of timers:
+
+ * Real time clock. Based on host time and used only for callbacks that
+   do not change the virtual machine state. For this reason real time
+   clock and timers does not affect deterministic replay at all.
+ * Virtual clock. These timers run only during the emulation. In icount
+   mode virtual clock value is calculated using executed instructions counter.
+   That is why it is completely deterministic and does not have to be recorded.
+ * Host clock. This clock is used by device models that simulate real time
+   sources (e.g. real time clock chip). Host clock is the one of the sources
+   of non-determinism. Host clock read operations should be logged to
+   make the execution deterministic.
+ * Virtual real time clock. This clock is similar to real time clock but
+   it is used only for increasing virtual clock while virtual machine is
+   sleeping. Due to its nature it is also non-deterministic as the host clock
+   and has to be logged too.
+
 All virtual devices should use virtual clock for timers that change the guest
 state. Virtual clock is deterministic, therefore such timers are deterministic
 too.
@@ -26,14 +155,50 @@ but its speed depends on the guest execution. This clock is used by
 the virtual devices (e.g., slirp routing device) that lie outside the
 replayed guest.
 
+Block devices
+-------------
+
+Block devices record/replay module (``blkreplay``) intercepts calls of
+bdrv coroutine functions at the top of block drivers stack.
+
+All block completion operations are added to the queue in the coroutines.
+When the queue is flushed the information about processed requests
+is recorded to the log. In replay phase the queue is matched with
+events read from the log. Therefore block devices requests are processed
+deterministically.
+
 Bottom halves
 -------------
 
 Bottom half callbacks, that affect the guest state, should be invoked through
-replay_bh_schedule_event or replay_bh_schedule_oneshot_event functions.
+``replay_bh_schedule_event`` or ``replay_bh_schedule_oneshot_event`` functions.
 Their invocations are saved in record mode and synchronized with the existing
 log in replay mode.
 
+Disk I/O events are completely deterministic in our model, because
+in both record and replay modes we start virtual machine from the same
+disk state. But callbacks that virtual disk controller uses for reading and
+writing the disk may occur at different moments of time in record and replay
+modes.
+
+Reading and writing requests are created by CPU thread of QEMU. Later these
+requests proceed to block layer which creates "bottom halves". Bottom
+halves consist of callback and its parameters. They are processed when
+main loop locks the global mutex. These locks are not synchronized with
+replaying process because main loop also processes the events that do not
+affect the virtual machine state (like user interaction with monitor).
+
+That is why we had to implement saving and replaying bottom halves callbacks
+synchronously to the CPU execution. When the callback is about to execute
+it is added to the queue in the replay module. This queue is written to the
+log when its callbacks are executed. In replay mode callbacks are not processed
+until the corresponding event is read from the events log file.
+
+Sometimes the block layer uses asynchronous callbacks for its internal purposes
+(like reading or writing VM snapshots or disk image cluster tables). In this
+case bottom halves are not marked as "replayable" and do not saved
+into the log.
+
 Saving/restoring the VM state
 -----------------------------
 
@@ -42,7 +207,7 @@ should be restored by loadvm to the same values they had before savevm.
 
 Avoid accessing other devices' state, because the order of saving/restoring
 is not defined. It means that you should not call functions like
-'update_irq' in post_load callback. Save everything explicitly to avoid
+``update_irq`` in ``post_load`` callback. Save everything explicitly to avoid
 the dependencies that may make restoring the VM state non-deterministic.
 
 Stopping the VM
@@ -52,3 +217,90 @@ Stopping the guest should not interfere with its state (with the exception
 of the network connections, that could be broken by the remote timeouts).
 VM can be stopped at any moment of replay by the user. Restarting the VM
 after that stop should not break the replay by the unneeded guest state change.
+
+Replay log format
+=================
+
+Record/replay log consists of the header and the sequence of execution
+events. The header includes 4-byte replay version id and 8-byte reserved
+field. Version is updated every time replay log format changes to prevent
+using replay log created by another build of qemu.
+
+The sequence of the events describes virtual machine state changes.
+It includes all non-deterministic inputs of VM, synchronization marks and
+instruction counts used to correctly inject inputs at replay.
+
+Synchronization marks (checkpoints) are used for synchronizing qemu threads
+that perform operations with virtual hardware. These operations may change
+system's state (e.g., change some register or generate interrupt) and
+therefore should execute synchronously with CPU thread.
+
+Every event in the log includes 1-byte event id and optional arguments.
+When argument is an array, it is stored as 4-byte array length
+and corresponding number of bytes with data.
+Here is the list of events that are written into the log:
+
+ - EVENT_INSTRUCTION. Instructions executed since last event. Followed by:
+
+   - 4-byte number of executed instructions.
+
+ - EVENT_INTERRUPT. Used to synchronize interrupt processing.
+ - EVENT_EXCEPTION. Used to synchronize exception handling.
+ - EVENT_ASYNC. This is a group of events. When such an event is generated,
+   it is stored in the queue and processed in icount_account_warp_timer().
+   Every such event has it's own id from the following list:
+
+     - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes
+       callbacks that affect virtual machine state, but normally called
+       asynchronously. Followed by:
+
+        - 8-byte operation id.
+
+     - REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains
+       parameters of keyboard and mouse input operations
+       (key press/release, mouse pointer movement). Followed by:
+
+        - 9-16 bytes depending of input event.
+
+     - REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event.
+     - REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device input
+       initiated by the sender. Followed by:
+
+        - 1-byte character device id.
+        - Array with bytes were read.
+
+     - REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchronize
+       operations with disk and flash drives with CPU. Followed by:
+
+        - 8-byte operation id.
+
+     - REPLAY_ASYNC_EVENT_NET. Incoming network packet. Followed by:
+
+        - 1-byte network adapter id.
+        - 4-byte packet flags.
+        - Array with packet bytes.
+
+ - EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu,
+   e.g., by closing the window.
+ - EVENT_CHAR_WRITE. Used to synchronize character output operations. Followed by:
+
+    - 4-byte output function return value.
+    - 4-byte offset in the output array.
+
+ - EVENT_CHAR_READ_ALL. Used to synchronize character input operations,
+   initiated by qemu. Followed by:
+
+    - Array with bytes that were read.
+
+ - EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation,
+   initiated by qemu. Followed by:
+
+    - 4-byte error code.
+
+ - EVENT_CLOCK + clock_id. Group of events for host clock read operations. Followed by:
+
+    - 8-byte clock value.
+
+ - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of
+   CPU, internal threads, and asynchronous input events.
+ - EVENT_END. Last event in the log.
diff --git a/docs/replay.txt b/docs/replay.txt
deleted file mode 100644
index 6c9fdff09d..0000000000
--- a/docs/replay.txt
+++ /dev/null
@@ -1,407 +0,0 @@
-Copyright (c) 2010-2015 Institute for System Programming
-                        of the Russian Academy of Sciences.
-
-This work is licensed under the terms of the GNU GPL, version 2 or later.
-See the COPYING file in the top-level directory.
-
-Record/replay
--------------
-
-Record/replay functions are used for the deterministic replay of qemu execution.
-Execution recording writes a non-deterministic events log, which can be later
-used for replaying the execution anywhere and for unlimited number of times.
-It also supports checkpointing for faster rewind to the specific replay moment.
-Execution replaying reads the log and replays all non-deterministic events
-including external input, hardware clocks, and interrupts.
-
-Deterministic replay has the following features:
- * Deterministically replays whole system execution and all contents of
-   the memory, state of the hardware devices, clocks, and screen of the VM.
- * Writes execution log into the file for later replaying for multiple times
-   on different machines.
- * Supports i386, x86_64, and Arm hardware platforms.
- * Performs deterministic replay of all operations with keyboard and mouse
-   input devices.
-
-Usage of the record/replay:
- * First, record the execution with the following command line:
-    qemu-system-i386 \
-     -icount shift=7,rr=record,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
-     -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
-     -device ide-hd,drive=img-blkreplay \
-     -netdev user,id=net1 -device rtl8139,netdev=net1 \
-     -object filter-replay,id=replay,netdev=net1
- * After recording, you can replay it by using another command line:
-    qemu-system-i386 \
-     -icount shift=7,rr=replay,rrfile=replay.bin \
-     -drive file=disk.qcow2,if=none,snapshot,id=img-direct \
-     -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
-     -device ide-hd,drive=img-blkreplay \
-     -netdev user,id=net1 -device rtl8139,netdev=net1 \
-     -object filter-replay,id=replay,netdev=net1
-   The only difference with recording is changing the rr option
-   from record to replay.
- * Block device images are not actually changed in the recording mode,
-   because all of the changes are written to the temporary overlay file.
-   This behavior is enabled by using blkreplay driver. It should be used
-   for every enabled block device, as described in 'Block devices' section.
- * '-net none' option should be specified when network is not used,
-   because QEMU adds network card by default. When network is needed,
-   it should be configured explicitly with replay filter, as described
-   in 'Network devices' section.
- * Interaction with audio devices and serial ports are recorded and replayed
-   automatically when such devices are enabled.
-
-Academic papers with description of deterministic replay implementation:
-http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
-http://dl.acm.org/citation.cfm?id=2786805.2803179
-
-Modifications of qemu include:
- * wrappers for clock and time functions to save their return values in the log
- * saving different asynchronous events (e.g. system shutdown) into the log
- * synchronization of the bottom halves execution
- * synchronization of the threads from thread pool
- * recording/replaying user input (mouse, keyboard, and microphone)
- * adding internal checkpoints for cpu and io synchronization
- * network filter for recording and replaying the packets
- * block driver for making block layer deterministic
- * serial port input record and replay
- * recording of random numbers obtained from the external sources
-
-Locking and thread synchronisation
-----------------------------------
-
-Previously the synchronisation of the main thread and the vCPU thread
-was ensured by the holding of the BQL. However the trend has been to
-reduce the time the BQL was held across the system including under TCG
-system emulation. As it is important that batches of events are kept
-in sequence (e.g. expiring timers and checkpoints in the main thread
-while instruction checkpoints are written by the vCPU thread) we need
-another lock to keep things in lock-step. This role is now handled by
-the replay_mutex_lock. It used to be held only for each event being
-written but now it is held for a whole execution period. This results
-in a deterministic ping-pong between the two main threads.
-
-As the BQL is now a finer grained lock than the replay_lock it is almost
-certainly a bug, and a source of deadlocks, to take the
-replay_mutex_lock while the BQL is held. This is enforced by an assert.
-While the unlocks are usually in the reverse order, this is not
-necessary; you can drop the replay_lock while holding the BQL, without
-doing a more complicated unlock_iothread/replay_unlock/lock_iothread
-sequence.
-
-Non-deterministic events
-------------------------
-
-Our record/replay system is based on saving and replaying non-deterministic
-events (e.g. keyboard input) and simulating deterministic ones (e.g. reading
-from HDD or memory of the VM). Saving only non-deterministic events makes
-log file smaller and simulation faster.
-
-The following non-deterministic data from peripheral devices is saved into
-the log: mouse and keyboard input, network packets, audio controller input,
-serial port input, and hardware clocks (they are non-deterministic
-too, because their values are taken from the host machine). Inputs from
-simulated hardware, memory of VM, software interrupts, and execution of
-instructions are not saved into the log, because they are deterministic and
-can be replayed by simulating the behavior of virtual machine starting from
-initial state.
-
-We had to solve three tasks to implement deterministic replay: recording
-non-deterministic events, replaying non-deterministic events, and checking
-that there is no divergence between record and replay modes.
-
-We changed several parts of QEMU to make event log recording and replaying.
-Devices' models that have non-deterministic input from external devices were
-changed to write every external event into the execution log immediately.
-E.g. network packets are written into the log when they arrive into the virtual
-network adapter.
-
-All non-deterministic events are coming from these devices. But to
-replay them we need to know at which moments they occur. We specify
-these moments by counting the number of instructions executed between
-every pair of consecutive events.
-
-Instruction counting
---------------------
-
-QEMU should work in icount mode to use record/replay feature. icount was
-designed to allow deterministic execution in absence of external inputs
-of the virtual machine. We also use icount to control the occurrence of the
-non-deterministic events. The number of instructions elapsed from the last event
-is written to the log while recording the execution. In replay mode we
-can predict when to inject that event using the instruction counter.
-
-Timers
-------
-
-Timers are used to execute callbacks from different subsystems of QEMU
-at the specified moments of time. There are several kinds of timers:
- * Real time clock. Based on host time and used only for callbacks that
-   do not change the virtual machine state. For this reason real time
-   clock and timers does not affect deterministic replay at all.
- * Virtual clock. These timers run only during the emulation. In icount
-   mode virtual clock value is calculated using executed instructions counter.
-   That is why it is completely deterministic and does not have to be recorded.
- * Host clock. This clock is used by device models that simulate real time
-   sources (e.g. real time clock chip). Host clock is the one of the sources
-   of non-determinism. Host clock read operations should be logged to
-   make the execution deterministic.
- * Virtual real time clock. This clock is similar to real time clock but
-   it is used only for increasing virtual clock while virtual machine is
-   sleeping. Due to its nature it is also non-deterministic as the host clock
-   and has to be logged too.
-
-Checkpoints
------------
-
-Replaying of the execution of virtual machine is bound by sources of
-non-determinism. These are inputs from clock and peripheral devices,
-and QEMU thread scheduling. Thread scheduling affect on processing events
-from timers, asynchronous input-output, and bottom halves.
-
-Invocations of timers are coupled with clock reads and changing the state
-of the virtual machine. Reads produce non-deterministic data taken from
-host clock. And VM state changes should preserve their order. Their relative
-order in replay mode must replicate the order of callbacks in record mode.
-To preserve this order we use checkpoints. When a specific clock is processed
-in record mode we save to the log special "checkpoint" event.
-Checkpoints here do not refer to virtual machine snapshots. They are just
-record/replay events used for synchronization.
-
-QEMU in replay mode will try to invoke timers processing in random moment
-of time. That's why we do not process a group of timers until the checkpoint
-event will be read from the log. Such an event allows synchronizing CPU
-execution and timer events.
-
-Two other checkpoints govern the "warping" of the virtual clock.
-While the virtual machine is idle, the virtual clock increments at
-1 ns per *real time* nanosecond.  This is done by setting up a timer
-(called the warp timer) on the virtual real time clock, so that the
-timer fires at the next deadline of the virtual clock; the virtual clock
-is then incremented (which is called "warping" the virtual clock) as
-soon as the timer fires or the CPUs need to go out of the idle state.
-Two functions are used for this purpose; because these actions change
-virtual machine state and must be deterministic, each of them creates a
-checkpoint.  icount_start_warp_timer checks if the CPUs are idle and if so
-starts accounting real time to virtual clock.  icount_account_warp_timer
-is called when the CPUs get an interrupt or when the warp timer fires,
-and it warps the virtual clock by the amount of real time that has passed
-since icount_start_warp_timer.
-
-Bottom halves
--------------
-
-Disk I/O events are completely deterministic in our model, because
-in both record and replay modes we start virtual machine from the same
-disk state. But callbacks that virtual disk controller uses for reading and
-writing the disk may occur at different moments of time in record and replay
-modes.
-
-Reading and writing requests are created by CPU thread of QEMU. Later these
-requests proceed to block layer which creates "bottom halves". Bottom
-halves consist of callback and its parameters. They are processed when
-main loop locks the global mutex. These locks are not synchronized with
-replaying process because main loop also processes the events that do not
-affect the virtual machine state (like user interaction with monitor).
-
-That is why we had to implement saving and replaying bottom halves callbacks
-synchronously to the CPU execution. When the callback is about to execute
-it is added to the queue in the replay module. This queue is written to the
-log when its callbacks are executed. In replay mode callbacks are not processed
-until the corresponding event is read from the events log file.
-
-Sometimes the block layer uses asynchronous callbacks for its internal purposes
-(like reading or writing VM snapshots or disk image cluster tables). In this
-case bottom halves are not marked as "replayable" and do not saved
-into the log.
-
-Block devices
--------------
-
-Block devices record/replay module intercepts calls of
-bdrv coroutine functions at the top of block drivers stack.
-To record and replay block operations the drive must be configured
-as following:
- -drive file=disk.qcow2,if=none,snapshot,id=img-direct
- -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
- -device ide-hd,drive=img-blkreplay
-
-blkreplay driver should be inserted between disk image and virtual driver
-controller. Therefore all disk requests may be recorded and replayed.
-
-All block completion operations are added to the queue in the coroutines.
-Queue is flushed at checkpoints and information about processed requests
-is recorded to the log. In replay phase the queue is matched with
-events read from the log. Therefore block devices requests are processed
-deterministically.
-
-Snapshotting
-------------
-
-New VM snapshots may be created in replay mode. They can be used later
-to recover the desired VM state. All VM states created in replay mode
-are associated with the moment of time in the replay scenario.
-After recovering the VM state replay will start from that position.
-
-Default starting snapshot name may be specified with icount field
-rrsnapshot as follows:
- -icount shift=7,rr=record,rrfile=replay.bin,rrsnapshot=snapshot_name
-
-This snapshot is created at start of recording and restored at start
-of replaying. It also can be loaded while replaying to roll back
-the execution.
-
-'snapshot' flag of the disk image must be removed to save the snapshots
-in the overlay (or original image) instead of using the temporary overlay.
- -drive file=disk.ovl,if=none,id=img-direct
- -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
- -device ide-hd,drive=img-blkreplay
-
-Use QEMU monitor to create additional snapshots. 'savevm <name>' command
-created the snapshot and 'loadvm <name>' restores it. To prevent corruption
-of the original disk image, use overlay files linked to the original images.
-Therefore all new snapshots (including the starting one) will be saved in
-overlays and the original image remains unchanged.
-
-When you need to use snapshots with diskless virtual machine,
-it must be started with 'orphan' qcow2 image. This image will be used
-for storing VM snapshots. Here is the example of the command line for this:
-
-  qemu-system-i386 -icount shift=3,rr=replay,rrfile=record.bin,rrsnapshot=init \
-    -net none -drive file=empty.qcow2,if=none,id=rr
-
-empty.qcow2 drive does not connected to any virtual block device and used
-for VM snapshots only.
-
-Network devices
----------------
-
-Record and replay for network interactions is performed with the network filter.
-Each backend must have its own instance of the replay filter as follows:
- -netdev user,id=net1 -device rtl8139,netdev=net1
- -object filter-replay,id=replay,netdev=net1
-
-Replay network filter is used to record and replay network packets. While
-recording the virtual machine this filter puts all packets coming from
-the outer world into the log. In replay mode packets from the log are
-injected into the network device. All interactions with network backend
-in replay mode are disabled.
-
-Audio devices
--------------
-
-Audio data is recorded and replay automatically. The command line for recording
-and replaying must contain identical specifications of audio hardware, e.g.:
- -soundhw ac97
-
-Serial ports
-------------
-
-Serial ports input is recorded and replay automatically. The command lines
-for recording and replaying must contain identical number of ports in record
-and replay modes, but their backends may differ.
-E.g., '-serial stdio' in record mode, and '-serial null' in replay mode.
-
-Reverse debugging
------------------
-
-Reverse debugging allows "executing" the program in reverse direction.
-GDB remote protocol supports "reverse step" and "reverse continue"
-commands. The first one steps single instruction backwards in time,
-and the second one finds the last breakpoint in the past.
-
-Recorded executions may be used to enable reverse debugging. QEMU can't
-execute the code in backwards direction, but can load a snapshot and
-replay forward to find the desired position or breakpoint.
-
-The following GDB commands are supported:
- - reverse-stepi (or rsi) - step one instruction backwards
- - reverse-continue (or rc) - find last breakpoint in the past
-
-Reverse step loads the nearest snapshot and replays the execution until
-the required instruction is met.
-
-Reverse continue may include several passes of examining the execution
-between the snapshots. Each of the passes include the following steps:
- 1. loading the snapshot
- 2. replaying to examine the breakpoints
- 3. if breakpoint or watchpoint was met
-    - loading the snapshot again
-    - replaying to the required breakpoint
- 4. else
-    - proceeding to the p.1 with the earlier snapshot
-
-Therefore usage of the reverse debugging requires at least one snapshot
-created in advance. This can be done by omitting 'snapshot' option
-for the block drives and adding 'rrsnapshot' for both record and replay
-command lines.
-See the "Snapshotting" section to learn more about running record/replay
-and creating the snapshot in these modes.
-
-Replay log format
------------------
-
-Record/replay log consists of the header and the sequence of execution
-events. The header includes 4-byte replay version id and 8-byte reserved
-field. Version is updated every time replay log format changes to prevent
-using replay log created by another build of qemu.
-
-The sequence of the events describes virtual machine state changes.
-It includes all non-deterministic inputs of VM, synchronization marks and
-instruction counts used to correctly inject inputs at replay.
-
-Synchronization marks (checkpoints) are used for synchronizing qemu threads
-that perform operations with virtual hardware. These operations may change
-system's state (e.g., change some register or generate interrupt) and
-therefore should execute synchronously with CPU thread.
-
-Every event in the log includes 1-byte event id and optional arguments.
-When argument is an array, it is stored as 4-byte array length
-and corresponding number of bytes with data.
-Here is the list of events that are written into the log:
-
- - EVENT_INSTRUCTION. Instructions executed since last event.
-   Argument: 4-byte number of executed instructions.
- - EVENT_INTERRUPT. Used to synchronize interrupt processing.
- - EVENT_EXCEPTION. Used to synchronize exception handling.
- - EVENT_ASYNC. This is a group of events. When such an event is generated,
-   it is stored in the queue and processed in icount_account_warp_timer().
-   Every such event has it's own id from the following list:
-     - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes
-       callbacks that affect virtual machine state, but normally called
-       asynchronously.
-       Argument: 8-byte operation id.
-     - REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains
-       parameters of keyboard and mouse input operations
-       (key press/release, mouse pointer movement).
-       Arguments: 9-16 bytes depending of input event.
-     - REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event.
-     - REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device input
-       initiated by the sender.
-       Arguments: 1-byte character device id.
-                  Array with bytes were read.
-     - REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchronize
-       operations with disk and flash drives with CPU.
-       Argument: 8-byte operation id.
-     - REPLAY_ASYNC_EVENT_NET. Incoming network packet.
-       Arguments: 1-byte network adapter id.
-                  4-byte packet flags.
-                  Array with packet bytes.
- - EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu,
-   e.g., by closing the window.
- - EVENT_CHAR_WRITE. Used to synchronize character output operations.
-   Arguments: 4-byte output function return value.
-              4-byte offset in the output array.
- - EVENT_CHAR_READ_ALL. Used to synchronize character input operations,
-   initiated by qemu.
-   Argument: Array with bytes that were read.
- - EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation,
-   initiated by qemu.
-   Argument: 4-byte error code.
- - EVENT_CLOCK + clock_id. Group of events for host clock read operations.
-   Argument: 8-byte clock value.
- - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of
-   CPU, internal threads, and asynchronous input events.
- - EVENT_END. Last event in the log.
diff --git a/docs/system/index.rst b/docs/system/index.rst
index 23e30e26e5..e3695649c5 100644
--- a/docs/system/index.rst
+++ b/docs/system/index.rst
@@ -27,6 +27,7 @@ or Hypervisor.Framework.
    secrets
    authz
    gdb
+   replay
    managed-startup
    bootindex
    cpu-hotplug
diff --git a/docs/system/replay.rst b/docs/system/replay.rst
new file mode 100644
index 0000000000..3105327423
--- /dev/null
+++ b/docs/system/replay.rst
@@ -0,0 +1,237 @@
+.. _replay:
+
+..
+    Copyright (c) 2010-2022 Institute for System Programming
+                        of the Russian Academy of Sciences.
+
+    This work is licensed under the terms of the GNU GPL, version 2 or later.
+    See the COPYING file in the top-level directory.
+
+Record/replay
+=============
+
+Record/replay functions are used for the deterministic replay of qemu execution.
+Execution recording writes a non-deterministic events log, which can be later
+used for replaying the execution anywhere and for unlimited number of times.
+It also supports checkpointing for faster rewind to the specific replay moment.
+Execution replaying reads the log and replays all non-deterministic events
+including external input, hardware clocks, and interrupts.
+
+Deterministic replay has the following features:
+
+ * Deterministically replays whole system execution and all contents of
+   the memory, state of the hardware devices, clocks, and screen of the VM.
+ * Writes execution log into the file for later replaying for multiple times
+   on different machines.
+ * Supports i386, x86_64, ARM, AArch64, Risc-V, MIPS, MIPS64, S390X, Alpha,
+   PowerPC, PowerPC64, M68000, Microblaze, OpenRISC, Nios II, SPARC,
+   and Xtensa hardware platforms.
+ * Performs deterministic replay of all operations with keyboard and mouse
+   input devices, serial ports, and network.
+
+Usage of the record/replay:
+
+ * First, record the execution with the following command line:
+
+    .. parsed-literal::
+        |qemu_system| \\
+        -icount shift=auto,rr=record,rrfile=replay.bin \\
+        -drive file=disk.qcow2,if=none,snapshot,id=img-direct \\
+        -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \\
+        -device ide-hd,drive=img-blkreplay \\
+        -netdev user,id=net1 -device rtl8139,netdev=net1 \\
+        -object filter-replay,id=replay,netdev=net1
+
+ * After recording, you can replay it by using another command line:
+
+    .. parsed-literal::
+        |qemu_system| \\
+        -icount shift=auto,rr=replay,rrfile=replay.bin \\
+        -drive file=disk.qcow2,if=none,snapshot,id=img-direct \\
+        -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \\
+        -device ide-hd,drive=img-blkreplay \\
+        -netdev user,id=net1 -device rtl8139,netdev=net1 \\
+        -object filter-replay,id=replay,netdev=net1
+
+   The only difference with recording is changing the rr option
+   from record to replay.
+ * Block device images are not actually changed in the recording mode,
+   because all of the changes are written to the temporary overlay file.
+   This behavior is enabled by using blkreplay driver. It should be used
+   for every enabled block device, as described in :ref:`block-label` section.
+ * ``-net none`` option should be specified when network is not used,
+   because QEMU adds network card by default. When network is needed,
+   it should be configured explicitly with replay filter, as described
+   in :ref:`network-label` section.
+ * Interaction with audio devices and serial ports are recorded and replayed
+   automatically when such devices are enabled.
+
+Core idea
+---------
+
+Record/replay system is based on saving and replaying non-deterministic
+events (e.g. keyboard input) and simulating deterministic ones (e.g. reading
+from HDD or memory of the VM). Saving only non-deterministic events makes
+log file smaller and simulation faster.
+
+The following non-deterministic data from peripheral devices is saved into
+the log: mouse and keyboard input, network packets, audio controller input,
+serial port input, and hardware clocks (they are non-deterministic
+too, because their values are taken from the host machine). Inputs from
+simulated hardware, memory of VM, software interrupts, and execution of
+instructions are not saved into the log, because they are deterministic and
+can be replayed by simulating the behavior of virtual machine starting from
+initial state.
+
+Instruction counting
+--------------------
+
+QEMU should work in icount mode to use record/replay feature. icount was
+designed to allow deterministic execution in absence of external inputs
+of the virtual machine. Record/replay feature is enabled through ``-icount``
+command-line option, making possible deterministic execution of the machine,
+interacting with user or network.
+
+.. _block-label:
+
+Block devices
+-------------
+
+Block devices record/replay module intercepts calls of
+bdrv coroutine functions at the top of block drivers stack.
+To record and replay block operations the drive must be configured
+as following:
+
+.. parsed-literal::
+    -drive file=disk.qcow2,if=none,snapshot,id=img-direct
+    -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+    -device ide-hd,drive=img-blkreplay
+
+blkreplay driver should be inserted between disk image and virtual driver
+controller. Therefore all disk requests may be recorded and replayed.
+
+.. _snapshotting-label:
+
+Snapshotting
+------------
+
+New VM snapshots may be created in replay mode. They can be used later
+to recover the desired VM state. All VM states created in replay mode
+are associated with the moment of time in the replay scenario.
+After recovering the VM state replay will start from that position.
+
+Default starting snapshot name may be specified with icount field
+rrsnapshot as follows:
+
+.. parsed-literal::
+    -icount shift=auto,rr=record,rrfile=replay.bin,rrsnapshot=snapshot_name
+
+This snapshot is created at start of recording and restored at start
+of replaying. It also can be loaded while replaying to roll back
+the execution.
+
+``snapshot`` flag of the disk image must be removed to save the snapshots
+in the overlay (or original image) instead of using the temporary overlay.
+
+.. parsed-literal::
+    -drive file=disk.ovl,if=none,id=img-direct
+    -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay
+    -device ide-hd,drive=img-blkreplay
+
+Use QEMU monitor to create additional snapshots. ``savevm <name>`` command
+created the snapshot and ``loadvm <name>`` restores it. To prevent corruption
+of the original disk image, use overlay files linked to the original images.
+Therefore all new snapshots (including the starting one) will be saved in
+overlays and the original image remains unchanged.
+
+When you need to use snapshots with diskless virtual machine,
+it must be started with "orphan" qcow2 image. This image will be used
+for storing VM snapshots. Here is the example of the command line for this:
+
+.. parsed-literal::
+    |qemu_system| \\
+      -icount shift=auto,rr=replay,rrfile=record.bin,rrsnapshot=init \\
+      -net none -drive file=empty.qcow2,if=none,id=rr
+
+``empty.qcow2`` drive does not connected to any virtual block device and used
+for VM snapshots only.
+
+.. _network-label:
+
+Network devices
+---------------
+
+Record and replay for network interactions is performed with the network filter.
+Each backend must have its own instance of the replay filter as follows:
+
+.. parsed-literal::
+    -netdev user,id=net1 -device rtl8139,netdev=net1
+    -object filter-replay,id=replay,netdev=net1
+
+Replay network filter is used to record and replay network packets. While
+recording the virtual machine this filter puts all packets coming from
+the outer world into the log. In replay mode packets from the log are
+injected into the network device. All interactions with network backend
+in replay mode are disabled.
+
+Audio devices
+-------------
+
+Audio data is recorded and replay automatically. The command line for recording
+and replaying must contain identical specifications of audio hardware, e.g.:
+
+.. parsed-literal::
+    -soundhw ac97
+
+Serial ports
+------------
+
+Serial ports input is recorded and replay automatically. The command lines
+for recording and replaying must contain identical number of ports in record
+and replay modes, but their backends may differ.
+E.g., ``-serial stdio`` in record mode, and ``-serial null`` in replay mode.
+
+Reverse debugging
+-----------------
+
+Reverse debugging allows "executing" the program in reverse direction.
+GDB remote protocol supports "reverse step" and "reverse continue"
+commands. The first one steps single instruction backwards in time,
+and the second one finds the last breakpoint in the past.
+
+Recorded executions may be used to enable reverse debugging. QEMU can't
+execute the code in backwards direction, but can load a snapshot and
+replay forward to find the desired position or breakpoint.
+
+The following GDB commands are supported:
+
+ - ``reverse-stepi`` (or ``rsi``) - step one instruction backwards
+ - ``reverse-continue`` (or ``rc``) - find last breakpoint in the past
+
+Reverse step loads the nearest snapshot and replays the execution until
+the required instruction is met.
+
+Reverse continue may include several passes of examining the execution
+between the snapshots. Each of the passes include the following steps:
+
+ #. loading the snapshot
+ #. replaying to examine the breakpoints
+ #. if breakpoint or watchpoint was met
+
+    * loading the snapshot again
+    * replaying to the required breakpoint
+
+ #. else
+
+    * proceeding to the p.1 with the earlier snapshot
+
+Therefore usage of the reverse debugging requires at least one snapshot
+created. This can be done by omitting ``snapshot`` option
+for the block drives and adding ``rrsnapshot`` for both record and replay
+command lines.
+See the :ref:`snapshotting-label` section to learn more about running record/replay
+and creating the snapshot in these modes.
+
+When ``rrsnapshot`` is not used, then snapshot named ``start_debugging``
+created in temporary overlay. This allows using reverse debugging, but with
+temporary snapshots (existing within the session).

[PATCH v3 7/9] tests/avocado: update replay_linux test

[Pavel Dovgalyuk]

This patch updates replay_linux test to make it compatible with
new LinuxTest class.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
---
 tests/avocado/replay_linux.py |   19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py
index 15953f9e49..1099b5647f 100644
--- a/tests/avocado/replay_linux.py
+++ b/tests/avocado/replay_linux.py
@@ -32,9 +32,16 @@ class ReplayLinux(LinuxTest):
     bus = 'ide'
 
     def setUp(self):
-        super(ReplayLinux, self).setUp()
+        # LinuxTest does many replay-incompatible things, but includes
+        # useful methods. Do not setup LinuxTest here and just
+        # call some functions.
+        super(LinuxTest, self).setUp()
+        self._set_distro()
         self.boot_path = self.download_boot()
-        self.cloudinit_path = self.prepare_cloudinit()
+        self.phone_server = cloudinit.PhoneHomeServer(('0.0.0.0', 0),
+                                                      self.name)
+        ssh_pubkey, self.ssh_key = self.set_up_existing_ssh_keys()
+        self.cloudinit_path = self.prepare_cloudinit(ssh_pubkey)
 
     def vm_add_disk(self, vm, path, id, device):
         bus_string = ''
@@ -50,7 +57,9 @@ def launch_and_wait(self, record, args, shift):
         vm = self.get_vm()
         vm.add_args('-smp', '1')
         vm.add_args('-m', '1024')
-        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
+        vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22',
+                    '-device', 'virtio-net,netdev=vnet')
+        vm.add_args('-object', 'filter-replay,id=replay,netdev=vnet')
         if args:
             vm.add_args(*args)
         self.vm_add_disk(vm, self.boot_path, 0, self.hdd)
@@ -75,8 +84,8 @@ def launch_and_wait(self, record, args, shift):
                                     stop_check=(lambda : not vm.is_running()))
         console_drainer.start()
         if record:
-            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port),
-                                          self.name)
+            while not self.phone_server.instance_phoned_back:
+                self.phone_server.handle_request()
             vm.shutdown()
             logger.info('finished the recording with log size %s bytes'
                 % os.path.getsize(replay_path))

[PATCH v3 8/9] tests/avocado: add replay Linux tests for virtio machine

[Pavel Dovgalyuk]

This patch adds two tests for replaying Linux boot process
on x86_64 virtio platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
---
 tests/avocado/replay_linux.py |   26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py
index 1099b5647f..3bb1bc8816 100644
--- a/tests/avocado/replay_linux.py
+++ b/tests/avocado/replay_linux.py
@@ -123,3 +123,29 @@ def test_pc_q35(self):
         :avocado: tags=machine:q35
         """
         self.run_rr(shift=3)
+
+@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+class ReplayLinuxX8664Virtio(ReplayLinux):
+    """
+    :avocado: tags=arch:x86_64
+    :avocado: tags=virtio
+    :avocado: tags=accel:tcg
+    """
+
+    hdd = 'virtio-blk-pci'
+    cd = 'virtio-blk-pci'
+    bus = None
+
+    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
+
+    def test_pc_i440fx(self):
+        """
+        :avocado: tags=machine:pc
+        """
+        self.run_rr(shift=1)
+
+    def test_pc_q35(self):
+        """
+        :avocado: tags=machine:q35
+        """
+        self.run_rr(shift=3)

[PATCH v3 9/9] tests/avocado: add replay Linux test for Aarch64 machines

[Pavel Dovgalyuk]

This patch adds two tests for replaying Linux boot process
on Aarch64 platform.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
---
 tests/avocado/replay_linux.py |   41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py
index 3bb1bc8816..e1f9981a34 100644
--- a/tests/avocado/replay_linux.py
+++ b/tests/avocado/replay_linux.py
@@ -13,6 +13,7 @@
 import time
 
 from avocado import skipUnless
+from avocado_qemu import BUILD_DIR
 from avocado.utils import cloudinit
 from avocado.utils import network
 from avocado.utils import vmimage
@@ -149,3 +150,43 @@ def test_pc_q35(self):
         :avocado: tags=machine:q35
         """
         self.run_rr(shift=3)
+
+@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+class ReplayLinuxAarch64(ReplayLinux):
+    """
+    :avocado: tags=accel:tcg
+    :avocado: tags=arch:aarch64
+    :avocado: tags=machine:virt
+    :avocado: tags=cpu:max
+    """
+
+    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
+
+    hdd = 'virtio-blk-device'
+    cd = 'virtio-blk-device'
+    bus = None
+
+    def get_common_args(self):
+        return ('-bios',
+                os.path.join(BUILD_DIR, 'pc-bios', 'edk2-aarch64-code.fd'),
+                "-cpu", "max,lpa2=off",
+                '-device', 'virtio-rng-pci,rng=rng0',
+                '-object', 'rng-builtin,id=rng0')
+
+    def test_virt_gicv2(self):
+        """
+        :avocado: tags=machine:gic-version=2
+        """
+
+        self.run_rr(shift=3,
+                    args=(*self.get_common_args(),
+                          "-machine", "virt,gic-version=2"))
+
+    def test_virt_gicv3(self):
+        """
+        :avocado: tags=machine:gic-version=3
+        """
+
+        self.run_rr(shift=3,
+                    args=(*self.get_common_args(),
+                          "-machine", "virt,gic-version=3"))

Re: [PATCH v3 2/9] replay: notify vCPU when BH is scheduled

[Paolo Bonzini]

On 5/26/22 10:45, Pavel Dovgalyuk wrote:
> vCPU execution should be suspended when new BH is scheduled.
> This is needed to avoid guest timeouts caused by the long cycles
> of the execution. In replay mode execution may hang when
> vCPU sleeps and block event comes to the queue.
> This patch adds notification which wakes up vCPU or interrupts
> execution of guest code.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

At least aio_bh_schedule_oneshot_full should have the same effect, so 
should this be done at a lower level, in aio_bh_enqueue() or even 
aio_notify()?

Paolo

> --
> 
> v2: changed first_cpu to current_cpu (suggested by Richard Henderson)
> ---
>   include/sysemu/cpu-timers.h |    1 +
>   softmmu/icount.c            |    8 ++++++++
>   stubs/icount.c              |    4 ++++
>   util/async.c                |    8 ++++++++
>   4 files changed, 21 insertions(+)
> 
> diff --git a/include/sysemu/cpu-timers.h b/include/sysemu/cpu-timers.h
> index ed6ee5c46c..2e786fe7fb 100644
> --- a/include/sysemu/cpu-timers.h
> +++ b/include/sysemu/cpu-timers.h
> @@ -59,6 +59,7 @@ int64_t icount_round(int64_t count);
>   /* if the CPUs are idle, start accounting real time to virtual clock. */
>   void icount_start_warp_timer(void);
>   void icount_account_warp_timer(void);
> +void icount_notify_exit(void);
>   
>   /*
>    * CPU Ticks and Clock
> diff --git a/softmmu/icount.c b/softmmu/icount.c
> index 5ca271620d..1cafec5014 100644
> --- a/softmmu/icount.c
> +++ b/softmmu/icount.c
> @@ -486,3 +486,11 @@ void icount_configure(QemuOpts *opts, Error **errp)
>                      qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) +
>                      NANOSECONDS_PER_SECOND / 10);
>   }
> +
> +void icount_notify_exit(void)
> +{
> +    if (icount_enabled() && current_cpu) {
> +        qemu_cpu_kick(current_cpu);
> +        qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
> +    }
> +}
> diff --git a/stubs/icount.c b/stubs/icount.c
> index f13c43568b..6df8c2bf7d 100644
> --- a/stubs/icount.c
> +++ b/stubs/icount.c
> @@ -43,3 +43,7 @@ void icount_account_warp_timer(void)
>   {
>       abort();
>   }
> +
> +void icount_notify_exit(void)
> +{
> +}
> diff --git a/util/async.c b/util/async.c
> index 554ba70cca..75f50f47c4 100644
> --- a/util/async.c
> +++ b/util/async.c
> @@ -33,6 +33,7 @@
>   #include "block/raw-aio.h"
>   #include "qemu/coroutine_int.h"
>   #include "qemu/coroutine-tls.h"
> +#include "sysemu/cpu-timers.h"
>   #include "trace.h"
>   
>   /***********************************************************/
> @@ -185,6 +186,13 @@ void qemu_bh_schedule_idle(QEMUBH *bh)
>   void qemu_bh_schedule(QEMUBH *bh)
>   {
>       aio_bh_enqueue(bh, BH_SCHEDULED);
> +    /*
> +     * Workaround for record/replay.
> +     * vCPU execution should be suspended when new BH is set.
> +     * This is needed to avoid guest timeouts caused
> +     * by the long cycles of the execution.
> +     */
> +    icount_notify_exit();
>   }
>   
>   /* This func is async.
> 
> 

Re: [PATCH v3 4/9] replay: simplify async event processing

[Paolo Bonzini]

On 5/26/22 10:45, Pavel Dovgalyuk wrote:
> +
> +typedef enum ReplayAsyncEventKind {
> +    REPLAY_ASYNC_EVENT_BH,
> +    REPLAY_ASYNC_EVENT_BH_ONESHOT,
> +    REPLAY_ASYNC_EVENT_INPUT,
> +    REPLAY_ASYNC_EVENT_INPUT_SYNC,
> +    REPLAY_ASYNC_EVENT_CHAR_READ,
> +    REPLAY_ASYNC_EVENT_BLOCK,
> +    REPLAY_ASYNC_EVENT_NET,
> +    REPLAY_ASYNC_COUNT
> +} ReplayAsyncEventKind;
> +
>   /* Any changes to order/number of events will need to bump REPLAY_VERSION */
>   enum ReplayEvents {
>       /* for instruction event */
> @@ -22,6 +35,7 @@ enum ReplayEvents {
>       EVENT_EXCEPTION,
>       /* for async events */
>       EVENT_ASYNC,
> +    EVENT_ASYNC_LAST = EVENT_ASYNC + REPLAY_ASYNC_COUNT - 1,

Why not unify the two enums into one?

Paolo

Re: [PATCH v3 0/9] Record/replay refactoring and stuff

[Paolo Bonzini]

On 5/26/22 10:45, Pavel Dovgalyuk wrote:
> The following series includes the following record/replay-related changes:
> - simplified async event processing
> - updated record/replay documentation, which was also converted to rst
> - avocado tests for record/replay of Linux for x86_64 and Aarch64
> - some bugfixes
> 
> v3 changes:
>   - rebased to master
> 
> v2 changes:
>   - rebased to master
>   - fixed some issues found by Richard Henderson

ACK except for patch 2 (the suggestion in the reply to patch 4 can be 
ignored or done separately).

Paolo

Re: [PATCH v3 2/9] replay: notify vCPU when BH is scheduled

[Pavel Dovgalyuk]

On 26.05.2022 12:37, Paolo Bonzini wrote:
> On 5/26/22 10:45, Pavel Dovgalyuk wrote:
>> vCPU execution should be suspended when new BH is scheduled.
>> This is needed to avoid guest timeouts caused by the long cycles
>> of the execution. In replay mode execution may hang when
>> vCPU sleeps and block event comes to the queue.
>> This patch adds notification which wakes up vCPU or interrupts
>> execution of guest code.
>>
>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> 
> At least aio_bh_schedule_oneshot_full should have the same effect, so 
> should this be done at a lower level, in aio_bh_enqueue() or even 
> aio_notify()?

Not sure about aio_notify. It can operate with different contexts.
Can some of them be not related to the VM state?
Moving the notification to aio_bh_enqueue.

> 
> Paolo
> 
>> -- 
>>
>> v2: changed first_cpu to current_cpu (suggested by Richard Henderson)
>> ---
>>   include/sysemu/cpu-timers.h |    1 +
>>   softmmu/icount.c            |    8 ++++++++
>>   stubs/icount.c              |    4 ++++
>>   util/async.c                |    8 ++++++++
>>   4 files changed, 21 insertions(+)
>>
>> diff --git a/include/sysemu/cpu-timers.h b/include/sysemu/cpu-timers.h
>> index ed6ee5c46c..2e786fe7fb 100644
>> --- a/include/sysemu/cpu-timers.h
>> +++ b/include/sysemu/cpu-timers.h
>> @@ -59,6 +59,7 @@ int64_t icount_round(int64_t count);
>>   /* if the CPUs are idle, start accounting real time to virtual 
>> clock. */
>>   void icount_start_warp_timer(void);
>>   void icount_account_warp_timer(void);
>> +void icount_notify_exit(void);
>>   /*
>>    * CPU Ticks and Clock
>> diff --git a/softmmu/icount.c b/softmmu/icount.c
>> index 5ca271620d..1cafec5014 100644
>> --- a/softmmu/icount.c
>> +++ b/softmmu/icount.c
>> @@ -486,3 +486,11 @@ void icount_configure(QemuOpts *opts, Error **errp)
>>                      qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) +
>>                      NANOSECONDS_PER_SECOND / 10);
>>   }
>> +
>> +void icount_notify_exit(void)
>> +{
>> +    if (icount_enabled() && current_cpu) {
>> +        qemu_cpu_kick(current_cpu);
>> +        qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
>> +    }
>> +}
>> diff --git a/stubs/icount.c b/stubs/icount.c
>> index f13c43568b..6df8c2bf7d 100644
>> --- a/stubs/icount.c
>> +++ b/stubs/icount.c
>> @@ -43,3 +43,7 @@ void icount_account_warp_timer(void)
>>   {
>>       abort();
>>   }
>> +
>> +void icount_notify_exit(void)
>> +{
>> +}
>> diff --git a/util/async.c b/util/async.c
>> index 554ba70cca..75f50f47c4 100644
>> --- a/util/async.c
>> +++ b/util/async.c
>> @@ -33,6 +33,7 @@
>>   #include "block/raw-aio.h"
>>   #include "qemu/coroutine_int.h"
>>   #include "qemu/coroutine-tls.h"
>> +#include "sysemu/cpu-timers.h"
>>   #include "trace.h"
>>   /***********************************************************/
>> @@ -185,6 +186,13 @@ void qemu_bh_schedule_idle(QEMUBH *bh)
>>   void qemu_bh_schedule(QEMUBH *bh)
>>   {
>>       aio_bh_enqueue(bh, BH_SCHEDULED);
>> +    /*
>> +     * Workaround for record/replay.
>> +     * vCPU execution should be suspended when new BH is set.
>> +     * This is needed to avoid guest timeouts caused
>> +     * by the long cycles of the execution.
>> +     */
>> +    icount_notify_exit();
>>   }
>>   /* This func is async.
>>
>>
> 

Re: [PATCH v3 4/9] replay: simplify async event processing

[Pavel Dovgalyuk]

On 26.05.2022 12:40, Paolo Bonzini wrote:
> On 5/26/22 10:45, Pavel Dovgalyuk wrote:
>> +
>> +typedef enum ReplayAsyncEventKind {
>> +    REPLAY_ASYNC_EVENT_BH,
>> +    REPLAY_ASYNC_EVENT_BH_ONESHOT,
>> +    REPLAY_ASYNC_EVENT_INPUT,
>> +    REPLAY_ASYNC_EVENT_INPUT_SYNC,
>> +    REPLAY_ASYNC_EVENT_CHAR_READ,
>> +    REPLAY_ASYNC_EVENT_BLOCK,
>> +    REPLAY_ASYNC_EVENT_NET,
>> +    REPLAY_ASYNC_COUNT
>> +} ReplayAsyncEventKind;
>> +
>>   /* Any changes to order/number of events will need to bump 
>> REPLAY_VERSION */
>>   enum ReplayEvents {
>>       /* for instruction event */
>> @@ -22,6 +35,7 @@ enum ReplayEvents {
>>       EVENT_EXCEPTION,
>>       /* for async events */
>>       EVENT_ASYNC,
>> +    EVENT_ASYNC_LAST = EVENT_ASYNC + REPLAY_ASYNC_COUNT - 1,
> 
> Why not unify the two enums into one?

ReplayAsyncEventKind is used as a variable type.
I think narrow type describes the value better than common enum.

> 
> Paolo

Re: [PATCH v3 2/9] replay: notify vCPU when BH is scheduled

[Paolo Bonzini]

On 5/26/22 11:51, Pavel Dovgalyuk wrote:
>>
>> At least aio_bh_schedule_oneshot_full should have the same effect, so 
>> should this be done at a lower level, in aio_bh_enqueue() or even 
>> aio_notify()?
> 
> Not sure about aio_notify. It can operate with different contexts.
> Can some of them be not related to the VM state?

All but the main AioContext one would have current_cpu == NULL.

Paolo

> Moving the notification to aio_bh_enqueue.

Re: [PATCH v3 2/9] replay: notify vCPU when BH is scheduled

[Pavel Dovgalyuk]

On 26.05.2022 15:10, Paolo Bonzini wrote:
> On 5/26/22 11:51, Pavel Dovgalyuk wrote:
>>>
>>> At least aio_bh_schedule_oneshot_full should have the same effect, so 
>>> should this be done at a lower level, in aio_bh_enqueue() or even 
>>> aio_notify()?
>>
>> Not sure about aio_notify. It can operate with different contexts.
>> Can some of them be not related to the VM state?
> 
> All but the main AioContext one would have current_cpu == NULL.

aio_bh_enqueue is better. Moving this code to aio_notify breaks the tests.