[Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Pavel Dovgalyuk]

This set of patches is related to the reverse execution and deterministic 
replay of qemu execution. This implementation of deterministic replay can 
be used for deterministic debugging of guest code through gdb remote
interface.

These patches include only core function of the replay,
excluding the support for replaying serial, audio, network, and USB devices'
operations. Reverse debugging and monitor commands were also excluded to
be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later 
used for replaying the execution anywhere and for unlimited number of times. 
It also supports checkpointing for faster rewinding during reverse debugging. 
Execution replaying reads the log and replays all non-deterministic events 
including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
 * Deterministically replays whole system execution and all contents of the memory,
   state of the hadrware devices, clocks, and screen of the VM.
 * Writes execution log into the file for latter replaying for multiple times 
   on different machines.
 * Supports i386, x86_64, and ARM hardware platforms.
 * Performs deterministic replay of all operations with keyboard and mouse
   input devices.
 * Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
 * First, record the execution, by adding the following string to the command line:
   '-icount shift=7,rr=record,rrfile=replay.bin -net none'. 
   Block devices' images are not actually changed in the recording mode, 
   because all of the changes are written to the temporary overlay file.
 * Then you can replay it for the multiple times by using another command
   line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
 * '-net none' option should also be specified if network replay patches
   are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
 * wrappers for clock and time functions to save their return values in the log
 * saving different asynchronous events (e.g. system shutdown) into the log
 * synchronization of the bottom halves execution
 * synchronization of the threads from thread pool
 * recording/replaying user input (mouse and keyboard)
 * adding internal events for cpu and io synchronization

v9 changes:
 * Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
 * Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
 * Removed one of the replay mutexes (as suggested by Paolo Bonzini)
 * Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
 * Updated command line options' names (as suggested by Paolo Bonzini)
 * Added design document for record/replay (as suggested by Paolo Bonzini)
 * Simplified checkpoints for the timers
 * Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
 * Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
 * Some functions renaming and extracting.

v8 changes:
 * Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
 * Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
 * Moved replay_submode out of the series (as suggested by Paolo Bonzini)
 * Moved suffix option out of the series
 * Converted some of the defines into enums (as suggested by Paolo Bonzini)
 * Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
 * Moved record/replay options to icount group (as suggested by Paolo Bonzini)
 * Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
 * Added mutex to protect replay log file (as suggested by Paolo Bonzini)
 * Minor cleanups

v7 changes:
 * Removed patches that were applied to upstream.

v6 changes:
 * Fixed replay stub return value (as suggested by Eric Blake)
 * Fixed icount warping.
 * Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
 * Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
 * Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
 * Removed applied patches.

v5 changes:
 * Minor changes.
 * Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
 * Moved savevm-related code out of the core.
 * Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
 * Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
 * Adopted Paolo's icount patches
 * Fixed hardware interrupts replaying

v4 changes:
 * Updated block drivers to support new bdrv_open interface.
 * Moved migration patches into separate series (as suggested by Paolo Bonzini)
 * Fixed a bug in replay_break operation.
 * Fixed rtl8139 migration for replay.
 * Fixed 'period' parameter processing for record mode.
 * Fixed bug in 'reverse-stepi' implementation.
 * Fixed replay without making any snapshots (even the starting one).
 * Moved core replay patches into the separate series.
 * Fixed reverse step and reverse continue support.
 * Fixed several bugs in icount subsystem.
 * Reusing native qemu icount for replay instructions counting.
 * Separated core patches into their own series.

v3 changes:
 * Fixed bug with replay of the aio write operations.
 * Added virtual clock based on replay icount.
 * Removed duplicated saving of interrupt_request CPU field.
 * Fixed some coding style issues.
 * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
 * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
 * Added description and example for replay-info command (as suggested by Eric Blake)
 * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
 * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
 * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
 * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
 * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
 * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
 * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
 * Added QMP versions of replay commands (as suggested by Eric Blake)
 * Removed some optional features of replay to make patches cleaner
 * Minor changes and code cleanup were made

---

Pavel Dovgalyuk (23):
      i386: partial revert of interrupt poll fix
      replay: global variables and function stubs
      sysemu: system functions for replay
      replay: internal functions for replay log
      replay: introduce mutex to protect the replay log
      replay: introduce icount event
      cpu-exec: allow temporary disabling icount
      cpu: replay instructions sequence
      replay: interrupts and exceptions
      replay: asynchronous events infrastructure
      replay: recording and replaying clock ticks
      timer: replace time() with QEMU_CLOCK_HOST
      replay: shutdown event
      replay: checkpoints
      aio: replace stack of bottom halves with queue
      replay: bottom halves
      replay: replay aio requests
      replay: thread pool
      typedef: add typedef for QemuOpts
      replay: initialization and deinitialization
      replay: replay blockers for devices
      replay: command line options
      replay: recording of the user input


 Makefile.target                |    1 
 async.c                        |   49 ++++--
 block.c                        |   93 ++++++++++-
 block/block-backend.c          |   30 +++
 block/qcow2.c                  |    4 
 block/raw-posix.c              |    6 -
 block/raw-win32.c              |    4 
 cpu-exec.c                     |   59 +++++--
 cpus.c                         |   51 ++++--
 dma-helpers.c                  |   10 +
 docs/replay.txt                |  161 ++++++++++++++++++
 exec.c                         |    1 
 hw/block/virtio-blk.c          |   10 +
 hw/bt/hci.c                    |    7 +
 hw/ide/ahci.c                  |    4 
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 +-
 hw/timer/arm_timer.c           |    2 
 hw/usb/hcd-uhci.c              |    2 
 include/block/aio.h            |   22 ++-
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4 
 include/exec/exec-all.h        |    2 
 include/qapi/qmp/qerror.h      |    3 
 include/qemu-common.h          |    2 
 include/qemu/main-loop.h       |    1 
 include/qemu/option.h          |    5 -
 include/qemu/queue.h           |    7 +
 include/qemu/typedefs.h        |    3 
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1 
 include/ui/input.h             |    2 
 main-loop.c                    |   11 +
 qapi-schema.json               |   18 ++
 qemu-io-cmds.c                 |    2 
 qemu-options.hx                |    8 +
 qemu-timer.c                   |   45 ++++-
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  287 +++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  159 ++++++++++++++++++
 replay/replay-internal.c       |  191 ++++++++++++++++++++++
 replay/replay-internal.h       |  144 ++++++++++++++++
 replay/replay-time.c           |   84 ++++++++++
 replay/replay.c                |  351 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  120 ++++++++++++++
 stubs/Makefile.objs            |    1 
 stubs/replay.c                 |   41 +++++
 target-i386/cpu.c              |   10 -
 target-i386/seg_helper.c       |    7 +
 tests/test-thread-pool.c       |    7 -
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2 
 translate-all.c                |   10 +
 ui/input.c                     |   27 ++-
 util/iov.c                     |    4 
 vl.c                           |   47 +++++
 57 files changed, 2100 insertions(+), 139 deletions(-)
 create mode 100755 docs/replay.txt
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v9 01/23] i386: partial revert of interrupt poll fix

[Pavel Dovgalyuk]

Processing CPU_INTERRUPT_POLL requests in cpu_has_work functions
break the determinism of cpu_exec. This patch is required to make
interrupts processing deterministic.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c        |    6 ++++++
 target-i386/cpu.c |   10 ++--------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 6738117..a048f85 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -335,6 +335,12 @@ int cpu_exec(CPUArchState *env)
     volatile bool have_tb_lock = false;
 
     if (cpu->halted) {
+#ifdef TARGET_I386
+        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+            apic_poll_irq(x86_cpu->apic_state);
+            cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
+        }
+#endif
         if (!cpu_has_work(cpu)) {
             return EXCP_HALTED;
         }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 3a9b32e..e90464d 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2969,14 +2969,8 @@ static bool x86_cpu_has_work(CPUState *cs)
     X86CPU *cpu = X86_CPU(cs);
     CPUX86State *env = &cpu->env;
 
-#if !defined(CONFIG_USER_ONLY)
-    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
-        apic_poll_irq(cpu->apic_state);
-        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
-    }
-#endif
-
-    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&
+    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
+                                      CPU_INTERRUPT_POLL)) &&
             (env->eflags & IF_MASK)) ||
            (cs->interrupt_request & (CPU_INTERRUPT_NMI |
                                      CPU_INTERRUPT_INIT |

[Qemu-devel] [RFC PATCH v9 02/23] replay: global variables and function stubs

[Pavel Dovgalyuk]

This patch adds global variables, defines, functions declarations,
and function stubs for deterministic VM replay used by external modules.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 Makefile.target      |    1 
 docs/replay.txt      |  161 ++++++++++++++++++++++++++++++++++++++++++++++++++
 qapi-schema.json     |   18 ++++++
 replay/Makefile.objs |    1 
 replay/replay.c      |   14 ++++
 replay/replay.h      |   19 ++++++
 stubs/Makefile.objs  |    1 
 stubs/replay.c       |    3 +
 8 files changed, 218 insertions(+), 0 deletions(-)
 create mode 100755 docs/replay.txt
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

diff --git a/Makefile.target b/Makefile.target
index 58c6ae1..cd939c1 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -84,6 +84,7 @@ all: $(PROGS) stap
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
 obj-y += tcg/tcg.o tcg/tcg-op.o tcg/optimize.o
+obj-y += replay/
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
 obj-y += fpu/softfloat.o
diff --git a/docs/replay.txt b/docs/replay.txt
new file mode 100755
index 0000000..084ca65
--- /dev/null
+++ b/docs/replay.txt
@@ -0,0 +1,161 @@
+Record/replay
+-------------
+
+Record/replay functions are used for the reverse execution and deterministic 
+replay of qemu execution. This implementation of deterministic replay can 
+be used for deterministic debugging of guest code through gdb remote
+interface.
+
+Execution recording writes non-deterministic events log, which can be later 
+used for replaying the execution anywhere and for unlimited number of times. 
+It also supports checkpointing for faster rewinding during reverse debugging. 
+Execution replaying reads the log and replays all non-deterministic events 
+including external input, hardware clocks, and interrupts.
+
+Deterministic replay has the following features:
+ * Deterministically replays whole system execution and all contents of 
+   the memory, state of the hadrware devices, clocks, and screen of the VM.
+ * Writes execution log into the file for latter replaying for multiple times 
+   on different machines.
+ * Supports i386, x86_64, and ARM hardware platforms.
+ * Performs deterministic replay of all operations with keyboard and mouse
+   input devices.
+
+Usage of the record/replay:
+ * First, record the execution, by adding the following string to the command line:
+   '-icount shift=7,rr=record,rrfile=replay.bin -net none'. 
+   Block devices' images are not actually changed in the recording mode, 
+   because all of the changes are written to the temporary overlay file.
+ * Then you can replay it for the multiple times by using another command
+   line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
+ * '-net none' option should also be specified if network replay patches
+   are not applied.
+
+Paper with short description of deterministic replay implementation:
+http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
+
+Modifications of qemu include:
+ * wrappers for clock and time functions to save their return values in the log
+ * saving different asynchronous events (e.g. system shutdown) into the log
+ * synchronization of the bottom halves execution
+ * synchronization of the threads from thread pool
+ * recording/replaying user input (mouse and keyboard)
+ * adding internal checkpoints for cpu and io synchronization
+
+Non-deterministic events
+------------------------
+
+Our record/replay system is based on saving and replaying non-deterministic 
+events (e.g. keyboard input) and simulating deterministic ones (e.g. reading 
+from HDD or memory of the VM). Saving only non-deterministic events makes 
+log file smaller, simulation faster, and allows using reverse debugging even 
+for realtime applications. 
+
+The following non-deterministic data from peripheral devices is saved into 
+the log: mouse and keyboard input, network packets, audio controller input, 
+USB packets, serial port input, and hardware clocks (they are non-deterministic 
+too, because their values are taken from the host machine). Inputs from 
+simulated hardware, memory of VM, software interrupts, and execution of 
+instructions are not saved into the log, because they are deterministic and 
+can be replayed by simulating the behavior of virtual machine starting from 
+initial state.
+
+We had to solve three tasks to implement deterministic replay: recording 
+non-deterministic events, replaying non-deterministic events, and checking 
+that there is no divergence between record and replay modes.
+
+We changed several parts of QEMU to make event log recording and replaying.
+Devices' models that have non-deterministic input from external devices were 
+changed to write every external event into the execution log immediately. 
+E.g. network packets are written into the log when they arrive into the virtual 
+network adapter.
+
+All non-deterministic events are coming from these devices. But to 
+replay them we need to know at which moments they occur. We specify 
+these moments by counting the number of instructions executed between 
+every pair of consecutive events.
+
+Instructions counting
+---------------------
+
+QEMU should work in icount mode to use record/replay feature. icount was
+designed to allow deterministic execution in absense of external inputs
+of the virtual machine. We also use icount to control the occurence of the
+non-deterministic events. Number of instruction passed from the last event
+is written to the log while recording the execution. In replay mode we
+can predict when to inject that event using the instructions counter.
+
+Timers
+------
+
+Timers are used to execute callbacks from different subsystems of QEMU
+at the specified moments of time. There are several kinds of timers: 
+ * Real time clock. Based on host time and used only for callbacks that 
+   do not change the virtual machine state. For this reason real time
+   clock and timers does not affect deterministic replay at all;
+ * Virtual clock. These timers run only during the emulation. In icount
+   mode virtual clock value is calculated using executed instructions counter.
+   That is why it is completely deterministic and does not have to be recorded;
+ * Host clock. This clock is used by device models that simulate real time
+   sources (e.g. real time clock chip). Host clock is the one of the sources
+   of non-determinism. Host clock read operations should be logged to
+   make the execution deterministic.
+ * Real time clock for icount. This clock is similar to real time clock but
+   it is used only for increasing virtual clock while virtual machine is
+   sleeping. Due to its nature it is also non-deterministic as the host clock
+   and has to be logged too.
+
+Checkpoints
+-----------
+
+Replaying of the execution of virtual machine is bound by sources of
+non-determinism. These are inputs from clock and peripheral devices,
+and QEMU thread scheduling. Thread scheduling affect on processing events
+from timers, asynchronous input-output, and bottom halves.
+
+Invocations of timers are coupled with clock reads and changing the state 
+of the virtual machine. Reads produce non-deterministic data taken from
+host clock. And VM state changes should preserve their order. Their relative
+order in replay mode must replicate the order of callbacks in record mode.
+To preserve this order we use checkpoints. When specific clock is processed
+in record mode we save to the log special ''checkpoint`` event.
+Checkpoints here do not refer to virtual machine snapshots. They are just
+record/replay events used for synchronization.
+
+QEMU in replay mode will try to invoke timers processing in random moment 
+of time. That's why we do not process group of timers until the checkpoint
+event will be read from the log. Such an event allows synchronizing CPU 
+execution and timer events.
+
+Another checkpoints application in record/replay is instructions counting
+while the virtual machine is idle. This function (qemu_clock_warp) is called
+from the wait loop. It changes virtual machine state and must be deterministic
+then. That is why we added checkpoint to this function to prevent its 
+operation in replay mode when it does not correspond to record mode.
+
+Bottom halves
+-------------
+
+Disk I/O events are completely deterministic in our model, because 
+in both record and replay modes we start virtual machine from the same 
+disk state. But callbacks that virtual disk controller uses for reading and
+writing the disk may occur at different moments of time in record and replay
+modes.
+
+Reading and writing requests are created by CPU thread of QEMU. Later these
+requests proceed to block layer which creates ''bottom halves``. Bottom
+halves consist of callback and its parameters. They are processed when
+main loop locks the global mutex. These locking are not synchronized with
+replaying process because main loop also processes the events that do not
+affect the virtual machine state (like user interation with monitor).
+
+That is why we had to implement saving and replaying bottom halves callbacks
+synchronously to the CPU execution. When the callback is about to execute
+it is added to the queue in the replay module. This queue is written to the
+log when its callbacks are executed. In replay mode callbacks do not processed
+until the corresponding event is read from the events log file.
+
+Sometimes block layer uses asynchronous callbacks for its internal purposes 
+(like reading or writing VM snapshots or disk image cluster tables). In this
+case bottom halves are not marked as ''replayable`` and do not saved 
+into the log.
diff --git a/qapi-schema.json b/qapi-schema.json
index e16f8eb..ec6200a 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3606,3 +3606,21 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
new file mode 100755
index 0000000..7ea860f
--- /dev/null
+++ b/replay/Makefile.objs
@@ -0,0 +1 @@
+obj-y += replay.o
diff --git a/replay/replay.c b/replay/replay.c
new file mode 100755
index 0000000..5ce066f
--- /dev/null
+++ b/replay/replay.c
@@ -0,0 +1,14 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
diff --git a/replay/replay.h b/replay/replay.h
new file mode 100755
index 0000000..d6b73c3
--- /dev/null
+++ b/replay/replay.h
@@ -0,0 +1,19 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+
+#endif
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 5e347d0..45a6c71 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
diff --git a/stubs/replay.c b/stubs/replay.c
new file mode 100755
index 0000000..563c777
--- /dev/null
+++ b/stubs/replay.c
@@ -0,0 +1,3 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;

[Qemu-devel] [RFC PATCH v9 03/23] sysemu: system functions for replay

[Pavel Dovgalyuk]

This patch removes "static" specifier from several qemu function to make
them visible to the replay module. It also invents several system functions
that will be used by replay.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 translate-all.c         |    8 ++++++++
 5 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/cpus.c b/cpus.c
index 0cdd1d7..04124ca 100644
--- a/cpus.c
+++ b/cpus.c
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1104,7 +1104,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 4a6237f..63fb1a3 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -214,6 +214,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 16dc2f7..b5e1170 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 3f162a9..86ae556 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
diff --git a/translate-all.c b/translate-all.c
index 9f47ce7..cf30303 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -812,6 +812,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)

[Qemu-devel] [RFC PATCH v9 04/23] replay: internal functions for replay log

[Pavel Dovgalyuk]

This patch adds functions to perform read and write operations
with replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  143 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   44 ++++++++++++++
 3 files changed, 188 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 7ea860f..1148f45 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
diff --git a/replay/replay-internal.c b/replay/replay-internal.c
new file mode 100755
index 0000000..e905de8
--- /dev/null
+++ b/replay/replay-internal.c
@@ -0,0 +1,143 @@
+/*
+ * replay-internal.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay-internal.h"
+
+unsigned int replay_data_kind = -1;
+unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+FILE *replay_file;
+
+void replay_put_byte(uint8_t byte)
+{
+    if (replay_file) {
+        putc(byte, replay_file);
+    }
+}
+
+void replay_put_event(uint8_t event)
+{
+    replay_put_byte(event);
+}
+
+
+void replay_put_word(uint16_t word)
+{
+    replay_put_byte(word >> 8);
+    replay_put_byte(word);
+}
+
+void replay_put_dword(uint32_t dword)
+{
+    replay_put_word(dword >> 16);
+    replay_put_word(dword);
+}
+
+void replay_put_qword(int64_t qword)
+{
+    replay_put_dword(qword >> 32);
+    replay_put_dword(qword);
+}
+
+void replay_put_array(const uint8_t *buf, size_t size)
+{
+    if (replay_file) {
+        replay_put_dword(size);
+        fwrite(buf, 1, size, replay_file);
+    }
+}
+
+uint8_t replay_get_byte(void)
+{
+    uint8_t byte = 0;
+    if (replay_file) {
+        byte = getc(replay_file);
+    }
+    return byte;
+}
+
+uint16_t replay_get_word(void)
+{
+    uint16_t word = 0;
+    if (replay_file) {
+        word = replay_get_byte();
+        word = (word << 8) + replay_get_byte();
+    }
+
+    return word;
+}
+
+uint32_t replay_get_dword(void)
+{
+    uint32_t dword = 0;
+    if (replay_file) {
+        dword = replay_get_word();
+        dword = (dword << 16) + replay_get_word();
+    }
+
+    return dword;
+}
+
+int64_t replay_get_qword(void)
+{
+    int64_t qword = 0;
+    if (replay_file) {
+        qword = replay_get_dword();
+        qword = (qword << 32) + replay_get_dword();
+    }
+
+    return qword;
+}
+
+void replay_get_array(uint8_t *buf, size_t *size)
+{
+    if (replay_file) {
+        *size = replay_get_dword();
+        fread(buf, 1, *size, replay_file);
+    }
+}
+
+void replay_get_array_alloc(uint8_t **buf, size_t *size)
+{
+    if (replay_file) {
+        *size = replay_get_dword();
+        *buf = g_malloc(*size);
+        fread(*buf, 1, *size, replay_file);
+    }
+}
+
+void replay_check_error(void)
+{
+    if (replay_file) {
+        if (feof(replay_file)) {
+            fprintf(stderr, "replay file is over\n");
+            qemu_system_vmstop_request_prepare();
+            qemu_system_vmstop_request(RUN_STATE_PAUSED);
+        } else if (ferror(replay_file)) {
+            fprintf(stderr, "replay file is over or something goes wrong\n");
+            qemu_system_vmstop_request_prepare();
+            qemu_system_vmstop_request(RUN_STATE_INTERNAL_ERROR);
+        }
+    }
+}
+
+void replay_fetch_data_kind(void)
+{
+    if (replay_file) {
+        if (!replay_has_unread_data) {
+            replay_data_kind = replay_get_byte();
+            replay_check_error();
+            replay_has_unread_data = 1;
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
new file mode 100755
index 0000000..d1b7575
--- /dev/null
+++ b/replay/replay-internal.h
@@ -0,0 +1,44 @@
+#ifndef REPLAY_INTERNAL_H
+#define REPLAY_INTERNAL_H
+
+/*
+ * replay-internal.h
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <stdio.h>
+
+extern unsigned int replay_data_kind;
+extern unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+extern FILE *replay_file;
+
+void replay_put_byte(uint8_t byte);
+void replay_put_event(uint8_t event);
+void replay_put_word(uint16_t word);
+void replay_put_dword(uint32_t dword);
+void replay_put_qword(int64_t qword);
+void replay_put_array(const uint8_t *buf, size_t size);
+
+uint8_t replay_get_byte(void);
+uint16_t replay_get_word(void);
+uint32_t replay_get_dword(void);
+int64_t replay_get_qword(void);
+void replay_get_array(uint8_t *buf, size_t *size);
+void replay_get_array_alloc(uint8_t **buf, size_t *size);
+
+/*! Checks error status of the file. */
+void replay_check_error(void);
+
+/*! Reads data type from the file and stores it in the
+    replay_data_kind variable. */
+void replay_fetch_data_kind(void);
+
+#endif

[Qemu-devel] [RFC PATCH v9 05/23] replay: introduce mutex to protect the replay log

[Pavel Dovgalyuk]

This mutex will protect read/write operations for replay log.
Using mutex is necessary because most of the events consist of
several fields stored in the log. The mutex will help to avoid races.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay-internal.c |   27 +++++++++++++++++++++++++++
 replay/replay-internal.h |    7 +++++++
 2 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/replay/replay-internal.c b/replay/replay-internal.c
index e905de8..ce7a1d8 100755
--- a/replay/replay-internal.c
+++ b/replay/replay-internal.c
@@ -15,6 +15,13 @@
 unsigned int replay_data_kind = -1;
 unsigned int replay_has_unread_data;
 
+/* Mutex to protect reading and writing events to the log.
+   replay_data_kind and replay_has_unread_data are also protected
+   by this mutex. 
+   It also protects replay events queue which stores events to be
+   written or read to the log. */
+static QemuMutex lock;
+
 /* File for replay writing */
 FILE *replay_file;
 
@@ -141,3 +148,23 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+void replay_mutex_init(void)
+{
+    qemu_mutex_init(&lock);
+}
+
+void replay_mutex_destroy(void)
+{
+    qemu_mutex_destroy(&lock);
+}
+
+void replay_mutex_lock(void)
+{
+    qemu_mutex_lock(&lock);
+}
+
+void replay_mutex_unlock(void)
+{
+    qemu_mutex_unlock(&lock);
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index d1b7575..d92410e 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -34,6 +34,13 @@ int64_t replay_get_qword(void);
 void replay_get_array(uint8_t *buf, size_t *size);
 void replay_get_array_alloc(uint8_t **buf, size_t *size);
 
+/* Mutex functions for protecting replay log file */
+
+void replay_mutex_init(void);
+void replay_mutex_destroy(void);
+void replay_mutex_lock(void);
+void replay_mutex_unlock(void);
+
 /*! Checks error status of the file. */
 void replay_check_error(void);
 

[Qemu-devel] [RFC PATCH v9 06/23] replay: introduce icount event

[Pavel Dovgalyuk]

This patch adds icount event to the replay subsystem. This event corresponds
to execution of several instructions and used to synchronize input events
in the replay phase.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay-internal.c |   21 +++++++++++++++++++++
 replay/replay-internal.h |   24 ++++++++++++++++++++++++
 replay/replay.c          |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 98 insertions(+), 0 deletions(-)

diff --git a/replay/replay-internal.c b/replay/replay-internal.c
index ce7a1d8..21bb42e 100755
--- a/replay/replay-internal.c
+++ b/replay/replay-internal.c
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 unsigned int replay_data_kind = -1;
@@ -34,6 +35,7 @@ void replay_put_byte(uint8_t byte)
 
 void replay_put_event(uint8_t event)
 {
+    assert(event < EVENT_COUNT);
     replay_put_byte(event);
 }
 
@@ -145,6 +147,10 @@ void replay_fetch_data_kind(void)
             replay_data_kind = replay_get_byte();
             replay_check_error();
             replay_has_unread_data = 1;
+            if (replay_data_kind >= EVENT_COUNT) {
+                error_report("Replay: unknown event kind %d", replay_data_kind);
+                exit(1);
+            }
         }
     }
 }
@@ -168,3 +174,18 @@ void replay_mutex_unlock(void)
 {
     qemu_mutex_unlock(&lock);
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+            replay_mutex_unlock();
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index d92410e..3588448 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,20 @@
 
 #include <stdio.h>
 
+enum ReplayEvents {
+    /* for instruction event */
+    EVENT_INSTRUCTION,
+    EVENT_COUNT
+};
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern unsigned int replay_data_kind;
 extern unsigned int replay_has_unread_data;
 
@@ -48,4 +62,14 @@ void replay_check_error(void);
     replay_data_kind variable. */
 void replay_fetch_data_kind(void);
 
+/*! Saves queued events (like instructions and sound). */
+void replay_save_instructions(void);
+
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
diff --git a/replay/replay.c b/replay/replay.c
index 5ce066f..a43bbbc 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -9,6 +9,52 @@
  *
  */
 
+#include "qemu-common.h"
 #include "replay.h"
+#include "replay-internal.h"
+#include "qemu/timer.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
+
+ReplayState replay_state;
+
+bool skip_async_events(int stop_event)
+{
+    bool res = false;
+
+    /* nothing to skip - not all instructions used */
+    if (replay_state.instructions_count != 0
+        && replay_has_unread_data) {
+        return stop_event == EVENT_INSTRUCTION;
+    }
+
+    while (true) {
+        replay_fetch_data_kind();
+        if (stop_event == replay_data_kind) {
+            res = true;
+        }
+        switch (replay_data_kind) {
+        case EVENT_INSTRUCTION:
+            replay_state.instructions_count = replay_get_dword();
+            return res;
+        default:
+            /* clock, time_t, checkpoint and other events */
+            return res;
+        }
+    }
+    return res;
+}
+
+void skip_async_events_until(unsigned int kind)
+{
+    if (!skip_async_events(kind)) {
+        fprintf(stderr, "%"PRId64": Read data kind %d instead of expected %d\n",
+            replay_get_current_step(), replay_data_kind, kind);
+        exit(1);
+    }
+}
+
+uint64_t replay_get_current_step(void)
+{
+    return cpu_get_icount_raw();
+}
diff --git a/replay/replay.h b/replay/replay.h
index d6b73c3..a03c748 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -12,8 +12,15 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif

[Qemu-devel] [RFC PATCH v9 07/23] cpu-exec: allow temporary disabling icount

[Pavel Dovgalyuk]

This patch is required for deterministic replay to generate an exception
by trying executing an instruction without changing icount.
It adds new flag to TB for disabling icount while translating it.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c              |    7 ++++---
 include/exec/exec-all.h |    1 +
 translate-all.c         |    2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index a048f85..71e9814 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -197,7 +197,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
 /* Execute the code without caching the generated code. An interpreter
    could be used if available. */
 static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
-                             TranslationBlock *orig_tb)
+                             TranslationBlock *orig_tb, bool ignore_icount)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
@@ -213,7 +213,8 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles | CF_NOCACHE);
+                     max_cycles | CF_NOCACHE
+                         | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
@@ -516,7 +517,7 @@ int cpu_exec(CPUArchState *env)
                             if (insns_left > 0) {
                                 /* Execute remaining instructions.  */
                                 tb = (TranslationBlock *)(next_tb & ~TB_EXIT_MASK);
-                                cpu_exec_nocache(env, insns_left, tb);
+                                cpu_exec_nocache(env, insns_left, tb, false);
                                 align_clocks(&sc, cpu);
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 63fb1a3..48a7541 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -148,6 +148,7 @@ struct TranslationBlock {
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_NOCACHE     0x10000 /* To be freed after execution */
 #define CF_USE_ICOUNT  0x20000
+#define CF_IGNORE_ICOUNT 0x40000 /* Do not generate icount code */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
diff --git a/translate-all.c b/translate-all.c
index cf30303..258782f 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -1053,7 +1053,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     int code_gen_size;
 
     phys_pc = get_page_addr_code(env, pc);
-    if (use_icount) {
+    if (use_icount && !(cflags & CF_IGNORE_ICOUNT)) {
         cflags |= CF_USE_ICOUNT;
     }
     tb = tb_alloc(pc);

[Qemu-devel] [RFC PATCH v9 08/23] cpu: replay instructions sequence

[Pavel Dovgalyuk]

This patch adds calls to replay functions into the icount setup block.
In record mode number of executed instructions is written to the log.
In replay mode number of istructions to execute is taken from the replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c          |   38 +++++++++++++++++++++++++-------------
 replay/replay.c |   26 ++++++++++++++++++++++++++
 replay/replay.h |    4 ++++
 3 files changed, 55 insertions(+), 13 deletions(-)

diff --git a/cpus.c b/cpus.c
index 04124ca..511a0c5 100644
--- a/cpus.c
+++ b/cpus.c
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1306,6 +1307,28 @@ int vm_stop_force_state(RunState state)
     }
 }
 
+static int64_t tcg_get_icount_limit(void)
+{
+    int64_t deadline;
+
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+
+        /* Maintain prior (possibly buggy) behaviour where if no deadline
+         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+         * nanoseconds.
+         */
+        if ((deadline < 0) || (deadline > INT32_MAX)) {
+            deadline = INT32_MAX;
+        }
+
+        return qemu_icount_round(deadline);
+    } else {
+        return replay_get_instructions();
+    }
+}
+
 static int tcg_cpu_exec(CPUArchState *env)
 {
     CPUState *cpu = ENV_GET_CPU(env);
@@ -1319,24 +1342,12 @@ static int tcg_cpu_exec(CPUArchState *env)
 #endif
     if (use_icount) {
         int64_t count;
-        int64_t deadline;
         int decr;
         timers_state.qemu_icount -= (cpu->icount_decr.u16.low
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
-
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
-
-        count = qemu_icount_round(deadline);
+        count = tcg_get_icount_limit();
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1354,6 +1365,7 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        replay_account_executed_instructions();
     }
     return ret;
 }
diff --git a/replay/replay.c b/replay/replay.c
index a43bbbc..a6a5145 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -58,3 +58,29 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_icount_raw();
 }
+
+int replay_get_instructions(void)
+{
+    int res = 0;
+    replay_mutex_lock();
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        res = replay_state.instructions_count;
+    }
+    replay_mutex_unlock();
+    return res;
+}
+
+void replay_account_executed_instructions(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY
+        && replay_state.instructions_count > 0) {
+        int count = (int)(replay_get_current_step()
+                          - replay_state.current_step);
+        replay_state.instructions_count -= count;
+        replay_state.current_step += count;
+        if (replay_state.instructions_count == 0) {
+            replay_has_unread_data = 0;
+            qemu_notify_event();
+        }
+    }
+}
diff --git a/replay/replay.h b/replay/replay.h
index a03c748..d19715f 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -22,5 +22,9 @@ extern ReplayMode replay_mode;
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_account_executed_instructions(void);
 
 #endif

[Qemu-devel] [RFC PATCH v9 09/23] replay: interrupts and exceptions

[Pavel Dovgalyuk]

This patch includes modifications of common cpu files. All interrupts and
exceptions occured during recording are written into the replay log.
These events allow correct replaying the execution by kicking cpu thread
when one of these events is found in the log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c               |   48 ++++++++++++++++++++++++++--------
 replay/replay-internal.h |    4 +++
 replay/replay.c          |   65 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 ++++++++++++
 target-i386/seg_helper.c |    7 +++++
 5 files changed, 130 insertions(+), 11 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 71e9814..ff3d140 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 /* -icount align implementation. */
 
@@ -335,22 +336,25 @@ int cpu_exec(CPUArchState *env)
     /* This must be volatile so it is not trashed by longjmp() */
     volatile bool have_tb_lock = false;
 
+    /* replay_interrupt may need current_cpu */
+    current_cpu = cpu;
+
     if (cpu->halted) {
 #ifdef TARGET_I386
-        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+        if ((cpu->interrupt_request & CPU_INTERRUPT_POLL)
+            && replay_interrupt()) {
             apic_poll_irq(x86_cpu->apic_state);
             cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
         }
 #endif
         if (!cpu_has_work(cpu)) {
+            current_cpu = NULL;
             return EXCP_HALTED;
         }
 
         cpu->halted = 0;
     }
 
-    current_cpu = cpu;
-
     /* As long as current_cpu is null, up to the assignment just above,
      * requests by other threads to exit the execution loop are expected to
      * be issued using the exit_request global. We must make sure that our
@@ -397,10 +401,21 @@ int cpu_exec(CPUArchState *env)
                     cpu->exception_index = -1;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_INTERRUPT;
+                        break;
+                    }
 #endif
                 }
+            } else if (replay_has_exception()
+                       && cpu->icount_decr.u16.low + cpu->icount_extra == 0) {
+                /* try to cause an exception pending in the log */
+                cpu_exec_nocache(env, 1, tb_find_fast(env), true);
+                break;
             }
 
             next_tb = 0; /* force lookup of first TB */
@@ -416,30 +431,40 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if (replay_mode == REPLAY_MODE_PLAY
+                        && !replay_has_interrupt()) {
+                        /* Do nothing */
+                    } else if (interrupt_request & CPU_INTERRUPT_HALT) {
+                        replay_interrupt();
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    else if (interrupt_request & CPU_INTERRUPT_INIT) {
+                        replay_interrupt();
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    else if (interrupt_request & CPU_INTERRUPT_RESET) {
+                        replay_interrupt();
                         cpu_reset(cpu);
+                        cpu_loop_exit(cpu);
                     }
 #endif
                     /* The target hook has 3 exit conditions:
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
-                        next_tb = 0;
+                    else {
+                        replay_interrupt();
+                        if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                            next_tb = 0;
+                        }
                     }
                     /* Don't use the cached interrupt_request value,
                        do_interrupt may have updated the EXITTB flag. */
@@ -450,7 +475,8 @@ int cpu_exec(CPUArchState *env)
                         next_tb = 0;
                     }
                 }
-                if (unlikely(cpu->exit_request)) {
+                if (unlikely(cpu->exit_request
+                             || replay_has_interrupt())) {
                     cpu->exit_request = 0;
                     cpu->exception_index = EXCP_INTERRUPT;
                     cpu_loop_exit(cpu);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 3588448..7d9f580 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -17,6 +17,10 @@
 enum ReplayEvents {
     /* for instruction event */
     EVENT_INSTRUCTION,
+    /* for software interrupt */
+    EVENT_INTERRUPT,
+    /* for emulated exceptions */
+    EVENT_EXCEPTION,
     EVENT_COUNT
 };
 
diff --git a/replay/replay.c b/replay/replay.c
index a6a5145..54a4fb8 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -84,3 +84,68 @@ void replay_account_executed_instructions(void)
         }
     }
 }
+
+bool replay_exception(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_mutex_lock();
+        replay_put_event(EVENT_EXCEPTION);
+        replay_mutex_unlock();
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        bool res = replay_has_exception();
+        if (res) {
+            replay_has_unread_data = 0;
+        }
+        return res;
+    }
+
+    return true;
+}
+
+bool replay_has_exception(void)
+{
+    bool res = false;
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_account_executed_instructions();
+        replay_mutex_lock();
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            res = true;
+        }
+        replay_mutex_unlock();
+    }
+
+    return res;
+}
+
+bool replay_interrupt(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_mutex_lock();
+        replay_put_event(EVENT_INTERRUPT);
+        replay_mutex_unlock();
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        bool res = replay_has_interrupt();
+        if (res) {
+            replay_has_unread_data = 0;
+        }
+        return res;
+    }
+
+    return true;
+}
+
+bool replay_has_interrupt(void)
+{
+    bool res = false;
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_account_executed_instructions();
+        replay_mutex_lock();
+        res = skip_async_events(EVENT_INTERRUPT);
+        replay_mutex_unlock();
+    }
+    return res;
+}
diff --git a/replay/replay.h b/replay/replay.h
index d19715f..8915523 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -27,4 +27,21 @@ int replay_get_instructions(void);
 /*! Updates instructions counter in replay mode. */
 void replay_account_executed_instructions(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Used to determine that exception is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif
diff --git a/target-i386/seg_helper.c b/target-i386/seg_helper.c
index fa374d0..46e4b7b 100644
--- a/target-i386/seg_helper.c
+++ b/target-i386/seg_helper.c
@@ -22,6 +22,7 @@
 #include "qemu/log.h"
 #include "exec/helper-proto.h"
 #include "exec/cpu_ldst.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PCALL
 
@@ -1294,6 +1295,12 @@ bool x86_cpu_exec_interrupt(CPUState *cs, int interrupt_request)
     if (interrupt_request & CPU_INTERRUPT_POLL) {
         cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
         apic_poll_irq(cpu->apic_state);
+        /* Process one interrupt for a single call.
+           In record/replay mode each call to this function
+           is coupled with an event in the log */
+        if (replay_mode != REPLAY_MODE_NONE) {
+            return true;
+        }
     }
 #endif
     if (interrupt_request & CPU_INTERRUPT_SIPI) {

[Qemu-devel] [RFC PATCH v9 10/23] replay: asynchronous events infrastructure

[Pavel Dovgalyuk]

This patch adds module for saving and replaying asynchronous events.
These events include network packets, keyboard and mouse input,
USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points
such as beginning of TB execution, or checkpoint in the iothread.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  229 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   31 ++++++
 replay/replay.h          |    6 +
 4 files changed, 267 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-events.c

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 1148f45..56da09c 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
diff --git a/replay/replay-events.c b/replay/replay-events.c
new file mode 100755
index 0000000..7b4d773
--- /dev/null
+++ b/replay/replay-events.c
@@ -0,0 +1,229 @@
+/*
+ * replay-events.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "qemu/error-report.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+typedef struct Event {
+    ReplayAsyncEventKind event_kind;
+    void *opaque;
+    void *opaque2;
+    uint64_t id;
+
+    QTAILQ_ENTRY(Event) events;
+} Event;
+
+static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list);
+static unsigned int read_event_kind = -1;
+static uint64_t read_id = -1;
+static int read_checkpoint = -1;
+
+static bool events_enabled = false;
+
+/* Functions */
+
+static void replay_run_event(Event *event)
+{
+    switch (event->event_kind) {
+    default:
+        error_report("Replay: invalid async event ID (%d) in the queue",
+                    event->event_kind);
+        exit(1);
+        break;
+    }
+}
+
+void replay_enable_events(void)
+{
+    events_enabled = true;
+}
+
+bool replay_has_events(void)
+{
+    return !QTAILQ_EMPTY(&events_list);
+}
+
+void replay_flush_events(void)
+{
+    replay_mutex_lock();
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_mutex_unlock();
+        replay_run_event(event);
+        replay_mutex_lock();
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    replay_mutex_unlock();
+}
+
+void replay_disable_events(void)
+{
+    if (replay_mode != REPLAY_MODE_NONE) {
+        events_enabled = false;
+        /* Flush events queue before waiting of completion */
+        replay_flush_events();
+    }
+}
+
+void replay_clear_events(void)
+{
+    replay_mutex_lock();
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        QTAILQ_REMOVE(&events_list, event, events);
+
+        g_free(event);
+    }
+    replay_mutex_unlock();
+}
+
+static void replay_add_event_internal(ReplayAsyncEventKind event_kind,
+                                      void *opaque,
+                                      void *opaque2, uint64_t id)
+{
+    if (event_kind >= REPLAY_ASYNC_COUNT) {
+        error_report("Replay: invalid async event ID (%d)", event_kind);
+        exit(1);
+    }
+    if (!replay_file || replay_mode == REPLAY_MODE_NONE
+        || !events_enabled) {
+        Event e;
+        e.event_kind = event_kind;
+        e.opaque = opaque;
+        e.opaque2 = opaque2;
+        e.id = id;
+        replay_run_event(&e);
+        return;
+    }
+
+    Event *event = g_malloc0(sizeof(Event));
+    event->event_kind = event_kind;
+    event->opaque = opaque;
+    event->opaque2 = opaque2;
+    event->id = id;
+
+    replay_mutex_lock();
+    QTAILQ_INSERT_TAIL(&events_list, event, events);
+    replay_mutex_unlock();
+}
+
+void replay_add_event(ReplayAsyncEventKind event_kind, void *opaque)
+{
+    replay_add_event_internal(event_kind, opaque, NULL, 0);
+}
+
+static void replay_save_event(Event *event, int checkpoint)
+{
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        /* put the event into the file */
+        replay_put_event(EVENT_ASYNC);
+        replay_put_byte(checkpoint);
+        replay_put_byte(event->event_kind);
+
+        /* save event-specific data */
+        switch (event->event_kind) {
+        }
+    }
+}
+
+/* Called with replay mutex locked */
+void replay_save_events(int checkpoint)
+{
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_save_event(event, checkpoint);
+
+        replay_mutex_unlock();
+        replay_run_event(event);
+        replay_mutex_lock();
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+}
+
+static Event *replay_read_event(int checkpoint)
+{
+    Event *event;
+    if (read_event_kind == -1) {
+        read_checkpoint = replay_get_byte();
+        read_event_kind = replay_get_byte();
+        read_id = -1;
+        replay_check_error();
+    }
+
+    if (checkpoint != read_checkpoint) {
+        return NULL;
+    }
+
+    /* Events that has not to be in the queue */
+    switch (read_event_kind) {
+    default:
+        error_report("Unknown ID %d of replay event", read_event_kind);
+        exit(1);
+        break;
+    }
+
+    QTAILQ_FOREACH(event, &events_list, events) {
+        if (event->event_kind == read_event_kind
+            && (read_id == -1 || read_id == event->id)) {
+            break;
+        }
+    }
+
+    if (event) {
+        QTAILQ_REMOVE(&events_list, event, events);
+    } else {
+        return NULL;
+    }
+
+    /* Read event-specific data */
+
+    return event;
+}
+
+/* Called with replay mutex locked */
+void replay_read_events(int checkpoint)
+{
+    replay_fetch_data_kind();
+    while (replay_data_kind == EVENT_ASYNC) {
+        Event *event = replay_read_event(checkpoint);
+        if (!event) {
+            break;
+        }
+        replay_mutex_unlock();
+        replay_run_event(event);
+        replay_mutex_lock();
+
+        g_free(event);
+        replay_has_unread_data = 0;
+        read_event_kind = -1;
+        replay_fetch_data_kind();
+    }
+}
+
+void replay_init_events(void)
+{
+    read_event_kind = -1;
+}
+
+void replay_finish_events(void)
+{
+    events_enabled = false;
+    replay_clear_events();
+}
+
+bool replay_events_enabled(void)
+{
+    return events_enabled;
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 7d9f580..8e6e35b 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -21,9 +21,19 @@ enum ReplayEvents {
     EVENT_INTERRUPT,
     /* for emulated exceptions */
     EVENT_EXCEPTION,
+    /* for async events */
+    EVENT_ASYNC,
     EVENT_COUNT
 };
 
+/* Asynchronous events IDs */
+
+enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_COUNT
+};
+
+typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -76,4 +86,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int checkpoint);
+/*! Read events from the file into the input queue */
+void replay_read_events(int checkpoint);
+/*! Adds specified async event to the queue */
+void replay_add_event(ReplayAsyncEventKind event_id, void *opaque);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index 8915523..c2a7651 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -43,5 +43,11 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
+/*! Returns true when saving events is enabled */
+bool replay_events_enabled(void);
 
 #endif

[Qemu-devel] [RFC PATCH v9 11/23] replay: recording and replaying clock ticks

[Pavel Dovgalyuk]

Clock ticks are considered as the sources of non-deterministic data for
virtual machine. This patch implements saving the clock values when they
are acquired (virtual, host clock).
When replaying the execution corresponding values are read from log and
transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes
it is not true - e.g. when timeouts for timer lists are checked. In this case
we use a cached value of the clock, passing it to the client code.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c                   |    3 +-
 qemu-timer.c             |    7 ++--
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   11 ++++++
 replay/replay-time.c     |   84 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   23 +++++++++++++
 stubs/replay.c           |    9 +++++
 7 files changed, 134 insertions(+), 4 deletions(-)
 create mode 100755 replay/replay-time.c

diff --git a/cpus.c b/cpus.c
index 511a0c5..b40b48c 100644
--- a/cpus.c
+++ b/cpus.c
@@ -336,7 +336,8 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = cpu_get_clock_locked();
+        int64_t clock = REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT,
+                                     cpu_get_clock_locked());
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
diff --git a/qemu-timer.c b/qemu-timer.c
index 5741f0d..d605afd 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -24,6 +24,7 @@
 
 #include "qemu/main-loop.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 #ifdef CONFIG_POSIX
 #include <pthread.h>
@@ -570,15 +571,15 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             return cpu_get_clock();
         }
     case QEMU_CLOCK_HOST:
-        now = get_clock_realtime();
+        now = REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime());
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
     case QEMU_CLOCK_VIRTUAL_RT:
-        return cpu_get_clock();
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, cpu_get_clock());
     }
 }
 
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 56da09c..257c320 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 8e6e35b..1a88fda 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -23,6 +23,10 @@ enum ReplayEvents {
     EVENT_EXCEPTION,
     /* for async events */
     EVENT_ASYNC,
+    /* for clock read/writes */
+    /* some of grteater codes are reserved for clocks */
+    EVENT_CLOCK,
+    EVENT_CLOCK_LAST = EVENT_CLOCK + REPLAY_CLOCK_COUNT - 1,
     EVENT_COUNT
 };
 
@@ -35,6 +39,8 @@ enum ReplayAsyncEventKind {
 typedef enum ReplayAsyncEventKind ReplayAsyncEventKind;
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -86,6 +92,11 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
diff --git a/replay/replay-time.c b/replay/replay-time.c
new file mode 100755
index 0000000..a083498
--- /dev/null
+++ b/replay/replay-time.c
@@ -0,0 +1,84 @@
+/*
+ * replay-time.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+
+int64_t replay_save_clock(ReplayClockKind kind, int64_t clock)
+{
+    replay_save_instructions();
+
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    if (replay_file) {
+        replay_mutex_lock();
+        replay_put_event(EVENT_CLOCK + kind);
+        replay_put_qword(clock);
+        replay_mutex_unlock();
+    }
+
+    return clock;
+}
+
+void replay_read_next_clock(ReplayClockKind kind)
+{
+    replay_fetch_data_kind();
+    if (replay_file) {
+        unsigned int read_kind = replay_data_kind - EVENT_CLOCK;
+
+        if (read_kind != kind) {
+            return;
+        }
+        if (read_kind >= REPLAY_CLOCK_COUNT) {
+            fprintf(stderr,
+                    "invalid clock ID %d was read from replay\n", read_kind);
+            exit(1);
+        }
+
+        int64_t clock = replay_get_qword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        replay_state.cached_clock[read_kind] = clock;
+    }
+}
+
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(ReplayClockKind kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_account_executed_instructions();
+
+    if (replay_file) {
+        int64_t ret;
+        replay_mutex_lock();
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        ret = replay_state.cached_clock[kind];
+        replay_mutex_unlock();
+
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
diff --git a/replay/replay.h b/replay/replay.h
index c2a7651..2398509 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -16,6 +16,16 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+enum ReplayClockKind {
+    /* host_clock */
+    REPLAY_CLOCK_HOST,
+    /* virtual_rt_clock */
+    REPLAY_CLOCK_VIRTUAL_RT,
+    REPLAY_CLOCK_COUNT
+};
+typedef enum ReplayClockKind ReplayClockKind;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -43,6 +53,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(ReplayClockKind kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(ReplayClockKind kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/stubs/replay.c b/stubs/replay.c
index 563c777..121bca6 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -1,3 +1,12 @@
 #include "replay/replay.h"
 
 ReplayMode replay_mode;
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}

[Qemu-devel] [RFC PATCH v9 12/23] timer: replace time() with QEMU_CLOCK_HOST

[Pavel Dovgalyuk]

This patch replaces time() function calls with calls to
qemu_clock_get_ns(QEMU_CLOCK_HOST). It makes such requests deterministic
in record/replay mode of icount.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 vl.c |   12 ++++++++----
 1 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/vl.c b/vl.c
index 8c8f142..00161f4 100644
--- a/vl.c
+++ b/vl.c
@@ -710,13 +710,17 @@ void vm_start(void)
 /***********************************************************/
 /* real time host monotonic timer */
 
+static time_t qemu_time(void)
+{
+    return qemu_clock_get_ms(QEMU_CLOCK_HOST) / 1000;
+}
+
 /***********************************************************/
 /* host time/date access */
 void qemu_get_timedate(struct tm *tm, int offset)
 {
-    time_t ti;
+    time_t ti = qemu_time();
 
-    time(&ti);
     ti += offset;
     if (rtc_date_offset == -1) {
         if (rtc_utc)
@@ -744,7 +748,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - qemu_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -782,7 +786,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = qemu_time() - rtc_start_date;
     }
 }
 

[Qemu-devel] [RFC PATCH v9 13/23] replay: shutdown event

[Pavel Dovgalyuk]

This patch records and replays simulator shutdown event.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   12 ++++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    1 +
 4 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 1a88fda..27a3b0a 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -23,6 +23,8 @@ enum ReplayEvents {
     EVENT_EXCEPTION,
     /* for async events */
     EVENT_ASYNC,
+    /* for shutdown request */
+    EVENT_SHUTDOWN,
     /* for clock read/writes */
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK,
diff --git a/replay/replay.c b/replay/replay.c
index 54a4fb8..3330432 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -13,6 +13,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 
@@ -34,6 +35,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -149,3 +154,10 @@ bool replay_has_interrupt(void)
     }
     return res;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
diff --git a/replay/replay.h b/replay/replay.h
index 2398509..fcc93d1 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -66,6 +66,11 @@ int64_t replay_read_clock(ReplayClockKind kind);
             ? replay_save_clock((clock), (value))                       \
         : (value))
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/vl.c b/vl.c
index 00161f4..58b9dd5 100644
--- a/vl.c
+++ b/vl.c
@@ -1710,6 +1710,7 @@ void qemu_system_killed(int signal, pid_t pid)
 void qemu_system_shutdown_request(void)
 {
     trace_qemu_system_shutdown_request();
+    replay_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }

[Qemu-devel] [RFC PATCH v9 14/23] replay: checkpoints

[Pavel Dovgalyuk]

This patch introduces checkpoints that synchronize cpu thread and iothread.
When checkpoint is met in the code all asynchronous events from the queue
are executed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |   12 ++++++++++++
 cpus.c                   |    5 +++++
 main-loop.c              |    6 ++++++
 qemu-timer.c             |   38 ++++++++++++++++++++++++++++++++++----
 replay/replay-internal.h |    4 ++++
 replay/replay.c          |   36 ++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   18 ++++++++++++++++++
 stubs/replay.c           |   11 +++++++++++
 vl.c                     |    5 ++++-
 9 files changed, 130 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index 210fd5f..37b9bd8 100644
--- a/block.c
+++ b/block.c
@@ -36,6 +36,7 @@
 #include "qmp-commands.h"
 #include "qemu/timer.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #ifdef CONFIG_BSD
 #include <sys/types.h>
@@ -1994,6 +1995,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -2004,6 +2010,12 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
diff --git a/cpus.c b/cpus.c
index b40b48c..5b2f896 100644
--- a/cpus.c
+++ b/cpus.c
@@ -391,6 +391,11 @@ void qemu_clock_warp(QEMUClockType type)
         return;
     }
 
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) {
+        return;
+    }
+
     /*
      * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
      * This ensures that the deadline for the timer is computed correctly below.
diff --git a/main-loop.c b/main-loop.c
index 981bcb5..06aad06 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -29,6 +29,7 @@
 #include "slirp/libslirp.h"
 #include "qemu/main-loop.h"
 #include "block/aio.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 
@@ -497,6 +498,11 @@ int main_loop_wait(int nonblocking)
     slirp_pollfds_poll(gpollfds, (ret < 0));
 #endif
 
+    /* CPU thread can infinitely wait for event after
+       missing the warp */
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    }
     qemu_clock_run_all_timers();
 
     return ret;
diff --git a/qemu-timer.c b/qemu-timer.c
index d605afd..37d9098 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -478,10 +478,33 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)
     void *opaque;
 
     qemu_event_reset(&timer_list->timers_done_ev);
-    if (!timer_list->clock->enabled) {
+    if (!timer_list->clock->enabled || !timer_list->active_timers) {
         goto out;
     }
 
+    switch (timer_list->clock->type) {
+    case QEMU_CLOCK_REALTIME:
+        break;
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(CHECKPOINT_CLOCK_VIRTUAL)) {
+            goto out;
+        }
+        break;
+    case QEMU_CLOCK_HOST:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(CHECKPOINT_CLOCK_HOST)) {
+            goto out;
+        }
+    case QEMU_CLOCK_VIRTUAL_RT:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(CHECKPOINT_CLOCK_VIRTUAL_RT)) {
+            goto out;
+        }
+        break;
+    }
+
     current_time = qemu_clock_get_ns(timer_list->clock->type);
     for(;;) {
         qemu_mutex_lock(&timer_list->active_timers_lock);
@@ -545,11 +568,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg)
 {
     int64_t deadline = -1;
     QEMUClockType type;
+    bool play = replay_mode == REPLAY_MODE_PLAY;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
         if (qemu_clock_use_for_deadline(tlg->tl[type]->clock->type)) {
-            deadline = qemu_soonest_timeout(deadline,
-                                            timerlist_deadline_ns(
-                                                tlg->tl[type]));
+            if (!play || tlg->tl[type]->clock->type == QEMU_CLOCK_REALTIME) {
+                deadline = qemu_soonest_timeout(deadline,
+                                                timerlist_deadline_ns(
+                                                    tlg->tl[type]));
+            } else {
+                /* Read clock from the replay file and
+                   do not calculate the deadline, based on virtual clock. */
+                qemu_clock_get_ns(tlg->tl[type]->clock->type);
+            }
         }
     }
     return deadline;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 27a3b0a..fb27b48 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -29,6 +29,10 @@ enum ReplayEvents {
     /* some of grteater codes are reserved for clocks */
     EVENT_CLOCK,
     EVENT_CLOCK_LAST = EVENT_CLOCK + REPLAY_CLOCK_COUNT - 1,
+    /* for checkpoint event */
+    /* some of grteater codes are reserved for checkpoints */
+    EVENT_CHECKPOINT,
+    EVENT_CHECKPOINT_LAST = EVENT_CHECKPOINT + CHECKPOINT_COUNT - 1,
     EVENT_COUNT
 };
 
diff --git a/replay/replay.c b/replay/replay.c
index 3330432..9cc8dd0 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -161,3 +161,39 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    bool res = false;
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_mutex_lock();
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    res = replay_data_kind != EVENT_ASYNC;
+                    replay_mutex_unlock();
+                    return res;
+                }
+                replay_mutex_unlock();
+                return res;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            res = replay_data_kind != EVENT_ASYNC;
+            replay_mutex_unlock();
+            return res;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_mutex_lock();
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+            replay_mutex_unlock();
+        }
+    }
+
+    return true;
+}
diff --git a/replay/replay.h b/replay/replay.h
index fcc93d1..f4bdb31 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -26,6 +26,18 @@ enum ReplayClockKind {
 };
 typedef enum ReplayClockKind ReplayClockKind;
 
+/* IDs of the checkpoints */
+enum ReplayCheckpoint {
+    CHECKPOINT_BDRV_DRAIN,
+    CHECKPOINT_CLOCK_WARP,
+    CHECKPOINT_RESET_REQUESTED,
+    CHECKPOINT_CLOCK_VIRTUAL,
+    CHECKPOINT_CLOCK_HOST,
+    CHECKPOINT_CLOCK_VIRTUAL_RT,
+    CHECKPOINT_COUNT
+};
+typedef enum ReplayCheckpoint ReplayCheckpoint;
+
 extern ReplayMode replay_mode;
 
 /* Processing the instructions */
@@ -70,6 +82,12 @@ int64_t replay_read_clock(ReplayClockKind kind);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+bool replay_checkpoint(ReplayCheckpoint checkpoint);
 
 /* Asynchronous events queue */
 
diff --git a/stubs/replay.c b/stubs/replay.c
index 121bca6..1be3575 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -10,3 +11,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+bool replay_checkpoint(ReplayCheckpoint checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
diff --git a/vl.c b/vl.c
index 58b9dd5..565bbb2 100644
--- a/vl.c
+++ b/vl.c
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -1757,7 +1758,9 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get()
+        && replay_checkpoint(CHECKPOINT_RESET_REQUESTED)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);

[Qemu-devel] [RFC PATCH v9 15/23] aio: replace stack of bottom halves with queue

[Pavel Dovgalyuk]

Bottom halves in AIO context are stored and removes
in LIFO order. It makes their execution non-deterministic.
This patch replaces the stack with queue to preserve the
order of bottom halves processing.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 async.c              |   25 +++++++++----------------
 include/block/aio.h  |    4 ++--
 include/qemu/queue.h |    7 +++++++
 3 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/async.c b/async.c
index 2be88cc..a84f122 100644
--- a/async.c
+++ b/async.c
@@ -35,7 +35,7 @@ struct QEMUBH {
     AioContext *ctx;
     QEMUBHFunc *cb;
     void *opaque;
-    QEMUBH *next;
+    QSIMPLEQ_ENTRY(QEMUBH) next;
     bool scheduled;
     bool idle;
     bool deleted;
@@ -51,10 +51,7 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
         .opaque = opaque,
     };
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    QSIMPLEQ_INSERT_TAIL_RCU(&ctx->bh_queue, bh, next);
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
@@ -62,16 +59,15 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
-    QEMUBH *bh, **bhp, *next;
+    QEMUBH *bh, *next;
     int ret;
 
     ctx->walking_bh++;
 
     ret = 0;
-    for (bh = ctx->first_bh; bh; bh = next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         /* Make sure that fetching bh happens before accessing its members */
         smp_read_barrier_depends();
-        next = bh->next;
         if (!bh->deleted && bh->scheduled) {
             bh->scheduled = 0;
             /* Paired with write barrier in bh schedule to ensure reading for
@@ -90,14 +86,10 @@ int aio_bh_poll(AioContext *ctx)
     /* remove deleted bhs */
     if (!ctx->walking_bh) {
         qemu_mutex_lock(&ctx->bh_lock);
-        bhp = &ctx->first_bh;
-        while (*bhp) {
-            bh = *bhp;
+        QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
             if (bh->deleted) {
-                *bhp = bh->next;
+                QSIMPLEQ_REMOVE(&ctx->bh_queue, bh, QEMUBH, next);
                 g_free(bh);
-            } else {
-                bhp = &bh->next;
             }
         }
         qemu_mutex_unlock(&ctx->bh_lock);
@@ -161,7 +153,7 @@ aio_compute_timeout(AioContext *ctx)
     int timeout = -1;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             if (bh->idle) {
                 /* idle bottom halves will be polled at least
@@ -204,7 +196,7 @@ aio_ctx_check(GSource *source)
     AioContext *ctx = (AioContext *) source;
     QEMUBH *bh;
 
-    for (bh = ctx->first_bh; bh; bh = bh->next) {
+    QSIMPLEQ_FOREACH(bh, &ctx->bh_queue, next) {
         if (!bh->deleted && bh->scheduled) {
             return true;
 	}
@@ -311,6 +303,7 @@ AioContext *aio_context_new(Error **errp)
     qemu_mutex_init(&ctx->bh_lock);
     rfifolock_init(&ctx->lock, aio_rfifolock_cb, ctx);
     timerlistgroup_init(&ctx->tlg, aio_timerlist_notify, ctx);
+    QSIMPLEQ_INIT(&ctx->bh_queue);
 
     return ctx;
 }
diff --git a/include/block/aio.h b/include/block/aio.h
index 7d1e26b..82cdf78 100644
--- a/include/block/aio.h
+++ b/include/block/aio.h
@@ -71,8 +71,8 @@ struct AioContext {
     /* lock to protect between bh's adders and deleter */
     QemuMutex bh_lock;
 
-    /* Anchor of the list of Bottom Halves belonging to the context */
-    struct QEMUBH *first_bh;
+    /* List of Bottom Halves belonging to the context */
+    QSIMPLEQ_HEAD(, QEMUBH) bh_queue;
 
     /* A simple lock used to protect the first_bh list, and ensure that
      * no callbacks are removed while we're walking and dispatching callbacks.
diff --git a/include/qemu/queue.h b/include/qemu/queue.h
index c602797..810056d 100644
--- a/include/qemu/queue.h
+++ b/include/qemu/queue.h
@@ -281,6 +281,13 @@ struct {                                                                \
     (head)->sqh_last = &(elm)->field.sqe_next;                          \
 } while (/*CONSTCOND*/0)
 
+#define QSIMPLEQ_INSERT_TAIL_RCU(head, elm, field) do {                 \
+    (elm)->field.sqe_next = NULL;                                       \
+    *(head)->sqh_last = (elm);                                          \
+    smp_wmb();                                                          \
+    (head)->sqh_last = &(elm)->field.sqe_next;                          \
+} while (/*CONSTCOND*/0)
+
 #define QSIMPLEQ_INSERT_AFTER(head, listelm, elm, field) do {           \
     if (((elm)->field.sqe_next = (listelm)->field.sqe_next) == NULL)    \
         (head)->sqh_last = &(elm)->field.sqe_next;                      \

[Qemu-devel] [RFC PATCH v9 16/23] replay: bottom halves

[Pavel Dovgalyuk]

This patch introduces bottom half event for replay queue. It saves the events
into the queue and process them at the checkpoints and instructions execution.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 async.c                  |   24 +++++++++++++++++++++++-
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   18 ++++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    1 +
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 81 insertions(+), 6 deletions(-)

diff --git a/async.c b/async.c
index a84f122..91f2ae6 100644
--- a/async.c
+++ b/async.c
@@ -27,6 +27,7 @@
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
 #include "qemu/atomic.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,6 +40,8 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
@@ -56,6 +59,21 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -77,7 +95,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 6918572..357d7e9 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -13,6 +13,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
diff --git a/hw/ide/ahci.c b/hw/ide/ahci.c
index 5651372..13d7f84 100644
--- a/hw/ide/ahci.c
+++ b/hw/ide/ahci.c
@@ -33,6 +33,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 #define DEBUG_AHCI 0
 
@@ -1243,7 +1244,8 @@ static void ahci_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 }
diff --git a/hw/ide/core.c b/hw/ide/core.c
index ac3f015..d6af700 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk,
 
     iocb = blk_aio_get(&trim_aiocb_info, blk, cb, opaque);
     iocb->blk = blk;
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
diff --git a/hw/timer/arm_timer.c b/hw/timer/arm_timer.c
index 1452910..97784a0 100644
--- a/hw/timer/arm_timer.c
+++ b/hw/timer/arm_timer.c
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
diff --git a/hw/usb/hcd-uhci.c b/hw/usb/hcd-uhci.c
index f903de7..47ff9f4 100644
--- a/hw/usb/hcd-uhci.c
+++ b/hw/usb/hcd-uhci.c
@@ -1221,7 +1221,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
diff --git a/include/block/aio.h b/include/block/aio.h
index 82cdf78..ed76b43 100644
--- a/include/block/aio.h
+++ b/include/block/aio.h
@@ -35,6 +35,8 @@ struct BlockAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
     int refcnt;
 };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
diff --git a/include/qemu/main-loop.h b/include/qemu/main-loop.h
index 62c68c0..f5a98fe 100644
--- a/include/qemu/main-loop.h
+++ b/include/qemu/main-loop.h
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
diff --git a/main-loop.c b/main-loop.c
index 06aad06..87ebad4 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -514,3 +514,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
+
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 7b4d773..2fb73be 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -35,6 +35,9 @@ static bool events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -133,10 +136,18 @@ static void replay_save_event(Event *event, int checkpoint)
 
         /* save event-specific data */
         switch (event->event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            replay_put_qword(event->id);
+            break;
         }
     }
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 /* Called with replay mutex locked */
 void replay_save_events(int checkpoint)
 {
@@ -168,6 +179,11 @@ static Event *replay_read_event(int checkpoint)
 
     /* Events that has not to be in the queue */
     switch (read_event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        if (read_id == -1) {
+            read_id = replay_get_qword();
+        }
+        break;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index fb27b48..91c966c 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -39,6 +39,7 @@ enum ReplayEvents {
 /* Asynchronous events IDs */
 
 enum ReplayAsyncEventKind {
+    REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_COUNT
 };
 
diff --git a/replay/replay.h b/replay/replay.h
index f4bdb31..0c41f4e 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -95,5 +95,7 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint);
 void replay_disable_events(void);
 /*! Returns true when saving events is enabled */
 bool replay_events_enabled(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index 1be3575..268f3e0 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -21,3 +21,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}

[Qemu-devel] [RFC PATCH v9 17/23] replay: replay aio requests

[Pavel Dovgalyuk]

This patch adds identifier to aio requests. ID is used for creating bottom
halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying
of the aio requests.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)

diff --git a/block.c b/block.c
index 37b9bd8..fec72a2 100644
--- a/block.c
+++ b/block.c
@@ -84,7 +84,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4447,7 +4448,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4457,7 +4470,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4468,7 +4493,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4617,7 +4642,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4655,7 +4681,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4800,7 +4826,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4811,7 +4842,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4822,6 +4854,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4835,7 +4872,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4855,6 +4897,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4905,6 +4966,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
diff --git a/block/block-backend.c b/block/block-backend.c
index c28e240..7cd28e4 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -447,6 +447,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -454,12 +462,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -479,7 +501,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
diff --git a/block/qcow2.c b/block/qcow2.c
index 7e614d7..f6d128e 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1141,6 +1141,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1298,6 +1300,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 357d7e9..4faf6d3 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 1a8a176..120fb80 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -342,11 +342,11 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
     }
 
     if (is_write) {
-        blk_aio_writev(blk, sector_num, qiov, nb_sectors,
-                       virtio_blk_rw_complete, mrb->reqs[start]);
+        blk_aio_writev_replay(blk, sector_num, qiov, nb_sectors,
+                              virtio_blk_rw_complete, mrb->reqs[start]);
     } else {
-        blk_aio_readv(blk, sector_num, qiov, nb_sectors,
-                      virtio_blk_rw_complete, mrb->reqs[start]);
+        blk_aio_readv_replay(blk, sector_num, qiov, nb_sectors,
+                             virtio_blk_rw_complete, mrb->reqs[start]);
     }
 }
 
@@ -438,7 +438,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
     if (mrb->is_write && mrb->num_reqs > 0) {
         virtio_blk_submit_multireq(req->dev->blk, mrb);
     }
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
diff --git a/hw/ide/atapi.c b/hw/ide/atapi.c
index 1bf8b34..547b7d9 100644
--- a/hw/ide/atapi.c
+++ b/hw/ide/atapi.c
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
diff --git a/hw/ide/core.c b/hw/ide/core.c
index d6af700..76bebb1 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -630,11 +630,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s, uint32_t tx_bytes)
@@ -888,11 +890,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -928,7 +932,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
diff --git a/include/block/block.h b/include/block/block.h
index 321295e..4c3e224 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -296,11 +296,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -321,7 +334,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index 644b46d..aed7d71 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -319,6 +319,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index aab12b9..8c9be1d 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index e708552..a7d8d4e 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
diff --git a/stubs/replay.c b/stubs/replay.c
index 268f3e0..95b43f3 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -25,3 +25,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
diff --git a/trace-events b/trace-events
index f87b077..551d50d 100644
--- a/trace-events
+++ b/trace-events
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
diff --git a/util/iov.c b/util/iov.c
index 2fb18e6..96517e7 100644
--- a/util/iov.c
+++ b/util/iov.c
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }

[Qemu-devel] [RFC PATCH v9 18/23] replay: thread pool

[Pavel Dovgalyuk]

This patch modifies thread pool to allow replaying asynchronous thread tasks
synchronously in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/raw-posix.c           |    6 ++++-
 block/raw-win32.c           |    4 +++-
 include/block/thread-pool.h |    4 +++-
 replay/replay-events.c      |   11 ++++++++++
 replay/replay-internal.h    |    1 +
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 ++++
 tests/test-thread-pool.c    |    7 ++++--
 thread-pool.c               |   49 ++++++++++++++++++++++++++++++-------------
 9 files changed, 66 insertions(+), 22 deletions(-)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index e474c17..4636b95 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1136,7 +1136,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -2062,7 +2064,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
diff --git a/block/raw-win32.c b/block/raw-win32.c
index 06243d7..7ac693b 100644
--- a/block/raw-win32.c
+++ b/block/raw-win32.c
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h
index 42eb5e8..801ac00 100644
--- a/include/block/thread-pool.h
+++ b/include/block/thread-pool.h
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque);
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 2fb73be..bfc1a08 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -13,6 +13,7 @@
 #include "qemu/error-report.h"
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     ReplayAsyncEventKind event_kind;
@@ -38,6 +39,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -137,6 +141,7 @@ static void replay_save_event(Event *event, int checkpoint)
         /* save event-specific data */
         switch (event->event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             replay_put_qword(event->id);
             break;
         }
@@ -148,6 +153,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 /* Called with replay mutex locked */
 void replay_save_events(int checkpoint)
 {
@@ -180,6 +190,7 @@ static Event *replay_read_event(int checkpoint)
     /* Events that has not to be in the queue */
     switch (read_event_kind) {
     case REPLAY_ASYNC_EVENT_BH:
+    case REPLAY_ASYNC_EVENT_THREAD:
         if (read_id == -1) {
             read_id = replay_get_qword();
         }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 91c966c..a92a92c 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -40,6 +40,7 @@ enum ReplayEvents {
 
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
+    REPLAY_ASYNC_EVENT_THREAD,
     REPLAY_ASYNC_COUNT
 };
 
diff --git a/replay/replay.h b/replay/replay.h
index 0c41f4e..a93f065 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -97,5 +97,7 @@ void replay_disable_events(void);
 bool replay_events_enabled(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index 95b43f3..81eddae 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -30,3 +30,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
diff --git a/tests/test-thread-pool.c b/tests/test-thread-pool.c
index 6a0b981..f32594c 100644
--- a/tests/test-thread-pool.c
+++ b/tests/test-thread-pool.c
@@ -56,7 +56,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -120,7 +120,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
diff --git a/thread-pool.c b/thread-pool.c
index e2cac8e..f5a4dac 100644
--- a/thread-pool.c
+++ b/thread-pool.c
@@ -22,6 +22,7 @@
 #include "trace.h"
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 static void do_spawn_thread(ThreadPool *pool);
 
@@ -74,6 +75,27 @@ struct ThreadPool {
     bool stopping;
 };
 
+void thread_pool_work(ThreadPool *pool, void *r)
+{
+    ThreadPoolElement *req = (ThreadPoolElement *)r;
+    int ret;
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_unlock(&pool->lock);
+    }
+
+    ret = req->func(req->arg);
+    req->ret = ret;
+    /* Write ret before state.  */
+    smp_wmb();
+    req->state = THREAD_DONE;
+
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_lock(&pool->lock);
+    }
+
+    qemu_bh_schedule(pool->completion_bh);
+}
+
 static void *worker_thread(void *opaque)
 {
     ThreadPool *pool = opaque;
@@ -100,18 +122,12 @@ static void *worker_thread(void *opaque)
         req = QTAILQ_FIRST(&pool->request_list);
         QTAILQ_REMOVE(&pool->request_list, req, reqs);
         req->state = THREAD_ACTIVE;
-        qemu_mutex_unlock(&pool->lock);
-
-        ret = req->func(req->arg);
-
-        req->ret = ret;
-        /* Write ret before state.  */
-        smp_wmb();
-        req->state = THREAD_DONE;
-
-        qemu_mutex_lock(&pool->lock);
 
-        qemu_bh_schedule(pool->completion_bh);
+        if (replay_mode != REPLAY_MODE_NONE && req->common.replay) {
+            replay_add_thread_event(pool, req, req->common.replay_step);
+        } else {
+            thread_pool_work(pool, req);
+        }
     }
 
     pool->cur_threads--;
@@ -235,7 +251,8 @@ static const AIOCBInfo thread_pool_aiocb_info = {
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque)
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step)
 {
     ThreadPoolElement *req;
 
@@ -244,6 +261,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
     req->arg = arg;
     req->state = THREAD_QUEUED;
     req->pool = pool;
+    req->common.replay = replay;
+    req->common.replay_step = replay_step;
 
     QLIST_INSERT_HEAD(&pool->head, req, all);
 
@@ -254,8 +273,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         spawn_thread(pool);
     }
     QTAILQ_INSERT_TAIL(&pool->request_list, req, reqs);
-    qemu_mutex_unlock(&pool->lock);
     qemu_sem_post(&pool->sem);
+    qemu_mutex_unlock(&pool->lock);
     return &req->common;
 }
 
@@ -277,14 +296,14 @@ int coroutine_fn thread_pool_submit_co(ThreadPool *pool, ThreadPoolFunc *func,
 {
     ThreadPoolCo tpc = { .co = qemu_coroutine_self(), .ret = -EINPROGRESS };
     assert(qemu_in_coroutine());
-    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc);
+    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc, false, 0);
     qemu_coroutine_yield();
     return tpc.ret;
 }
 
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg)
 {
-    thread_pool_submit_aio(pool, func, arg, NULL, NULL);
+    thread_pool_submit_aio(pool, func, arg, NULL, NULL, false, 0);
 }
 
 static void thread_pool_init_one(ThreadPool *pool, AioContext *ctx)

[Qemu-devel] [RFC PATCH v9 19/23] typedef: add typedef for QemuOpts

[Pavel Dovgalyuk]

This patch moves typedefs for QemuOpts and related types
to qemu/typedefs.h file.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/qemu/option.h   |    5 +----
 include/qemu/typedefs.h |    3 +++
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/qemu/option.h b/include/qemu/option.h
index 58c0157..4b0d336 100644
--- a/include/qemu/option.h
+++ b/include/qemu/option.h
@@ -30,6 +30,7 @@
 #include "qemu/queue.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
+#include "qemu/typedefs.h"
 
 const char *get_opt_name(char *buf, int buf_size, const char *p, char delim);
 const char *get_opt_value(char *buf, int buf_size, const char *p);
@@ -44,10 +45,6 @@ void parse_option_size(const char *name, const char *value,
 bool has_help_option(const char *param);
 bool is_valid_option_list(const char *param);
 
-typedef struct QemuOpt QemuOpt;
-typedef struct QemuOpts QemuOpts;
-typedef struct QemuOptsList QemuOptsList;
-
 enum QemuOptType {
     QEMU_OPT_STRING = 0,  /* no parsing (use string as-is)                        */
     QEMU_OPT_BOOL,        /* on/off                                               */
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index cde3314..a5f8d51 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -64,6 +64,9 @@ typedef struct QEMUBH QEMUBH;
 typedef struct QemuConsole QemuConsole;
 typedef struct QEMUFile QEMUFile;
 typedef struct QEMUMachine QEMUMachine;
+typedef struct QemuOpt QemuOpt;
+typedef struct QemuOpts QemuOpts;
+typedef struct QemuOptsList QemuOptsList;
 typedef struct QEMUSGList QEMUSGList;
 typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct QEMUTimerListGroup QEMUTimerListGroup;

[Qemu-devel] [RFC PATCH v9 20/23] replay: initialization and deinitialization

[Pavel Dovgalyuk]

This patch introduces the functions for enabling the record/replay and for
freeing the resources when simulator closes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.h |    2 +
 replay/replay.c          |  138 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   10 +++
 stubs/replay.c           |    5 ++
 vl.c                     |    4 +
 7 files changed, 161 insertions(+), 1 deletions(-)

diff --git a/block.c b/block.c
index fec72a2..63e4e6a 100644
--- a/block.c
+++ b/block.c
@@ -2011,7 +2011,7 @@ void bdrv_drain_all(void)
             aio_context_release(aio_context);
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_events_enabled()) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
             /* Nothing */
diff --git a/exec.c b/exec.c
index 6dff7bc..1d9094d 100644
--- a/exec.c
+++ b/exec.c
@@ -783,6 +783,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index a92a92c..0fbde78 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -33,6 +33,8 @@ enum ReplayEvents {
     /* some of grteater codes are reserved for checkpoints */
     EVENT_CHECKPOINT,
     EVENT_CHECKPOINT_LAST = EVENT_CHECKPOINT + CHECKPOINT_COUNT - 1,
+    /* end of log event */
+    EVENT_END,
     EVENT_COUNT
 };
 
diff --git a/replay/replay.c b/replay/replay.c
index 9cc8dd0..0c1f4f3 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -15,8 +15,16 @@
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002
+/* Size of replay log header */
+#define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
+
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 
+/* Name of replay file  */
+static char *replay_filename;
 ReplayState replay_state;
 
 bool skip_async_events(int stop_event)
@@ -165,6 +173,10 @@ void replay_shutdown_request(void)
 bool replay_checkpoint(ReplayCheckpoint checkpoint)
 {
     bool res = false;
+    if (!replay_events_enabled()) {
+        return true;
+    }
+
     replay_save_instructions();
 
     if (replay_file) {
@@ -197,3 +209,129 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint)
 
     return true;
 }
+
+static void replay_enable(const char *fname, int mode)
+{
+    const char *fmode = NULL;
+    assert(!replay_file);
+
+    switch (mode) {
+    case REPLAY_MODE_RECORD:
+        fmode = "wb";
+        break;
+    case REPLAY_MODE_PLAY:
+        fmode = "rb";
+        break;
+    default:
+        fprintf(stderr, "Replay: internal error: invalid replay mode\n");
+        exit(1);
+    }
+
+    atexit(replay_finish);
+
+    replay_mutex_init();
+
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        unsigned int version = replay_get_dword();
+        uint64_t offset = replay_get_qword();
+        if (version != REPLAY_VERSION) {
+            fprintf(stderr, "Replay: invalid input log file version\n");
+            exit(1);
+        }
+        /* go to the beginning */
+        fseek(replay_file, 12, SEEK_SET);
+    }
+
+    replay_init_events();
+}
+
+void replay_configure(QemuOpts *opts)
+{
+    const char *fname;
+    const char *rr;
+    ReplayMode mode = REPLAY_MODE_NONE;
+
+    rr = qemu_opt_get(opts, "rr");
+    if (!rr) {
+        /* Just enabling icount */
+        return;
+    } else if (!strcmp(rr, "record")) {
+        mode = REPLAY_MODE_RECORD;
+    } else if (!strcmp(rr, "replay")) {
+        mode = REPLAY_MODE_PLAY;
+    } else {
+        fprintf(stderr, "Invalid icount rr option: %s\n", rr);
+        exit(1);
+    }
+
+    fname = qemu_opt_get(opts, "rrfile");
+    if (!fname) {
+        fprintf(stderr, "File name not specified for replay\n");
+        exit(1);
+    }
+
+    replay_enable(fname, mode);
+}
+
+void replay_init_timer(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    /* Timer for snapshotting will be set up here. */
+
+    replay_enable_events();
+}
+
+void replay_finish(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_save_instructions();
+
+    /* finalize the file */
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            /* Just zero in this version.
+               But will be used later for snapshots table. */
+            replay_put_qword(offset);
+        }
+
+        fclose(replay_file);
+        replay_file = NULL;
+    }
+    if (replay_filename) {
+        g_free(replay_filename);
+        replay_filename = NULL;
+    }
+
+    replay_finish_events();
+    replay_mutex_destroy();
+}
diff --git a/replay/replay.h b/replay/replay.h
index a93f065..88c6a60 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -15,6 +15,7 @@
 #include <stdbool.h>
 #include <stdint.h>
 #include "qapi-types.h"
+#include "qemu/typedefs.h"
 
 /* replay clock kinds */
 enum ReplayClockKind {
@@ -40,6 +41,15 @@ typedef enum ReplayCheckpoint ReplayCheckpoint;
 
 extern ReplayMode replay_mode;
 
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
+
 /* Processing the instructions */
 
 /*! Returns number of executed instructions. */
diff --git a/stubs/replay.c b/stubs/replay.c
index 81eddae..19fabe9 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -34,3 +34,8 @@ uint64_t replay_get_current_step(void)
 void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
 {
 }
+
+bool replay_events_enabled(void)
+{
+    return false;
+}
diff --git a/vl.c b/vl.c
index 565bbb2..a02e831 100644
--- a/vl.c
+++ b/vl.c
@@ -4323,6 +4323,8 @@ int main(int argc, char **argv, char **envp)
     /* Done notifiers can load ROMs */
     rom_load_done();
 
+    replay_init_timer();
+
     qemu_system_reset(VMRESET_SILENT);
     if (loadvm) {
         if (load_vmstate(loadvm) < 0) {
@@ -4359,6 +4361,8 @@ int main(int argc, char **argv, char **envp)
     }
 
     main_loop();
+    replay_disable_events();
+
     bdrv_close_all();
     pause_all_vcpus();
     res_free();

[Qemu-devel] [RFC PATCH v9 21/23] replay: replay blockers for devices

[Pavel Dovgalyuk]

Some devices are not supported by record/replay subsystem.
This patch introduces replay blocker which denies starting record/replay
if such devices are included into the configuration.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/bt/hci.c               |    7 +++++++
 include/qapi/qmp/qerror.h |    3 +++
 replay/replay.c           |   14 ++++++++++++++
 replay/replay.h           |    3 +++
 vl.c                      |    9 +++++++++
 5 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/hw/bt/hci.c b/hw/bt/hci.c
index 7ea3dc6..9be30e1 100644
--- a/hw/bt/hci.c
+++ b/hw/bt/hci.c
@@ -23,6 +23,8 @@
 #include "hw/usb.h"
 #include "sysemu/bt.h"
 #include "hw/bt.h"
+#include "qapi/qmp/qerror.h"
+#include "replay/replay.h"
 
 struct bt_hci_s {
     uint8_t *(*evt_packet)(void *opaque);
@@ -72,6 +74,8 @@ struct bt_hci_s {
 
     struct HCIInfo info;
     struct bt_device_s device;
+
+    Error *replay_blocker;
 };
 
 #define DEFAULT_RSSI_DBM	20
@@ -2191,6 +2195,9 @@ struct HCIInfo *bt_new_hci(struct bt_scatternet_s *net)
 
     s->device.handle_destroy = bt_hci_destroy;
 
+    error_set(&s->replay_blocker, QERR_REPLAY_NOT_SUPPORTED, "bt hci");
+    replay_add_blocker(s->replay_blocker);
+
     return &s->info;
 }
 
diff --git a/include/qapi/qmp/qerror.h b/include/qapi/qmp/qerror.h
index 986260f..a9f5e24 100644
--- a/include/qapi/qmp/qerror.h
+++ b/include/qapi/qmp/qerror.h
@@ -136,4 +136,7 @@ void qerror_report_err(Error *err);
 #define QERR_UNSUPPORTED \
     ERROR_CLASS_GENERIC_ERROR, "this feature or command is not currently supported"
 
+#define QERR_REPLAY_NOT_SUPPORTED \
+    ERROR_CLASS_GENERIC_ERROR, "Record/replay feature is not supported for '%s'"
+
 #endif /* QERROR_H */
diff --git a/replay/replay.c b/replay/replay.c
index 0c1f4f3..eb730a9 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -14,6 +14,7 @@
 #include "replay-internal.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
 
 /* Current version of the replay mechanism.
    Increase it when file format changes. */
@@ -27,6 +28,8 @@ ReplayMode replay_mode = REPLAY_MODE_NONE;
 static char *replay_filename;
 ReplayState replay_state;
 
+static GSList *replay_blockers;
+
 bool skip_async_events(int stop_event)
 {
     bool res = false;
@@ -296,6 +299,12 @@ void replay_init_timer(void)
         return;
     }
 
+    if (replay_blockers) {
+        error_report("Record/replay: %s", 
+                     error_get_pretty(replay_blockers->data));
+        exit(1);
+    }
+
     /* Timer for snapshotting will be set up here. */
 
     replay_enable_events();
@@ -335,3 +344,8 @@ void replay_finish(void)
     replay_finish_events();
     replay_mutex_destroy();
 }
+
+void replay_add_blocker(Error *reason)
+{
+    replay_blockers = g_slist_prepend(replay_blockers, reason);
+}
diff --git a/replay/replay.h b/replay/replay.h
index 88c6a60..f0ef494 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -15,6 +15,7 @@
 #include <stdbool.h>
 #include <stdint.h>
 #include "qapi-types.h"
+#include "qapi/error.h"
 #include "qemu/typedefs.h"
 
 /* replay clock kinds */
@@ -49,6 +50,8 @@ void replay_configure(struct QemuOpts *opts);
 void replay_init_timer(void);
 /*! Closes replay log file and frees other resources. */
 void replay_finish(void);
+/*! Adds replay blocker with the specified error description */
+void replay_add_blocker(Error *reason);
 
 /* Processing the instructions */
 
diff --git a/vl.c b/vl.c
index a02e831..a5555fb 100644
--- a/vl.c
+++ b/vl.c
@@ -800,7 +800,11 @@ static void configure_rtc(QemuOpts *opts)
         if (!strcmp(value, "utc")) {
             rtc_utc = 1;
         } else if (!strcmp(value, "localtime")) {
+            Error *blocker = NULL;
             rtc_utc = 0;
+            error_set(&blocker, QERR_REPLAY_NOT_SUPPORTED,
+                      "-rtc base=localtime");
+            replay_add_blocker(blocker);
         } else {
             configure_rtc_date_offset(value, 0);
         }
@@ -1206,6 +1210,11 @@ static void smp_parse(QemuOpts *opts)
         exit(1);
     }
 
+    if (smp_cpus > 1 || smp_cores > 1 || smp_threads > 1) {
+        Error *blocker = NULL;
+        error_set(&blocker, QERR_REPLAY_NOT_SUPPORTED, "smp");
+        replay_add_blocker(blocker);
+    }
 }
 
 static void realtime_init(void)

[Qemu-devel] [RFC PATCH v9 22/23] replay: command line options

[Pavel Dovgalyuk]

This patch introduces command line options for enabling recording or replaying
virtual machine behavior. "-record" option starts recording of the execution
and saves it into the log, specified with "fname" parameter. "-replay" option
is intended for replaying previously saved log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c          |    1 +
 qemu-options.hx |    8 ++++++--
 vl.c            |   16 +++++++++++++++-
 3 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/cpus.c b/cpus.c
index 5b2f896..e289a9d 100644
--- a/cpus.c
+++ b/cpus.c
@@ -895,6 +895,7 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
     while (all_cpu_threads_idle()) {
        /* Start accounting real time to the virtual clock if the CPUs
diff --git a/qemu-options.hx b/qemu-options.hx
index 85ca3ad..c62661a 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3038,11 +3038,11 @@ re-inject them.
 ETEXI
 
 DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
-    "-icount [shift=N|auto][,align=on|off]\n" \
+    "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfile=<filename>]\n" \
     "                enable virtual instruction counter with 2^N clock ticks per\n" \
     "                instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL)
 STEXI
-@item -icount [shift=@var{N}|auto]
+@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfile=@var{filename}]
 @findex -icount
 Enable virtual instruction counter.  The virtual cpu will execute one
 instruction every 2^@var{N} ns of virtual time.  If @code{auto} is specified
@@ -3064,6 +3064,10 @@ Currently this option does not work when @option{shift} is @code{auto}.
 Note: The sync algorithm will work for those shift values for which
 the guest clock runs ahead of the host clock. Typically this happens
 when the shift value is high (how high depends on the host machine).
+
+When @option{rr} option is specified deterministic record/replay is enabled.
+Replay log is written into @var{filename} file in record mode and
+read from this file in replay mode.
 ETEXI
 
 DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
diff --git a/vl.c b/vl.c
index a5555fb..f12d3db 100644
--- a/vl.c
+++ b/vl.c
@@ -472,6 +472,12 @@ static QemuOptsList qemu_icount_opts = {
         }, {
             .name = "align",
             .type = QEMU_OPT_BOOL,
+        }, {
+            .name = "rr",
+            .type = QEMU_OPT_STRING,
+        }, {
+            .name = "rrfile",
+            .type = QEMU_OPT_STRING,
         },
         { /* end of list */ }
     },
@@ -2751,6 +2757,7 @@ int main(int argc, char **argv, char **envp)
 {
     int i;
     int snapshot, linux_boot;
+    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;
@@ -3770,6 +3777,8 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_configure(icount_opts);
+
     opts = qemu_get_machine_opts();
     optarg = qemu_opt_get(opts, "type");
     if (optarg) {
@@ -4137,6 +4146,11 @@ int main(int argc, char **argv, char **envp)
         qemu_opts_del(icount_opts);
     }
 
+    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
+        fprintf(stderr, "Please enable icount to use record/replay\n");
+        exit(1);
+    }
+
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
 
@@ -4173,7 +4187,7 @@ int main(int argc, char **argv, char **envp)
     }
 
     /* open the virtual block devices */
-    if (snapshot)
+    if (snapshot || replay_mode != REPLAY_MODE_NONE)
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);
     if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
                           &machine_class->block_default_type, 1) != 0) {

[Qemu-devel] [RFC PATCH v9 23/23] replay: recording of the user input

[Pavel Dovgalyuk]

This records user input (keyboard and mouse events) in record mode and replays
these input events in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   31 +++++++++
 replay/replay-input.c    |  159 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   13 ++++
 replay/replay.h          |    4 +
 ui/input.c               |   27 +++++---
 7 files changed, 229 insertions(+), 8 deletions(-)
 create mode 100755 replay/replay-input.c

diff --git a/include/ui/input.h b/include/ui/input.h
index 5d5ac00..d06a12d 100644
--- a/include/ui/input.h
+++ b/include/ui/input.h
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 257c320..3936296 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
diff --git a/replay/replay-events.c b/replay/replay-events.c
index bfc1a08..ae7aaa1 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -14,6 +14,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "block/thread-pool.h"
+#include "ui/input.h"
 
 typedef struct Event {
     ReplayAsyncEventKind event_kind;
@@ -42,6 +43,13 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_THREAD:
         thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
         break;
+    case REPLAY_ASYNC_EVENT_INPUT:
+        qemu_input_event_send_impl(NULL, (InputEvent *)event->opaque);
+        qapi_free_InputEvent((InputEvent *)event->opaque);
+        break;
+    case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+        qemu_input_event_sync_impl();
+        break;
     default:
         error_report("Replay: invalid async event ID (%d) in the queue",
                     event->event_kind);
@@ -144,6 +152,9 @@ static void replay_save_event(Event *event, int checkpoint)
         case REPLAY_ASYNC_EVENT_THREAD:
             replay_put_qword(event->id);
             break;
+        case REPLAY_ASYNC_EVENT_INPUT:
+            replay_save_input_event(event->opaque);
+            break;
         }
     }
 }
@@ -158,6 +169,16 @@ void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
 }
 
+void replay_add_input_event(struct InputEvent *event)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT, event, NULL, 0);
+}
+
+void replay_add_input_sync_event(void)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT_SYNC, NULL, NULL, 0);
+}
+
 /* Called with replay mutex locked */
 void replay_save_events(int checkpoint)
 {
@@ -195,6 +216,16 @@ static Event *replay_read_event(int checkpoint)
             read_id = replay_get_qword();
         }
         break;
+    case REPLAY_ASYNC_EVENT_INPUT:
+        event = g_malloc0(sizeof(Event));
+        event->event_kind = read_event_kind;
+        event->opaque = replay_read_input_event();
+        return event;
+    case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+        event = g_malloc0(sizeof(Event));
+        event->event_kind = read_event_kind;
+        event->opaque = 0;
+        return event;
     default:
         error_report("Unknown ID %d of replay event", read_event_kind);
         exit(1);
diff --git a/replay/replay-input.c b/replay/replay-input.c
new file mode 100755
index 0000000..54923b9
--- /dev/null
+++ b/replay/replay-input.c
@@ -0,0 +1,159 @@
+/*
+ * replay-input.c
+ *
+ * Copyright (c) 2010-2015 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+#include "ui/input.h"
+#include "qapi/qmp-output-visitor.h"
+#include "qapi/qmp-input-visitor.h"
+#include "qapi-visit.h"
+
+static InputEvent *qapi_clone_InputEvent(InputEvent *src)
+{
+    QmpOutputVisitor *qov;
+    QmpInputVisitor *qiv;
+    Visitor *ov, *iv;
+    QObject *obj;
+    InputEvent *dst = NULL;
+
+    qov = qmp_output_visitor_new();
+    ov = qmp_output_get_visitor(qov);
+    visit_type_InputEvent(ov, &src, NULL, &error_abort);
+    obj = qmp_output_get_qobject(qov);
+    qmp_output_visitor_cleanup(qov);
+    if (!obj) {
+        return NULL;
+    }
+
+    qiv = qmp_input_visitor_new(obj);
+    iv = qmp_input_get_visitor(qiv);
+    visit_type_InputEvent(iv, &dst, NULL, &error_abort);
+    qmp_input_visitor_cleanup(qiv);
+    qobject_decref(obj);
+
+    return dst;
+}
+
+void replay_save_input_event(InputEvent *evt)
+{
+    replay_put_dword(evt->kind);
+
+    switch (evt->kind) {
+    case INPUT_EVENT_KIND_KEY:
+        replay_put_dword(evt->key->key->kind);
+
+        switch (evt->key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            replay_put_qword(evt->key->key->number);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            replay_put_dword(evt->key->key->qcode);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        replay_put_dword(evt->btn->button);
+        replay_put_byte(evt->btn->down);
+        break;
+    case INPUT_EVENT_KIND_REL:
+        replay_put_dword(evt->rel->axis);
+        replay_put_qword(evt->rel->value);
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        replay_put_dword(evt->abs->axis);
+        replay_put_qword(evt->abs->value);
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+}
+
+InputEvent *replay_read_input_event(void)
+{
+    InputEvent evt;
+    KeyValue keyValue;
+    InputKeyEvent key;
+    key.key = &keyValue;
+    InputBtnEvent btn;
+    InputMoveEvent rel;
+    InputMoveEvent abs;
+
+    evt.kind = replay_get_dword();
+    switch (evt.kind) {
+    case INPUT_EVENT_KIND_KEY:
+        evt.key = &key;
+        evt.key->key->kind = replay_get_dword();
+
+        switch (evt.key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            evt.key->key->number = replay_get_qword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            evt.key->key->qcode = (QKeyCode)replay_get_dword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        evt.btn = &btn;
+        evt.btn->button = (InputButton)replay_get_dword();
+        evt.btn->down = replay_get_byte();
+        break;
+    case INPUT_EVENT_KIND_REL:
+        evt.rel = &rel;
+        evt.rel->axis = (InputAxis)replay_get_dword();
+        evt.rel->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        evt.abs = &abs;
+        evt.abs->axis = (InputAxis)replay_get_dword();
+        evt.abs->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+
+    return qapi_clone_InputEvent(&evt);
+}
+
+void replay_input_event(QemuConsole *src, InputEvent *evt)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(qapi_clone_InputEvent(evt));
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }
+}
+
+void replay_input_sync_event(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_sync_event();
+    } else {
+        qemu_input_event_sync_impl();
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 0fbde78..85674e2 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -43,6 +43,8 @@ enum ReplayEvents {
 enum ReplayAsyncEventKind {
     REPLAY_ASYNC_EVENT_BH,
     REPLAY_ASYNC_EVENT_THREAD,
+    REPLAY_ASYNC_EVENT_INPUT,
+    REPLAY_ASYNC_EVENT_INPUT_SYNC,
     REPLAY_ASYNC_COUNT
 };
 
@@ -128,4 +130,15 @@ void replay_read_events(int checkpoint);
 /*! Adds specified async event to the queue */
 void replay_add_event(ReplayAsyncEventKind event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index f0ef494..f5a1d7e 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -112,5 +112,9 @@ bool replay_events_enabled(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_input_event(QemuConsole *src, InputEvent *evt);
+/*! Adds input sync event to the queue */
+void replay_input_sync_event(void);
 
 #endif
diff --git a/ui/input.c b/ui/input.c
index 7ba99e5..85cb819 100644
--- a/ui/input.c
+++ b/ui/input.c
@@ -5,6 +5,7 @@
 #include "trace.h"
 #include "ui/input.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 struct QemuInputHandlerState {
     DeviceState       *dev;
@@ -298,14 +299,10 @@ static void qemu_input_queue_sync(struct QemuInputEventQueueHead *queue)
     QTAILQ_INSERT_TAIL(queue, item, node);
 }
 
-void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt)
 {
     QemuInputHandlerState *s;
 
-    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
-        return;
-    }
-
     qemu_input_event_trace(src, evt);
 
     /* pre processing */
@@ -322,14 +319,19 @@ void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
     s->events++;
 }
 
-void qemu_input_event_sync(void)
+void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
-    QemuInputHandlerState *s;
-
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
         return;
     }
 
+    replay_input_event(src, evt);
+}
+
+void qemu_input_event_sync_impl(void)
+{
+    QemuInputHandlerState *s;
+
     trace_input_event_sync();
 
     QTAILQ_FOREACH(s, &handlers, node) {
@@ -343,6 +345,15 @@ void qemu_input_event_sync(void)
     }
 }
 
+void qemu_input_event_sync(void)
+{
+    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
+        return;
+    }
+
+    replay_input_sync_event();
+}
+
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 {
     InputEvent *evt = g_new0(InputEvent, 1);

Re: [Qemu-devel] [RFC PATCH v9 04/23] replay: internal functions for replay log

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> This patch adds functions to perform read and write operations
> with replay log.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  replay/Makefile.objs     |    1 
>  replay/replay-internal.c |  143 ++++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-internal.h |   44 ++++++++++++++
>  3 files changed, 188 insertions(+), 0 deletions(-)
>  create mode 100755 replay/replay-internal.c
>  create mode 100755 replay/replay-internal.h
> 
> diff --git a/replay/Makefile.objs b/replay/Makefile.objs
> index 7ea860f..1148f45 100755
> --- a/replay/Makefile.objs
> +++ b/replay/Makefile.objs
> @@ -1 +1,2 @@
>  obj-y += replay.o
> +obj-y += replay-internal.o
> diff --git a/replay/replay-internal.c b/replay/replay-internal.c
> new file mode 100755
> index 0000000..e905de8
> --- /dev/null
> +++ b/replay/replay-internal.c
> @@ -0,0 +1,143 @@
> +/*
> + * replay-internal.c
> + *
> + * Copyright (c) 2010-2015 Institute for System Programming
> + *                         of the Russian Academy of Sciences.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu-common.h"
> +#include "replay-internal.h"
> +
> +unsigned int replay_data_kind = -1;
> +unsigned int replay_has_unread_data;
> +
> +/* File for replay writing */
> +FILE *replay_file;
> +
> +void replay_put_byte(uint8_t byte)
> +{
> +    if (replay_file) {
> +        putc(byte, replay_file);
> +    }
> +}
> +
> +void replay_put_event(uint8_t event)
> +{
> +    replay_put_byte(event);
> +}
> +
> +
> +void replay_put_word(uint16_t word)
> +{
> +    replay_put_byte(word >> 8);
> +    replay_put_byte(word);
> +}
> +
> +void replay_put_dword(uint32_t dword)
> +{
> +    replay_put_word(dword >> 16);
> +    replay_put_word(dword);
> +}
> +
> +void replay_put_qword(int64_t qword)
> +{
> +    replay_put_dword(qword >> 32);
> +    replay_put_dword(qword);
> +}
> +
> +void replay_put_array(const uint8_t *buf, size_t size)
> +{
> +    if (replay_file) {
> +        replay_put_dword(size);
> +        fwrite(buf, 1, size, replay_file);
> +    }
> +}
> +
> +uint8_t replay_get_byte(void)
> +{
> +    uint8_t byte = 0;
> +    if (replay_file) {
> +        byte = getc(replay_file);
> +    }
> +    return byte;
> +}
> +
> +uint16_t replay_get_word(void)
> +{
> +    uint16_t word = 0;
> +    if (replay_file) {
> +        word = replay_get_byte();
> +        word = (word << 8) + replay_get_byte();
> +    }
> +
> +    return word;
> +}
> +
> +uint32_t replay_get_dword(void)
> +{
> +    uint32_t dword = 0;
> +    if (replay_file) {
> +        dword = replay_get_word();
> +        dword = (dword << 16) + replay_get_word();
> +    }
> +
> +    return dword;
> +}
> +
> +int64_t replay_get_qword(void)
> +{
> +    int64_t qword = 0;
> +    if (replay_file) {
> +        qword = replay_get_dword();
> +        qword = (qword << 32) + replay_get_dword();
> +    }
> +
> +    return qword;
> +}
> +
> +void replay_get_array(uint8_t *buf, size_t *size)
> +{
> +    if (replay_file) {
> +        *size = replay_get_dword();
> +        fread(buf, 1, *size, replay_file);
> +    }
> +}
> +
> +void replay_get_array_alloc(uint8_t **buf, size_t *size)
> +{
> +    if (replay_file) {
> +        *size = replay_get_dword();
> +        *buf = g_malloc(*size);
> +        fread(*buf, 1, *size, replay_file);
> +    }
> +}
> +
> +void replay_check_error(void)
> +{
> +    if (replay_file) {
> +        if (feof(replay_file)) {
> +            fprintf(stderr, "replay file is over\n");

I suggest using error_report (that will also require removal of the \n)
but apart from this nit:

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

> +            qemu_system_vmstop_request_prepare();
> +            qemu_system_vmstop_request(RUN_STATE_PAUSED);
> +        } else if (ferror(replay_file)) {
> +            fprintf(stderr, "replay file is over or something goes wrong\n");
> +            qemu_system_vmstop_request_prepare();
> +            qemu_system_vmstop_request(RUN_STATE_INTERNAL_ERROR);
> +        }
> +    }
> +}
> +
> +void replay_fetch_data_kind(void)
> +{
> +    if (replay_file) {
> +        if (!replay_has_unread_data) {
> +            replay_data_kind = replay_get_byte();
> +            replay_check_error();
> +            replay_has_unread_data = 1;
> +        }
> +    }
> +}
> diff --git a/replay/replay-internal.h b/replay/replay-internal.h
> new file mode 100755
> index 0000000..d1b7575
> --- /dev/null
> +++ b/replay/replay-internal.h
> @@ -0,0 +1,44 @@
> +#ifndef REPLAY_INTERNAL_H
> +#define REPLAY_INTERNAL_H
> +
> +/*
> + * replay-internal.h
> + *
> + * Copyright (c) 2010-2015 Institute for System Programming
> + *                         of the Russian Academy of Sciences.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include <stdio.h>
> +
> +extern unsigned int replay_data_kind;
> +extern unsigned int replay_has_unread_data;
> +
> +/* File for replay writing */
> +extern FILE *replay_file;
> +
> +void replay_put_byte(uint8_t byte);
> +void replay_put_event(uint8_t event);
> +void replay_put_word(uint16_t word);
> +void replay_put_dword(uint32_t dword);
> +void replay_put_qword(int64_t qword);
> +void replay_put_array(const uint8_t *buf, size_t size);
> +
> +uint8_t replay_get_byte(void);
> +uint16_t replay_get_word(void);
> +uint32_t replay_get_dword(void);
> +int64_t replay_get_qword(void);
> +void replay_get_array(uint8_t *buf, size_t *size);
> +void replay_get_array_alloc(uint8_t **buf, size_t *size);
> +
> +/*! Checks error status of the file. */
> +void replay_check_error(void);
> +
> +/*! Reads data type from the file and stores it in the
> +    replay_data_kind variable. */
> +void replay_fetch_data_kind(void);
> +
> +#endif
> 

Re: [Qemu-devel] [RFC PATCH v9 08/23] cpu: replay instructions sequence

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> This patch adds calls to replay functions into the icount setup block.
> In record mode number of executed instructions is written to the log.
> In replay mode number of istructions to execute is taken from the replay log.

So much better!  If there is a v10, please add a comment about how the
iothread is woken up---before the call to qemu_notify_event() and here
in the commit message.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo

> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpus.c          |   38 +++++++++++++++++++++++++-------------
>  replay/replay.c |   26 ++++++++++++++++++++++++++
>  replay/replay.h |    4 ++++
>  3 files changed, 55 insertions(+), 13 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 04124ca..511a0c5 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -41,6 +41,7 @@
>  #include "qemu/seqlock.h"
>  #include "qapi-event.h"
>  #include "hw/nmi.h"
> +#include "replay/replay.h"
>  
>  #ifndef _WIN32
>  #include "qemu/compatfd.h"
> @@ -1306,6 +1307,28 @@ int vm_stop_force_state(RunState state)
>      }
>  }
>  
> +static int64_t tcg_get_icount_limit(void)
> +{
> +    int64_t deadline;
> +
> +    if (replay_mode != REPLAY_MODE_PLAY) {
> +        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
> +
> +        /* Maintain prior (possibly buggy) behaviour where if no deadline
> +         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
> +         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
> +         * nanoseconds.
> +         */
> +        if ((deadline < 0) || (deadline > INT32_MAX)) {
> +            deadline = INT32_MAX;
> +        }
> +
> +        return qemu_icount_round(deadline);
> +    } else {
> +        return replay_get_instructions();
> +    }
> +}
> +
>  static int tcg_cpu_exec(CPUArchState *env)
>  {
>      CPUState *cpu = ENV_GET_CPU(env);
> @@ -1319,24 +1342,12 @@ static int tcg_cpu_exec(CPUArchState *env)
>  #endif
>      if (use_icount) {
>          int64_t count;
> -        int64_t deadline;
>          int decr;
>          timers_state.qemu_icount -= (cpu->icount_decr.u16.low
>                                      + cpu->icount_extra);
>          cpu->icount_decr.u16.low = 0;
>          cpu->icount_extra = 0;
> -        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
> -
> -        /* Maintain prior (possibly buggy) behaviour where if no deadline
> -         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
> -         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
> -         * nanoseconds.
> -         */
> -        if ((deadline < 0) || (deadline > INT32_MAX)) {
> -            deadline = INT32_MAX;
> -        }
> -
> -        count = qemu_icount_round(deadline);
> +        count = tcg_get_icount_limit();
>          timers_state.qemu_icount += count;
>          decr = (count > 0xffff) ? 0xffff : count;
>          count -= decr;
> @@ -1354,6 +1365,7 @@ static int tcg_cpu_exec(CPUArchState *env)
>                          + cpu->icount_extra);
>          cpu->icount_decr.u32 = 0;
>          cpu->icount_extra = 0;
> +        replay_account_executed_instructions();
>      }
>      return ret;
>  }
> diff --git a/replay/replay.c b/replay/replay.c
> index a43bbbc..a6a5145 100755
> --- a/replay/replay.c
> +++ b/replay/replay.c
> @@ -58,3 +58,29 @@ uint64_t replay_get_current_step(void)
>  {
>      return cpu_get_icount_raw();
>  }
> +
> +int replay_get_instructions(void)
> +{
> +    int res = 0;
> +    replay_mutex_lock();
> +    if (skip_async_events(EVENT_INSTRUCTION)) {
> +        res = replay_state.instructions_count;
> +    }
> +    replay_mutex_unlock();
> +    return res;
> +}
> +
> +void replay_account_executed_instructions(void)
> +{
> +    if (replay_mode == REPLAY_MODE_PLAY
> +        && replay_state.instructions_count > 0) {
> +        int count = (int)(replay_get_current_step()
> +                          - replay_state.current_step);
> +        replay_state.instructions_count -= count;
> +        replay_state.current_step += count;
> +        if (replay_state.instructions_count == 0) {
> +            replay_has_unread_data = 0;
> +            qemu_notify_event();
> +        }
> +    }
> +}
> diff --git a/replay/replay.h b/replay/replay.h
> index a03c748..d19715f 100755
> --- a/replay/replay.h
> +++ b/replay/replay.h
> @@ -22,5 +22,9 @@ extern ReplayMode replay_mode;
>  
>  /*! Returns number of executed instructions. */
>  uint64_t replay_get_current_step(void);
> +/*! Returns number of instructions to execute in replay mode. */
> +int replay_get_instructions(void);
> +/*! Updates instructions counter in replay mode. */
> +void replay_account_executed_instructions(void);
>  
>  #endif
> 
> 
> 

Re: [Qemu-devel] [RFC PATCH v9 12/23] timer: replace time() with QEMU_CLOCK_HOST

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> This patch replaces time() function calls with calls to
> qemu_clock_get_ns(QEMU_CLOCK_HOST). It makes such requests deterministic
> in record/replay mode of icount.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

> ---
>  vl.c |   12 ++++++++----
>  1 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/vl.c b/vl.c
> index 8c8f142..00161f4 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -710,13 +710,17 @@ void vm_start(void)
>  /***********************************************************/
>  /* real time host monotonic timer */
>  
> +static time_t qemu_time(void)
> +{
> +    return qemu_clock_get_ms(QEMU_CLOCK_HOST) / 1000;
> +}
> +
>  /***********************************************************/
>  /* host time/date access */
>  void qemu_get_timedate(struct tm *tm, int offset)
>  {
> -    time_t ti;
> +    time_t ti = qemu_time();
>  
> -    time(&ti);
>      ti += offset;
>      if (rtc_date_offset == -1) {
>          if (rtc_utc)
> @@ -744,7 +748,7 @@ int qemu_timedate_diff(struct tm *tm)
>      else
>          seconds = mktimegm(tm) + rtc_date_offset;
>  
> -    return seconds - time(NULL);
> +    return seconds - qemu_time();
>  }
>  
>  static void configure_rtc_date_offset(const char *startdate, int legacy)
> @@ -782,7 +786,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
>                              "'2006-06-17T16:01:21' or '2006-06-17'\n");
>              exit(1);
>          }
> -        rtc_date_offset = time(NULL) - rtc_start_date;
> +        rtc_date_offset = qemu_time() - rtc_start_date;
>      }
>  }
>  
> 
> 
> 

Re: [Qemu-devel] [RFC PATCH v9 15/23] aio: replace stack of bottom halves with queue

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> +        QSIMPLEQ_FOREACH_SAFE(bh, &ctx->bh_queue, next, next) {
>              if (bh->deleted) {
> -                *bhp = bh->next;
> +                QSIMPLEQ_REMOVE(&ctx->bh_queue, bh, QEMUBH, next);

QSIMPLEQ_REMOVE is very inefficient, so that this loop can become
O(n^2).  You could use QTAILQ or introduce QSIMPLE_REMOVE_AFTER.

Paolo

>                  g_free(bh);
> -            } else {
> -                bhp = &bh->next;
>              }

Re: [Qemu-devel] [RFC PATCH v9 15/23] aio: replace stack of bottom halves with queue

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> +#define QSIMPLEQ_INSERT_TAIL_RCU(head, elm, field) do {                 \
> +    (elm)->field.sqe_next = NULL;                                       \

Also, smp_wmb() goes here.

My fault: in the previous review I wrote "smp_wmb() must be after the
(elm) assignment". I meant after the assignment of elm's fields (i.e.
write elm's fields before publishing it) but it wasn't clear at all.

Even better, you can replace the following two lines with:

    atomic_rcu_set((head)->sqh_last, (elm));

Paolo

> +    *(head)->sqh_last = (elm);                                          \
> +    smp_wmb();                                                          \

> +    (head)->sqh_last = &(elm)->field.sqe_next;                          \
> +} while (/*CONSTCOND*/0)
> +

Re: [Qemu-devel] [RFC PATCH v9 19/23] typedef: add typedef for QemuOpts

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> This patch moves typedefs for QemuOpts and related types
> to qemu/typedefs.h file.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  include/qemu/option.h   |    5 +----
>  include/qemu/typedefs.h |    3 +++
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/qemu/option.h b/include/qemu/option.h
> index 58c0157..4b0d336 100644
> --- a/include/qemu/option.h
> +++ b/include/qemu/option.h
> @@ -30,6 +30,7 @@
>  #include "qemu/queue.h"
>  #include "qapi/error.h"
>  #include "qapi/qmp/qdict.h"
> +#include "qemu/typedefs.h"
>  
>  const char *get_opt_name(char *buf, int buf_size, const char *p, char delim);
>  const char *get_opt_value(char *buf, int buf_size, const char *p);
> @@ -44,10 +45,6 @@ void parse_option_size(const char *name, const char *value,
>  bool has_help_option(const char *param);
>  bool is_valid_option_list(const char *param);
>  
> -typedef struct QemuOpt QemuOpt;
> -typedef struct QemuOpts QemuOpts;
> -typedef struct QemuOptsList QemuOptsList;
> -
>  enum QemuOptType {
>      QEMU_OPT_STRING = 0,  /* no parsing (use string as-is)                        */
>      QEMU_OPT_BOOL,        /* on/off                                               */
> diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
> index cde3314..a5f8d51 100644
> --- a/include/qemu/typedefs.h
> +++ b/include/qemu/typedefs.h
> @@ -64,6 +64,9 @@ typedef struct QEMUBH QEMUBH;
>  typedef struct QemuConsole QemuConsole;
>  typedef struct QEMUFile QEMUFile;
>  typedef struct QEMUMachine QEMUMachine;
> +typedef struct QemuOpt QemuOpt;
> +typedef struct QemuOpts QemuOpts;
> +typedef struct QemuOptsList QemuOptsList;
>  typedef struct QEMUSGList QEMUSGList;
>  typedef struct QEMUSizedBuffer QEMUSizedBuffer;
>  typedef struct QEMUTimerListGroup QEMUTimerListGroup;
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [RFC PATCH v9 20/23] replay: initialization and deinitialization

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> +void replay_init_timer(void)
> +{

You can call this replay_start.

> +        fprintf(stderr, "Invalid icount rr option: %s\n", rr);
> +        exit(1);
> +    }
> +
> +    fname = qemu_opt_get(opts, "rrfile");
> +    if (!fname) {
> +        fprintf(stderr, "File name not specified for replay\n");
> +        exit(1);

error_report here too, apart from this:

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [RFC PATCH v9 22/23] replay: command line options

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> This patch introduces command line options for enabling recording or replaying
> virtual machine behavior. "-record" option starts recording of the execution
> and saves it into the log, specified with "fname" parameter. "-replay" option
> is intended for replaying previously saved log.

Inaccurate commit message...

> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpus.c          |    1 +
>  qemu-options.hx |    8 ++++++--
>  vl.c            |   16 +++++++++++++++-
>  3 files changed, 22 insertions(+), 3 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 5b2f896..e289a9d 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -895,6 +895,7 @@ static void qemu_wait_io_event_common(CPUState *cpu)
>  static void qemu_tcg_wait_io_event(void)
>  {
>      CPUState *cpu;
> +    GMainContext *context = g_main_context_default();

... and unnecessary change (I think---shouldn't this give a warning?).
>  
>      while (all_cpu_threads_idle()) {
>         /* Start accounting real time to the virtual clock if the CPUs
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 85ca3ad..c62661a 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -3038,11 +3038,11 @@ re-inject them.
>  ETEXI
>  
>  DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
> -    "-icount [shift=N|auto][,align=on|off]\n" \
> +    "-icount [shift=N|auto][,align=on|off][,rr=record|replay,rrfile=<filename>]\n" \
>      "                enable virtual instruction counter with 2^N clock ticks per\n" \
>      "                instruction and enable aligning the host and virtual clocks\n", QEMU_ARCH_ALL)
>  STEXI
> -@item -icount [shift=@var{N}|auto]
> +@item -icount [shift=@var{N}|auto][,rr=record|replay,rrfile=@var{filename}]
>  @findex -icount
>  Enable virtual instruction counter.  The virtual cpu will execute one
>  instruction every 2^@var{N} ns of virtual time.  If @code{auto} is specified
> @@ -3064,6 +3064,10 @@ Currently this option does not work when @option{shift} is @code{auto}.
>  Note: The sync algorithm will work for those shift values for which
>  the guest clock runs ahead of the host clock. Typically this happens
>  when the shift value is high (how high depends on the host machine).
> +
> +When @option{rr} option is specified deterministic record/replay is enabled.
> +Replay log is written into @var{filename} file in record mode and
> +read from this file in replay mode.
>  ETEXI
>  
>  DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
> diff --git a/vl.c b/vl.c
> index a5555fb..f12d3db 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -472,6 +472,12 @@ static QemuOptsList qemu_icount_opts = {
>          }, {
>              .name = "align",
>              .type = QEMU_OPT_BOOL,
> +        }, {
> +            .name = "rr",
> +            .type = QEMU_OPT_STRING,
> +        }, {
> +            .name = "rrfile",
> +            .type = QEMU_OPT_STRING,
>          },
>          { /* end of list */ }
>      },
> @@ -2751,6 +2757,7 @@ int main(int argc, char **argv, char **envp)
>  {
>      int i;
>      int snapshot, linux_boot;
> +    const char *icount_option = NULL;
>      const char *initrd_filename;
>      const char *kernel_filename, *kernel_cmdline;
>      const char *boot_order;
> @@ -3770,6 +3777,8 @@ int main(int argc, char **argv, char **envp)
>          }
>      }
>  
> +    replay_configure(icount_opts);

Is it possible to call this together with configure_icount (or even from
configure_icount)?

>      opts = qemu_get_machine_opts();
>      optarg = qemu_opt_get(opts, "type");
>      if (optarg) {
> @@ -4137,6 +4146,11 @@ int main(int argc, char **argv, char **envp)
>          qemu_opts_del(icount_opts);
>      }
>  
> +    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
> +        fprintf(stderr, "Please enable icount to use record/replay\n");
> +        exit(1);
> +    }

Can be checked in configure_icount and/or replay_configure.

>      /* clean up network at qemu process termination */
>      atexit(&net_cleanup);
>  
> @@ -4173,7 +4187,7 @@ int main(int argc, char **argv, char **envp)
>      }
>  
>      /* open the virtual block devices */
> -    if (snapshot)
> +    if (snapshot || replay_mode != REPLAY_MODE_NONE)
>          qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);

Please add braces around "if" body, since you are at it.

>      if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
>                            &machine_class->block_default_type, 1) != 0) {
> 

Otherwise looks good.

Paolo

Re: [Qemu-devel] [RFC PATCH v9 08/23] cpu: replay instructions sequence

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> +        if (replay_state.instructions_count == 0) {

Please add an assertion that replay_data_kind is EVENT_INSTRUCTION,
since you are consuming it below.

> +            replay_has_unread_data = 0;
> +            qemu_notify_event();

Re: [Qemu-devel] [RFC PATCH v9 06/23] replay: introduce icount event

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> +    /* nothing to skip - not all instructions used */
> +    if (replay_state.instructions_count != 0
> +        && replay_has_unread_data) {
> +        return stop_event == EVENT_INSTRUCTION;

Same here, please assert that replay_data_kind == EVENT_INSTRUCTION.
Assertions communicate invariants, and thus help the person reading the
code.

Paolo

> +    }

Re: [Qemu-devel] [RFC PATCH v9 09/23] replay: interrupts and exceptions

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> +{
> +    bool res = false;
> +    if (replay_mode == REPLAY_MODE_PLAY) {
> +        replay_account_executed_instructions();
> +        replay_mutex_lock();
> +        if (skip_async_events(EVENT_EXCEPTION)) {
> +            res = true;
> +        }

Just "res = skip_async_events(EVENT_EXCEPTION)" is simpler.

Paolo

> +        replay_mutex_unlock();
> +    }
> +

Re: [Qemu-devel] [RFC PATCH v9 11/23] replay: recording and replaying clock ticks

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> +void replay_read_next_clock(ReplayClockKind kind)

This function is called like this:

> +    if (replay_file) {
> +        int64_t ret;
> +        replay_mutex_lock();
> +        if (skip_async_events(EVENT_CLOCK + kind)) {
> +            replay_read_next_clock(kind);
> +        }

So it seems to me that this section is unnecessary:

> +    replay_fetch_data_kind();
> +    if (replay_file) {
> +        unsigned int read_kind = replay_data_kind - EVENT_CLOCK;
> +
> +        if (read_kind != kind) {
> +            return;
> +        }

1) replay_file must be non-NULL

2) skip_async_events has fetched replay_data_kind

3) replay_data_kind - EVENT_CLOCK must match read_kind

It can be replaced simply with

    assert(replay_has_unread_data);
    assert(replay_data_kind - EVENT_CLOCK == kind);

I suspect there are other cases like this where you have unnecessary
calls to replay_fetch_data_kind().

Paolo

Re: [Qemu-devel] [RFC PATCH v9 14/23] replay: checkpoints

[Paolo Bonzini]

On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> 
> +
> +    if (replay_file) {
> +        if (replay_mode == REPLAY_MODE_PLAY) {
> +            replay_mutex_lock();
> +            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
> +                if (replay_data_kind == EVENT_ASYNC) {
> +                    replay_read_events(checkpoint);
> +                    replay_fetch_data_kind();
> +                    res = replay_data_kind != EVENT_ASYNC;
> +                    replay_mutex_unlock();
> +                    return res;
> +                }
> +                replay_mutex_unlock();
> +                return res;
> +            }
> +            replay_has_unread_data = 0;
> +            replay_read_events(checkpoint);
> +            replay_fetch_data_kind();
> +            res = replay_data_kind != EVENT_ASYNC;
> +            replay_mutex_unlock();
> +            return res;
> +        } else if (replay_mode == REPLAY_MODE_RECORD) {
> +            replay_mutex_lock();
> +            replay_put_event(EVENT_CHECKPOINT + checkpoint);
> +            replay_save_events(checkpoint);
> +            replay_mutex_unlock();
> +        }
> +    }
> +
> +    return true;

This is cleaner:

    if (!replay_file) {
        return true;
    }

    replay_mutex_lock();

    if (replay_mode == REPLAY_MODE_PLAY) {
        if (skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
            replay_has_unread_data = 0;
        } else if (replay_data_kind != EVENT_ASYNC) {
            res = false;
            goto out;
        }
        replay_read_events(checkpoint);
        replay_fetch_data_kind();
        res = replay_data_kind != EVENT_ASYNC;
    } else if (replay_mode == REPLAY_MODE_RECORD) {
        replay_put_event(EVENT_CHECKPOINT + checkpoint);
        replay_save_events(checkpoint);
        res = true;
    }
out:
    replay_mutex_unlock();
    return res;

I'm not sure however of the meaning of "res = replay_data_kind !=
EVENT_ASYNC;".  Why not just return true since skip_async_events has
returned true?  Can you add a comment?

Paolo


> +        if (!replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
> +            /* Do not wait anymore, we stopped at some place in
> +               the middle of execution during replay */
> +            return;
> +        }
>          busy = false;
>  
>          QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
> @@ -2004,6 +2010,12 @@ void bdrv_drain_all(void)
>              aio_context_release(aio_context);
>          }
>      }
> +    if (replay_mode == REPLAY_MODE_PLAY) {
> +        /* Skip checkpoints from the log */
> +        while (replay_checkpoint(CHECKPOINT_BDRV_DRAIN)) {
> +            /* Nothing */
> +        }
> +    }
>  }

Can you explain this?  Otherwise the patch looks great.

Re: [Qemu-devel] [RFC PATCH v9 06/23] replay: introduce icount event

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
> +/*! Skips async events until some sync event will be found. */
> +bool skip_async_events(int stop_event);
> +/*! Skips async events invocations from the input,
> +    until required data kind is found. If the requested data is not found
> +    reports an error and stops the execution. */
> +void skip_async_events_until(unsigned int kind);

skip_async_events_until is unused, please remove it.

Also, "skip_async_events" is not skipping async events anymore. :)
Perhaps rename it to replay_next_event_is?

Paolo

Re: [Qemu-devel] [RFC PATCH v9 09/23] replay: interrupts and exceptions

[Paolo Bonzini]

On 18/02/2015 12:56, Pavel Dovgalyuk wrote:
>      if (interrupt_request & CPU_INTERRUPT_POLL) {
>          cs->interrupt_request &= ~CPU_INTERRUPT_POLL;
>          apic_poll_irq(cpu->apic_state);
> +        /* Process one interrupt for a single call.
> +           In record/replay mode each call to this function
> +           is coupled with an event in the log */
> +        if (replay_mode != REPLAY_MODE_NONE) {
> +            return true;
> +        }

Is it okay to just return true always, even outside replay mode?  If so,
make it a separate patch.

Paolo

Re: [Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Paolo Bonzini]

On 18/02/2015 12:55, Pavel Dovgalyuk wrote:
> This set of patches is related to the reverse execution and deterministic 
> replay of qemu execution. This implementation of deterministic replay can 
> be used for deterministic debugging of guest code through gdb remote
> interface.
> 
> These patches include only core function of the replay,
> excluding the support for replaying serial, audio, network, and USB devices'
> operations. Reverse debugging and monitor commands were also excluded to
> be submitted later as separate patches.
> 
> Execution recording writes non-deterministic events log, which can be later 
> used for replaying the execution anywhere and for unlimited number of times. 
> It also supports checkpointing for faster rewinding during reverse debugging. 
> Execution replaying reads the log and replays all non-deterministic events 
> including external input, hardware clocks, and interrupts.
> 
> Deterministic replay has the following features:
>  * Deterministically replays whole system execution and all contents of the memory,
>    state of the hadrware devices, clocks, and screen of the VM.
>  * Writes execution log into the file for latter replaying for multiple times 
>    on different machines.
>  * Supports i386, x86_64, and ARM hardware platforms.
>  * Performs deterministic replay of all operations with keyboard and mouse
>    input devices.
>  * Supports auto-checkpointing for convenient reverse debugging.
> 
> Usage of the record/replay:
>  * First, record the execution, by adding the following string to the command line:
>    '-icount shift=7,rr=record,rrfile=replay.bin -net none'. 
>    Block devices' images are not actually changed in the recording mode, 
>    because all of the changes are written to the temporary overlay file.
>  * Then you can replay it for the multiple times by using another command
>    line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
>  * '-net none' option should also be specified if network replay patches
>    are not applied.
> 
> Paper with short description of deterministic replay implementation:
> http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
> 
> Modifications of qemu include:
>  * wrappers for clock and time functions to save their return values in the log
>  * saving different asynchronous events (e.g. system shutdown) into the log
>  * synchronization of the bottom halves execution
>  * synchronization of the threads from thread pool
>  * recording/replaying user input (mouse and keyboard)
>  * adding internal events for cpu and io synchronization
> 
> v9 changes:
>  * Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
>  * Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
>  * Removed one of the replay mutexes (as suggested by Paolo Bonzini)
>  * Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
>  * Updated command line options' names (as suggested by Paolo Bonzini)
>  * Added design document for record/replay (as suggested by Paolo Bonzini)
>  * Simplified checkpoints for the timers
>  * Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
>  * Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
>  * Some functions renaming and extracting.

I haven't yet reviewed patch 23 completely, and I have to think (a lot
:)) more about block devices.  In the meanwhile I understand the replay
code much better so I had some suggestions.

In general, the handling of replay_has_unread_data / replay_data_kind is
a bit messy.  It would be nice if you could call
replay_fetch_data_kind() only when replay_has_unread_data == 0.  Or,
even, remove replay_has_unread_data altogether: just call
replay_fetch_data_kind() when you'd set it to zero.  That would simplify
a lot the code for readers.

Paolo

> v8 changes:
>  * Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
>  * Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
>  * Moved replay_submode out of the series (as suggested by Paolo Bonzini)
>  * Moved suffix option out of the series
>  * Converted some of the defines into enums (as suggested by Paolo Bonzini)
>  * Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
>  * Moved record/replay options to icount group (as suggested by Paolo Bonzini)
>  * Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
>  * Added mutex to protect replay log file (as suggested by Paolo Bonzini)
>  * Minor cleanups
> 
> v7 changes:
>  * Removed patches that were applied to upstream.
> 
> v6 changes:
>  * Fixed replay stub return value (as suggested by Eric Blake)
>  * Fixed icount warping.
>  * Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
>  * Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
>  * Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
>  * Removed applied patches.
> 
> v5 changes:
>  * Minor changes.
>  * Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
>  * Moved savevm-related code out of the core.
>  * Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
>  * Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
>  * Adopted Paolo's icount patches
>  * Fixed hardware interrupts replaying
> 
> v4 changes:
>  * Updated block drivers to support new bdrv_open interface.
>  * Moved migration patches into separate series (as suggested by Paolo Bonzini)
>  * Fixed a bug in replay_break operation.
>  * Fixed rtl8139 migration for replay.
>  * Fixed 'period' parameter processing for record mode.
>  * Fixed bug in 'reverse-stepi' implementation.
>  * Fixed replay without making any snapshots (even the starting one).
>  * Moved core replay patches into the separate series.
>  * Fixed reverse step and reverse continue support.
>  * Fixed several bugs in icount subsystem.
>  * Reusing native qemu icount for replay instructions counting.
>  * Separated core patches into their own series.
> 
> v3 changes:
>  * Fixed bug with replay of the aio write operations.
>  * Added virtual clock based on replay icount.
>  * Removed duplicated saving of interrupt_request CPU field.
>  * Fixed some coding style issues.
>  * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
>  * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
>  * Added description and example for replay-info command (as suggested by Eric Blake)
>  * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
>  * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
>  * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
>  * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
>  * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
>  * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)
> 
> v2 changes:
>  * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
>  * Added QMP versions of replay commands (as suggested by Eric Blake)
>  * Removed some optional features of replay to make patches cleaner
>  * Minor changes and code cleanup were made
> 
> ---
> 
> Pavel Dovgalyuk (23):
>       i386: partial revert of interrupt poll fix
>       replay: global variables and function stubs
>       sysemu: system functions for replay
>       replay: internal functions for replay log
>       replay: introduce mutex to protect the replay log
>       replay: introduce icount event
>       cpu-exec: allow temporary disabling icount
>       cpu: replay instructions sequence
>       replay: interrupts and exceptions
>       replay: asynchronous events infrastructure
>       replay: recording and replaying clock ticks
>       timer: replace time() with QEMU_CLOCK_HOST
>       replay: shutdown event
>       replay: checkpoints
>       aio: replace stack of bottom halves with queue
>       replay: bottom halves
>       replay: replay aio requests
>       replay: thread pool
>       typedef: add typedef for QemuOpts
>       replay: initialization and deinitialization
>       replay: replay blockers for devices
>       replay: command line options
>       replay: recording of the user input
> 
> 
>  Makefile.target                |    1 
>  async.c                        |   49 ++++--
>  block.c                        |   93 ++++++++++-
>  block/block-backend.c          |   30 +++
>  block/qcow2.c                  |    4 
>  block/raw-posix.c              |    6 -
>  block/raw-win32.c              |    4 
>  cpu-exec.c                     |   59 +++++--
>  cpus.c                         |   51 ++++--
>  dma-helpers.c                  |   10 +
>  docs/replay.txt                |  161 ++++++++++++++++++
>  exec.c                         |    1 
>  hw/block/virtio-blk.c          |   10 +
>  hw/bt/hci.c                    |    7 +
>  hw/ide/ahci.c                  |    4 
>  hw/ide/atapi.c                 |   10 +
>  hw/ide/core.c                  |   18 +-
>  hw/timer/arm_timer.c           |    2 
>  hw/usb/hcd-uhci.c              |    2 
>  include/block/aio.h            |   22 ++-
>  include/block/block.h          |   15 ++
>  include/block/thread-pool.h    |    4 
>  include/exec/exec-all.h        |    2 
>  include/qapi/qmp/qerror.h      |    3 
>  include/qemu-common.h          |    2 
>  include/qemu/main-loop.h       |    1 
>  include/qemu/option.h          |    5 -
>  include/qemu/queue.h           |    7 +
>  include/qemu/typedefs.h        |    3 
>  include/qom/cpu.h              |   10 +
>  include/sysemu/block-backend.h |   10 +
>  include/sysemu/cpus.h          |    1 
>  include/ui/input.h             |    2 
>  main-loop.c                    |   11 +
>  qapi-schema.json               |   18 ++
>  qemu-io-cmds.c                 |    2 
>  qemu-options.hx                |    8 +
>  qemu-timer.c                   |   45 ++++-
>  replay/Makefile.objs           |    5 +
>  replay/replay-events.c         |  287 +++++++++++++++++++++++++++++++++
>  replay/replay-input.c          |  159 ++++++++++++++++++
>  replay/replay-internal.c       |  191 ++++++++++++++++++++++
>  replay/replay-internal.h       |  144 ++++++++++++++++
>  replay/replay-time.c           |   84 ++++++++++
>  replay/replay.c                |  351 ++++++++++++++++++++++++++++++++++++++++
>  replay/replay.h                |  120 ++++++++++++++
>  stubs/Makefile.objs            |    1 
>  stubs/replay.c                 |   41 +++++
>  target-i386/cpu.c              |   10 -
>  target-i386/seg_helper.c       |    7 +
>  tests/test-thread-pool.c       |    7 -
>  thread-pool.c                  |   49 ++++--
>  trace-events                   |    2 
>  translate-all.c                |   10 +
>  ui/input.c                     |   27 ++-
>  util/iov.c                     |    4 
>  vl.c                           |   47 +++++
>  57 files changed, 2100 insertions(+), 139 deletions(-)
>  create mode 100755 docs/replay.txt
>  create mode 100755 replay/Makefile.objs
>  create mode 100755 replay/replay-events.c
>  create mode 100755 replay/replay-input.c
>  create mode 100755 replay/replay-internal.c
>  create mode 100755 replay/replay-internal.h
>  create mode 100755 replay/replay-time.c
>  create mode 100755 replay/replay.c
>  create mode 100755 replay/replay.h
>  create mode 100755 stubs/replay.c
> 

Re: [Qemu-devel] [RFC PATCH v9 22/23] replay: command line options

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
> On 18/02/2015 12:57, Pavel Dovgalyuk wrote:
> > @@ -2751,6 +2757,7 @@ int main(int argc, char **argv, char **envp)
> >  {
> >      int i;
> >      int snapshot, linux_boot;
> > +    const char *icount_option = NULL;
> >      const char *initrd_filename;
> >      const char *kernel_filename, *kernel_cmdline;
> >      const char *boot_order;
> > @@ -3770,6 +3777,8 @@ int main(int argc, char **argv, char **envp)
> >          }
> >      }
> >
> > +    replay_configure(icount_opts);
> 
> Is it possible to call this together with configure_icount (or even from
> configure_icount)?

I think that it is the best place for replay_configure.
It sets replay_mode which may be checked while configuring the virtual machine.
That's why this function is called before any initialization actions.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v9 22/23] replay: command line options

[Paolo Bonzini]

On 20/02/2015 09:02, Pavel Dovgaluk wrote:
>> > 
>> > Is it possible to call this together with configure_icount (or even from
>> > configure_icount)?
> I think that it is the best place for replay_configure.
> It sets replay_mode which may be checked while configuring the virtual machine.
> That's why this function is called before any initialization actions.

Ok, that was only a minor remark.

Paolo

Re: [Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
> On 18/02/2015 12:55, Pavel Dovgalyuk wrote:
> > This set of patches is related to the reverse execution and deterministic
> > replay of qemu execution. This implementation of deterministic replay can
> > be used for deterministic debugging of guest code through gdb remote
> > interface.
> >
> > These patches include only core function of the replay,
> > excluding the support for replaying serial, audio, network, and USB devices'
> > operations. Reverse debugging and monitor commands were also excluded to
> > be submitted later as separate patches.
> >
> > Execution recording writes non-deterministic events log, which can be later
> > used for replaying the execution anywhere and for unlimited number of times.
> > It also supports checkpointing for faster rewinding during reverse debugging.
> > Execution replaying reads the log and replays all non-deterministic events
> > including external input, hardware clocks, and interrupts.
> >
> > Deterministic replay has the following features:
> >  * Deterministically replays whole system execution and all contents of the memory,
> >    state of the hadrware devices, clocks, and screen of the VM.
> >  * Writes execution log into the file for latter replaying for multiple times
> >    on different machines.
> >  * Supports i386, x86_64, and ARM hardware platforms.
> >  * Performs deterministic replay of all operations with keyboard and mouse
> >    input devices.
> >  * Supports auto-checkpointing for convenient reverse debugging.
> >
> > Usage of the record/replay:
> >  * First, record the execution, by adding the following string to the command line:
> >    '-icount shift=7,rr=record,rrfile=replay.bin -net none'.
> >    Block devices' images are not actually changed in the recording mode,
> >    because all of the changes are written to the temporary overlay file.
> >  * Then you can replay it for the multiple times by using another command
> >    line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
> >  * '-net none' option should also be specified if network replay patches
> >    are not applied.
> >
> > Paper with short description of deterministic replay implementation:
> > http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
> >
> > Modifications of qemu include:
> >  * wrappers for clock and time functions to save their return values in the log
> >  * saving different asynchronous events (e.g. system shutdown) into the log
> >  * synchronization of the bottom halves execution
> >  * synchronization of the threads from thread pool
> >  * recording/replaying user input (mouse and keyboard)
> >  * adding internal events for cpu and io synchronization
> >
> > v9 changes:
> >  * Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
> >  * Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
> >  * Removed one of the replay mutexes (as suggested by Paolo Bonzini)
> >  * Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
> >  * Updated command line options' names (as suggested by Paolo Bonzini)
> >  * Added design document for record/replay (as suggested by Paolo Bonzini)
> >  * Simplified checkpoints for the timers
> >  * Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
> >  * Added replay blockers instead of checking the command line (as suggested by Paolo
> Bonzini)
> >  * Some functions renaming and extracting.
> 
> I haven't yet reviewed patch 23 completely, and I have to think (a lot
> :)) more about block devices.  In the meanwhile I understand the replay
> code much better so I had some suggestions.

Do you have more comments? I'm ready to submit a new version.

> 
> In general, the handling of replay_has_unread_data / replay_data_kind is
> a bit messy.  It would be nice if you could call
> replay_fetch_data_kind() only when replay_has_unread_data == 0.  Or,
> even, remove replay_has_unread_data altogether: just call
> replay_fetch_data_kind() when you'd set it to zero.  That would simplify
> a lot the code for readers.

Fixed.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Paolo Bonzini]

On 27/02/2015 10:23, Pavel Dovgaluk wrote:
>> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
>> On 18/02/2015 12:55, Pavel Dovgalyuk wrote:
>>> This set of patches is related to the reverse execution and deterministic
>>> replay of qemu execution. This implementation of deterministic replay can
>>> be used for deterministic debugging of guest code through gdb remote
>>> interface.
>>>
>>> These patches include only core function of the replay,
>>> excluding the support for replaying serial, audio, network, and USB devices'
>>> operations. Reverse debugging and monitor commands were also excluded to
>>> be submitted later as separate patches.
>>>
>>> Execution recording writes non-deterministic events log, which can be later
>>> used for replaying the execution anywhere and for unlimited number of times.
>>> It also supports checkpointing for faster rewinding during reverse debugging.
>>> Execution replaying reads the log and replays all non-deterministic events
>>> including external input, hardware clocks, and interrupts.
>>>
>>> Deterministic replay has the following features:
>>>  * Deterministically replays whole system execution and all contents of the memory,
>>>    state of the hadrware devices, clocks, and screen of the VM.
>>>  * Writes execution log into the file for latter replaying for multiple times
>>>    on different machines.
>>>  * Supports i386, x86_64, and ARM hardware platforms.
>>>  * Performs deterministic replay of all operations with keyboard and mouse
>>>    input devices.
>>>  * Supports auto-checkpointing for convenient reverse debugging.
>>>
>>> Usage of the record/replay:
>>>  * First, record the execution, by adding the following string to the command line:
>>>    '-icount shift=7,rr=record,rrfile=replay.bin -net none'.
>>>    Block devices' images are not actually changed in the recording mode,
>>>    because all of the changes are written to the temporary overlay file.
>>>  * Then you can replay it for the multiple times by using another command
>>>    line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
>>>  * '-net none' option should also be specified if network replay patches
>>>    are not applied.
>>>
>>> Paper with short description of deterministic replay implementation:
>>> http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
>>>
>>> Modifications of qemu include:
>>>  * wrappers for clock and time functions to save their return values in the log
>>>  * saving different asynchronous events (e.g. system shutdown) into the log
>>>  * synchronization of the bottom halves execution
>>>  * synchronization of the threads from thread pool
>>>  * recording/replaying user input (mouse and keyboard)
>>>  * adding internal events for cpu and io synchronization
>>>
>>> v9 changes:
>>>  * Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
>>>  * Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
>>>  * Removed one of the replay mutexes (as suggested by Paolo Bonzini)
>>>  * Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
>>>  * Updated command line options' names (as suggested by Paolo Bonzini)
>>>  * Added design document for record/replay (as suggested by Paolo Bonzini)
>>>  * Simplified checkpoints for the timers
>>>  * Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
>>>  * Added replay blockers instead of checking the command line (as suggested by Paolo
>> Bonzini)
>>>  * Some functions renaming and extracting.
>>
>> I haven't yet reviewed patch 23 completely, and I have to think (a lot
>> :)) more about block devices.  In the meanwhile I understand the replay
>> code much better so I had some suggestions.
> 
> Do you have more comments? I'm ready to submit a new version.

23 is fine.  Block devices probably are best left as is for now.

Paolo

>>
>> In general, the handling of replay_has_unread_data / replay_data_kind is
>> a bit messy.  It would be nice if you could call
>> replay_fetch_data_kind() only when replay_has_unread_data == 0.  Or,
>> even, remove replay_has_unread_data altogether: just call
>> replay_fetch_data_kind() when you'd set it to zero.  That would simplify
>> a lot the code for readers.
> 
> Fixed.
> 
> Pavel Dovgalyuk
>