[Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Pavel Dovgalyuk]

This set of patches is related to the reverse execution and deterministic 
replay of qemu execution  Our implementation of deterministic replay can 
be used for deterministic and reverse debugging of guest code through gdb 
remote interface.

Execution recording writes non-deterministic events log, which can be later 
used for replaying the execution anywhere and for unlimited number of times. 
It also supports checkpointing for faster rewinding during reverse debugging. 
Execution replaying reads the log and replays all non-deterministic events 
including external input, hardware clocks, and interrupts.

Reverse execution has the following features:
 * Deterministically replays whole system execution and all contents of the memory,
   state of the hadrware devices, clocks, and screen of the VM.
 * Writes execution log into the file for latter replaying for multiple times 
   on different machines.
 * Supports i386, x86_64, and ARM hardware platforms.
 * Performs deterministic replay of all operations with keyboard, mouse, network adapters,
   audio devices, serial interfaces, and physical USB devices connected to the emulator.
 * Provides support for gdb reverse debugging commands like reverse-step and reverse-continue.
 * Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
 * First, record the execution, by adding '-record fname=replay.bin' to the
   command line.
 * Then you can replay it for the multiple times by using another command
   line option: '-replay fname=replay.bin'
 * Virtual machine should have at least one virtual disk, which is used to
   store checkpoints. If you want to enable automatic checkpointing, simply
   add ',period=XX' to record options, where XX is the checkpointing period
   in seconds.
 * Using of the network adapters in record/replay mode is possible with 
   the following command-line options:
   - '-net user' (or another host adapter) in record mode
   - '-net replay' in replay mode. Every host network adapter should be
     replaced by 'replay' when replaying the execution.
 * Reverse debugging can be used through gdb remote interface.
   reverse-stepi and reverse-continue commands are supported. Other reverse
   commands should also work, because they reuse these ones.
 * Monitor is extended by the following commands:
   - replay_info - prints information about replay mode and current step
     (number of instructions executed)
   - replay_break - sets "breakpoint" at the specified instructions count.
   - replay_seek - rewinds (using the checkpoints, if possible) to the
     specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
 * adding missed fields of the virtual devices' states to the vmstate 
   structures to allow deterministic saving and restoring the VM state
 * adding virtual clock-based timers to vmstate structures, because virtual 
   clock is the part of the virtual machine state
 * modification of block layer to support automatic creation of the overlay
   files to store the changes and snapshots while recording
 * disabling of system reset while loading VM state to avoid generating of
   interrupts by reset handlers
 * adding warpers for clock and time functions to save their return
   values in the log
 * saving different asynchronous events (e.g. system shutdown) into the log
 * synchronization of the bottom halves execution
 * synchronization of the threads from thread pool
 * recording/replaying user input (mouse and keyboard), input from virtual
   serial ports, incoming network packets, input from connected USB devices
 * adding HMP/QMP commands to monitor for controlling replay execution

v4 changes:
 * Updated block drivers to support new bdrv_open interface.
 * Moved migration patches into separate series (as suggested by Paolo Bonzini)
 * Fixed a bug in replay_break operation.
 * Fixed rtl8139 migration for replay.
 * Fixed 'period' parameter processing for record mode.
 * Fixed bug in 'reverse-stepi' implementation.
 * Fixed replay without making any snapshots (even the starting one).
 * Moved core replay patches into the separate series.
 * Fixed reverse step and reverse continue support.

v3 changes:
 * Fixed bug with replay of the aio write operations.
 * Added virtual clock based on replay icount.
 * Removed duplicated saving of interrupt_request CPU field.
 * Fixed some coding style issues.
 * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
 * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
 * Added description and example for replay-info command (as suggested by Eric Blake)
 * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
 * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
 * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
 * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
 * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
 * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
 * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
 * Added QMP versions of replay commands (as suggested by Eric Blake)
 * Removed some optional features of replay to make patches cleaner
 * Minor changes and code cleanup were made

---

Paolo Bonzini (4):
      From 7abf2f72777958d395cfd01d97fe707cc06152b5 Mon Sep 17 00:00:00 2001
      From 185a3a47d08857a66332ae862b372a153ce92bb9 Mon Sep 17 00:00:00 2001
      From a0cb9e80ba0de409b5ad556109a1c71ce4d8ce19 Mon Sep 17 00:00:00 2001
      From 04bbd21134dd2c6b7309a7f5f2b780aae2757003 Mon Sep 17 00:00:00 2001

Pavel Dovgalyuk (27):
      cpu-exec: fix cpu_exec_nocache
      acpi: accurate overflow check
      replay: global variables and function stubs
      sysemu: system functions for replay
      replay: internal functions for replay log
      cpu-exec: reset exception_index correctly
      icount: implement icount requesting
      icount: improve enable/disable ticks
      replay: introduce icount event
      i386: do not cross the pages boundaries in replay mode
      cpu-exec: allow temporary disabling icount
      cpu-exec: invalidate nocache translation if they are interrupted
      replay: interrupts and exceptions
      replay: asynchronous events infrastructure
      cpu: replay instructions sequence
      replay: recording and replaying clock ticks
      replay: recording and replaying different timers
      timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock
      cpus: make icount warp deterministic in replay mode
      replay: shutdown event
      replay: checkpoints
      replay: bottom halves
      replay: replay aio requests
      replay: thread pool
      replay: initialization and deinitialization
      replay: command line options
      replay: recording of the user input


 Makefile.target                |    1 
 async.c                        |   46 +++++-
 block.c                        |   92 +++++++++++-
 block/block-backend.c          |   30 ++++
 block/qcow2.c                  |    4 +
 block/raw-posix.c              |    6 +
 block/raw-win32.c              |    4 -
 cpu-exec.c                     |   50 +++++-
 cpus.c                         |   97 +++++++++---
 dma-helpers.c                  |   10 +
 exec.c                         |    1 
 hw/acpi/core.c                 |    7 +
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4 -
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 ++
 hw/timer/arm_timer.c           |    2 
 hw/timer/mc146818rtc.c         |   10 +
 hw/timer/pl031.c               |   10 +
 hw/usb/hcd-uhci.c              |    2 
 include/block/aio.h            |   18 ++
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4 -
 include/exec/exec-all.h        |    8 +
 include/exec/gen-icount.h      |    6 -
 include/qemu-common.h          |    3 
 include/qemu/main-loop.h       |    1 
 include/qemu/timer.h           |   34 ++++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1 
 include/sysemu/sysemu.h        |    1 
 include/ui/input.h             |    2 
 main-loop.c                    |   10 +
 qapi-schema.json               |   32 ++++
 qemu-io-cmds.c                 |    2 
 qemu-options.hx                |   27 +++
 qemu-timer.c                   |   51 +++++-
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  292 +++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  108 ++++++++++++++
 replay/replay-internal.c       |  155 ++++++++++++++++++++
 replay/replay-internal.h       |  132 +++++++++++++++++
 replay/replay-time.c           |  191 ++++++++++++++++++++++++
 replay/replay.c                |  314 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  119 +++++++++++++++
 stubs/Makefile.objs            |    1 
 stubs/replay.c                 |   42 +++++
 target-alpha/translate.c       |   10 +
 target-arm/translate-a64.c     |    6 -
 target-arm/translate.c         |    6 -
 target-cris/translate.c        |    2 
 target-i386/cpu.h              |    3 
 target-i386/translate.c        |   62 +++++---
 target-lm32/translate.c        |   10 +
 target-m68k/translate.c        |    2 
 target-microblaze/translate.c  |    2 
 target-mips/translate.c        |   26 ++-
 target-moxie/translate.c       |    2 
 target-openrisc/translate.c    |    2 
 target-ppc/cpu.h               |   13 +-
 target-ppc/translate.c         |   12 +-
 target-ppc/translate_init.c    |  271 +++++++++++++++++------------------
 target-s390x/translate.c       |    2 
 target-sh4/translate.c         |    2 
 target-sparc/translate.c       |    2 
 target-tricore/translate.c     |    2 
 target-unicore32/translate.c   |    2 
 target-xtensa/translate.c      |    2 
 tests/test-thread-pool.c       |    7 +
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2 
 translate-all.c                |   19 ++
 ui/input.c                     |   80 ++++++++--
 util/iov.c                     |    4 +
 vl.c                           |  112 +++++++++++++-
 76 files changed, 2365 insertions(+), 357 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v5 01/31] cpu-exec: fix cpu_exec_nocache

[Pavel Dovgalyuk]

In icount mode cpu_exec_nocache function is used to execute part of the
existing TB. At the end of cpu_exec_nocache newly created TB is deleted.
Sometimes io_read function needs to recompile current TB and restart TB
lookup and execution. After that tb_find_fast function finds old (bigger)
TB again. This TB cannot be executed (because icount is not big enough)
and cpu_exec_nocache is called again. Such a loop continues over and over.
This patch deletes old TB and avoids finding it in the TB cache.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 3913de0..8830255 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -202,13 +202,18 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
+    target_ulong pc = orig_tb->pc;
+    target_ulong cs_base = orig_tb->cs_base;
+    uint64_t flags = orig_tb->flags;
 
     /* Should never happen.
        We only end up here when an existing TB is too long.  */
     if (max_cycles > CF_COUNT_MASK)
         max_cycles = CF_COUNT_MASK;
 
-    tb = tb_gen_code(cpu, orig_tb->pc, orig_tb->cs_base, orig_tb->flags,
+    /* tb_gen_code can flush our orig_tb, invalidate it now */
+    tb_phys_invalidate(orig_tb, -1);
+    tb = tb_gen_code(cpu, pc, cs_base, flags,
                      max_cycles);
     cpu->current_tb = tb;
     /* execute the generated code */

[Qemu-devel] [RFC PATCH v5 02/31] acpi: accurate overflow check

[Pavel Dovgalyuk]

Compare clock in ns, because acpi_pm_tmr_update uses rounded
to ns value instead of ticks.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/acpi/core.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/hw/acpi/core.c b/hw/acpi/core.c
index a7368fb..51913d6 100644
--- a/hw/acpi/core.c
+++ b/hw/acpi/core.c
@@ -376,8 +376,11 @@ static void acpi_notify_wakeup(Notifier *notifier, void *data)
 /* ACPI PM1a EVT */
 uint16_t acpi_pm1_evt_get_sts(ACPIREGS *ar)
 {
-    int64_t d = acpi_pm_tmr_get_clock();
-    if (d >= ar->tmr.overflow_time) {
+    /* Compare ns-clock, not PM timer ticks, because
+       acpi_pm_tmr_update function uses ns for setting the timer. */
+    int64_t d = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    if (d >= muldiv64(ar->tmr.overflow_time,
+                      get_ticks_per_sec(), PM_TIMER_FREQUENCY)) {
         ar->pm1.evt.sts |= ACPI_BITMASK_TIMER_STATUS;
     }
     return ar->pm1.evt.sts;

[Qemu-devel] [RFC PATCH v5 03/31] replay: global variables and function stubs

[Pavel Dovgalyuk]

This patch adds global variables, defines, functions declarations,
and function stubs for deterministic VM replay used by external modules.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 Makefile.target      |    1 +
 qapi-schema.json     |   32 ++++++++++++++++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   25 +++++++++++++++++++++++++
 replay/replay.h      |   23 +++++++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    8 ++++++++
 7 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

diff --git a/Makefile.target b/Makefile.target
index e9ff1ee..a45378f 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -83,6 +83,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
diff --git a/qapi-schema.json b/qapi-schema.json
index 24379ab..797600e 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3508,3 +3508,35 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
+
+##
+# ReplaySubmode:
+#
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.3
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
new file mode 100755
index 0000000..7ea860f
--- /dev/null
+++ b/replay/Makefile.objs
@@ -0,0 +1 @@
+obj-y += replay.o
diff --git a/replay/replay.c b/replay/replay.c
new file mode 100755
index 0000000..ac976b2
--- /dev/null
+++ b/replay/replay.c
@@ -0,0 +1,25 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
+/*! Stores current submode for PLAY mode */
+ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
+
+/* Suffix for the disk images filenames */
+char *replay_image_suffix;
+
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return play_submode;
+}
diff --git a/replay/replay.h b/replay/replay.h
new file mode 100755
index 0000000..51a18fe
--- /dev/null
+++ b/replay/replay.h
@@ -0,0 +1,23 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+extern char *replay_image_suffix;
+
+/*! Returns replay play submode */
+ReplaySubmode replay_get_play_submode(void);
+
+#endif
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 5e347d0..45a6c71 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
diff --git a/stubs/replay.c b/stubs/replay.c
new file mode 100755
index 0000000..b146d55
--- /dev/null
+++ b/stubs/replay.c
@@ -0,0 +1,8 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return 0;
+}

[Qemu-devel] [RFC PATCH v5 04/31] sysemu: system functions for replay

[Pavel Dovgalyuk]

This patch removes "static" specifier from several qemu function to make
them visible to the replay module. It also invents several system functions
that will be used by replay.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 translate-all.c         |    8 ++++++++
 5 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/cpus.c b/cpus.c
index 0c33458..e53d605 100644
--- a/cpus.c
+++ b/cpus.c
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1112,7 +1112,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index ab956f2..1d17f75 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -212,6 +212,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 2098f1c..5afb44c 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 3f162a9..86ae556 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
diff --git a/translate-all.c b/translate-all.c
index ba5c840..7177b71 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -806,6 +806,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)

[Qemu-devel] [RFC PATCH v5 05/31] replay: internal functions for replay log

[Pavel Dovgalyuk]

This patch adds functions to perform read and write operations
with replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 3 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 7ea860f..1148f45 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
diff --git a/replay/replay-internal.c b/replay/replay-internal.c
new file mode 100755
index 0000000..429b13c
--- /dev/null
+++ b/replay/replay-internal.c
@@ -0,0 +1,141 @@
+/*
+ * replay-internal.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay-internal.h"
+
+volatile unsigned int replay_data_kind = -1;
+volatile unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+FILE *replay_file;
+
+void replay_put_byte(uint8_t byte)
+{
+    if (replay_file) {
+        fwrite(&byte, sizeof(byte), 1, replay_file);
+    }
+}
+
+void replay_put_event(uint8_t event)
+{
+    replay_put_byte(event);
+}
+
+
+void replay_put_word(uint16_t word)
+{
+    if (replay_file) {
+        fwrite(&word, sizeof(word), 1, replay_file);
+    }
+}
+
+void replay_put_dword(uint32_t dword)
+{
+    if (replay_file) {
+        fwrite(&dword, sizeof(dword), 1, replay_file);
+    }
+}
+
+void replay_put_qword(int64_t qword)
+{
+    if (replay_file) {
+        fwrite(&qword, sizeof(qword), 1, replay_file);
+    }
+}
+
+void replay_put_array(const uint8_t *buf, size_t size)
+{
+    if (replay_file) {
+        fwrite(&size, sizeof(size), 1, replay_file);
+        fwrite(buf, 1, size, replay_file);
+    }
+}
+
+uint8_t replay_get_byte(void)
+{
+    uint8_t byte;
+    if (replay_file) {
+        fread(&byte, sizeof(byte), 1, replay_file);
+    }
+    return byte;
+}
+
+uint16_t replay_get_word(void)
+{
+    uint16_t word;
+    if (replay_file) {
+        fread(&word, sizeof(word), 1, replay_file);
+    }
+
+    return word;
+}
+
+uint32_t replay_get_dword(void)
+{
+    uint32_t dword;
+    if (replay_file) {
+        fread(&dword, sizeof(dword), 1, replay_file);
+    }
+
+    return dword;
+}
+
+int64_t replay_get_qword(void)
+{
+    int64_t qword;
+    if (replay_file) {
+        fread(&qword, sizeof(qword), 1, replay_file);
+    }
+
+    return qword;
+}
+
+void replay_get_array(uint8_t *buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        fread(buf, 1, *size, replay_file);
+    }
+}
+
+void replay_get_array_alloc(uint8_t **buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        *buf = g_malloc(*size);
+        fread(*buf, 1, *size, replay_file);
+    }
+}
+
+void replay_check_error(void)
+{
+    if (replay_file) {
+        if (feof(replay_file)) {
+            fprintf(stderr, "replay file is over\n");
+            exit(1);
+        } else if (ferror(replay_file)) {
+            fprintf(stderr, "replay file is over or something goes wrong\n");
+            exit(1);
+        }
+    }
+}
+
+void replay_fetch_data_kind(void)
+{
+    if (replay_file) {
+        if (!replay_has_unread_data) {
+            replay_data_kind = replay_get_byte();
+            replay_check_error();
+            replay_has_unread_data = 1;
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
new file mode 100755
index 0000000..927f7c7
--- /dev/null
+++ b/replay/replay-internal.h
@@ -0,0 +1,50 @@
+#ifndef REPLAY_INTERNAL_H
+#define REPLAY_INTERNAL_H
+
+/*
+ * replay-internal.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <stdio.h>
+
+extern volatile unsigned int replay_data_kind;
+extern volatile unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+extern FILE *replay_file;
+
+void replay_put_byte(uint8_t byte);
+void replay_put_event(uint8_t event);
+void replay_put_word(uint16_t word);
+void replay_put_dword(uint32_t dword);
+void replay_put_qword(int64_t qword);
+void replay_put_array(const uint8_t *buf, size_t size);
+
+uint8_t replay_get_byte(void);
+uint16_t replay_get_word(void);
+uint32_t replay_get_dword(void);
+int64_t replay_get_qword(void);
+void replay_get_array(uint8_t *buf, size_t *size);
+void replay_get_array_alloc(uint8_t **buf, size_t *size);
+
+/*! Checks error status of the file. */
+void replay_check_error(void);
+
+/*! Reads data type from the file and stores it in the
+    replay_data_kind variable. */
+void replay_fetch_data_kind(void);
+
+/*! Saves queued events (like instructions and sound). */
+void replay_save_instructions(void);
+/*! Checks that the next data is corresponding to the desired kind.
+    Terminates the program in case of error. */
+void validate_data_kind(int kind);
+
+#endif

[Qemu-devel] [RFC PATCH v5 06/31] cpu-exec: reset exception_index correctly

[Pavel Dovgalyuk]

Exception index is reset at every entry at every entry into cpu_exec()
function. This may cause missing the exceptions while replaying them.
This patch moves exception_index reset to the locations where they are
processed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c |    3 ++-
 cpus.c     |    3 +++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 8830255..4df9856 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -358,7 +358,6 @@ int cpu_exec(CPUArchState *env)
     }
 
     cc->cpu_exec_enter(cpu);
-    cpu->exception_index = -1;
 
     /* Calculate difference between guest clock and host clock.
      * This delay includes the delay of the last cycle, so
@@ -378,6 +377,7 @@ int cpu_exec(CPUArchState *env)
                     if (ret == EXCP_DEBUG) {
                         cpu_handle_debug_exception(env);
                     }
+                    cpu->exception_index = -1;
                     break;
                 } else {
 #if defined(CONFIG_USER_ONLY)
@@ -388,6 +388,7 @@ int cpu_exec(CPUArchState *env)
                     cc->do_interrupt(cpu);
 #endif
                     ret = cpu->exception_index;
+                    cpu->exception_index = -1;
                     break;
 #else
                     cc->do_interrupt(cpu);
diff --git a/cpus.c b/cpus.c
index e53d605..a7b6c53 100644
--- a/cpus.c
+++ b/cpus.c
@@ -934,6 +934,7 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
     qemu_mutex_lock(&qemu_global_mutex);
     qemu_thread_get_self(cpu->thread);
     cpu->thread_id = qemu_get_thread_id();
+    cpu->exception_index = -1;
     current_cpu = cpu;
 
     r = kvm_init_vcpu(cpu);
@@ -974,6 +975,7 @@ static void *qemu_dummy_cpu_thread_fn(void *arg)
     qemu_mutex_lock_iothread();
     qemu_thread_get_self(cpu->thread);
     cpu->thread_id = qemu_get_thread_id();
+    cpu->exception_index = -1;
 
     sigemptyset(&waitset);
     sigaddset(&waitset, SIG_IPI);
@@ -1016,6 +1018,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
     CPU_FOREACH(cpu) {
         cpu->thread_id = qemu_get_thread_id();
         cpu->created = true;
+        cpu->exception_index = -1;
     }
     qemu_cond_signal(&qemu_cpu_cond);
 

[Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Pavel Dovgalyuk]

Replay uses number of executed instructions to determine corrent events
injection moments. This patch introduces new function for querying the
instructions counter.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c               |   26 +++++++++++++++++++++++---
 include/qemu/timer.h |    1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/cpus.c b/cpus.c
index a7b6c53..492e19a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -136,8 +136,7 @@ typedef struct TimersState {
 
 static TimersState timers_state;
 
-/* Return the virtual CPU time, based on the instruction counter.  */
-static int64_t cpu_get_icount_locked(void)
+static int64_t cpu_get_instructions_counter_locked(void)
 {
     int64_t icount;
     CPUState *cpu = current_cpu;
@@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
     icount = timers_state.qemu_icount;
     if (cpu) {
         if (!cpu_can_do_io(cpu)) {
-            fprintf(stderr, "Bad clock read\n");
+            fprintf(stderr, "Bad icount read\n");
+            exit(1);
         }
         icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
     }
+    return icount;
+}
+
+int64_t cpu_get_instructions_counter(void)
+{
+    /* This function calls are synchnonized to timer changes,
+       calling cpu_get_instructions_counter_locked without lock is safe */
+    int64_t icount = timers_state.qemu_icount;
+    CPUState *cpu = current_cpu;
+
+    if (cpu) {
+        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
+    }
+    return icount;
+}
+
+/* Return the virtual CPU time, based on the instruction counter.  */
+static int64_t cpu_get_icount_locked(void)
+{
+    int64_t icount = cpu_get_instructions_counter_locked();
     return timers_state.qemu_icount_bias + cpu_icount_to_ns(icount);
 }
 
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 5f5210d..38a02c5 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -743,6 +743,7 @@ static inline int64_t get_clock(void)
 #endif
 
 /* icount */
+int64_t cpu_get_instructions_counter(void);
 int64_t cpu_get_icount(void);
 int64_t cpu_get_clock(void);
 int64_t cpu_get_clock_offset(void);

[Qemu-devel] [RFC PATCH v5 08/31] icount: improve enable/disable ticks

[Pavel Dovgalyuk]

This patch eliminates call of the cpu_get_real_ticks while enabling
or disabling the virtual timer in icount mode. These calls are used
for cpu_ticks_offset which is not needed in this mode.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c |   12 ++++++++----
 1 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cpus.c b/cpus.c
index 492e19a..43ae7fc 100644
--- a/cpus.c
+++ b/cpus.c
@@ -267,8 +267,10 @@ void cpu_enable_ticks(void)
     /* Here, the really thing protected by seqlock is cpu_clock_offset. */
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (!timers_state.cpu_ticks_enabled) {
-        timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
-        timers_state.cpu_clock_offset -= get_clock();
+        if (!use_icount) {
+            timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
+            timers_state.cpu_clock_offset -= get_clock();
+        }
         timers_state.cpu_ticks_enabled = 1;
     }
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
@@ -283,8 +285,10 @@ void cpu_disable_ticks(void)
     /* Here, the really thing protected by seqlock is cpu_clock_offset. */
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (timers_state.cpu_ticks_enabled) {
-        timers_state.cpu_ticks_offset += cpu_get_real_ticks();
-        timers_state.cpu_clock_offset = cpu_get_clock_locked();
+        if (!use_icount) {
+            timers_state.cpu_ticks_offset += cpu_get_real_ticks();
+            timers_state.cpu_clock_offset = cpu_get_clock_locked();
+        }
         timers_state.cpu_ticks_enabled = 0;
     }
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);

[Qemu-devel] [RFC PATCH v5 09/31] replay: introduce icount event

[Pavel Dovgalyuk]

This patch adds icount event to the replay subsystem. This event corresponds
to execution of several instructions and used to synchronize input events
in the replay phase.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay-internal.c |   14 ++++++++++++++
 replay/replay-internal.h |   18 ++++++++++++++++++
 replay/replay.c          |   45 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 84 insertions(+), 0 deletions(-)

diff --git a/replay/replay-internal.c b/replay/replay-internal.c
index 429b13c..83a53bd 100755
--- a/replay/replay-internal.c
+++ b/replay/replay-internal.c
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 volatile unsigned int replay_data_kind = -1;
@@ -139,3 +140,16 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 927f7c7..582b44c 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,17 @@
 
 #include <stdio.h>
 
+/* for instruction event */
+#define EVENT_INSTRUCTION           32
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
 
@@ -47,4 +58,11 @@ void replay_save_instructions(void);
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
diff --git a/replay/replay.c b/replay/replay.c
index ac976b2..c305e0c 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -9,7 +9,10 @@
  *
  */
 
+#include "qemu-common.h"
 #include "replay.h"
+#include "replay-internal.h"
+#include "qemu/timer.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 /*! Stores current submode for PLAY mode */
@@ -18,8 +21,50 @@ ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
 /* Suffix for the disk images filenames */
 char *replay_image_suffix;
 
+ReplayState replay_state;
 
 ReplaySubmode replay_get_play_submode(void)
 {
     return play_submode;
 }
+
+bool skip_async_events(int stop_event)
+{
+    /* nothing to skip - not all instructions used */
+    if (replay_state.instructions_count != 0
+        && replay_has_unread_data) {
+        return stop_event == EVENT_INSTRUCTION;
+    }
+
+    bool res = false;
+    while (true) {
+        replay_fetch_data_kind();
+        if (stop_event == replay_data_kind) {
+            res = true;
+        }
+        switch (replay_data_kind) {
+        case EVENT_INSTRUCTION:
+            replay_state.instructions_count = replay_get_dword();
+            return res;
+        default:
+            /* clock, time_t, checkpoint and other events */
+            return res;
+        }
+    }
+
+    return res;
+}
+
+void skip_async_events_until(unsigned int kind)
+{
+    if (!skip_async_events(kind)) {
+        fprintf(stderr, "%"PRId64": Read data kind %d instead of expected %d\n",
+            replay_get_current_step(), replay_data_kind, kind);
+        exit(1);
+    }
+}
+
+uint64_t replay_get_current_step(void)
+{
+    return cpu_get_instructions_counter();
+}
diff --git a/replay/replay.h b/replay/replay.h
index 51a18fe..e40daf5 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -12,6 +12,8 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
@@ -20,4 +22,9 @@ extern char *replay_image_suffix;
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif

[Qemu-devel] [RFC PATCH v5 10/31] i386: do not cross the pages boundaries in replay mode

[Pavel Dovgalyuk]

This patch denies crossing the boundary of the pages in the replay mode,
because it can cause an exception. Do it only when boundary is
crossed by the first instruction in the block.
If current instruction already crossed the bound - it's ok,
because an exception hasn't stopped this code.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 target-i386/cpu.h       |    3 +++
 target-i386/translate.c |   14 ++++++++++++++
 2 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index 2968749..204aaf1 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -28,6 +28,9 @@
 #define TARGET_LONG_BITS 32
 #endif
 
+/* Maximum instruction code size */
+#define TARGET_MAX_INSN_SIZE 16
+
 /* target supports implicit self modifying code */
 #define TARGET_HAS_SMC
 /* support for self modifying code even if the modified instruction is
diff --git a/target-i386/translate.c b/target-i386/translate.c
index 4d5dfb3..a264908 100644
--- a/target-i386/translate.c
+++ b/target-i386/translate.c
@@ -8035,6 +8035,20 @@ static inline void gen_intermediate_code_internal(X86CPU *cpu,
             gen_eob(dc);
             break;
         }
+        /* Do not cross the boundary of the pages in icount mode,
+           it can cause an exception. Do it only when boundary is
+           crossed by the first instruction in the block.
+           If current instruction already crossed the bound - it's ok,
+           because an exception hasn't stopped this code.
+         */
+        if (use_icount
+            && ((pc_ptr & TARGET_PAGE_MASK)
+                != ((pc_ptr + TARGET_MAX_INSN_SIZE - 1) & TARGET_PAGE_MASK)
+                || (pc_ptr & ~TARGET_PAGE_MASK) == 0)) {
+            gen_jmp_im(pc_ptr - dc->cs_base);
+            gen_eob(dc);
+            break;
+        }
         /* if too long translation, stop generation too */
         if (tcg_ctx.gen_opc_ptr >= gen_opc_end ||
             (pc_ptr - pc_start) >= (TARGET_PAGE_SIZE - 32) ||

[Qemu-devel] [RFC PATCH v5 11/31] From 7abf2f72777958d395cfd01d97fe707cc06152b5 Mon Sep 17 00:00:00 2001

[Pavel Dovgalyuk]

From: Paolo Bonzini <pbonzini@redhat.com>

Subject: [PATCH] target-ppc: pass DisasContext to SPR generator functions

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 target-ppc/cpu.h            |   13 +-
 target-ppc/translate.c      |   10 +-
 target-ppc/translate_init.c |  247 +++++++++++++++++++++----------------------
 3 files changed, 133 insertions(+), 137 deletions(-)

diff --git a/target-ppc/cpu.h b/target-ppc/cpu.h
index 8724561..21deddf 100644
--- a/target-ppc/cpu.h
+++ b/target-ppc/cpu.h
@@ -320,6 +320,7 @@ typedef struct opc_handler_t opc_handler_t;
 /*****************************************************************************/
 /* Types used to describe some PowerPC registers */
 typedef struct CPUPPCState CPUPPCState;
+typedef struct DisasContext DisasContext;
 typedef struct ppc_tb_t ppc_tb_t;
 typedef struct ppc_spr_t ppc_spr_t;
 typedef struct ppc_dcr_t ppc_dcr_t;
@@ -328,13 +329,13 @@ typedef union ppc_tlb_t ppc_tlb_t;
 
 /* SPR access micro-ops generations callbacks */
 struct ppc_spr_t {
-    void (*uea_read)(void *opaque, int gpr_num, int spr_num);
-    void (*uea_write)(void *opaque, int spr_num, int gpr_num);
+    void (*uea_read)(DisasContext *ctx, int gpr_num, int spr_num);
+    void (*uea_write)(DisasContext *ctx, int spr_num, int gpr_num);
 #if !defined(CONFIG_USER_ONLY)
-    void (*oea_read)(void *opaque, int gpr_num, int spr_num);
-    void (*oea_write)(void *opaque, int spr_num, int gpr_num);
-    void (*hea_read)(void *opaque, int gpr_num, int spr_num);
-    void (*hea_write)(void *opaque, int spr_num, int gpr_num);
+    void (*oea_read)(DisasContext *ctx, int gpr_num, int spr_num);
+    void (*oea_write)(DisasContext *ctx, int spr_num, int gpr_num);
+    void (*hea_read)(DisasContext *ctx, int gpr_num, int spr_num);
+    void (*hea_write)(DisasContext *ctx, int spr_num, int gpr_num);
 #endif
     const char *name;
     target_ulong default_value;
diff --git a/target-ppc/translate.c b/target-ppc/translate.c
index d03daea..439f7f0 100644
--- a/target-ppc/translate.c
+++ b/target-ppc/translate.c
@@ -183,7 +183,7 @@ void ppc_translate_init(void)
 }
 
 /* internal defines */
-typedef struct DisasContext {
+struct DisasContext {
     struct TranslationBlock *tb;
     target_ulong nip;
     uint32_t opcode;
@@ -206,7 +206,7 @@ typedef struct DisasContext {
     int singlestep_enabled;
     uint64_t insns_flags;
     uint64_t insns_flags2;
-} DisasContext;
+};
 
 /* Return true iff byteswap is needed in a scalar memop */
 static inline bool need_byteswap(const DisasContext *ctx)
@@ -4221,7 +4221,7 @@ static void gen_mfmsr(DisasContext *ctx)
 #endif
 }
 
-static void spr_noaccess(void *opaque, int gprn, int sprn)
+static void spr_noaccess(DisasContext *ctx, int gprn, int sprn)
 {
 #if 0
     sprn = ((sprn >> 5) & 0x1F) | ((sprn & 0x1F) << 5);
@@ -4233,7 +4233,7 @@ static void spr_noaccess(void *opaque, int gprn, int sprn)
 /* mfspr */
 static inline void gen_op_mfspr(DisasContext *ctx)
 {
-    void (*read_cb)(void *opaque, int gprn, int sprn);
+    void (*read_cb)(DisasContext *ctx, int gprn, int sprn);
     uint32_t sprn = SPR(ctx->opcode);
 
 #if !defined(CONFIG_USER_ONLY)
@@ -4384,7 +4384,7 @@ static void gen_mtmsr(DisasContext *ctx)
 /* mtspr */
 static void gen_mtspr(DisasContext *ctx)
 {
-    void (*write_cb)(void *opaque, int sprn, int gprn);
+    void (*write_cb)(DisasContext *ctx, int sprn, int gprn);
     uint32_t sprn = SPR(ctx->opcode);
 
 #if !defined(CONFIG_USER_ONLY)
diff --git a/target-ppc/translate_init.c b/target-ppc/translate_init.c
index 33fb4cc..c7ca95e 100644
--- a/target-ppc/translate_init.c
+++ b/target-ppc/translate_init.c
@@ -65,7 +65,7 @@ static void spr_load_dump_spr(int sprn)
 #endif
 }
 
-static void spr_read_generic (void *opaque, int gprn, int sprn)
+static void spr_read_generic (DisasContext *ctx, int gprn, int sprn)
 {
     gen_load_spr(cpu_gpr[gprn], sprn);
     spr_load_dump_spr(sprn);
@@ -80,14 +80,14 @@ static void spr_store_dump_spr(int sprn)
 #endif
 }
 
-static void spr_write_generic (void *opaque, int sprn, int gprn)
+static void spr_write_generic (DisasContext *ctx, int sprn, int gprn)
 {
     gen_store_spr(sprn, cpu_gpr[gprn]);
     spr_store_dump_spr(sprn);
 }
 
 #if !defined(CONFIG_USER_ONLY)
-static void spr_write_generic32(void *opaque, int sprn, int gprn)
+static void spr_write_generic32(DisasContext *ctx, int sprn, int gprn)
 {
 #ifdef TARGET_PPC64
     TCGv t0 = tcg_temp_new();
@@ -96,11 +96,11 @@ static void spr_write_generic32(void *opaque, int sprn, int gprn)
     tcg_temp_free(t0);
     spr_store_dump_spr(sprn);
 #else
-    spr_write_generic(opaque, sprn, gprn);
+    spr_write_generic(ctx, sprn, gprn);
 #endif
 }
 
-static void spr_write_clear (void *opaque, int sprn, int gprn)
+static void spr_write_clear (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
     TCGv t1 = tcg_temp_new();
@@ -112,7 +112,7 @@ static void spr_write_clear (void *opaque, int sprn, int gprn)
     tcg_temp_free(t1);
 }
 
-static void spr_access_nop(void *opaque, int sprn, int gprn)
+static void spr_access_nop(DisasContext *ctx, int sprn, int gprn)
 {
 }
 
@@ -120,47 +120,47 @@ static void spr_access_nop(void *opaque, int sprn, int gprn)
 
 /* SPR common to all PowerPC */
 /* XER */
-static void spr_read_xer (void *opaque, int gprn, int sprn)
+static void spr_read_xer (DisasContext *ctx, int gprn, int sprn)
 {
     gen_read_xer(cpu_gpr[gprn]);
 }
 
-static void spr_write_xer (void *opaque, int sprn, int gprn)
+static void spr_write_xer (DisasContext *ctx, int sprn, int gprn)
 {
     gen_write_xer(cpu_gpr[gprn]);
 }
 
 /* LR */
-static void spr_read_lr (void *opaque, int gprn, int sprn)
+static void spr_read_lr (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_mov_tl(cpu_gpr[gprn], cpu_lr);
 }
 
-static void spr_write_lr (void *opaque, int sprn, int gprn)
+static void spr_write_lr (DisasContext *ctx, int sprn, int gprn)
 {
     tcg_gen_mov_tl(cpu_lr, cpu_gpr[gprn]);
 }
 
 /* CFAR */
 #if defined(TARGET_PPC64) && !defined(CONFIG_USER_ONLY)
-static void spr_read_cfar (void *opaque, int gprn, int sprn)
+static void spr_read_cfar (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_mov_tl(cpu_gpr[gprn], cpu_cfar);
 }
 
-static void spr_write_cfar (void *opaque, int sprn, int gprn)
+static void spr_write_cfar (DisasContext *ctx, int sprn, int gprn)
 {
     tcg_gen_mov_tl(cpu_cfar, cpu_gpr[gprn]);
 }
 #endif /* defined(TARGET_PPC64) && !defined(CONFIG_USER_ONLY) */
 
 /* CTR */
-static void spr_read_ctr (void *opaque, int gprn, int sprn)
+static void spr_read_ctr (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_mov_tl(cpu_gpr[gprn], cpu_ctr);
 }
 
-static void spr_write_ctr (void *opaque, int sprn, int gprn)
+static void spr_write_ctr (DisasContext *ctx, int sprn, int gprn)
 {
     tcg_gen_mov_tl(cpu_ctr, cpu_gpr[gprn]);
 }
@@ -171,13 +171,13 @@ static void spr_write_ctr (void *opaque, int sprn, int gprn)
 /* UPMCx */
 /* USIA */
 /* UDECR */
-static void spr_read_ureg (void *opaque, int gprn, int sprn)
+static void spr_read_ureg (DisasContext *ctx, int gprn, int sprn)
 {
     gen_load_spr(cpu_gpr[gprn], sprn + 0x10);
 }
 
 #if defined(TARGET_PPC64) && !defined(CONFIG_USER_ONLY)
-static void spr_write_ureg(void *opaque, int sprn, int gprn)
+static void spr_write_ureg(DisasContext *ctx, int sprn, int gprn)
 {
     gen_store_spr(sprn + 0x10, cpu_gpr[gprn]);
 }
@@ -186,7 +186,7 @@ static void spr_write_ureg(void *opaque, int sprn, int gprn)
 /* SPR common to all non-embedded PowerPC */
 /* DECR */
 #if !defined(CONFIG_USER_ONLY)
-static void spr_read_decr (void *opaque, int gprn, int sprn)
+static void spr_read_decr (DisasContext *ctx, int gprn, int sprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -194,11 +194,11 @@ static void spr_read_decr (void *opaque, int gprn, int sprn)
     gen_helper_load_decr(cpu_gpr[gprn], cpu_env);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 
-static void spr_write_decr (void *opaque, int sprn, int gprn)
+static void spr_write_decr (DisasContext *ctx, int sprn, int gprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -206,14 +206,14 @@ static void spr_write_decr (void *opaque, int sprn, int gprn)
     gen_helper_store_decr(cpu_env, cpu_gpr[gprn]);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 #endif
 
 /* SPR common to all non-embedded PowerPC, except 601 */
 /* Time base */
-static void spr_read_tbl (void *opaque, int gprn, int sprn)
+static void spr_read_tbl (DisasContext *ctx, int gprn, int sprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -221,11 +221,11 @@ static void spr_read_tbl (void *opaque, int gprn, int sprn)
     gen_helper_load_tbl(cpu_gpr[gprn], cpu_env);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 
-static void spr_read_tbu (void *opaque, int gprn, int sprn)
+static void spr_read_tbu (DisasContext *ctx, int gprn, int sprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -233,24 +233,24 @@ static void spr_read_tbu (void *opaque, int gprn, int sprn)
     gen_helper_load_tbu(cpu_gpr[gprn], cpu_env);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 
 __attribute__ (( unused ))
-static void spr_read_atbl (void *opaque, int gprn, int sprn)
+static void spr_read_atbl (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_atbl(cpu_gpr[gprn], cpu_env);
 }
 
 __attribute__ (( unused ))
-static void spr_read_atbu (void *opaque, int gprn, int sprn)
+static void spr_read_atbu (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_atbu(cpu_gpr[gprn], cpu_env);
 }
 
 #if !defined(CONFIG_USER_ONLY)
-static void spr_write_tbl (void *opaque, int sprn, int gprn)
+static void spr_write_tbl (DisasContext *ctx, int sprn, int gprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -258,11 +258,11 @@ static void spr_write_tbl (void *opaque, int sprn, int gprn)
     gen_helper_store_tbl(cpu_env, cpu_gpr[gprn]);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 
-static void spr_write_tbu (void *opaque, int sprn, int gprn)
+static void spr_write_tbu (DisasContext *ctx, int sprn, int gprn)
 {
     if (use_icount) {
         gen_io_start();
@@ -270,25 +270,25 @@ static void spr_write_tbu (void *opaque, int sprn, int gprn)
     gen_helper_store_tbu(cpu_env, cpu_gpr[gprn]);
     if (use_icount) {
         gen_io_end();
-        gen_stop_exception(opaque);
+        gen_stop_exception(ctx);
     }
 }
 
 __attribute__ (( unused ))
-static void spr_write_atbl (void *opaque, int sprn, int gprn)
+static void spr_write_atbl (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_atbl(cpu_env, cpu_gpr[gprn]);
 }
 
 __attribute__ (( unused ))
-static void spr_write_atbu (void *opaque, int sprn, int gprn)
+static void spr_write_atbu (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_atbu(cpu_env, cpu_gpr[gprn]);
 }
 
 #if defined(TARGET_PPC64)
 __attribute__ (( unused ))
-static void spr_read_purr (void *opaque, int gprn, int sprn)
+static void spr_read_purr (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_purr(cpu_gpr[gprn], cpu_env);
 }
@@ -298,38 +298,38 @@ static void spr_read_purr (void *opaque, int gprn, int sprn)
 #if !defined(CONFIG_USER_ONLY)
 /* IBAT0U...IBAT0U */
 /* IBAT0L...IBAT7L */
-static void spr_read_ibat (void *opaque, int gprn, int sprn)
+static void spr_read_ibat (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, IBAT[sprn & 1][(sprn - SPR_IBAT0U) / 2]));
 }
 
-static void spr_read_ibat_h (void *opaque, int gprn, int sprn)
+static void spr_read_ibat_h (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, IBAT[sprn & 1][(sprn - SPR_IBAT4U) / 2]));
 }
 
-static void spr_write_ibatu (void *opaque, int sprn, int gprn)
+static void spr_write_ibatu (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_IBAT0U) / 2);
     gen_helper_store_ibatu(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_ibatu_h (void *opaque, int sprn, int gprn)
+static void spr_write_ibatu_h (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(((sprn - SPR_IBAT4U) / 2) + 4);
     gen_helper_store_ibatu(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_ibatl (void *opaque, int sprn, int gprn)
+static void spr_write_ibatl (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_IBAT0L) / 2);
     gen_helper_store_ibatl(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_ibatl_h (void *opaque, int sprn, int gprn)
+static void spr_write_ibatl_h (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(((sprn - SPR_IBAT4L) / 2) + 4);
     gen_helper_store_ibatl(cpu_env, t0, cpu_gpr[gprn]);
@@ -338,38 +338,38 @@ static void spr_write_ibatl_h (void *opaque, int sprn, int gprn)
 
 /* DBAT0U...DBAT7U */
 /* DBAT0L...DBAT7L */
-static void spr_read_dbat (void *opaque, int gprn, int sprn)
+static void spr_read_dbat (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, DBAT[sprn & 1][(sprn - SPR_DBAT0U) / 2]));
 }
 
-static void spr_read_dbat_h (void *opaque, int gprn, int sprn)
+static void spr_read_dbat_h (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, DBAT[sprn & 1][((sprn - SPR_DBAT4U) / 2) + 4]));
 }
 
-static void spr_write_dbatu (void *opaque, int sprn, int gprn)
+static void spr_write_dbatu (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_DBAT0U) / 2);
     gen_helper_store_dbatu(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_dbatu_h (void *opaque, int sprn, int gprn)
+static void spr_write_dbatu_h (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(((sprn - SPR_DBAT4U) / 2) + 4);
     gen_helper_store_dbatu(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_dbatl (void *opaque, int sprn, int gprn)
+static void spr_write_dbatl (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_DBAT0L) / 2);
     gen_helper_store_dbatl(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_dbatl_h (void *opaque, int sprn, int gprn)
+static void spr_write_dbatl_h (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(((sprn - SPR_DBAT4L) / 2) + 4);
     gen_helper_store_dbatl(cpu_env, t0, cpu_gpr[gprn]);
@@ -377,19 +377,19 @@ static void spr_write_dbatl_h (void *opaque, int sprn, int gprn)
 }
 
 /* SDR1 */
-static void spr_write_sdr1 (void *opaque, int sprn, int gprn)
+static void spr_write_sdr1 (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_sdr1(cpu_env, cpu_gpr[gprn]);
 }
 
 /* 64 bits PowerPC specific SPRs */
 #if defined(TARGET_PPC64)
-static void spr_read_hior (void *opaque, int gprn, int sprn)
+static void spr_read_hior (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, excp_prefix));
 }
 
-static void spr_write_hior (void *opaque, int sprn, int gprn)
+static void spr_write_hior (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
     tcg_gen_andi_tl(t0, cpu_gpr[gprn], 0x3FFFFF00000ULL);
@@ -401,31 +401,29 @@ static void spr_write_hior (void *opaque, int sprn, int gprn)
 
 /* PowerPC 601 specific registers */
 /* RTC */
-static void spr_read_601_rtcl (void *opaque, int gprn, int sprn)
+static void spr_read_601_rtcl (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_601_rtcl(cpu_gpr[gprn], cpu_env);
 }
 
-static void spr_read_601_rtcu (void *opaque, int gprn, int sprn)
+static void spr_read_601_rtcu (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_601_rtcu(cpu_gpr[gprn], cpu_env);
 }
 
 #if !defined(CONFIG_USER_ONLY)
-static void spr_write_601_rtcu (void *opaque, int sprn, int gprn)
+static void spr_write_601_rtcu (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_601_rtcu(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_601_rtcl (void *opaque, int sprn, int gprn)
+static void spr_write_601_rtcl (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_601_rtcl(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_hid0_601 (void *opaque, int sprn, int gprn)
+static void spr_write_hid0_601 (DisasContext *ctx, int sprn, int gprn)
 {
-    DisasContext *ctx = opaque;
-
     gen_helper_store_hid0_601(cpu_env, cpu_gpr[gprn]);
     /* Must stop the translation as endianness may have changed */
     gen_stop_exception(ctx);
@@ -434,19 +432,19 @@ static void spr_write_hid0_601 (void *opaque, int sprn, int gprn)
 
 /* Unified bats */
 #if !defined(CONFIG_USER_ONLY)
-static void spr_read_601_ubat (void *opaque, int gprn, int sprn)
+static void spr_read_601_ubat (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, IBAT[sprn & 1][(sprn - SPR_IBAT0U) / 2]));
 }
 
-static void spr_write_601_ubatu (void *opaque, int sprn, int gprn)
+static void spr_write_601_ubatu (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_IBAT0U) / 2);
     gen_helper_store_601_batl(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_601_ubatl (void *opaque, int sprn, int gprn)
+static void spr_write_601_ubatl (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32((sprn - SPR_IBAT0U) / 2);
     gen_helper_store_601_batu(cpu_env, t0, cpu_gpr[gprn]);
@@ -456,36 +454,34 @@ static void spr_write_601_ubatl (void *opaque, int sprn, int gprn)
 
 /* PowerPC 40x specific registers */
 #if !defined(CONFIG_USER_ONLY)
-static void spr_read_40x_pit (void *opaque, int gprn, int sprn)
+static void spr_read_40x_pit (DisasContext *ctx, int gprn, int sprn)
 {
     gen_helper_load_40x_pit(cpu_gpr[gprn], cpu_env);
 }
 
-static void spr_write_40x_pit (void *opaque, int sprn, int gprn)
+static void spr_write_40x_pit (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_40x_pit(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_40x_dbcr0 (void *opaque, int sprn, int gprn)
+static void spr_write_40x_dbcr0 (DisasContext *ctx, int sprn, int gprn)
 {
-    DisasContext *ctx = opaque;
-
     gen_helper_store_40x_dbcr0(cpu_env, cpu_gpr[gprn]);
     /* We must stop translation as we may have rebooted */
     gen_stop_exception(ctx);
 }
 
-static void spr_write_40x_sler (void *opaque, int sprn, int gprn)
+static void spr_write_40x_sler (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_40x_sler(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_booke_tcr (void *opaque, int sprn, int gprn)
+static void spr_write_booke_tcr (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_booke_tcr(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_booke_tsr (void *opaque, int sprn, int gprn)
+static void spr_write_booke_tsr (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_store_booke_tsr(cpu_env, cpu_gpr[gprn]);
 }
@@ -494,19 +490,19 @@ static void spr_write_booke_tsr (void *opaque, int sprn, int gprn)
 /* PowerPC 403 specific registers */
 /* PBL1 / PBU1 / PBL2 / PBU2 */
 #if !defined(CONFIG_USER_ONLY)
-static void spr_read_403_pbr (void *opaque, int gprn, int sprn)
+static void spr_read_403_pbr (DisasContext *ctx, int gprn, int sprn)
 {
     tcg_gen_ld_tl(cpu_gpr[gprn], cpu_env, offsetof(CPUPPCState, pb[sprn - SPR_403_PBL1]));
 }
 
-static void spr_write_403_pbr (void *opaque, int sprn, int gprn)
+static void spr_write_403_pbr (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(sprn - SPR_403_PBL1);
     gen_helper_store_403_pbr(cpu_env, t0, cpu_gpr[gprn]);
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_pir (void *opaque, int sprn, int gprn)
+static void spr_write_pir (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
     tcg_gen_andi_tl(t0, cpu_gpr[gprn], 0xF);
@@ -516,7 +512,7 @@ static void spr_write_pir (void *opaque, int sprn, int gprn)
 #endif
 
 /* SPE specific registers */
-static void spr_read_spefscr (void *opaque, int gprn, int sprn)
+static void spr_read_spefscr (DisasContext *ctx, int gprn, int sprn)
 {
     TCGv_i32 t0 = tcg_temp_new_i32();
     tcg_gen_ld_i32(t0, cpu_env, offsetof(CPUPPCState, spe_fscr));
@@ -524,7 +520,7 @@ static void spr_read_spefscr (void *opaque, int gprn, int sprn)
     tcg_temp_free_i32(t0);
 }
 
-static void spr_write_spefscr (void *opaque, int sprn, int gprn)
+static void spr_write_spefscr (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_temp_new_i32();
     tcg_gen_trunc_tl_i32(t0, cpu_gpr[gprn]);
@@ -534,7 +530,7 @@ static void spr_write_spefscr (void *opaque, int sprn, int gprn)
 
 #if !defined(CONFIG_USER_ONLY)
 /* Callback used to write the exception vector base */
-static void spr_write_excp_prefix (void *opaque, int sprn, int gprn)
+static void spr_write_excp_prefix (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
     tcg_gen_ld_tl(t0, cpu_env, offsetof(CPUPPCState, ivpr_mask));
@@ -544,9 +540,8 @@ static void spr_write_excp_prefix (void *opaque, int sprn, int gprn)
     tcg_temp_free(t0);
 }
 
-static void spr_write_excp_vector (void *opaque, int sprn, int gprn)
+static void spr_write_excp_vector (DisasContext *ctx, int sprn, int gprn)
 {
-    DisasContext *ctx = opaque;
     int sprn_offs;
 
     if (sprn >= SPR_BOOKE_IVOR0 && sprn <= SPR_BOOKE_IVOR15) {
@@ -604,12 +599,12 @@ static inline void vscr_init (CPUPPCState *env, uint32_t val)
 
 static inline void _spr_register(CPUPPCState *env, int num,
                                  const char *name,
-                                 void (*uea_read)(void *opaque, int gprn, int sprn),
-                                 void (*uea_write)(void *opaque, int sprn, int gprn),
+                                 void (*uea_read)(DisasContext *ctx, int gprn, int sprn),
+                                 void (*uea_write)(DisasContext *ctx, int sprn, int gprn),
 #if !defined(CONFIG_USER_ONLY)
 
-                                 void (*oea_read)(void *opaque, int gprn, int sprn),
-                                 void (*oea_write)(void *opaque, int sprn, int gprn),
+                                 void (*oea_read)(DisasContext *ctx, int gprn, int sprn),
+                                 void (*oea_write)(DisasContext *ctx, int sprn, int gprn),
 #endif
 #if defined(CONFIG_KVM)
                                  uint64_t one_reg_id,
@@ -1040,19 +1035,19 @@ static void gen_spr_7xx (CPUPPCState *env)
 
 #ifdef TARGET_PPC64
 #ifndef CONFIG_USER_ONLY
-static void spr_read_uamr (void *opaque, int gprn, int sprn)
+static void spr_read_uamr (DisasContext *ctx, int gprn, int sprn)
 {
     gen_load_spr(cpu_gpr[gprn], SPR_AMR);
     spr_load_dump_spr(SPR_AMR);
 }
 
-static void spr_write_uamr (void *opaque, int sprn, int gprn)
+static void spr_write_uamr (DisasContext *ctx, int sprn, int gprn)
 {
     gen_store_spr(SPR_AMR, cpu_gpr[gprn]);
     spr_store_dump_spr(SPR_AMR);
 }
 
-static void spr_write_uamr_pr (void *opaque, int sprn, int gprn)
+static void spr_write_uamr_pr (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
 
@@ -1454,7 +1449,7 @@ static void gen_74xx_soft_tlb (CPUPPCState *env, int nb_tlbs, int nb_ways)
 }
 
 #if !defined(CONFIG_USER_ONLY)
-static void spr_write_e500_l1csr0 (void *opaque, int sprn, int gprn)
+static void spr_write_e500_l1csr0 (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
 
@@ -1463,7 +1458,7 @@ static void spr_write_e500_l1csr0 (void *opaque, int sprn, int gprn)
     tcg_temp_free(t0);
 }
 
-static void spr_write_e500_l1csr1(void *opaque, int sprn, int gprn)
+static void spr_write_e500_l1csr1(DisasContext *ctx, int sprn, int gprn)
 {
     TCGv t0 = tcg_temp_new();
 
@@ -1472,12 +1467,12 @@ static void spr_write_e500_l1csr1(void *opaque, int sprn, int gprn)
     tcg_temp_free(t0);
 }
 
-static void spr_write_booke206_mmucsr0 (void *opaque, int sprn, int gprn)
+static void spr_write_booke206_mmucsr0 (DisasContext *ctx, int sprn, int gprn)
 {
     gen_helper_booke206_tlbflush(cpu_env, cpu_gpr[gprn]);
 }
 
-static void spr_write_booke_pid (void *opaque, int sprn, int gprn)
+static void spr_write_booke_pid (DisasContext *ctx, int sprn, int gprn)
 {
     TCGv_i32 t0 = tcg_const_i32(sprn);
     gen_helper_booke_setpid(cpu_env, t0, cpu_gpr[gprn]);
@@ -1693,7 +1688,7 @@ static void gen_spr_BookE206(CPUPPCState *env, uint32_t mas_mask,
     /* TLB assist registers */
     /* XXX : not implemented */
     for (i = 0; i < 8; i++) {
-        void (*uea_write)(void *o, int sprn, int gprn) = &spr_write_generic32;
+        void (*uea_write)(DisasContext *ctx, int sprn, int gprn) = &spr_write_generic32;
         if (i == 2 && (mas_mask & (1 << i)) && (env->insns_flags & PPC_64B)) {
             uea_write = &spr_write_generic;
         }
@@ -4642,7 +4637,7 @@ POWERPC_FAMILY(e300)(ObjectClass *oc, void *data)
 }
 
 #if !defined(CONFIG_USER_ONLY)
-static void spr_write_mas73(void *opaque, int sprn, int gprn)
+static void spr_write_mas73(DisasContext *ctx, int sprn, int gprn)
 {
     TCGv val = tcg_temp_new();
     tcg_gen_ext32u_tl(val, cpu_gpr[gprn]);
@@ -4652,7 +4647,7 @@ static void spr_write_mas73(void *opaque, int sprn, int gprn)
     tcg_temp_free(val);
 }
 
-static void spr_read_mas73(void *opaque, int gprn, int sprn)
+static void spr_read_mas73(DisasContext *ctx, int gprn, int sprn)
 {
     TCGv mas7 = tcg_temp_new();
     TCGv mas3 = tcg_temp_new();
@@ -7283,14 +7278,14 @@ enum BOOK3S_CPU_TYPE {
     BOOK3S_CPU_POWER8
 };
 
-static void gen_fscr_facility_check(void *opaque, int facility_sprn, int bit,
-                                    int sprn, int cause)
+static void gen_fscr_facility_check(DisasContext *ctx, int facility_sprn,
+                                    int bit, int sprn, int cause)
 {
     TCGv_i32 t1 = tcg_const_i32(bit);
     TCGv_i32 t2 = tcg_const_i32(sprn);
     TCGv_i32 t3 = tcg_const_i32(cause);
 
-    gen_update_current_nip(opaque);
+    gen_update_current_nip(ctx);
     gen_helper_fscr_facility_check(cpu_env, t1, t2, t3);
 
     tcg_temp_free_i32(t3);
@@ -7298,14 +7293,14 @@ static void gen_fscr_facility_check(void *opaque, int facility_sprn, int bit,
     tcg_temp_free_i32(t1);
 }
 
-static void gen_msr_facility_check(void *opaque, int facility_sprn, int bit,
-                                   int sprn, int cause)
+static void gen_msr_facility_check(DisasContext *ctx, int facility_sprn,
+                                   int bit, int sprn, int cause)
 {
     TCGv_i32 t1 = tcg_const_i32(bit);
     TCGv_i32 t2 = tcg_const_i32(sprn);
     TCGv_i32 t3 = tcg_const_i32(cause);
 
-    gen_update_current_nip(opaque);
+    gen_update_current_nip(ctx);
     gen_helper_msr_facility_check(cpu_env, t1, t2, t3);
 
     tcg_temp_free_i32(t3);
@@ -7313,7 +7308,7 @@ static void gen_msr_facility_check(void *opaque, int facility_sprn, int bit,
     tcg_temp_free_i32(t1);
 }
 
-static void spr_read_prev_upper32(void *opaque, int gprn, int sprn)
+static void spr_read_prev_upper32(DisasContext *ctx, int gprn, int sprn)
 {
     TCGv spr_up = tcg_temp_new();
     TCGv spr = tcg_temp_new();
@@ -7326,7 +7321,7 @@ static void spr_read_prev_upper32(void *opaque, int gprn, int sprn)
     tcg_temp_free(spr_up);
 }
 
-static void spr_write_prev_upper32(void *opaque, int sprn, int gprn)
+static void spr_write_prev_upper32(DisasContext *ctx, int sprn, int gprn)
 {
     TCGv spr = tcg_temp_new();
 
@@ -7665,16 +7660,16 @@ static void gen_spr_power6_common(CPUPPCState *env)
                  0x00000000);
 }
 
-static void spr_read_tar(void *opaque, int gprn, int sprn)
+static void spr_read_tar(DisasContext *ctx, int gprn, int sprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_TAR, sprn, FSCR_IC_TAR);
-    spr_read_generic(opaque, gprn, sprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_TAR, sprn, FSCR_IC_TAR);
+    spr_read_generic(ctx, gprn, sprn);
 }
 
-static void spr_write_tar(void *opaque, int sprn, int gprn)
+static void spr_write_tar(DisasContext *ctx, int sprn, int gprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_TAR, sprn, FSCR_IC_TAR);
-    spr_write_generic(opaque, sprn, gprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_TAR, sprn, FSCR_IC_TAR);
+    spr_write_generic(ctx, sprn, gprn);
 }
 
 static void gen_spr_power8_tce_address_control(CPUPPCState *env)
@@ -7685,28 +7680,28 @@ static void gen_spr_power8_tce_address_control(CPUPPCState *env)
                  0x00000000);
 }
 
-static void spr_read_tm(void *opaque, int gprn, int sprn)
+static void spr_read_tm(DisasContext *ctx, int gprn, int sprn)
 {
-    gen_msr_facility_check(opaque, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
-    spr_read_generic(opaque, gprn, sprn);
+    gen_msr_facility_check(ctx, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
+    spr_read_generic(ctx, gprn, sprn);
 }
 
-static void spr_write_tm(void *opaque, int sprn, int gprn)
+static void spr_write_tm(DisasContext *ctx, int sprn, int gprn)
 {
-    gen_msr_facility_check(opaque, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
-    spr_write_generic(opaque, sprn, gprn);
+    gen_msr_facility_check(ctx, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
+    spr_write_generic(ctx, sprn, gprn);
 }
 
-static void spr_read_tm_upper32(void *opaque, int gprn, int sprn)
+static void spr_read_tm_upper32(DisasContext *ctx, int gprn, int sprn)
 {
-    gen_msr_facility_check(opaque, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
-    spr_read_prev_upper32(opaque, gprn, sprn);
+    gen_msr_facility_check(ctx, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
+    spr_read_prev_upper32(ctx, gprn, sprn);
 }
 
-static void spr_write_tm_upper32(void *opaque, int sprn, int gprn)
+static void spr_write_tm_upper32(DisasContext *ctx, int sprn, int gprn)
 {
-    gen_msr_facility_check(opaque, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
-    spr_write_prev_upper32(opaque, sprn, gprn);
+    gen_msr_facility_check(ctx, SPR_FSCR, MSR_TM, sprn, FSCR_IC_TM);
+    spr_write_prev_upper32(ctx, sprn, gprn);
 }
 
 static void gen_spr_power8_tm(CPUPPCState *env)
@@ -7729,28 +7724,28 @@ static void gen_spr_power8_tm(CPUPPCState *env)
                  0x00000000);
 }
 
-static void spr_read_ebb(void *opaque, int gprn, int sprn)
+static void spr_read_ebb(DisasContext *ctx, int gprn, int sprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
-    spr_read_generic(opaque, gprn, sprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
+    spr_read_generic(ctx, gprn, sprn);
 }
 
-static void spr_write_ebb(void *opaque, int sprn, int gprn)
+static void spr_write_ebb(DisasContext *ctx, int sprn, int gprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
-    spr_write_generic(opaque, sprn, gprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
+    spr_write_generic(ctx, sprn, gprn);
 }
 
-static void spr_read_ebb_upper32(void *opaque, int gprn, int sprn)
+static void spr_read_ebb_upper32(DisasContext *ctx, int gprn, int sprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
-    spr_read_prev_upper32(opaque, gprn, sprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
+    spr_read_prev_upper32(ctx, gprn, sprn);
 }
 
-static void spr_write_ebb_upper32(void *opaque, int sprn, int gprn)
+static void spr_write_ebb_upper32(DisasContext *ctx, int sprn, int gprn)
 {
-    gen_fscr_facility_check(opaque, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
-    spr_write_prev_upper32(opaque, sprn, gprn);
+    gen_fscr_facility_check(ctx, SPR_FSCR, FSCR_EBB, sprn, FSCR_IC_EBB);
+    spr_write_prev_upper32(ctx, sprn, gprn);
 }
 
 static void gen_spr_power8_ebb(CPUPPCState *env)

[Qemu-devel] [RFC PATCH v5 12/31] From 185a3a47d08857a66332ae862b372a153ce92bb9 Mon Sep 17 00:00:00 2001

[Pavel Dovgalyuk]

From: Paolo Bonzini <pbonzini@redhat.com>

Subject: [PATCH] cpu-exec: add a new CF_USE_ICOUNT cflag

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/exec/exec-all.h |    5 +++--
 translate-all.c         |    3 +++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 1d17f75..3d19e72 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -142,9 +142,11 @@ struct TranslationBlock {
     uint64_t flags; /* flags defining in which context the code was generated */
     uint16_t size;      /* size of target code for this block (1 <=
                            size <= TARGET_PAGE_SIZE) */
-    uint16_t cflags;    /* compile flags */
+    uint16_t icount;
+    uint32_t cflags;    /* compile flags */
 #define CF_COUNT_MASK  0x7fff
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
+#define CF_USE_ICOUNT  0x10000
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
@@ -168,7 +170,6 @@ struct TranslationBlock {
        jmp_first */
     struct TranslationBlock *jmp_next[2];
     struct TranslationBlock *jmp_first;
-    uint32_t icount;
 };
 
 #include "exec/spinlock.h"
diff --git a/translate-all.c b/translate-all.c
index 7177b71..e9f5178 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -1047,6 +1047,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     int code_gen_size;
 
     phys_pc = get_page_addr_code(env, pc);
+    if (use_icount) {
+        cflags |= CF_USE_ICOUNT;
+    }
     tb = tb_alloc(pc);
     if (!tb) {
         /* flush must be done */

[Qemu-devel] [RFC PATCH v5 13/31] From a0cb9e80ba0de409b5ad556109a1c71ce4d8ce19 Mon Sep 17 00:00:00 2001

[Pavel Dovgalyuk]

From: Paolo Bonzini <pbonzini@redhat.com>

Subject: [PATCH] translate: check cflags instead of use_icount global

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 target-alpha/translate.c    |    8 ++++---
 target-arm/translate-a64.c  |    4 ++--
 target-arm/translate.c      |    4 ++--
 target-i386/translate.c     |   46 ++++++++++++++++++++++++++-----------------
 target-lm32/translate.c     |    8 ++++---
 target-mips/translate.c     |   24 +++++++++++++---------
 target-ppc/translate_init.c |   24 +++++++++++-----------
 translate-all.c             |    2 +-
 8 files changed, 67 insertions(+), 53 deletions(-)

diff --git a/target-alpha/translate.c b/target-alpha/translate.c
index 76658a0..5387b93 100644
--- a/target-alpha/translate.c
+++ b/target-alpha/translate.c
@@ -1285,7 +1285,7 @@ static int cpu_pr_data(int pr)
     return 0;
 }
 
-static ExitStatus gen_mfpr(TCGv va, int regno)
+static ExitStatus gen_mfpr(DisasContext *ctx, TCGv va, int regno)
 {
     int data = cpu_pr_data(regno);
 
@@ -1295,7 +1295,7 @@ static ExitStatus gen_mfpr(TCGv va, int regno)
 	if (regno == 249) {
 		helper = gen_helper_get_vmtime;
 	}
-        if (use_icount) {
+        if (ctx->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
             helper(va);
             gen_io_end();
@@ -2283,7 +2283,7 @@ static ExitStatus translate_one(DisasContext *ctx, uint32_t insn)
         case 0xC000:
             /* RPCC */
             va = dest_gpr(ctx, ra);
-            if (use_icount) {
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_start();
                 gen_helper_load_pcc(va, cpu_env);
                 gen_io_end();
@@ -2317,7 +2317,7 @@ static ExitStatus translate_one(DisasContext *ctx, uint32_t insn)
 #ifndef CONFIG_USER_ONLY
         REQUIRE_TB_FLAG(TB_FLAGS_PAL_MODE);
         va = dest_gpr(ctx, ra);
-        ret = gen_mfpr(va, insn & 0xffff);
+        ret = gen_mfpr(ctx, va, insn & 0xffff);
         break;
 #else
         goto invalid_opc;
diff --git a/target-arm/translate-a64.c b/target-arm/translate-a64.c
index 3a3c48a..ba6a85c 100644
--- a/target-arm/translate-a64.c
+++ b/target-arm/translate-a64.c
@@ -1373,7 +1373,7 @@ static void handle_sys(DisasContext *s, uint32_t insn, bool isread,
         break;
     }
 
-    if (use_icount && (ri->type & ARM_CP_IO)) {
+    if ((s->tb->cflags & CF_USE_ICOUNT) && (ri->type & ARM_CP_IO)) {
         gen_io_start();
     }
 
@@ -1404,7 +1404,7 @@ static void handle_sys(DisasContext *s, uint32_t insn, bool isread,
         }
     }
 
-    if (use_icount && (ri->type & ARM_CP_IO)) {
+    if ((s->tb->cflags & CF_USE_ICOUNT) && (ri->type & ARM_CP_IO)) {
         /* I/O operations must end the TB here (whether read or write) */
         gen_io_end();
         s->is_jmp = DISAS_UPDATE;
diff --git a/target-arm/translate.c b/target-arm/translate.c
index 9436650..d7343df 100644
--- a/target-arm/translate.c
+++ b/target-arm/translate.c
@@ -7153,7 +7153,7 @@ static int disas_coproc_insn(CPUARMState * env, DisasContext *s, uint32_t insn)
             break;
         }
 
-        if (use_icount && (ri->type & ARM_CP_IO)) {
+        if ((s->tb->cflags & CF_USE_ICOUNT) && (ri->type & ARM_CP_IO)) {
             gen_io_start();
         }
 
@@ -7244,7 +7244,7 @@ static int disas_coproc_insn(CPUARMState * env, DisasContext *s, uint32_t insn)
             }
         }
 
-        if (use_icount && (ri->type & ARM_CP_IO)) {
+        if ((s->tb->cflags & CF_USE_ICOUNT) && (ri->type & ARM_CP_IO)) {
             /* I/O operations must end the TB here (whether read or write) */
             gen_io_end();
             gen_lookup_tb(s);
diff --git a/target-i386/translate.c b/target-i386/translate.c
index a264908..b7631b7 100644
--- a/target-i386/translate.c
+++ b/target-i386/translate.c
@@ -1168,8 +1168,9 @@ static inline void gen_cmps(DisasContext *s, TCGMemOp ot)
 
 static inline void gen_ins(DisasContext *s, TCGMemOp ot)
 {
-    if (use_icount)
+    if (s->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
+    }
     gen_string_movl_A0_EDI(s);
     /* Note: we must do this dummy write first to be restartable in
        case of page fault. */
@@ -1181,14 +1182,16 @@ static inline void gen_ins(DisasContext *s, TCGMemOp ot)
     gen_op_st_v(s, ot, cpu_T[0], cpu_A0);
     gen_op_movl_T0_Dshift(ot);
     gen_op_add_reg_T0(s->aflag, R_EDI);
-    if (use_icount)
+    if (s->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
+    }
 }
 
 static inline void gen_outs(DisasContext *s, TCGMemOp ot)
 {
-    if (use_icount)
+    if (s->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
+    }
     gen_string_movl_A0_ESI(s);
     gen_op_ld_v(s, ot, cpu_T[0], cpu_A0);
 
@@ -1199,8 +1202,9 @@ static inline void gen_outs(DisasContext *s, TCGMemOp ot)
 
     gen_op_movl_T0_Dshift(ot);
     gen_op_add_reg_T0(s->aflag, R_ESI);
-    if (use_icount)
+    if (s->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
+    }
 }
 
 /* same method as Valgrind : we generate jumps to current or next
@@ -6278,7 +6282,7 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
             gen_repz_ins(s, ot, pc_start - s->cs_base, s->pc - s->cs_base);
         } else {
             gen_ins(s, ot);
-            if (use_icount) {
+            if (s->tb->cflags & CF_USE_ICOUNT) {
                 gen_jmp(s, s->pc - s->cs_base);
             }
         }
@@ -6293,7 +6297,7 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
             gen_repz_outs(s, ot, pc_start - s->cs_base, s->pc - s->cs_base);
         } else {
             gen_outs(s, ot);
-            if (use_icount) {
+            if (s->tb->cflags & CF_USE_ICOUNT) {
                 gen_jmp(s, s->pc - s->cs_base);
             }
         }
@@ -6309,12 +6313,13 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
         tcg_gen_movi_tl(cpu_T[0], val);
         gen_check_io(s, ot, pc_start - s->cs_base,
                      SVM_IOIO_TYPE_MASK | svm_is_rep(prefixes));
-        if (use_icount)
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
+	}
         tcg_gen_movi_i32(cpu_tmp2_i32, val);
         gen_helper_in_func(ot, cpu_T[1], cpu_tmp2_i32);
         gen_op_mov_reg_v(ot, R_EAX, cpu_T[1]);
-        if (use_icount) {
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
             gen_jmp(s, s->pc - s->cs_base);
         }
@@ -6328,12 +6333,13 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
                      svm_is_rep(prefixes));
         gen_op_mov_v_reg(ot, cpu_T[1], R_EAX);
 
-        if (use_icount)
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
+	}
         tcg_gen_movi_i32(cpu_tmp2_i32, val);
         tcg_gen_trunc_tl_i32(cpu_tmp3_i32, cpu_T[1]);
         gen_helper_out_func(ot, cpu_tmp2_i32, cpu_tmp3_i32);
-        if (use_icount) {
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
             gen_jmp(s, s->pc - s->cs_base);
         }
@@ -6344,12 +6350,13 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
         tcg_gen_ext16u_tl(cpu_T[0], cpu_regs[R_EDX]);
         gen_check_io(s, ot, pc_start - s->cs_base,
                      SVM_IOIO_TYPE_MASK | svm_is_rep(prefixes));
-        if (use_icount)
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
+	}
         tcg_gen_trunc_tl_i32(cpu_tmp2_i32, cpu_T[0]);
         gen_helper_in_func(ot, cpu_T[1], cpu_tmp2_i32);
         gen_op_mov_reg_v(ot, R_EAX, cpu_T[1]);
-        if (use_icount) {
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
             gen_jmp(s, s->pc - s->cs_base);
         }
@@ -6362,12 +6369,13 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
                      svm_is_rep(prefixes));
         gen_op_mov_v_reg(ot, cpu_T[1], R_EAX);
 
-        if (use_icount)
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
+	}
         tcg_gen_trunc_tl_i32(cpu_tmp2_i32, cpu_T[0]);
         tcg_gen_trunc_tl_i32(cpu_tmp3_i32, cpu_T[1]);
         gen_helper_out_func(ot, cpu_tmp2_i32, cpu_tmp3_i32);
-        if (use_icount) {
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
             gen_jmp(s, s->pc - s->cs_base);
         }
@@ -7065,10 +7073,11 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
     case 0x131: /* rdtsc */
         gen_update_cc_op(s);
         gen_jmp_im(pc_start - s->cs_base);
-        if (use_icount)
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
+	}
         gen_helper_rdtsc(cpu_env);
-        if (use_icount) {
+        if (s->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
             gen_jmp(s, s->pc - s->cs_base);
         }
@@ -7451,10 +7460,11 @@ static target_ulong disas_insn(CPUX86State *env, DisasContext *s,
                         goto illegal_op;
                     gen_update_cc_op(s);
                     gen_jmp_im(pc_start - s->cs_base);
-                    if (use_icount)
+                    if (s->tb->cflags & CF_USE_ICOUNT) {
                         gen_io_start();
+		    }
                     gen_helper_rdtscp(cpu_env);
-                    if (use_icount) {
+                    if (s->tb->cflags & CF_USE_ICOUNT) {
                         gen_io_end();
                         gen_jmp(s, s->pc - s->cs_base);
                     }
diff --git a/target-lm32/translate.c b/target-lm32/translate.c
index 8454e8b..f748f96 100644
--- a/target-lm32/translate.c
+++ b/target-lm32/translate.c
@@ -865,24 +865,24 @@ static void dec_wcsr(DisasContext *dc)
         break;
     case CSR_IM:
         /* mark as an io operation because it could cause an interrupt */
-        if (use_icount) {
+        if (dc->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
         }
         gen_helper_wcsr_im(cpu_env, cpu_R[dc->r1]);
         tcg_gen_movi_tl(cpu_pc, dc->pc + 4);
-        if (use_icount) {
+        if (dc->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
         }
         dc->is_jmp = DISAS_UPDATE;
         break;
     case CSR_IP:
         /* mark as an io operation because it could cause an interrupt */
-        if (use_icount) {
+        if (dc->tb->cflags & CF_USE_ICOUNT) {
             gen_io_start();
         }
         gen_helper_wcsr_ip(cpu_env, cpu_R[dc->r1]);
         tcg_gen_movi_tl(cpu_pc, dc->pc + 4);
-        if (use_icount) {
+        if (dc->tb->cflags & CF_USE_ICOUNT) {
             gen_io_end();
         }
         dc->is_jmp = DISAS_UPDATE;
diff --git a/target-mips/translate.c b/target-mips/translate.c
index 446eb8a..c34558c 100644
--- a/target-mips/translate.c
+++ b/target-mips/translate.c
@@ -4804,10 +4804,11 @@ static void gen_mfc0(DisasContext *ctx, TCGv arg, int reg, int sel)
         switch (sel) {
         case 0:
             /* Mark as an IO operation because we read the time.  */
-            if (use_icount)
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_start();
+	    }
             gen_helper_mfc0_count(arg, cpu_env);
-            if (use_icount) {
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_end();
             }
             /* Break the TB to be able to take timer interrupts immediately
@@ -5172,8 +5173,9 @@ static void gen_mtc0(DisasContext *ctx, TCGv arg, int reg, int sel)
     if (sel != 0)
         check_insn(ctx, ISA_MIPS32);
 
-    if (use_icount)
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
+    }
 
     switch (reg) {
     case 0:
@@ -5769,7 +5771,7 @@ static void gen_mtc0(DisasContext *ctx, TCGv arg, int reg, int sel)
     (void)rn; /* avoid a compiler warning */
     LOG_DISAS("mtc0 %s (reg %d sel %d)\n", rn, reg, sel);
     /* For simplicity assume that all writes can cause interrupts.  */
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         ctx->bstate = BS_STOP;
     }
@@ -6013,10 +6015,11 @@ static void gen_dmfc0(DisasContext *ctx, TCGv arg, int reg, int sel)
         switch (sel) {
         case 0:
             /* Mark as an IO operation because we read the time.  */
-            if (use_icount)
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_start();
+            }
             gen_helper_mfc0_count(arg, cpu_env);
-            if (use_icount) {
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_end();
             }
             /* Break the TB to be able to take timer interrupts immediately
@@ -6367,8 +6370,9 @@ static void gen_dmtc0(DisasContext *ctx, TCGv arg, int reg, int sel)
     if (sel != 0)
         check_insn(ctx, ISA_MIPS64);
 
-    if (use_icount)
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
+    }
 
     switch (reg) {
     case 0:
@@ -6661,11 +6665,11 @@ static void gen_dmtc0(DisasContext *ctx, TCGv arg, int reg, int sel)
             save_cpu_state(ctx, 1);
             /* Mark as an IO operation because we may trigger a software
                interrupt.  */
-            if (use_icount) {
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_start();
             }
             gen_helper_mtc0_cause(cpu_env, arg);
-            if (use_icount) {
+            if (ctx->tb->cflags & CF_USE_ICOUNT) {
                 gen_io_end();
             }
             /* Stop translation as we may have triggered an intetrupt */
@@ -6955,7 +6959,7 @@ static void gen_dmtc0(DisasContext *ctx, TCGv arg, int reg, int sel)
     (void)rn; /* avoid a compiler warning */
     LOG_DISAS("dmtc0 %s (reg %d sel %d)\n", rn, reg, sel);
     /* For simplicity assume that all writes can cause interrupts.  */
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         ctx->bstate = BS_STOP;
     }
diff --git a/target-ppc/translate_init.c b/target-ppc/translate_init.c
index c7ca95e..ee074ac 100644
--- a/target-ppc/translate_init.c
+++ b/target-ppc/translate_init.c
@@ -188,11 +188,11 @@ static void spr_write_ureg(DisasContext *ctx, int sprn, int gprn)
 #if !defined(CONFIG_USER_ONLY)
 static void spr_read_decr (DisasContext *ctx, int gprn, int sprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_load_decr(cpu_gpr[gprn], cpu_env);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
@@ -200,11 +200,11 @@ static void spr_read_decr (DisasContext *ctx, int gprn, int sprn)
 
 static void spr_write_decr (DisasContext *ctx, int sprn, int gprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_store_decr(cpu_env, cpu_gpr[gprn]);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
@@ -215,11 +215,11 @@ static void spr_write_decr (DisasContext *ctx, int sprn, int gprn)
 /* Time base */
 static void spr_read_tbl (DisasContext *ctx, int gprn, int sprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_load_tbl(cpu_gpr[gprn], cpu_env);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
@@ -227,11 +227,11 @@ static void spr_read_tbl (DisasContext *ctx, int gprn, int sprn)
 
 static void spr_read_tbu (DisasContext *ctx, int gprn, int sprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_load_tbu(cpu_gpr[gprn], cpu_env);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
@@ -252,11 +252,11 @@ static void spr_read_atbu (DisasContext *ctx, int gprn, int sprn)
 #if !defined(CONFIG_USER_ONLY)
 static void spr_write_tbl (DisasContext *ctx, int sprn, int gprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_store_tbl(cpu_env, cpu_gpr[gprn]);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
@@ -264,11 +264,11 @@ static void spr_write_tbl (DisasContext *ctx, int sprn, int gprn)
 
 static void spr_write_tbu (DisasContext *ctx, int sprn, int gprn)
 {
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_start();
     }
     gen_helper_store_tbu(cpu_env, cpu_gpr[gprn]);
-    if (use_icount) {
+    if (ctx->tb->cflags & CF_USE_ICOUNT) {
         gen_io_end();
         gen_stop_exception(ctx);
     }
diff --git a/translate-all.c b/translate-all.c
index e9f5178..c256d58 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -218,7 +218,7 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
 
     gen_intermediate_code_pc(env, tb);
 
-    if (use_icount) {
+    if (tb->cflags & CF_USE_ICOUNT) {
         /* Reset the cycle counter to the start of the block.  */
         cpu->icount_decr.u16.low += tb->icount;
         /* Clear the IO flag.  */

[Qemu-devel] [RFC PATCH v5 14/31] From 04bbd21134dd2c6b7309a7f5f2b780aae2757003 Mon Sep 17 00:00:00 2001

[Pavel Dovgalyuk]

From: Paolo Bonzini <pbonzini@redhat.com>

Subject: [PATCH] gen-icount: check cflags instead of use_icount global

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/exec/gen-icount.h     |    6 +++---
 target-alpha/translate.c      |    2 +-
 target-arm/translate-a64.c    |    2 +-
 target-arm/translate.c        |    2 +-
 target-cris/translate.c       |    2 +-
 target-i386/translate.c       |    2 +-
 target-lm32/translate.c       |    2 +-
 target-m68k/translate.c       |    2 +-
 target-microblaze/translate.c |    2 +-
 target-mips/translate.c       |    2 +-
 target-moxie/translate.c      |    2 +-
 target-openrisc/translate.c   |    2 +-
 target-ppc/translate.c        |    2 +-
 target-s390x/translate.c      |    2 +-
 target-sh4/translate.c        |    2 +-
 target-sparc/translate.c      |    2 +-
 target-tricore/translate.c    |    2 +-
 target-unicore32/translate.c  |    2 +-
 target-xtensa/translate.c     |    2 +-
 19 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/include/exec/gen-icount.h b/include/exec/gen-icount.h
index da53395..221aad0 100644
--- a/include/exec/gen-icount.h
+++ b/include/exec/gen-icount.h
@@ -9,7 +9,7 @@ static TCGArg *icount_arg;
 static int icount_label;
 static int exitreq_label;
 
-static inline void gen_tb_start(void)
+static inline void gen_tb_start(TranslationBlock *tb)
 {
     TCGv_i32 count;
     TCGv_i32 flag;
@@ -21,7 +21,7 @@ static inline void gen_tb_start(void)
     tcg_gen_brcondi_i32(TCG_COND_NE, flag, 0, exitreq_label);
     tcg_temp_free_i32(flag);
 
-    if (!use_icount)
+    if (!(tb->cflags & CF_USE_ICOUNT))
         return;
 
     icount_label = gen_new_label();
@@ -43,7 +43,7 @@ static void gen_tb_end(TranslationBlock *tb, int num_insns)
     gen_set_label(exitreq_label);
     tcg_gen_exit_tb((uintptr_t)tb + TB_EXIT_REQUESTED);
 
-    if (use_icount) {
+    if (tb->cflags & CF_USE_ICOUNT) {
         *icount_arg = num_insns;
         gen_set_label(icount_label);
         tcg_gen_exit_tb((uintptr_t)tb + TB_EXIT_ICOUNT_EXPIRED);
diff --git a/target-alpha/translate.c b/target-alpha/translate.c
index 5387b93..f888367 100644
--- a/target-alpha/translate.c
+++ b/target-alpha/translate.c
@@ -2828,7 +2828,7 @@ static inline void gen_intermediate_code_internal(AlphaCPU *cpu,
         pc_mask = ~TARGET_PAGE_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-arm/translate-a64.c b/target-arm/translate-a64.c
index ba6a85c..7f17a0c 100644
--- a/target-arm/translate-a64.c
+++ b/target-arm/translate-a64.c
@@ -10963,7 +10963,7 @@ void gen_intermediate_code_internal_a64(ARMCPU *cpu,
         max_insns = CF_COUNT_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
 
     tcg_clear_temp_count();
 
diff --git a/target-arm/translate.c b/target-arm/translate.c
index d7343df..f3bef91 100644
--- a/target-arm/translate.c
+++ b/target-arm/translate.c
@@ -11049,7 +11049,7 @@ static inline void gen_intermediate_code_internal(ARMCPU *cpu,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
 
-    gen_tb_start();
+    gen_tb_start(tb);
 
     tcg_clear_temp_count();
 
diff --git a/target-cris/translate.c b/target-cris/translate.c
index e37b04e..19a452d 100644
--- a/target-cris/translate.c
+++ b/target-cris/translate.c
@@ -3206,7 +3206,7 @@ gen_intermediate_code_internal(CRISCPU *cpu, TranslationBlock *tb,
         max_insns = CF_COUNT_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         check_breakpoint(env, dc);
 
diff --git a/target-i386/translate.c b/target-i386/translate.c
index b7631b7..ceaaa54 100644
--- a/target-i386/translate.c
+++ b/target-i386/translate.c
@@ -8003,7 +8003,7 @@ static inline void gen_intermediate_code_internal(X86CPU *cpu,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
 
-    gen_tb_start();
+    gen_tb_start(tb);
     for(;;) {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-lm32/translate.c b/target-lm32/translate.c
index f748f96..a7579dc 100644
--- a/target-lm32/translate.c
+++ b/target-lm32/translate.c
@@ -1095,7 +1095,7 @@ void gen_intermediate_code_internal(LM32CPU *cpu,
         max_insns = CF_COUNT_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         check_breakpoint(env, dc);
 
diff --git a/target-m68k/translate.c b/target-m68k/translate.c
index efd4cfc..47edc7a 100644
--- a/target-m68k/translate.c
+++ b/target-m68k/translate.c
@@ -3010,7 +3010,7 @@ gen_intermediate_code_internal(M68kCPU *cpu, TranslationBlock *tb,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         pc_offset = dc->pc - pc_start;
         gen_throws_exception = NULL;
diff --git a/target-microblaze/translate.c b/target-microblaze/translate.c
index fd2b771..69ce4df 100644
--- a/target-microblaze/translate.c
+++ b/target-microblaze/translate.c
@@ -1720,7 +1720,7 @@ gen_intermediate_code_internal(MicroBlazeCPU *cpu, TranslationBlock *tb,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do
     {
 #if SIM_COMPAT
diff --git a/target-mips/translate.c b/target-mips/translate.c
index c34558c..94d624b 100644
--- a/target-mips/translate.c
+++ b/target-mips/translate.c
@@ -17432,7 +17432,7 @@ gen_intermediate_code_internal(MIPSCPU *cpu, TranslationBlock *tb,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
     LOG_DISAS("\ntb %p idx %d hflags %04x\n", tb, ctx.mem_idx, ctx.hflags);
-    gen_tb_start();
+    gen_tb_start(tb);
     while (ctx.bstate == BS_NONE) {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-moxie/translate.c b/target-moxie/translate.c
index 4541b9b..564f3ee 100644
--- a/target-moxie/translate.c
+++ b/target-moxie/translate.c
@@ -843,7 +843,7 @@ gen_intermediate_code_internal(MoxieCPU *cpu, TranslationBlock *tb,
     ctx.bstate = BS_NONE;
     num_insns = 0;
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-openrisc/translate.c b/target-openrisc/translate.c
index 407bd97..7dea405 100644
--- a/target-openrisc/translate.c
+++ b/target-openrisc/translate.c
@@ -1675,7 +1675,7 @@ static inline void gen_intermediate_code_internal(OpenRISCCPU *cpu,
         max_insns = CF_COUNT_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
 
     do {
         check_breakpoint(cpu, dc);
diff --git a/target-ppc/translate.c b/target-ppc/translate.c
index 439f7f0..4b56088 100644
--- a/target-ppc/translate.c
+++ b/target-ppc/translate.c
@@ -11344,7 +11344,7 @@ static inline void gen_intermediate_code_internal(PowerPCCPU *cpu,
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
 
-    gen_tb_start();
+    gen_tb_start(tb);
     tcg_clear_temp_count();
     /* Set env in case of segfault during code fetch */
     while (ctx.exception == POWERPC_EXCP_NONE
diff --git a/target-s390x/translate.c b/target-s390x/translate.c
index 0cb036f..dcc792e 100644
--- a/target-s390x/translate.c
+++ b/target-s390x/translate.c
@@ -4767,7 +4767,7 @@ static inline void gen_intermediate_code_internal(S390CPU *cpu,
         max_insns = CF_COUNT_MASK;
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
 
     do {
         if (search_pc) {
diff --git a/target-sh4/translate.c b/target-sh4/translate.c
index 3088edc..7010740 100644
--- a/target-sh4/translate.c
+++ b/target-sh4/translate.c
@@ -1890,7 +1890,7 @@ gen_intermediate_code_internal(SuperHCPU *cpu, TranslationBlock *tb,
     max_insns = tb->cflags & CF_COUNT_MASK;
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
-    gen_tb_start();
+    gen_tb_start(tb);
     while (ctx.bstate == BS_NONE && tcg_ctx.gen_opc_ptr < gen_opc_end) {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-sparc/translate.c b/target-sparc/translate.c
index 78c4e21..25d1bd6 100644
--- a/target-sparc/translate.c
+++ b/target-sparc/translate.c
@@ -5271,7 +5271,7 @@ static inline void gen_intermediate_code_internal(SPARCCPU *cpu,
     max_insns = tb->cflags & CF_COUNT_MASK;
     if (max_insns == 0)
         max_insns = CF_COUNT_MASK;
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-tricore/translate.c b/target-tricore/translate.c
index d5a9596..7adbd07 100644
--- a/target-tricore/translate.c
+++ b/target-tricore/translate.c
@@ -2447,7 +2447,7 @@ gen_intermediate_code_internal(TriCoreCPU *cpu, struct TranslationBlock *tb,
     ctx.mem_idx = cpu_mmu_index(env);
 
     tcg_clear_temp_count();
-    gen_tb_start();
+    gen_tb_start(tb);
     while (ctx.bstate == BS_NONE) {
         ctx.opcode = cpu_ldl_code(env, ctx.pc);
         decode_opc(env, &ctx, 0);
diff --git a/target-unicore32/translate.c b/target-unicore32/translate.c
index 653c225..ab7e96f 100644
--- a/target-unicore32/translate.c
+++ b/target-unicore32/translate.c
@@ -1917,7 +1917,7 @@ static inline void gen_intermediate_code_internal(UniCore32CPU *cpu,
     }
 #endif
 
-    gen_tb_start();
+    gen_tb_start(tb);
     do {
         if (unlikely(!QTAILQ_EMPTY(&cs->breakpoints))) {
             QTAILQ_FOREACH(bp, &cs->breakpoints, entry) {
diff --git a/target-xtensa/translate.c b/target-xtensa/translate.c
index badca19..7ea820f 100644
--- a/target-xtensa/translate.c
+++ b/target-xtensa/translate.c
@@ -3019,7 +3019,7 @@ void gen_intermediate_code_internal(XtensaCPU *cpu,
         dc.next_icount = tcg_temp_local_new_i32();
     }
 
-    gen_tb_start();
+    gen_tb_start(tb);
 
     if (tb->flags & XTENSA_TBFLAG_EXCEPTION) {
         tcg_gen_movi_i32(cpu_pc, dc.pc);

[Qemu-devel] [RFC PATCH v5 15/31] cpu-exec: allow temporary disabling icount

[Pavel Dovgalyuk]

This patch is required for deterministic replay to generate an exception
by trying executing an instruction without changing icount.
It adds new flag to TB for disabling icount while translating it.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c              |    6 +++---
 include/exec/exec-all.h |    1 +
 translate-all.c         |    2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 4df9856..05341cf 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -198,7 +198,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
 /* Execute the code without caching the generated code. An interpreter
    could be used if available. */
 static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
-                             TranslationBlock *orig_tb)
+                             TranslationBlock *orig_tb, bool ignore_icount)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
@@ -214,7 +214,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles);
+                     max_cycles | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
@@ -517,7 +517,7 @@ int cpu_exec(CPUArchState *env)
                         } else {
                             if (insns_left > 0) {
                                 /* Execute remaining instructions.  */
-                                cpu_exec_nocache(env, insns_left, tb);
+                                cpu_exec_nocache(env, insns_left, tb, false);
                                 align_clocks(&sc, cpu);
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 3d19e72..1e6d7e8 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -147,6 +147,7 @@ struct TranslationBlock {
 #define CF_COUNT_MASK  0x7fff
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_USE_ICOUNT  0x10000
+#define CF_IGNORE_ICOUNT 0x20000 /* Do not generate icount code */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
diff --git a/translate-all.c b/translate-all.c
index c256d58..21a78a4 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -1047,7 +1047,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     int code_gen_size;
 
     phys_pc = get_page_addr_code(env, pc);
-    if (use_icount) {
+    if (use_icount && !(cflags & CF_IGNORE_ICOUNT)) {
         cflags |= CF_USE_ICOUNT;
     }
     tb = tb_alloc(pc);

[Qemu-devel] [RFC PATCH v5 16/31] cpu-exec: invalidate nocache translation if they are interrupted

[Pavel Dovgalyuk]

In this case, QEMU might longjmp out of cpu-exec.c and miss the final
cleanup in cpu_exec_nocache.  Do this manually through a new compile
flag.  This is important once we add no-icount translations.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c              |    2 +-
 include/exec/exec-all.h |    1 +
 translate-all.c         |    6 ++++++
 3 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 05341cf..65cbeb7 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -214,7 +214,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
+                     max_cycles | CF_NOCACHE | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 1e6d7e8..d409565 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -148,6 +148,7 @@ struct TranslationBlock {
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_USE_ICOUNT  0x10000
 #define CF_IGNORE_ICOUNT 0x20000 /* Do not generate icount code */
+#define CF_NOCACHE     0x40000 /* To be freed after execution */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
diff --git a/translate-all.c b/translate-all.c
index 21a78a4..269c4ba 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -264,6 +264,12 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t retaddr)
     tb = tb_find_pc(retaddr);
     if (tb) {
         cpu_restore_state_from_tb(cpu, tb, retaddr);
+        if (tb->cflags & CF_NOCACHE) {
+            /* one-shot translation, invalidate it immediately */
+            cpu->current_tb = NULL;
+            tb_phys_invalidate(tb, -1);
+            tb_free(tb);
+        }
         return true;
     }
     return false;

[Qemu-devel] [RFC PATCH v5 17/31] replay: interrupts and exceptions

[Pavel Dovgalyuk]

This patch includes modifications of common cpu files. All interrupts and
exceptions occured during recording are written into the replay log.
These events allow correct replaying the execution by kicking cpu thread
when one of these events is found in the log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c               |   33 +++++++++++++++++++++++------
 replay/replay-internal.h |    4 +++
 replay/replay.c          |   53 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 +++++++++++++++
 4 files changed, 100 insertions(+), 7 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 65cbeb7..05cca50 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 /* -icount align implementation. */
 
@@ -391,10 +392,21 @@ int cpu_exec(CPUArchState *env)
                     cpu->exception_index = -1;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_INTERRUPT;
+                        break;
+                    }
 #endif
                 }
+            } else if (replay_has_exception()
+                       && cpu->icount_decr.u16.low + cpu->icount_extra == 0) {
+                /* try to cause an exception pending in the log */
+                cpu_exec_nocache(env, 1, tb_find_fast(env), true);
+                break;
             }
 
             next_tb = 0; /* force lookup of first TB */
@@ -410,21 +422,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif
@@ -432,7 +447,10 @@ int cpu_exec(CPUArchState *env)
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                    if ((replay_mode != REPLAY_MODE_PLAY
+                            || replay_has_interrupt())
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                        replay_interrupt();
                         next_tb = 0;
                     }
                     /* Don't use the cached interrupt_request value,
@@ -444,7 +462,8 @@ int cpu_exec(CPUArchState *env)
                         next_tb = 0;
                     }
                 }
-                if (unlikely(cpu->exit_request)) {
+                if (unlikely(cpu->exit_request
+                             || replay_has_interrupt())) {
                     cpu->exit_request = 0;
                     cpu->exception_index = EXCP_INTERRUPT;
                     cpu_loop_exit(cpu);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 582b44c..fd5c230 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for software interrupt */
+#define EVENT_INTERRUPT             15
+/* for emulated exceptions */
+#define EVENT_EXCEPTION             23
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
diff --git a/replay/replay.c b/replay/replay.c
index c305e0c..c275794 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -68,3 +68,56 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_instructions_counter();
 }
+
+bool replay_exception(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_EXCEPTION);
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            replay_has_unread_data = 0;
+            return true;
+        }
+        return false;
+    }
+
+    return true;
+}
+
+bool replay_has_exception(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+bool replay_interrupt(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_INTERRUPT);
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_INTERRUPT)) {
+            replay_has_unread_data = 0;
+            return true;
+        }
+        return false;
+    }
+
+    return true;
+}
+
+bool replay_has_interrupt(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        return skip_async_events(EVENT_INTERRUPT);
+    }
+    return false;
+}
diff --git a/replay/replay.h b/replay/replay.h
index e40daf5..d7abaee 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -27,4 +27,21 @@ ReplaySubmode replay_get_play_submode(void);
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Used to determine that exception is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif

[Qemu-devel] [RFC PATCH v5 18/31] replay: asynchronous events infrastructure

[Pavel Dovgalyuk]

This patch adds module for saving and replaying asynchronous events.
These events include network packets, keyboard and mouse input,
USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points
such as beginning of TB execution, or checkpoint in the iothread.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  217 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   27 ++++++
 replay/replay.h          |    4 +
 4 files changed, 249 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-events.c

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 1148f45..56da09c 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
diff --git a/replay/replay-events.c b/replay/replay-events.c
new file mode 100755
index 0000000..f3c9b16
--- /dev/null
+++ b/replay/replay-events.c
@@ -0,0 +1,217 @@
+/*
+ * replay-events.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+typedef struct Event {
+    int event_kind;
+    void *opaque;
+    void *opaque2;
+    uint64_t id;
+
+    QTAILQ_ENTRY(Event) events;
+} Event;
+
+static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list);
+
+static QemuMutex lock;
+static unsigned int read_event_kind = -1;
+static uint64_t read_id = -1;
+static int read_opt = -1;
+
+static bool replay_events_enabled = false;
+
+/* Functions */
+
+static void replay_run_event(Event *event)
+{
+    switch (event->event_kind) {
+    default:
+        fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
+                event->event_kind);
+        exit(1);
+        break;
+    }
+}
+
+void replay_enable_events(void)
+{
+    replay_events_enabled = true;
+}
+
+bool replay_has_events(void)
+{
+    return !QTAILQ_EMPTY(&events_list);
+}
+
+void replay_flush_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_disable_events(void)
+{
+    replay_events_enabled = false;
+    /* Flush events queue before waiting of completion */
+    replay_flush_events();
+}
+
+void replay_clear_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        QTAILQ_REMOVE(&events_list, event, events);
+
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+static void replay_add_event_internal(int event_kind, void *opaque,
+                                      void *opaque2, uint64_t id)
+{
+    if (event_kind >= REPLAY_ASYNC_COUNT) {
+        fprintf(stderr, "Replay: invalid async event ID (%d)\n", event_kind);
+        exit(1);
+    }
+    if (!replay_file || replay_mode == REPLAY_MODE_NONE
+        || !replay_events_enabled) {
+        Event e;
+        e.event_kind = event_kind;
+        e.opaque = opaque;
+        e.opaque2 = opaque2;
+        e.id = id;
+        replay_run_event(&e);
+        return;
+    }
+
+    Event *event = g_malloc0(sizeof(Event));
+    event->event_kind = event_kind;
+    event->opaque = opaque;
+    event->opaque2 = opaque2;
+    event->id = id;
+
+    qemu_mutex_lock(&lock);
+    QTAILQ_INSERT_TAIL(&events_list, event, events);
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_add_event(int event_kind, void *opaque)
+{
+    replay_add_event_internal(event_kind, opaque, NULL, 0);
+}
+
+void replay_save_events(int opt)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            /* put the event into the file */
+            replay_put_event(EVENT_ASYNC_OPT);
+            replay_put_byte(opt);
+            replay_put_byte(event->event_kind);
+
+            /* save event-specific data */
+            switch (event->event_kind) {
+            }
+        }
+
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_read_events(int opt)
+{
+    replay_fetch_data_kind();
+    while (replay_data_kind == EVENT_ASYNC_OPT) {
+        if (read_event_kind == -1) {
+            read_opt = replay_get_byte();
+            read_event_kind = replay_get_byte();
+            read_id = -1;
+            replay_check_error();
+        }
+
+        if (opt != read_opt) {
+            break;
+        }
+        /* Execute some events without searching them in the queue */
+        switch (read_event_kind) {
+        default:
+            fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
+            exit(1);
+            break;
+        }
+
+        qemu_mutex_lock(&lock);
+
+        Event *event = NULL;
+        Event *curr = NULL;
+        QTAILQ_FOREACH(curr, &events_list, events) {
+            if (curr->event_kind == read_event_kind
+                && (read_id == -1 || read_id == curr->id)) {
+                event = curr;
+                break;
+            }
+        }
+
+        if (event) {
+            /* read event-specific reading data */
+
+            QTAILQ_REMOVE(&events_list, event, events);
+
+            qemu_mutex_unlock(&lock);
+
+            /* reset unread data and other parameters to allow
+               reading other data from the log while
+               running the event */
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_id = -1;
+            read_opt = -1;
+
+            replay_run_event(event);
+            g_free(event);
+
+            replay_fetch_data_kind();
+        } else {
+            qemu_mutex_unlock(&lock);
+            /* No such event found in the queue */
+            break;
+        }
+    }
+}
+
+void replay_init_events(void)
+{
+    read_event_kind = -1;
+    qemu_mutex_init(&lock);
+}
+
+void replay_finish_events(void)
+{
+    replay_events_enabled = false;
+    replay_clear_events();
+    qemu_mutex_destroy(&lock);
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index fd5c230..fcba977 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -18,9 +18,15 @@
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
+/* for async events */
+#define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
+/* Asynchronous events IDs */
+
+#define REPLAY_ASYNC_COUNT             0
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -69,4 +75,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(int event_id, void *opaque);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index d7abaee..0cfd71a 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -43,5 +43,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif

[Qemu-devel] [RFC PATCH v5 19/31] cpu: replay instructions sequence

[Pavel Dovgalyuk]

This patch adds calls to replay functions into the icount setup block.
In record mode number of executed instructions is written to the log.
In replay mode number of istructions to execute is taken from the replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   18 ++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 41 insertions(+), 10 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 05cca50..8938c89 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -541,6 +541,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();
                             cpu_loop_exit(cpu);
                         }
                         break;
diff --git a/cpus.c b/cpus.c
index 43ae7fc..707bf34 100644
--- a/cpus.c
+++ b/cpus.c
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1360,18 +1361,22 @@ static int tcg_cpu_exec(CPUArchState *env)
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
 
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
+            /* Maintain prior (possibly buggy) behaviour where if no deadline
+             * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+             * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+             * nanoseconds.
+             */
+            if ((deadline < 0) || (deadline > INT32_MAX)) {
+                deadline = INT32_MAX;
+            }
 
-        count = qemu_icount_round(deadline);
+            count = qemu_icount_round(deadline);
+        } else {
+            count = replay_get_instructions();
+        }
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1389,6 +1394,9 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_exec_instructions();
+        }
     }
     return ret;
 }
diff --git a/replay/replay.c b/replay/replay.c
index c275794..a6de6a1 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -69,6 +69,24 @@ uint64_t replay_get_current_step(void)
     return cpu_get_instructions_counter();
 }
 
+int replay_get_instructions(void)
+{
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        return replay_state.instructions_count;
+    }
+    return 0;
+}
+
+void replay_exec_instructions(void)
+{
+    int count = (int)(replay_get_current_step() - replay_state.current_step);
+    replay_state.instructions_count -= count;
+    replay_state.current_step += count;
+    if (replay_state.instructions_count == 0 && count != 0) {
+        replay_has_unread_data = 0;
+    }
+}
+
 bool replay_exception(void)
 {
     if (replay_mode == REPLAY_MODE_RECORD) {
diff --git a/replay/replay.h b/replay/replay.h
index 0cfd71a..90a949b 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -26,6 +26,10 @@ ReplaySubmode replay_get_play_submode(void);
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_exec_instructions(void);
 
 /* Interrupts and exceptions */
 

[Qemu-devel] [RFC PATCH v5 20/31] replay: recording and replaying clock ticks

[Pavel Dovgalyuk]

Clock ticks are considered as the sources of non-deterministic data for
virtual machine. This patch implements saving the clock values when they
are acquired (virtual, host clock, rdtsc, and some other timers).
When replaying the execution corresponding values are read from log and
transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes
it is not true - e.g. when timeouts for timer lists are checked. In this case
we use a cached value of the clock, passing it to the client code.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/qemu/timer.h     |   20 ++++++++++--
 qemu-timer.c             |    3 +-
 replay/Makefile.objs     |    1 +
 replay/replay-internal.h |   11 ++++++
 replay/replay-time.c     |   79 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   21 ++++++++++++
 stubs/replay.c           |    9 +++++
 7 files changed, 141 insertions(+), 3 deletions(-)
 create mode 100755 replay/replay-time.c

diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 38a02c5..7b43331 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -699,8 +700,8 @@ static inline int64_t get_ticks_per_sec(void)
  * Low level clock functions
  */
 
-/* real time host monotonic timer */
-static inline int64_t get_clock_realtime(void)
+/* real time host monotonic timer implementation */
+static inline int64_t get_clock_realtime_impl(void)
 {
     struct timeval tv;
 
@@ -708,6 +709,12 @@ static inline int64_t get_clock_realtime(void)
     return tv.tv_sec * 1000000000LL + (tv.tv_usec * 1000);
 }
 
+/* real time host monotonic timer interface */
+static inline int64_t get_clock_realtime(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime_impl());
+}
+
 /* Warning: don't insert tracepoints into these functions, they are
    also used by simpletrace backend and tracepoints would cause
    an infinite recursion! */
@@ -752,6 +759,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -905,6 +914,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
diff --git a/qemu-timer.c b/qemu-timer.c
index 00a5d35..8307913 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -562,7 +563,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
         now = get_clock_realtime();
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 56da09c..257c320 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
+obj-y += replay-time.o
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index fcba977..c36d7de 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -22,12 +22,17 @@
 #define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
+/* for clock read/writes */
+#define EVENT_CLOCK                 64
+/* some of grteater codes are reserved for clocks */
 
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -75,6 +80,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
diff --git a/replay/replay-time.c b/replay/replay-time.c
new file mode 100755
index 0000000..3f94f4e
--- /dev/null
+++ b/replay/replay-time.c
@@ -0,0 +1,79 @@
+/*
+ * replay-time.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+    replay_save_instructions();
+
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    if (replay_file) {
+        replay_put_event(EVENT_CLOCK + kind);
+        replay_put_qword(clock);
+    }
+
+    return clock;
+}
+
+void replay_read_next_clock(unsigned int kind)
+{
+    replay_fetch_data_kind();
+    if (replay_file) {
+        unsigned int read_kind = replay_data_kind - EVENT_CLOCK;
+
+        if (kind != -1 && read_kind != kind) {
+            return;
+        }
+        if (read_kind >= REPLAY_CLOCK_COUNT) {
+            fprintf(stderr,
+                    "invalid clock ID %d was read from replay\n", read_kind);
+            exit(1);
+        }
+
+        int64_t clock = replay_get_qword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        replay_state.cached_clock[read_kind] = clock;
+    }
+}
+
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
diff --git a/replay/replay.h b/replay/replay.h
index 90a949b..cba3b18 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -16,6 +16,14 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_HOST       1
+
+#define REPLAY_CLOCK_COUNT      2
+
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
 
@@ -47,6 +55,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/stubs/replay.c b/stubs/replay.c
index b146d55..f3f60fe 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -6,3 +6,12 @@ ReplaySubmode replay_get_play_submode(void)
 {
     return 0;
 }
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}

[Qemu-devel] [RFC PATCH v5 21/31] replay: recording and replaying different timers

[Pavel Dovgalyuk]

This patch introduces functions for recording and replaying realtime sources,
that do not use qemu-clock interface. These include return value of time()
function in time_t and struct tm forms. Patch also adds warning to
get_timedate function to prevent its usage in recording mode, because it may
lead to non-determinism.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/timer/mc146818rtc.c   |   10 ++++
 hw/timer/pl031.c         |   10 ++++
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    4 ++
 replay/replay-time.c     |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 ++++++-
 7 files changed, 157 insertions(+), 5 deletions(-)

diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
index f18d128..7e27931 100644
--- a/hw/timer/mc146818rtc.c
+++ b/hw/timer/mc146818rtc.c
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
@@ -703,7 +704,14 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
diff --git a/hw/timer/pl031.c b/hw/timer/pl031.c
index 34d9b44..19264f2 100644
--- a/hw/timer/pl031.c
+++ b/hw/timer/pl031.c
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,14 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
diff --git a/include/qemu-common.h b/include/qemu-common.h
index b87e9c2..d18f033 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index c36d7de..009029d 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for time_t event */
+#define EVENT_TIME_T                1
+/* for tm event */
+#define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
diff --git a/replay/replay-time.c b/replay/replay-time.c
index 3f94f4e..5d944d3 100755
--- a/replay/replay-time.c
+++ b/replay/replay-time.c
@@ -77,3 +77,115 @@ int64_t replay_read_clock(unsigned int kind)
     fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
     exit(1);
 }
+
+/*! Saves time_t value to the log */
+static void replay_save_time_t(time_t tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_put_event(EVENT_TIME_T);
+        if (sizeof(tm) == 4) {
+            replay_put_dword(tm);
+        } else if (sizeof(tm) == 8) {
+            replay_put_qword(tm);
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+    }
+}
+
+/*! Reads time_t value from the log. Stops execution in case of error */
+static time_t replay_read_time_t(void)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        time_t tm;
+
+        skip_async_events_until(EVENT_TIME_T);
+
+        if (sizeof(tm) == 4) {
+            tm = replay_get_dword();
+        } else if (sizeof(tm) == 8) {
+            tm = replay_get_qword();
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+
+        replay_check_error();
+
+        replay_has_unread_data = 0;
+
+        return tm;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+void replay_save_tm(struct tm *tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_put_event(EVENT_TM);
+
+        replay_put_dword(tm->tm_sec);
+        replay_put_dword(tm->tm_min);
+        replay_put_dword(tm->tm_hour);
+        replay_put_dword(tm->tm_mday);
+        replay_put_dword(tm->tm_mon);
+        replay_put_dword(tm->tm_year);
+        replay_put_dword(tm->tm_wday);
+        replay_put_dword(tm->tm_yday);
+        replay_put_dword(tm->tm_isdst);
+    }
+}
+
+void replay_read_tm(struct tm *tm)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        skip_async_events_until(EVENT_TM);
+
+        tm->tm_sec = replay_get_dword();
+        tm->tm_min = replay_get_dword();
+        tm->tm_hour = replay_get_dword();
+        tm->tm_mday = replay_get_dword();
+        tm->tm_mon = replay_get_dword();
+        tm->tm_year = replay_get_dword();
+        tm->tm_wday = replay_get_dword();
+        tm->tm_yday = replay_get_dword();
+        tm->tm_isdst = replay_get_dword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        return;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+time_t replay_time(void)
+{
+    time_t systime;
+
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        systime = time(NULL);
+        replay_save_time_t(systime);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        systime = replay_read_time_t();
+    } else {
+        systime = time(NULL);
+    }
+
+    return systime;
+}
diff --git a/replay/replay.h b/replay/replay.h
index cba3b18..143fe85 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 #include "qapi-types.h"
 
 /* replay clock kinds */
@@ -67,6 +68,13 @@ int64_t replay_read_clock(unsigned int kind);
         : replay_mode == REPLAY_MODE_RECORD                             \
             ? replay_save_clock((clock), (value))                       \
         : (value))
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! Saves struct tm value to the log */
+void replay_save_tm(struct tm *tm);
+/*! Reads struct tm value from the log. Stops execution in case of error */
+void replay_read_tm(struct tm *tm);
 
 /* Asynchronous events queue */
 
diff --git a/vl.c b/vl.c
index f6b3546..37c6616 100644
--- a/vl.c
+++ b/vl.c
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -767,7 +768,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -784,6 +785,16 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate "
+                        "function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -799,7 +810,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -837,7 +848,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 

[Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Pavel Dovgalyuk]

This patch introduces new QEMU_CLOCK_VIRTUAL_RT clock, which
should be used for icount warping. Separate timer is needed
for replaying the execution, because warping callbacks should
be deterministic. We cannot make realtime clock deterministic
because it is used for screen updates and other simulator-specific
actions. That is why we added new clock which is recorded and
replayed when needed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/qemu/timer.h |    7 +++++++
 qemu-timer.c         |    2 ++
 replay/replay.h      |    4 +++-
 3 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 7b43331..df27157 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -37,12 +37,19 @@
  * is suspended, and it will reflect system time changes the host may
  * undergo (e.g. due to NTP). The host clock has the same precision as
  * the virtual clock.
+ *
+ * @QEMU_CLOCK_VIRTUAL_RT: realtime clock used for icount warp
+ *
+ * This clock runs as a realtime clock, but is used for icount warp
+ * and thus should be traced with record/replay to make warp function
+ * behave deterministically.
  */
 
 typedef enum {
     QEMU_CLOCK_REALTIME = 0,
     QEMU_CLOCK_VIRTUAL = 1,
     QEMU_CLOCK_HOST = 2,
+    QEMU_CLOCK_VIRTUAL_RT = 3,
     QEMU_CLOCK_MAX
 } QEMUClockType;
 
diff --git a/qemu-timer.c b/qemu-timer.c
index 8307913..3f99af5 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -567,6 +567,8 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
+    case QEMU_CLOCK_VIRTUAL_RT:
+        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, get_clock());
     }
 }
 
diff --git a/replay/replay.h b/replay/replay.h
index 143fe85..0c02e03 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -22,8 +22,10 @@
 #define REPLAY_CLOCK_REAL_TICKS 0
 /* host_clock */
 #define REPLAY_CLOCK_HOST       1
+/* virtual_rt_clock */
+#define REPLAY_CLOCK_VIRTUAL_RT 2
 
-#define REPLAY_CLOCK_COUNT      2
+#define REPLAY_CLOCK_COUNT      3
 
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;

[Qemu-devel] [RFC PATCH v5 23/31] cpus: make icount warp deterministic in replay mode

[Pavel Dovgalyuk]

This patch adds saving and replaying warping parameters in record and replay
modes. These parameters affect on virtual clock values and therefore should
be deterministic.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/cpus.c b/cpus.c
index 707bf34..f6a6319 100644
--- a/cpus.c
+++ b/cpus.c
@@ -370,7 +370,7 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
+        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
@@ -444,7 +444,7 @@ void qemu_clock_warp(QEMUClockType type)
     }
 
     /* We want to use the earliest deadline from ALL vm_clocks */
-    clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
+    clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
     deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline < 0) {
         return;
@@ -537,8 +537,8 @@ void configure_icount(QemuOpts *opts, Error **errp)
         return;
     }
     icount_align_option = qemu_opt_get_bool(opts, "align", false);
-    icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
-                                          icount_warp_rt, NULL);
+    icount_warp_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT,
+                                     icount_warp_rt, NULL);
     if (strcmp(option, "auto") != 0) {
         errno = 0;
         icount_time_shift = strtol(option, &rem_str, 0);
@@ -562,10 +562,10 @@ void configure_icount(QemuOpts *opts, Error **errp)
        the virtual time trigger catches emulated time passing too fast.
        Realtime triggers occur even when idle, so use them less frequently
        than VM triggers.  */
-    icount_rt_timer = timer_new_ms(QEMU_CLOCK_REALTIME,
-                                        icount_adjust_rt, NULL);
+    icount_rt_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_RT,
+                                   icount_adjust_rt, NULL);
     timer_mod(icount_rt_timer,
-                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
+                   qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + 1000);
     icount_vm_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                                         icount_adjust_vm, NULL);
     timer_mod(icount_vm_timer,

[Qemu-devel] [RFC PATCH v5 24/31] replay: shutdown event

[Pavel Dovgalyuk]

This patch records and replays simulator shutdown event.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/sysemu/sysemu.h  |    1 +
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   11 +++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    8 +++++++-
 5 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 0037a69..bf4b1bd 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -57,6 +57,7 @@ void qemu_register_suspend_notifier(Notifier *notifier);
 void qemu_system_wakeup_request(WakeupReason reason);
 void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
 void qemu_register_wakeup_notifier(Notifier *notifier);
+void qemu_system_shutdown_request_impl(void);
 void qemu_system_shutdown_request(void);
 void qemu_system_powerdown_request(void);
 void qemu_register_powerdown_notifier(Notifier *notifier);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 009029d..ec6973d 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -20,6 +20,8 @@
 #define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
+/* for shutdown request */
+#define EVENT_SHUTDOWN              20
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
 /* for async events */
diff --git a/replay/replay.c b/replay/replay.c
index a6de6a1..c118f62 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -43,6 +43,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request_impl();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -139,3 +143,10 @@ bool replay_has_interrupt(void)
     }
     return false;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
diff --git a/replay/replay.h b/replay/replay.h
index 0c02e03..00c9906 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -78,6 +78,11 @@ void replay_save_tm(struct tm *tm);
 /*! Reads struct tm value from the log. Stops execution in case of error */
 void replay_read_tm(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/vl.c b/vl.c
index 37c6616..4155342 100644
--- a/vl.c
+++ b/vl.c
@@ -1792,13 +1792,19 @@ void qemu_system_killed(int signal, pid_t pid)
     qemu_system_shutdown_request();
 }
 
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_impl(void)
 {
     trace_qemu_system_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
 
+void qemu_system_shutdown_request(void)
+{
+    replay_shutdown_request();
+    qemu_system_shutdown_request_impl();
+}
+
 static void qemu_system_powerdown(void)
 {
     qapi_event_send_powerdown(&error_abort);

[Qemu-devel] [RFC PATCH v5 25/31] replay: checkpoints

[Pavel Dovgalyuk]

This patch introduces checkpoints that synchronize cpu thread and iothread.
When checkpoint is met in the code all asynchronous events from the queue
are executed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |   11 +++++++++++
 cpus.c                   |    7 ++++++-
 include/qemu/timer.h     |    6 ++++--
 main-loop.c              |    5 +++++
 qemu-timer.c             |   46 ++++++++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    3 +++
 replay/replay.c          |   28 ++++++++++++++++++++++++++++
 replay/replay.h          |    6 ++++++
 stubs/replay.c           |   11 +++++++++++
 vl.c                     |    3 ++-
 10 files changed, 114 insertions(+), 12 deletions(-)

diff --git a/block.c b/block.c
index 88f6d9b..cc84050 100644
--- a/block.c
+++ b/block.c
@@ -1919,6 +1919,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(8)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -1935,6 +1940,12 @@ void bdrv_drain_all(void)
             busy |= bs_busy;
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(8)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
diff --git a/cpus.c b/cpus.c
index f6a6319..8780eee 100644
--- a/cpus.c
+++ b/cpus.c
@@ -405,7 +405,7 @@ void qtest_clock_warp(int64_t dest)
         timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
@@ -425,6 +425,11 @@ void qemu_clock_warp(QEMUClockType type)
         return;
     }
 
+    /* warp clock deterministically in record/replay mode */
+    if (!replay_checkpoint(4)) {
+        return;
+    }
+
     /*
      * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now.
      * This ensures that the deadline for the timer is computed correctly below.
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index df27157..853eed5 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -239,13 +239,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -336,12 +337,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
diff --git a/main-loop.c b/main-loop.c
index 981bcb5..d6e93c3 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -497,6 +497,11 @@ int main_loop_wait(int nonblocking)
     slirp_pollfds_poll(gpollfds, (ret < 0));
 #endif
 
+    /* CPU thread can infinitely wait for event after
+       missing the warp */
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+    }
     qemu_clock_run_all_timers();
 
     return ret;
diff --git a/qemu-timer.c b/qemu-timer.c
index 3f99af5..dbd6c5e 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -458,7 +458,7 @@ bool timer_expired(QEMUTimer *timer_head, int64_t current_time)
     return timer_expired_ns(timer_head, current_time * timer_head->scale);
 }
 
-bool timerlist_run_timers(QEMUTimerList *timer_list)
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all)
 {
     QEMUTimer *ts;
     int64_t current_time;
@@ -466,6 +466,29 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)
     QEMUTimerCB *cb;
     void *opaque;
 
+    switch (timer_list->clock->type) {
+    case QEMU_CLOCK_REALTIME:
+        break;
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 2 : 3)) {
+            return false;
+        }
+        break;
+    case QEMU_CLOCK_HOST:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 5 : 6)) {
+            return false;
+        }
+    case QEMU_CLOCK_VIRTUAL_RT:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 10 : 11)) {
+            return false;
+        }
+        break;
+    }
+
     qemu_event_reset(&timer_list->timers_done_ev);
     if (!timer_list->clock->enabled) {
         goto out;
@@ -498,9 +521,9 @@ out:
     return progress;
 }
 
-bool qemu_clock_run_timers(QEMUClockType type)
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all)
 {
-    return timerlist_run_timers(main_loop_tlg.tl[type]);
+    return timerlist_run_timers(main_loop_tlg.tl[type], run_all);
 }
 
 void timerlistgroup_init(QEMUTimerListGroup *tlg,
@@ -525,7 +548,7 @@ bool timerlistgroup_run_timers(QEMUTimerListGroup *tlg)
     QEMUClockType type;
     bool progress = false;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= timerlist_run_timers(tlg->tl[type]);
+        progress |= timerlist_run_timers(tlg->tl[type], false);
     }
     return progress;
 }
@@ -534,11 +557,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg)
 {
     int64_t deadline = -1;
     QEMUClockType type;
+    bool play = replay_mode == REPLAY_MODE_PLAY;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
         if (qemu_clock_use_for_deadline(tlg->tl[type]->clock->type)) {
-            deadline = qemu_soonest_timeout(deadline,
-                                            timerlist_deadline_ns(
-                                                tlg->tl[type]));
+            if (!play || tlg->tl[type]->clock->type == QEMU_CLOCK_REALTIME) {
+                deadline = qemu_soonest_timeout(deadline,
+                                                timerlist_deadline_ns(
+                                                    tlg->tl[type]));
+            } else {
+                /* Read clock from the replay file and
+                   do not calculate the deadline, based on virtual clock. */
+                qemu_clock_get_ns(tlg->tl[type]->clock->type);
+            }
         }
     }
     return deadline;
@@ -608,7 +638,7 @@ bool qemu_clock_run_all_timers(void)
     QEMUClockType type;
 
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= qemu_clock_run_timers(type);
+        progress |= qemu_clock_run_timers(type, true);
     }
 
     return progress;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index ec6973d..5dad566 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -32,6 +32,9 @@
 #define EVENT_CLOCK                 64
 /* some of grteater codes are reserved for clocks */
 
+/* for checkpoint event */
+#define EVENT_CHECKPOINT            96
+
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
diff --git a/replay/replay.c b/replay/replay.c
index c118f62..07ede73 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -150,3 +150,31 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+/* Used checkpoints: 2 3 4 5 6 7 8 9 10 11 */
+int replay_checkpoint(unsigned int checkpoint)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC_OPT) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    return replay_data_kind != EVENT_ASYNC_OPT;
+                }
+                return 0;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            return replay_data_kind != EVENT_ASYNC_OPT;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+        }
+    }
+
+    return 1;
+}
diff --git a/replay/replay.h b/replay/replay.h
index 00c9906..6961751 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -82,6 +82,12 @@ void replay_read_tm(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+int replay_checkpoint(unsigned int checkpoint);
 
 /* Asynchronous events queue */
 
diff --git a/stubs/replay.c b/stubs/replay.c
index f3f60fe..ab9ede9 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -15,3 +16,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+int replay_checkpoint(unsigned int checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
diff --git a/vl.c b/vl.c
index 4155342..717d67e 100644
--- a/vl.c
+++ b/vl.c
@@ -1847,7 +1847,8 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get() && replay_checkpoint(7)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);

[Qemu-devel] [RFC PATCH v5 26/31] replay: bottom halves

[Pavel Dovgalyuk]

This patch introduces bottom half event for replay queue. It saves the events
into the queue and process them at the checkpoints and instructions execution.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 async.c                  |   46 ++++++++++++++++++++++++++++++++++++++++------
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   18 ++++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    3 ++-
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 99 insertions(+), 12 deletions(-)

diff --git a/async.c b/async.c
index 6e1b282..97111c0 100644
--- a/async.c
+++ b/async.c
@@ -27,6 +27,7 @@
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
 #include "qemu/atomic.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,24 +40,53 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 {
-    QEMUBH *bh;
+    QEMUBH *bh, **last;
     bh = g_malloc0(sizeof(QEMUBH));
     bh->ctx = ctx;
     bh->cb = cb;
     bh->opaque = opaque;
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    if (replay_mode != REPLAY_MODE_NONE) {
+        /* Slower way, but this is a queue and not a stack.
+           Replay will process the BH in the same order they
+           came into the queue. */
+        last = &ctx->first_bh;
+        while (*last) {
+            last = &(*last)->next;
+        }
+        smp_wmb();
+        *last = bh;
+    } else {
+        bh->next = ctx->first_bh;
+        /* Make sure that the members are ready before putting bh into list */
+        smp_wmb();
+        ctx->first_bh = bh;
+    }
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -79,7 +109,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 6918572..357d7e9 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -13,6 +13,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
diff --git a/hw/ide/ahci.c b/hw/ide/ahci.c
index 70958e3..fbefc52 100644
--- a/hw/ide/ahci.c
+++ b/hw/ide/ahci.c
@@ -33,6 +33,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 /* #define DEBUG_AHCI */
 
@@ -1192,7 +1193,8 @@ static void ahci_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 }
diff --git a/hw/ide/core.c b/hw/ide/core.c
index 44e3d50..b622aae 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk,
 
     iocb = blk_aio_get(&trim_aiocb_info, blk, cb, opaque);
     iocb->blk = blk;
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
diff --git a/hw/timer/arm_timer.c b/hw/timer/arm_timer.c
index 1452910..97784a0 100644
--- a/hw/timer/arm_timer.c
+++ b/hw/timer/arm_timer.c
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
diff --git a/hw/usb/hcd-uhci.c b/hw/usb/hcd-uhci.c
index 5b88f30..dc17e5f 100644
--- a/hw/usb/hcd-uhci.c
+++ b/hw/usb/hcd-uhci.c
@@ -1224,7 +1224,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
diff --git a/include/block/aio.h b/include/block/aio.h
index 6bf0e04..5a77431 100644
--- a/include/block/aio.h
+++ b/include/block/aio.h
@@ -35,6 +35,8 @@ struct BlockAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
     int refcnt;
 };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
diff --git a/include/qemu/main-loop.h b/include/qemu/main-loop.h
index 62c68c0..f5a98fe 100644
--- a/include/qemu/main-loop.h
+++ b/include/qemu/main-loop.h
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
diff --git a/main-loop.c b/main-loop.c
index d6e93c3..df99bcf 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -513,3 +513,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
+
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
diff --git a/replay/replay-events.c b/replay/replay-events.c
index f3c9b16..1aee0a4 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -36,6 +36,9 @@ static bool replay_events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -119,6 +122,11 @@ void replay_add_event(int event_kind, void *opaque)
     replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -132,6 +140,9 @@ void replay_save_events(int opt)
 
             /* save event-specific data */
             switch (event->event_kind) {
+            case REPLAY_ASYNC_EVENT_BH:
+                replay_put_qword(event->id);
+                break;
             }
         }
 
@@ -158,6 +169,11 @@ void replay_read_events(int opt)
         }
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            if (read_id == -1) {
+                read_id = replay_get_qword();
+            }
+            break;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 5dad566..6e0c2e9 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -37,7 +37,8 @@
 
 /* Asynchronous events IDs */
 
-#define REPLAY_ASYNC_COUNT             0
+#define REPLAY_ASYNC_EVENT_BH          0
+#define REPLAY_ASYNC_COUNT             1
 
 typedef struct ReplayState {
     /*! Cached clock values. */
diff --git a/replay/replay.h b/replay/replay.h
index 6961751..d3f168d 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -93,5 +93,7 @@ int replay_checkpoint(unsigned int checkpoint);
 
 /*! Disables storing events in the queue */
 void replay_disable_events(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index ab9ede9..6d6c2eb 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -26,3 +26,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}

[Qemu-devel] [RFC PATCH v5 27/31] replay: replay aio requests

[Pavel Dovgalyuk]

This patch adds identifier to aio requests. ID is used for creating bottom
halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying
of the aio requests.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)

diff --git a/block.c b/block.c
index cc84050..02c6a78 100644
--- a/block.c
+++ b/block.c
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4342,7 +4343,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4352,7 +4365,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4363,7 +4388,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4505,7 +4530,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4543,7 +4569,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4688,7 +4714,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4699,7 +4730,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4710,6 +4742,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4723,7 +4760,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4743,6 +4785,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4793,6 +4854,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
diff --git a/block/block-backend.c b/block/block-backend.c
index d0692b1..a3f9e67 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -451,6 +451,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -458,12 +466,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -483,7 +505,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
diff --git a/block/qcow2.c b/block/qcow2.c
index d031515..96a607e 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 357d7e9..4faf6d3 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index b19b102..2413cbe 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = blk_aio_multiwrite(blk, mrb->blkreq, mrb->num_writes);
+    ret = blk_aio_multiwrite_replay(blk, mrb->blkreq, mrb->num_writes);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->blk, mrb);
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
 
     block_acct_start(blk_get_stats(req->dev->blk), &req->acct, req->qiov.size,
                      BLOCK_ACCT_READ);
-    blk_aio_readv(req->dev->blk, sector, &req->qiov,
-                  req->qiov.size / BDRV_SECTOR_SIZE,
-                  virtio_blk_rw_complete, req);
+    blk_aio_readv_replay(req->dev->blk, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
diff --git a/hw/ide/atapi.c b/hw/ide/atapi.c
index c63b7e5..02b11df 100644
--- a/hw/ide/atapi.c
+++ b/hw/ide/atapi.c
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
diff --git a/hw/ide/core.c b/hw/ide/core.c
index b622aae..1886364 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -629,11 +629,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s)
@@ -881,11 +883,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -921,7 +925,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
diff --git a/include/block/block.h b/include/block/block.h
index 341054d..17211ff 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -284,11 +284,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -309,7 +322,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index d18f033..1fa65d0 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 52d13c1..1f8a7c3 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index d94fb1e..b4e484c 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
diff --git a/stubs/replay.c b/stubs/replay.c
index 6d6c2eb..507ea32 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -30,3 +30,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
diff --git a/trace-events b/trace-events
index 6c3a400..7c9828c 100644
--- a/trace-events
+++ b/trace-events
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
diff --git a/util/iov.c b/util/iov.c
index 24566c8..b03f83f 100644
--- a/util/iov.c
+++ b/util/iov.c
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }

[Qemu-devel] [RFC PATCH v5 28/31] replay: thread pool

[Pavel Dovgalyuk]

This patch modifies thread pool to allow replaying asynchronous thread tasks
synchronously in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/raw-posix.c           |    6 ++++-
 block/raw-win32.c           |    4 +++-
 include/block/thread-pool.h |    4 +++-
 replay/replay-events.c      |   11 ++++++++++
 replay/replay-internal.h    |    3 ++-
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 ++++
 tests/test-thread-pool.c    |    7 ++++--
 thread-pool.c               |   49 ++++++++++++++++++++++++++++++-------------
 9 files changed, 67 insertions(+), 23 deletions(-)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index 475cf74..5733f4f 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1078,7 +1078,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1970,7 +1972,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
diff --git a/block/raw-win32.c b/block/raw-win32.c
index 7b58881..39f2fa0 100644
--- a/block/raw-win32.c
+++ b/block/raw-win32.c
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h
index 42eb5e8..801ac00 100644
--- a/include/block/thread-pool.h
+++ b/include/block/thread-pool.h
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque);
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 1aee0a4..4da5de0 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -12,6 +12,7 @@
 #include "qemu-common.h"
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     int event_kind;
@@ -39,6 +40,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -127,6 +131,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -141,6 +150,7 @@ void replay_save_events(int opt)
             /* save event-specific data */
             switch (event->event_kind) {
             case REPLAY_ASYNC_EVENT_BH:
+            case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
             }
@@ -170,6 +180,7 @@ void replay_read_events(int opt)
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             if (read_id == -1) {
                 read_id = replay_get_qword();
             }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 6e0c2e9..c32bd9c 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -38,7 +38,8 @@
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_EVENT_BH          0
-#define REPLAY_ASYNC_COUNT             1
+#define REPLAY_ASYNC_EVENT_THREAD      1
+#define REPLAY_ASYNC_COUNT             2
 
 typedef struct ReplayState {
     /*! Cached clock values. */
diff --git a/replay/replay.h b/replay/replay.h
index d3f168d..c72a3fd 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -95,5 +95,7 @@ int replay_checkpoint(unsigned int checkpoint);
 void replay_disable_events(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index 507ea32..f2d4ed6 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -35,3 +35,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
diff --git a/tests/test-thread-pool.c b/tests/test-thread-pool.c
index 6a0b981..f32594c 100644
--- a/tests/test-thread-pool.c
+++ b/tests/test-thread-pool.c
@@ -56,7 +56,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -120,7 +120,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
diff --git a/thread-pool.c b/thread-pool.c
index e2cac8e..f5a4dac 100644
--- a/thread-pool.c
+++ b/thread-pool.c
@@ -22,6 +22,7 @@
 #include "trace.h"
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 static void do_spawn_thread(ThreadPool *pool);
 
@@ -74,6 +75,27 @@ struct ThreadPool {
     bool stopping;
 };
 
+void thread_pool_work(ThreadPool *pool, void *r)
+{
+    ThreadPoolElement *req = (ThreadPoolElement *)r;
+    int ret;
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_unlock(&pool->lock);
+    }
+
+    ret = req->func(req->arg);
+    req->ret = ret;
+    /* Write ret before state.  */
+    smp_wmb();
+    req->state = THREAD_DONE;
+
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_lock(&pool->lock);
+    }
+
+    qemu_bh_schedule(pool->completion_bh);
+}
+
 static void *worker_thread(void *opaque)
 {
     ThreadPool *pool = opaque;
@@ -100,18 +122,12 @@ static void *worker_thread(void *opaque)
         req = QTAILQ_FIRST(&pool->request_list);
         QTAILQ_REMOVE(&pool->request_list, req, reqs);
         req->state = THREAD_ACTIVE;
-        qemu_mutex_unlock(&pool->lock);
-
-        ret = req->func(req->arg);
-
-        req->ret = ret;
-        /* Write ret before state.  */
-        smp_wmb();
-        req->state = THREAD_DONE;
-
-        qemu_mutex_lock(&pool->lock);
 
-        qemu_bh_schedule(pool->completion_bh);
+        if (replay_mode != REPLAY_MODE_NONE && req->common.replay) {
+            replay_add_thread_event(pool, req, req->common.replay_step);
+        } else {
+            thread_pool_work(pool, req);
+        }
     }
 
     pool->cur_threads--;
@@ -235,7 +251,8 @@ static const AIOCBInfo thread_pool_aiocb_info = {
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque)
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step)
 {
     ThreadPoolElement *req;
 
@@ -244,6 +261,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
     req->arg = arg;
     req->state = THREAD_QUEUED;
     req->pool = pool;
+    req->common.replay = replay;
+    req->common.replay_step = replay_step;
 
     QLIST_INSERT_HEAD(&pool->head, req, all);
 
@@ -254,8 +273,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         spawn_thread(pool);
     }
     QTAILQ_INSERT_TAIL(&pool->request_list, req, reqs);
-    qemu_mutex_unlock(&pool->lock);
     qemu_sem_post(&pool->sem);
+    qemu_mutex_unlock(&pool->lock);
     return &req->common;
 }
 
@@ -277,14 +296,14 @@ int coroutine_fn thread_pool_submit_co(ThreadPool *pool, ThreadPoolFunc *func,
 {
     ThreadPoolCo tpc = { .co = qemu_coroutine_self(), .ret = -EINPROGRESS };
     assert(qemu_in_coroutine());
-    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc);
+    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc, false, 0);
     qemu_coroutine_yield();
     return tpc.ret;
 }
 
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg)
 {
-    thread_pool_submit_aio(pool, func, arg, NULL, NULL);
+    thread_pool_submit_aio(pool, func, arg, NULL, NULL, false, 0);
 }
 
 static void thread_pool_init_one(ThreadPool *pool, AioContext *ctx)

[Qemu-devel] [RFC PATCH v5 29/31] replay: initialization and deinitialization

[Pavel Dovgalyuk]

This patch introduces the functions for enabling the record/replay and for
freeing the resources when simulator closes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.h |    2 +
 replay/replay.c          |  134 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   13 ++++
 stubs/replay.c           |    1 
 vl.c                     |    5 ++
 7 files changed, 157 insertions(+), 1 deletions(-)

diff --git a/block.c b/block.c
index 02c6a78..08b6b3f 100644
--- a/block.c
+++ b/block.c
@@ -1941,7 +1941,7 @@ void bdrv_drain_all(void)
             busy |= bs_busy;
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_checkpoints) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(8)) {
             /* Nothing */
diff --git a/exec.c b/exec.c
index 759055d..2215984 100644
--- a/exec.c
+++ b/exec.c
@@ -794,6 +794,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index c32bd9c..142e09a 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -34,6 +34,8 @@
 
 /* for checkpoint event */
 #define EVENT_CHECKPOINT            96
+/* end of log event */
+#define EVENT_END                   127
 
 /* Asynchronous events IDs */
 
diff --git a/replay/replay.c b/replay/replay.c
index 07ede73..14e650c 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -14,14 +14,23 @@
 #include "replay-internal.h"
 #include "qemu/timer.h"
 
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002
+/* Size of replay log header */
+#define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
+
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 /*! Stores current submode for PLAY mode */
 ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
 
+/* Name of replay file  */
+static char *replay_filename;
 /* Suffix for the disk images filenames */
 char *replay_image_suffix;
 
 ReplayState replay_state;
+bool replay_checkpoints;
 
 ReplaySubmode replay_get_play_submode(void)
 {
@@ -154,6 +163,10 @@ void replay_shutdown_request(void)
 /* Used checkpoints: 2 3 4 5 6 7 8 9 10 11 */
 int replay_checkpoint(unsigned int checkpoint)
 {
+    if (!replay_checkpoints) {
+        return 1;
+    }
+
     replay_save_instructions();
 
     if (replay_file) {
@@ -178,3 +191,124 @@ int replay_checkpoint(unsigned int checkpoint)
 
     return 1;
 }
+
+static void replay_enable(const char *fname, int mode)
+{
+    const char *fmode = NULL;
+    if (replay_file) {
+        fprintf(stderr,
+                "Replay: some record/replay operation is already started\n");
+        return;
+    }
+
+    switch (mode) {
+    case REPLAY_MODE_RECORD:
+        fmode = "wb";
+        break;
+    case REPLAY_MODE_PLAY:
+        fmode = "rb";
+        play_submode = REPLAY_SUBMODE_NORMAL;
+        break;
+    default:
+        fprintf(stderr, "Replay: internal error: invalid replay mode\n");
+        exit(1);
+    }
+
+    atexit(replay_finish);
+
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        unsigned int version = replay_get_dword();
+        uint64_t offset = replay_get_qword();
+        if (version != REPLAY_VERSION) {
+            fprintf(stderr, "Replay: invalid input log file version\n");
+            exit(1);
+        }
+        /* go to the beginning */
+        fseek(replay_file, 12, SEEK_SET);
+    }
+
+    replay_init_events();
+}
+
+void replay_configure(QemuOpts *opts, int mode)
+{
+    const char *fname;
+
+    fname = qemu_opt_get(opts, "fname");
+    if (!fname) {
+        fprintf(stderr, "File name not specified for replay\n");
+        exit(1);
+    }
+
+    const char *suffix = qemu_opt_get(opts, "suffix");
+    if (suffix) {
+        replay_image_suffix = g_strdup(suffix);
+    } else {
+        replay_image_suffix = g_strdup("replay_qcow");
+    }
+
+    replay_enable(fname, mode);
+}
+
+void replay_init_timer(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_checkpoints = true;
+    replay_enable_events();
+}
+
+void replay_finish(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_save_instructions();
+
+    /* finalize the file */
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            replay_put_qword(offset);
+        }
+
+        fclose(replay_file);
+        replay_file = NULL;
+    }
+    if (replay_filename) {
+        g_free(replay_filename);
+        replay_filename = NULL;
+    }
+    if (replay_image_suffix) {
+        g_free(replay_image_suffix);
+        replay_image_suffix = NULL;
+    }
+
+    replay_finish_events();
+}
diff --git a/replay/replay.h b/replay/replay.h
index c72a3fd..9c40648 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -17,6 +17,8 @@
 #include <time.h>
 #include "qapi-types.h"
 
+struct QemuOpts;
+
 /* replay clock kinds */
 /* rdtsc */
 #define REPLAY_CLOCK_REAL_TICKS 0
@@ -29,10 +31,21 @@
 
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
+/*! Is set to true after finishing initialization */
+extern bool replay_checkpoints;
 
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts, int mode);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
+
 /* Processing the instructions */
 
 /*! Returns number of executed instructions. */
diff --git a/stubs/replay.c b/stubs/replay.c
index f2d4ed6..4827ef1 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -2,6 +2,7 @@
 #include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
+bool replay_checkpoints;
 
 ReplaySubmode replay_get_play_submode(void)
 {
diff --git a/vl.c b/vl.c
index 717d67e..877a77c 100644
--- a/vl.c
+++ b/vl.c
@@ -4363,7 +4363,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();
+
     main_loop();
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_disable_events();
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();

[Qemu-devel] [RFC PATCH v5 30/31] replay: command line options

[Pavel Dovgalyuk]

This patch introduces command line options for enabling recording or replaying
virtual machine behavior. "-record" option starts recording of the execution
and saves it into the log, specified with "fname" parameter. "-replay" option
is intended for replaying previously saved log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c          |    3 +-
 qemu-options.hx |   27 +++++++++++++++++++
 vl.c            |   79 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 106 insertions(+), 3 deletions(-)

diff --git a/cpus.c b/cpus.c
index 8780eee..c8fca6c 100644
--- a/cpus.c
+++ b/cpus.c
@@ -929,9 +929,10 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
     while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
+        /* Start accounting real time to the virtual clock if the CPUs
           are idle.  */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
diff --git a/qemu-options.hx b/qemu-options.hx
index 22cf3b9..44ad8fc 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3380,6 +3380,33 @@ Dump json-encoded vmstate information for current machine type to file
 in @var{file}
 ETEXI
 
+DEF("record", HAS_ARG, QEMU_OPTION_record,
+    "-record fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                writes replay file for latter replaying\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -record fname=@var{file}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Writes compact execution trace into @var{file}.
+Changes for disk images are written
+into separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
+DEF("replay", HAS_ARG, QEMU_OPTION_replay,
+    "-replay fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                plays saved replay file\n", QEMU_ARCH_ALL)
+STEXI
+@item -replay fname=@var{filename}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Plays compact execution trace from @var{filename}.
+Changes for disk images and VM states are read
+from separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
 HXCOMM This is the last statement. Insert new options before this line!
 STEXI
 @end table
diff --git a/vl.c b/vl.c
index 877a77c..fb5003e 100644
--- a/vl.c
+++ b/vl.c
@@ -548,6 +548,42 @@ static QemuOptsList qemu_icount_opts = {
     },
 };
 
+static QemuOptsList qemu_record_opts = {
+    .name = "record",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_record_opts.head),
+    .desc = {
+        {
+            .name = "fname",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "suffix",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "snapshot",
+            .type = QEMU_OPT_BOOL,
+        },
+        { /* end of list */ }
+    },
+};
+
+static QemuOptsList qemu_replay_opts = {
+    .name = "replay",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_replay_opts.head),
+    .desc = {
+        {
+            .name = "fname",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "suffix",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "snapshot",
+            .type = QEMU_OPT_BOOL,
+        },
+        { /* end of list */ }
+    },
+};
+
 /**
  * Get machine options
  *
@@ -2711,7 +2747,9 @@ out:
 int main(int argc, char **argv, char **envp)
 {
     int i;
-    int snapshot, linux_boot;
+    int snapshot, linux_boot, replay_snapshot;
+    int not_compatible_replay_param = 0;
+    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;
@@ -2784,6 +2822,8 @@ int main(int argc, char **argv, char **envp)
     qemu_add_opts(&qemu_name_opts);
     qemu_add_opts(&qemu_numa_opts);
     qemu_add_opts(&qemu_icount_opts);
+    qemu_add_opts(&qemu_replay_opts);
+    qemu_add_opts(&qemu_record_opts);
 
     runstate_init();
 
@@ -2797,6 +2837,7 @@ int main(int argc, char **argv, char **envp)
     cpu_model = NULL;
     ram_size = default_ram_size;
     snapshot = 0;
+    replay_snapshot = 1;
     cyls = heads = secs = 0;
     translation = BIOS_ATA_TRANSLATION_AUTO;
 
@@ -2914,6 +2955,7 @@ int main(int argc, char **argv, char **envp)
                 break;
             case QEMU_OPTION_pflash:
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
@@ -3070,6 +3112,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3284,6 +3327,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_fsdev:
                 olist = qemu_find_opts("fsdev");
@@ -3412,6 +3456,7 @@ int main(int argc, char **argv, char **envp)
                 if (strncmp(optarg, "mon:", 4) == 0) {
                     default_monitor = 0;
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_debugcon:
                 add_device_config(DEV_DEBUGCON, optarg);
@@ -3532,6 +3577,7 @@ int main(int argc, char **argv, char **envp)
                 if (!qemu_opts_parse(qemu_find_opts("smp-opts"), optarg, 1)) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
 	    case QEMU_OPTION_vnc:
 #ifdef CONFIG_VNC
@@ -3770,6 +3816,24 @@ int main(int argc, char **argv, char **envp)
                     exit(1);
                 }
                 break;
+            case QEMU_OPTION_record:
+                opts = qemu_opts_parse(qemu_find_opts("record"), optarg, 0);
+                if (!opts) {
+                    fprintf(stderr, "Invalid record options: %s\n", optarg);
+                    exit(1);
+                }
+                replay_configure(opts, REPLAY_MODE_RECORD);
+                replay_snapshot = qemu_opt_get_bool(opts, "snapshot", 1);
+                break;
+            case QEMU_OPTION_replay:
+                opts = qemu_opts_parse(qemu_find_opts("replay"), optarg, 0);
+                if (!opts) {
+                    fprintf(stderr, "Invalid replay options: %s\n", optarg);
+                    exit(1);
+                }
+                replay_configure(opts, REPLAY_MODE_PLAY);
+                replay_snapshot = qemu_opt_get_bool(opts, "snapshot", 1);
+                break;
             default:
                 os_parse_cmd_args(popt->index, optarg);
             }
@@ -3784,6 +3848,12 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+    if (not_compatible_replay_param && (replay_mode != REPLAY_MODE_NONE)) {
+        fprintf(stderr, "options -smp, -pflash, -chardev, -bt, -parallel "
+                        "are not compatible with record/replay\n");
+        exit(1);
+    }
+
     if (qemu_opts_foreach(qemu_find_opts("sandbox"), parse_sandbox, NULL, 0)) {
         exit(1);
     }
@@ -4125,6 +4195,11 @@ int main(int argc, char **argv, char **envp)
         qemu_opts_del(icount_opts);
     }
 
+    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
+        fprintf(stderr, "Please enable icount to use record/replay\n");
+        exit(1);
+    }
+
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
 
@@ -4161,7 +4236,7 @@ int main(int argc, char **argv, char **envp)
     }
 
     /* open the virtual block devices */
-    if (snapshot)
+    if (snapshot || (replay_mode != REPLAY_MODE_NONE && replay_snapshot))
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);
     if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
                           &machine_class->block_default_type, 1) != 0) {

[Qemu-devel] [RFC PATCH v5 31/31] replay: recording of the user input

[Pavel Dovgalyuk]

This records user input (keyboard and mouse events) in record mode and replays
these input events in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   48 ++++++++++++++++++++
 replay/replay-input.c    |  108 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   11 ++++-
 replay/replay.h          |    5 ++
 ui/input.c               |   80 ++++++++++++++++++++++++++--------
 7 files changed, 235 insertions(+), 20 deletions(-)
 create mode 100755 replay/replay-input.c

diff --git a/include/ui/input.h b/include/ui/input.h
index 5d5ac00..d06a12d 100644
--- a/include/ui/input.h
+++ b/include/ui/input.h
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 257c320..3936296 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 4da5de0..308186b 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -13,6 +13,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "block/thread-pool.h"
+#include "ui/input.h"
 
 typedef struct Event {
     int event_kind;
@@ -43,6 +44,16 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_THREAD:
         thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
         break;
+    case REPLAY_ASYNC_EVENT_INPUT:
+        qemu_input_event_send_impl(NULL, (InputEvent *)event->opaque);
+        /* Using local variables, when replaying. Do not free them. */
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            qapi_free_InputEvent((InputEvent *)event->opaque);
+        }
+        break;
+    case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+        qemu_input_event_sync_impl();
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -136,6 +147,16 @@ void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
 }
 
+void replay_add_input_event(struct InputEvent *event)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT, event, NULL, 0);
+}
+
+void replay_add_input_sync_event(void)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT_SYNC, NULL, NULL, 0);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -153,6 +174,9 @@ void replay_save_events(int opt)
             case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
+            case REPLAY_ASYNC_EVENT_INPUT:
+                replay_save_input_event(event->opaque);
+                break;
             }
         }
 
@@ -178,6 +202,7 @@ void replay_read_events(int opt)
             break;
         }
         /* Execute some events without searching them in the queue */
+        Event e;
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
         case REPLAY_ASYNC_EVENT_THREAD:
@@ -185,6 +210,29 @@ void replay_read_events(int opt)
                 read_id = replay_get_qword();
             }
             break;
+        case REPLAY_ASYNC_EVENT_INPUT:
+            e.event_kind = read_event_kind;
+            e.opaque = replay_read_input_event();
+
+            replay_run_event(&e);
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+            /* continue with the next event */
+            continue;
+        case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+            e.event_kind = read_event_kind;
+            e.opaque = 0;
+            replay_run_event(&e);
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+            /* continue with the next event */
+            continue;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
diff --git a/replay/replay-input.c b/replay/replay-input.c
new file mode 100755
index 0000000..f5d1482
--- /dev/null
+++ b/replay/replay-input.c
@@ -0,0 +1,108 @@
+/*
+ * replay-input.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+#include "ui/input.h"
+
+void replay_save_input_event(InputEvent *evt)
+{
+    replay_put_dword(evt->kind);
+
+    switch (evt->kind) {
+    case INPUT_EVENT_KIND_KEY:
+        replay_put_dword(evt->key->key->kind);
+
+        switch (evt->key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            replay_put_qword(evt->key->key->number);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            replay_put_dword(evt->key->key->qcode);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        replay_put_dword(evt->btn->button);
+        replay_put_byte(evt->btn->down);
+        break;
+    case INPUT_EVENT_KIND_REL:
+        replay_put_dword(evt->rel->axis);
+        replay_put_qword(evt->rel->value);
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        replay_put_dword(evt->abs->axis);
+        replay_put_qword(evt->abs->value);
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+}
+
+InputEvent *replay_read_input_event(void)
+{
+    static InputEvent evt;
+    static KeyValue keyValue;
+    static InputKeyEvent key;
+    key.key = &keyValue;
+    static InputBtnEvent btn;
+    static InputMoveEvent rel;
+    static InputMoveEvent abs;
+
+    evt.kind = replay_get_dword();
+    switch (evt.kind) {
+    case INPUT_EVENT_KIND_KEY:
+        evt.key = &key;
+        evt.key->key->kind = replay_get_dword();
+
+        switch (evt.key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            evt.key->key->number = replay_get_qword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            evt.key->key->qcode = (QKeyCode)replay_get_dword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        evt.btn = &btn;
+        evt.btn->button = (InputButton)replay_get_dword();
+        evt.btn->down = replay_get_byte();
+        break;
+    case INPUT_EVENT_KIND_REL:
+        evt.rel = &rel;
+        evt.rel->axis = (InputAxis)replay_get_dword();
+        evt.rel->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        evt.abs = &abs;
+        evt.abs->axis = (InputAxis)replay_get_dword();
+        evt.abs->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+
+    return &evt;
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 142e09a..ac3ba82 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -41,7 +41,9 @@
 
 #define REPLAY_ASYNC_EVENT_BH          0
 #define REPLAY_ASYNC_EVENT_THREAD      1
-#define REPLAY_ASYNC_COUNT             2
+#define REPLAY_ASYNC_EVENT_INPUT       2
+#define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
+#define REPLAY_ASYNC_COUNT             4
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -120,4 +122,11 @@ void replay_read_events(int opt);
 /*! Adds specified async event to the queue */
 void replay_add_event(int event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index 9c40648..f8bf4c7 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -18,6 +18,7 @@
 #include "qapi-types.h"
 
 struct QemuOpts;
+struct InputEvent;
 
 /* replay clock kinds */
 /* rdtsc */
@@ -110,5 +111,9 @@ void replay_disable_events(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
 
 #endif
diff --git a/ui/input.c b/ui/input.c
index 002831e..4e49fe5 100644
--- a/ui/input.c
+++ b/ui/input.c
@@ -5,6 +5,7 @@
 #include "trace.h"
 #include "ui/input.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 struct QemuInputHandlerState {
     DeviceState       *dev;
@@ -295,14 +296,10 @@ static void qemu_input_queue_sync(struct QemuInputEventQueueHead *queue)
     QTAILQ_INSERT_TAIL(queue, item, node);
 }
 
-void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt)
 {
     QemuInputHandlerState *s;
 
-    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
-        return;
-    }
-
     qemu_input_event_trace(src, evt);
 
     /* pre processing */
@@ -319,14 +316,25 @@ void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
     s->events++;
 }
 
-void qemu_input_event_sync(void)
+void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
-    QemuInputHandlerState *s;
-
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
         return;
     }
 
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }
+}
+
+void qemu_input_event_sync_impl(void)
+{
+    QemuInputHandlerState *s;
+
     trace_input_event_sync();
 
     QTAILQ_FOREACH(s, &handlers, node) {
@@ -340,6 +348,21 @@ void qemu_input_event_sync(void)
     }
 }
 
+void qemu_input_event_sync(void)
+{
+    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
+        return;
+    }
+
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_sync_event();
+    } else {
+        qemu_input_event_sync_impl();
+    }
+}
+
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 {
     InputEvent *evt = g_new0(InputEvent, 1);
@@ -353,14 +376,23 @@ InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down)
 {
     InputEvent *evt;
-    evt = qemu_input_event_new_key(key, down);
-    if (QTAILQ_EMPTY(&kbd_queue)) {
-        qemu_input_event_send(src, evt);
-        qemu_input_event_sync();
-        qapi_free_InputEvent(evt);
-    } else {
-        qemu_input_queue_event(&kbd_queue, src, evt);
-        qemu_input_queue_sync(&kbd_queue);
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        evt = qemu_input_event_new_key(key, down);
+        if (QTAILQ_EMPTY(&kbd_queue)) {
+            qemu_input_event_send(src, evt);
+            qemu_input_event_sync();
+            if (replay_mode != REPLAY_MODE_RECORD) {
+                qapi_free_InputEvent(evt);
+            }
+        } else {
+            if (replay_mode != REPLAY_MODE_NONE) {
+                fprintf(stderr, "Input queue is not supported "
+                                "in record/replay mode\n");
+                exit(1);
+            }
+            qemu_input_queue_event(&kbd_queue, src, evt);
+            qemu_input_queue_sync(&kbd_queue);
+        }
     }
 }
 
@@ -386,6 +418,10 @@ void qemu_input_event_send_key_delay(uint32_t delay_ms)
         kbd_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, qemu_input_queue_process,
                                  &kbd_queue);
     }
+    if (replay_mode != REPLAY_MODE_NONE) {
+        fprintf(stderr, "Input queue is not supported in record/replay mode\n");
+        exit(1);
+    }
     qemu_input_queue_delay(&kbd_queue, kbd_timer,
                            delay_ms ? delay_ms : kbd_default_delay_ms);
 }
@@ -405,7 +441,9 @@ void qemu_input_queue_btn(QemuConsole *src, InputButton btn, bool down)
     InputEvent *evt;
     evt = qemu_input_event_new_btn(btn, down);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_update_buttons(QemuConsole *src, uint32_t *button_map,
@@ -458,7 +496,9 @@ void qemu_input_queue_rel(QemuConsole *src, InputAxis axis, int value)
     InputEvent *evt;
     evt = qemu_input_event_new_move(INPUT_EVENT_KIND_REL, axis, value);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_queue_abs(QemuConsole *src, InputAxis axis, int value, int size)
@@ -467,7 +507,9 @@ void qemu_input_queue_abs(QemuConsole *src, InputAxis axis, int value, int size)
     int scaled = qemu_input_scale_axis(value, size, INPUT_EVENT_ABS_SIZE);
     evt = qemu_input_event_new_move(INPUT_EVENT_KIND_ABS, axis, scaled);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_check_mode_change(void)

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Pavel Dovgaluk]

That covermail was wrong. Here is the correct one:




This set of patches is related to the reverse execution and deterministic 
replay of qemu execution  This implementation of deterministic replay can 
be used for deterministic debugging of guest code through gdb remote
interface.

These patches include only core function of the replay,
excluding the support for replaying serial, audio, network, and USB devices'
operations. Reverse debugging and monitor commands were also excluded to
be submitted later as separate patches.

Execution recording writes non-deterministic events log, which can be later 
used for replaying the execution anywhere and for unlimited number of times. 
It also supports checkpointing for faster rewinding during reverse debugging. 
Execution replaying reads the log and replays all non-deterministic events 
including external input, hardware clocks, and interrupts.

Deterministic replay has the following features:
 * Deterministically replays whole system execution and all contents of the memory,
   state of the hadrware devices, clocks, and screen of the VM.
 * Writes execution log into the file for latter replaying for multiple times 
   on different machines.
 * Supports i386, x86_64, and ARM hardware platforms.
 * Performs deterministic replay of all operations with keyboard and mouse
   input devices.
 * Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
 * First, record the execution, by adding the following string to the command line:
   '-record fname=replay.bin -icount 7 -net none'. Block devices' images are not
   actually changed in the recording mode, because all of the changes are
   written to the temporary overlay file.
 * Then you can replay it for the multiple times by using another command
   line option: '-replay fname=replay.bin -icount 7 -net none'
 * '-net none' option should also be specified if network replay patches
   are not applied.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
 * wrappers for clock and time functions to save their return values in the log
 * saving different asynchronous events (e.g. system shutdown) into the log
 * synchronization of the bottom halves execution
 * synchronization of the threads from thread pool
 * recording/replaying user input (mouse and keyboard)
 * adding internal events for cpu and io synchronization

v5 changes:
 * Minor changes.
 * Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
 * Moved savevm-related code out of the core.
 * Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
 * Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
 * Adopted Paolo's icount patches
 * Fixed hardware interrupts replaying

v4 changes:
 * Updated block drivers to support new bdrv_open interface.
 * Moved migration patches into separate series (as suggested by Paolo Bonzini)
 * Fixed a bug in replay_break operation.
 * Fixed rtl8139 migration for replay.
 * Fixed 'period' parameter processing for record mode.
 * Fixed bug in 'reverse-stepi' implementation.
 * Fixed replay without making any snapshots (even the starting one).
 * Moved core replay patches into the separate series.
 * Fixed reverse step and reverse continue support.
 * Fixed several bugs in icount subsystem.
 * Reusing native qemu icount for replay instructions counting.
 * Separated core patches into their own series.

v3 changes:
 * Fixed bug with replay of the aio write operations.
 * Added virtual clock based on replay icount.
 * Removed duplicated saving of interrupt_request CPU field.
 * Fixed some coding style issues.
 * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
 * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
 * Added description and example for replay-info command (as suggested by Eric Blake)
 * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
 * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
 * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
 * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
 * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
 * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
 * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
 * Added QMP versions of replay commands (as suggested by Eric Blake)
 * Removed some optional features of replay to make patches cleaner
 * Minor changes and code cleanup were made

Re: [Qemu-devel] [RFC PATCH v5 20/31] replay: recording and replaying clock ticks

[Paolo Bonzini]

On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> +/* real time host monotonic timer implementation */
> +static inline int64_t get_clock_realtime_impl(void)
>  {
>      struct timeval tv;
>  
> @@ -708,6 +709,12 @@ static inline int64_t get_clock_realtime(void)
>      return tv.tv_sec * 1000000000LL + (tv.tv_usec * 1000);
>  }
>  
> +/* real time host monotonic timer interface */
> +static inline int64_t get_clock_realtime(void)
> +{
> +    return REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime_impl());
> +}
> +

Any reason to do this instead of using REPLAY_CLOCK in qemu_get_clock,
like you do for QEMU_CLOCK_VIRTUAL_RT?

Paolo

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Paolo Bonzini]

On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> This patch introduces new QEMU_CLOCK_VIRTUAL_RT clock, which
> should be used for icount warping. Separate timer is needed
> for replaying the execution, because warping callbacks should
> be deterministic. We cannot make realtime clock deterministic
> because it is used for screen updates and other simulator-specific
> actions. That is why we added new clock which is recorded and
> replayed when needed.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  include/qemu/timer.h |    7 +++++++
>  qemu-timer.c         |    2 ++
>  replay/replay.h      |    4 +++-
>  3 files changed, 12 insertions(+), 1 deletions(-)
> 
> diff --git a/include/qemu/timer.h b/include/qemu/timer.h
> index 7b43331..df27157 100644
> --- a/include/qemu/timer.h
> +++ b/include/qemu/timer.h
> @@ -37,12 +37,19 @@
>   * is suspended, and it will reflect system time changes the host may
>   * undergo (e.g. due to NTP). The host clock has the same precision as
>   * the virtual clock.
> + *
> + * @QEMU_CLOCK_VIRTUAL_RT: realtime clock used for icount warp
> + *
> + * This clock runs as a realtime clock, but is used for icount warp
> + * and thus should be traced with record/replay to make warp function
> + * behave deterministically.
>   */

I think it should also stop/restart across "stop" and "cont" commands,
similar to QEMU_CLOCK_VIRTUAL.  This is as simple as changing
get_clock() to cpu_get_clock().

This way, QEMU_CLOCK_VIRTUAL_RT is "what QEMU_CLOCK_VIRTUAL does without
-icount".  This makes a lot of sense and can be merged in 2.3
independent of the rest of the series.

Paolo

>  typedef enum {
>      QEMU_CLOCK_REALTIME = 0,
>      QEMU_CLOCK_VIRTUAL = 1,
>      QEMU_CLOCK_HOST = 2,
> +    QEMU_CLOCK_VIRTUAL_RT = 3,
>      QEMU_CLOCK_MAX
>  } QEMUClockType;
>  
> diff --git a/qemu-timer.c b/qemu-timer.c
> index 8307913..3f99af5 100644
> --- a/qemu-timer.c
> +++ b/qemu-timer.c
> @@ -567,6 +567,8 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
>              notifier_list_notify(&clock->reset_notifiers, &now);
>          }
>          return now;
> +    case QEMU_CLOCK_VIRTUAL_RT:
> +        return REPLAY_CLOCK(REPLAY_CLOCK_VIRTUAL_RT, get_clock());
>      }
>  }
>  
> diff --git a/replay/replay.h b/replay/replay.h
> index 143fe85..0c02e03 100755
> --- a/replay/replay.h
> +++ b/replay/replay.h
> @@ -22,8 +22,10 @@
>  #define REPLAY_CLOCK_REAL_TICKS 0
>  /* host_clock */
>  #define REPLAY_CLOCK_HOST       1
> +/* virtual_rt_clock */
> +#define REPLAY_CLOCK_VIRTUAL_RT 2
>  
> -#define REPLAY_CLOCK_COUNT      2
> +#define REPLAY_CLOCK_COUNT      3
>  
>  extern ReplayMode replay_mode;
>  extern char *replay_image_suffix;
> 

Re: [Qemu-devel] [RFC PATCH v5 23/31] cpus: make icount warp deterministic in replay mode

[Paolo Bonzini]

On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> This patch adds saving and replaying warping parameters in record and replay
> modes. These parameters affect on virtual clock values and therefore should
> be deterministic.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>

I think this makes warping behave better when you "stop" and "cont" the
VM.  We should apply this independent of the rest of the series.

Paolo

> ---
>  cpus.c |   14 +++++++-------
>  1 files changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 707bf34..f6a6319 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -370,7 +370,7 @@ static void icount_warp_rt(void *opaque)
>  
>      seqlock_write_lock(&timers_state.vm_clock_seqlock);
>      if (runstate_is_running()) {
> -        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> +        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
>          int64_t warp_delta;
>  
>          warp_delta = clock - vm_clock_warp_start;
> @@ -444,7 +444,7 @@ void qemu_clock_warp(QEMUClockType type)
>      }
>  
>      /* We want to use the earliest deadline from ALL vm_clocks */
> -    clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> +    clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);
>      deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
>      if (deadline < 0) {
>          return;
> @@ -537,8 +537,8 @@ void configure_icount(QemuOpts *opts, Error **errp)
>          return;
>      }
>      icount_align_option = qemu_opt_get_bool(opts, "align", false);
> -    icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
> -                                          icount_warp_rt, NULL);
> +    icount_warp_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT,
> +                                     icount_warp_rt, NULL);
>      if (strcmp(option, "auto") != 0) {
>          errno = 0;
>          icount_time_shift = strtol(option, &rem_str, 0);
> @@ -562,10 +562,10 @@ void configure_icount(QemuOpts *opts, Error **errp)
>         the virtual time trigger catches emulated time passing too fast.
>         Realtime triggers occur even when idle, so use them less frequently
>         than VM triggers.  */
> -    icount_rt_timer = timer_new_ms(QEMU_CLOCK_REALTIME,
> -                                        icount_adjust_rt, NULL);
> +    icount_rt_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_RT,
> +                                   icount_adjust_rt, NULL);
>      timer_mod(icount_rt_timer,
> -                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
> +                   qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + 1000);
>      icount_vm_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
>                                          icount_adjust_vm, NULL);
>      timer_mod(icount_vm_timer,
> 

Re: [Qemu-devel] [RFC PATCH v5 20/31] replay: recording and replaying clock ticks

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> > +/* real time host monotonic timer implementation */
> > +static inline int64_t get_clock_realtime_impl(void)
> >  {
> >      struct timeval tv;
> >
> > @@ -708,6 +709,12 @@ static inline int64_t get_clock_realtime(void)
> >      return tv.tv_sec * 1000000000LL + (tv.tv_usec * 1000);
> >  }
> >
> > +/* real time host monotonic timer interface */
> > +static inline int64_t get_clock_realtime(void)
> > +{
> > +    return REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime_impl());
> > +}
> > +
> 
> Any reason to do this instead of using REPLAY_CLOCK in qemu_get_clock,
> like you do for QEMU_CLOCK_VIRTUAL_RT?

hw/ppc.c uses this functions in pre_save and post_load function.
It seems that these calls' results also should be logged by replay.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> > This patch introduces new QEMU_CLOCK_VIRTUAL_RT clock, which
> > should be used for icount warping. Separate timer is needed
> > for replaying the execution, because warping callbacks should
> > be deterministic. We cannot make realtime clock deterministic
> > because it is used for screen updates and other simulator-specific
> > actions. That is why we added new clock which is recorded and
> > replayed when needed.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  include/qemu/timer.h |    7 +++++++
> >  qemu-timer.c         |    2 ++
> >  replay/replay.h      |    4 +++-
> >  3 files changed, 12 insertions(+), 1 deletions(-)
> >
> > diff --git a/include/qemu/timer.h b/include/qemu/timer.h
> > index 7b43331..df27157 100644
> > --- a/include/qemu/timer.h
> > +++ b/include/qemu/timer.h
> > @@ -37,12 +37,19 @@
> >   * is suspended, and it will reflect system time changes the host may
> >   * undergo (e.g. due to NTP). The host clock has the same precision as
> >   * the virtual clock.
> > + *
> > + * @QEMU_CLOCK_VIRTUAL_RT: realtime clock used for icount warp
> > + *
> > + * This clock runs as a realtime clock, but is used for icount warp
> > + * and thus should be traced with record/replay to make warp function
> > + * behave deterministically.
> >   */
> 
> I think it should also stop/restart across "stop" and "cont" commands,
> similar to QEMU_CLOCK_VIRTUAL.  This is as simple as changing
> get_clock() to cpu_get_clock().

Ok, then I'll have to remove !use_icount check from here and retest the series.

void cpu_enable_ticks(void)
{
    /* Here, the really thing protected by seqlock is cpu_clock_offset. */
    seqlock_write_lock(&timers_state.vm_clock_seqlock);
    if (!timers_state.cpu_ticks_enabled) {
        if (!use_icount) {
            timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
            timers_state.cpu_clock_offset -= get_clock();
        }
        timers_state.cpu_ticks_enabled = 1;
    }
    seqlock_write_unlock(&timers_state.vm_clock_seqlock);
}

> This way, QEMU_CLOCK_VIRTUAL_RT is "what QEMU_CLOCK_VIRTUAL does without
> -icount".  This makes a lot of sense and can be merged in 2.3
> independent of the rest of the series.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 20/31] replay: recording and replaying clock ticks

[Paolo Bonzini]

On 26/11/2014 13:22, Pavel Dovgaluk wrote:
>> > Any reason to do this instead of using REPLAY_CLOCK in qemu_get_clock,
>> > like you do for QEMU_CLOCK_VIRTUAL_RT?
> hw/ppc.c uses this functions in pre_save and post_load function.
> It seems that these calls' results also should be logged by replay.

It should use qemu_get_clock_ns(QEMU_CLOCK_REALTIME) instead; same for
block/raw-posix.c and target-mips/kvm.c.

Paolo

Re: [Qemu-devel] [RFC PATCH v5 02/31] acpi: accurate overflow check

[Paolo Bonzini]

On 26/11/2014 11:38, Pavel Dovgalyuk wrote:
> Compare clock in ns, because acpi_pm_tmr_update uses rounded
> to ns value instead of ticks.
> 
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  hw/acpi/core.c |    7 +++++--
>  1 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/acpi/core.c b/hw/acpi/core.c
> index a7368fb..51913d6 100644
> --- a/hw/acpi/core.c
> +++ b/hw/acpi/core.c
> @@ -376,8 +376,11 @@ static void acpi_notify_wakeup(Notifier *notifier, void *data)
>  /* ACPI PM1a EVT */
>  uint16_t acpi_pm1_evt_get_sts(ACPIREGS *ar)
>  {
> -    int64_t d = acpi_pm_tmr_get_clock();
> -    if (d >= ar->tmr.overflow_time) {
> +    /* Compare ns-clock, not PM timer ticks, because
> +       acpi_pm_tmr_update function uses ns for setting the timer. */
> +    int64_t d = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
> +    if (d >= muldiv64(ar->tmr.overflow_time,
> +                      get_ticks_per_sec(), PM_TIMER_FREQUENCY)) {
>          ar->pm1.evt.sts |= ACPI_BITMASK_TIMER_STATUS;
>      }
>      return ar->pm1.evt.sts;

This is commit 3ef0eab178e5120a0e1c079d163d5c71689d9b71. :)

With it, Windows can boot in -icount mode (at least I tested XP).

Paolo

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Paolo Bonzini]

On 26/11/2014 11:38, Pavel Dovgalyuk wrote:
> This set of patches is related to the reverse execution and deterministic 
> replay of qemu execution  Our implementation of deterministic replay can 
> be used for deterministic and reverse debugging of guest code through gdb 
> remote interface.

Lots of progress!

I think these patches are now mergeable:

Paolo Bonzini (4):
      target-ppc: pass DisasContext to SPR generator functions
      cpu-exec: add a new CF_USE_ICOUNT cflag
      translate: check cflags instead of use_icount global
      gen-icount: check cflags instead of use_icount global

Pavel Dovgalyuk (6):
      cpu-exec: fix cpu_exec_nocache
      cpu-exec: reset exception_index correctly
      i386: do not cross the pages boundaries in replay mode
      cpu-exec: invalidate nocache translation if they are interrupted
      timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock
      cpus: make icount warp behave well with respect to stop/cont

(The last is "cpus: make icount warp deterministic in replay mode".

Alex, can you ACK "target-ppc: pass DisasContext to SPR generator
functions" please?  In general a review of the other three patches by me
would be welcome.

Thanks,

Paolo

Re: [Qemu-devel] [RFC PATCH v5 03/31] replay: global variables and function stubs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On 11/26/2014 03:39 AM, Pavel Dovgalyuk wrote:
> This patch adds global variables, defines, functions declarations,
> and function stubs for deterministic VM replay used by external modules.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---


> +# Since: 2.3
> +##
> +{ 'enum': 'ReplaySubmode',
> +  'data': [ 'unknown', 'normal' ] }
> diff --git a/replay/Makefile.objs b/replay/Makefile.objs

> +++ b/stubs/replay.c
> @@ -0,0 +1,8 @@
> +#include "replay/replay.h"
> +
> +ReplayMode replay_mode;
> +
> +ReplaySubmode replay_get_play_submode(void)
> +{
> +    return 0;

Although QMP code generation happens to assign 0 to the first listed
enum (in this case, REPLAY_SUBMODE_UNKNOWN), it is safer to explicitly
use an enum value rather than an open-coded '0' in this function.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> > + * @QEMU_CLOCK_VIRTUAL_RT: realtime clock used for icount warp
> > + *
> > + * This clock runs as a realtime clock, but is used for icount warp
> > + * and thus should be traced with record/replay to make warp function
> > + * behave deterministically.
> >   */
> 
> I think it should also stop/restart across "stop" and "cont" commands,
> similar to QEMU_CLOCK_VIRTUAL.  This is as simple as changing
> get_clock() to cpu_get_clock().
> 
> This way, QEMU_CLOCK_VIRTUAL_RT is "what QEMU_CLOCK_VIRTUAL does without
> -icount".  This makes a lot of sense and can be merged in 2.3
> independent of the rest of the series.

I've updated QEMU to master and started testing changed that you proposed.

And one more problem came out here. The problem is related to patch 60e68042cf70f271308dc6b4b22b609d054af929

It changes x86_cpu_has_work function. And this function instead of just checking the CPU state changes it:

-    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
-                                      CPU_INTERRUPT_POLL)) &&
+#if !defined(CONFIG_USER_ONLY)
+    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
+        apic_poll_irq(cpu->apic_state);
+        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
+    }
+#endif
+
+    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&

These changes break the deterministic execution, because cpu_has_work() may be called at
any moment of the execution.

When POLL interrupt request is processed by x86_cpu_exec_interrupt function, as it were before,
everything is ok, because I ensure that these calls occur at the same moments in record/replay.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Artyom Tarasenko]

On Wed, Nov 26, 2014 at 11:47 AM, Pavel Dovgaluk
<Pavel.Dovgaluk@ispras.ru> wrote:
> That covermail was wrong. Here is the correct one:
>
> This set of patches is related to the reverse execution and deterministic
> replay of qemu execution  This implementation of deterministic replay can
> be used for deterministic debugging of guest code through gdb remote
> interface.
>
> These patches include only core function of the replay,
> excluding the support for replaying serial, audio, network, and USB devices'
> operations. Reverse debugging and monitor commands were also excluded to
> be submitted later as separate patches.
>
> Execution recording writes non-deterministic events log, which can be later
> used for replaying the execution anywhere and for unlimited number of times.
> It also supports checkpointing for faster rewinding during reverse debugging.
> Execution replaying reads the log and replays all non-deterministic events
> including external input, hardware clocks, and interrupts.
>
> Deterministic replay has the following features:
>  * Deterministically replays whole system execution and all contents of the memory,
>    state of the hadrware devices, clocks, and screen of the VM.
>  * Writes execution log into the file for latter replaying for multiple times
>    on different machines.
>  * Supports i386, x86_64, and ARM hardware platforms.

This patch series is really impressive, hats off!

I wonder what would have to be done to add support for SPARC-64? Since
has a PCI bus, I guess the support in atapi and usb devices is already
there. Is anything beyond translate.c has to be adjusted?

Artyom

>  * Performs deterministic replay of all operations with keyboard and mouse
>    input devices.
>  * Supports auto-checkpointing for convenient reverse debugging.
>
> Usage of the record/replay:
>  * First, record the execution, by adding the following string to the command line:
>    '-record fname=replay.bin -icount 7 -net none'. Block devices' images are not
>    actually changed in the recording mode, because all of the changes are
>    written to the temporary overlay file.
>  * Then you can replay it for the multiple times by using another command
>    line option: '-replay fname=replay.bin -icount 7 -net none'
>  * '-net none' option should also be specified if network replay patches
>    are not applied.
>
> Paper with short description of deterministic replay implementation:
> http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
>
> Modifications of qemu include:
>  * wrappers for clock and time functions to save their return values in the log
>  * saving different asynchronous events (e.g. system shutdown) into the log
>  * synchronization of the bottom halves execution
>  * synchronization of the threads from thread pool
>  * recording/replaying user input (mouse and keyboard)
>  * adding internal events for cpu and io synchronization
>
> v5 changes:
>  * Minor changes.
>  * Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
>  * Moved savevm-related code out of the core.
>  * Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
>  * Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
>  * Adopted Paolo's icount patches
>  * Fixed hardware interrupts replaying
>
> v4 changes:
>  * Updated block drivers to support new bdrv_open interface.
>  * Moved migration patches into separate series (as suggested by Paolo Bonzini)
>  * Fixed a bug in replay_break operation.
>  * Fixed rtl8139 migration for replay.
>  * Fixed 'period' parameter processing for record mode.
>  * Fixed bug in 'reverse-stepi' implementation.
>  * Fixed replay without making any snapshots (even the starting one).
>  * Moved core replay patches into the separate series.
>  * Fixed reverse step and reverse continue support.
>  * Fixed several bugs in icount subsystem.
>  * Reusing native qemu icount for replay instructions counting.
>  * Separated core patches into their own series.
>
> v3 changes:
>  * Fixed bug with replay of the aio write operations.
>  * Added virtual clock based on replay icount.
>  * Removed duplicated saving of interrupt_request CPU field.
>  * Fixed some coding style issues.
>  * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
>  * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
>  * Added description and example for replay-info command (as suggested by Eric Blake)
>  * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
>  * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
>  * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
>  * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
>  * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
>  * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)
>
> v2 changes:
>  * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
>  * Added QMP versions of replay commands (as suggested by Eric Blake)
>  * Removed some optional features of replay to make patches cleaner
>  * Minor changes and code cleanup were made
>
>
>



-- 
Regards,
Artyom Tarasenko

SPARC and PPC PReP under qemu blog: http://tyom.blogspot.com/search/label/qemu

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Paolo Bonzini]

On 27/11/2014 10:11, Pavel Dovgaluk wrote:
> When POLL interrupt request is processed by x86_cpu_exec_interrupt function, as it were before,
> everything is ok, because I ensure that these calls occur at the same moments in record/replay.

Does this partial revert work?

Paolo

diff --git a/cpu-exec.c b/cpu-exec.c
index 3913de0..c976095 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -331,6 +331,12 @@ int cpu_exec(CPUArchState *env)
     volatile bool have_tb_lock = false;
 
     if (cpu->halted) {
+#ifdef TARGET_I386
+        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
+            apic_poll_irq(x86_cpu->apic_state);
+            cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
+        }
+#endif
         if (!cpu_has_work(cpu)) {
             return EXCP_HALTED;
         }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index e9df33e..3f13dfe 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2912,14 +2912,8 @@ static bool x86_cpu_has_work(CPUState *cs)
     X86CPU *cpu = X86_CPU(cs);
     CPUX86State *env = &cpu->env;
 
-#if !defined(CONFIG_USER_ONLY)
-    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
-        apic_poll_irq(cpu->apic_state);
-        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
-    }
-#endif
-
-    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&
+    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
+                                      CPU_INTERRUPT_POLL)) &&
             (env->eflags & IF_MASK)) ||
            (cs->interrupt_request & (CPU_INTERRUPT_NMI |
                                      CPU_INTERRUPT_INIT |

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Paolo Bonzini]

On 27/11/2014 10:53, Artyom Tarasenko wrote:
>> >
>> > Deterministic replay has the following features:
>> >  * Deterministically replays whole system execution and all contents of the memory,
>> >    state of the hadrware devices, clocks, and screen of the VM.
>> >  * Writes execution log into the file for latter replaying for multiple times
>> >    on different machines.
>> >  * Supports i386, x86_64, and ARM hardware platforms.
> This patch series is really impressive, hats off!
> 
> I wonder what would have to be done to add support for SPARC-64? Since
> has a PCI bus, I guess the support in atapi and usb devices is already
> there. Is anything beyond translate.c has to be adjusted?

Does -icount work reliably on SPARC?

Paolo

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Pavel Dovgaluk]

> From: Artyom Tarasenko [mailto:atar4qemu@gmail.com]
> On Wed, Nov 26, 2014 at 11:47 AM, Pavel Dovgaluk
> <Pavel.Dovgaluk@ispras.ru> wrote:
> > That covermail was wrong. Here is the correct one:
> >
> > This set of patches is related to the reverse execution and deterministic
> > replay of qemu execution  This implementation of deterministic replay can
> > be used for deterministic debugging of guest code through gdb remote
> > interface.
> >
> > These patches include only core function of the replay,
> > excluding the support for replaying serial, audio, network, and USB devices'
> > operations. Reverse debugging and monitor commands were also excluded to
> > be submitted later as separate patches.
> >
> > Execution recording writes non-deterministic events log, which can be later
> > used for replaying the execution anywhere and for unlimited number of times.
> > It also supports checkpointing for faster rewinding during reverse debugging.
> > Execution replaying reads the log and replays all non-deterministic events
> > including external input, hardware clocks, and interrupts.
> >
> > Deterministic replay has the following features:
> >  * Deterministically replays whole system execution and all contents of the memory,
> >    state of the hadrware devices, clocks, and screen of the VM.
> >  * Writes execution log into the file for latter replaying for multiple times
> >    on different machines.
> >  * Supports i386, x86_64, and ARM hardware platforms.
> 
> This patch series is really impressive, hats off!
> 
> I wonder what would have to be done to add support for SPARC-64? Since
> has a PCI bus, I guess the support in atapi and usb devices is already
> there. Is anything beyond translate.c has to be adjusted?

This implementation of replay does not requires changing the translation.
But there could be some issues anyway:
 - icount can have some bugs for this CPU
 - there could be translation non-determinism when it crosses pages boundary
 - virtual devices can create non-synchronized bottom halves that will bring non-determinism
 - clocks and timers can be incorrectly used by virtual devices

It will be great if you can try replay for SPARC and report about your experience.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 27/11/2014 10:11, Pavel Dovgaluk wrote:
> > When POLL interrupt request is processed by x86_cpu_exec_interrupt function, as it were
> before,
> > everything is ok, because I ensure that these calls occur at the same moments in
> record/replay.
> 
> Does this partial revert work?

This one is better because I can place synchronization point for replay here.

Pavel Dovgalyuk

> 
> Paolo
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 3913de0..c976095 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -331,6 +331,12 @@ int cpu_exec(CPUArchState *env)
>      volatile bool have_tb_lock = false;
> 
>      if (cpu->halted) {
> +#ifdef TARGET_I386
> +        if (cpu->interrupt_request & CPU_INTERRUPT_POLL) {
> +            apic_poll_irq(x86_cpu->apic_state);
> +            cpu_reset_interrupt(cpu, CPU_INTERRUPT_POLL);
> +        }
> +#endif
>          if (!cpu_has_work(cpu)) {
>              return EXCP_HALTED;
>          }
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index e9df33e..3f13dfe 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -2912,14 +2912,8 @@ static bool x86_cpu_has_work(CPUState *cs)
>      X86CPU *cpu = X86_CPU(cs);
>      CPUX86State *env = &cpu->env;
> 
> -#if !defined(CONFIG_USER_ONLY)
> -    if (cs->interrupt_request & CPU_INTERRUPT_POLL) {
> -        apic_poll_irq(cpu->apic_state);
> -        cpu_reset_interrupt(cs, CPU_INTERRUPT_POLL);
> -    }
> -#endif
> -
> -    return ((cs->interrupt_request & CPU_INTERRUPT_HARD) &&
> +    return ((cs->interrupt_request & (CPU_INTERRUPT_HARD |
> +                                      CPU_INTERRUPT_POLL)) &&
>              (env->eflags & IF_MASK)) ||
>             (cs->interrupt_request & (CPU_INTERRUPT_NMI |
>                                       CPU_INTERRUPT_INIT |

Re: [Qemu-devel] [RFC PATCH v5 00/31] Deterministic replay and reverse execution

[Artyom Tarasenko]

On Thu, Nov 27, 2014 at 6:45 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 27/11/2014 10:53, Artyom Tarasenko wrote:
>>> >
>>> > Deterministic replay has the following features:
>>> >  * Deterministically replays whole system execution and all contents of the memory,
>>> >    state of the hadrware devices, clocks, and screen of the VM.
>>> >  * Writes execution log into the file for latter replaying for multiple times
>>> >    on different machines.
>>> >  * Supports i386, x86_64, and ARM hardware platforms.
>> This patch series is really impressive, hats off!
>>
>> I wonder what would have to be done to add support for SPARC-64? Since
>> has a PCI bus, I guess the support in atapi and usb devices is already
>> there. Is anything beyond translate.c has to be adjusted?
>
> Does -icount work reliably on SPARC?

No. Currently using -icount 1 or -icount 2 is reliably producing the
"Bad clock read"  message, and using -icount 20
reliably crashes qemu with the "qemu: fatal: Raised interrupt while
not in I/O function" message.
So, I guess I see the starting point...

Artyom

-- 
Regards,
Artyom Tarasenko

SPARC and PPC PReP under qemu blog: http://tyom.blogspot.com/search/label/qemu

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 26/11/2014 11:40, Pavel Dovgalyuk wrote:
> > This patch introduces new QEMU_CLOCK_VIRTUAL_RT clock, which
> > should be used for icount warping. Separate timer is needed
> > for replaying the execution, because warping callbacks should
> > be deterministic. We cannot make realtime clock deterministic
> > because it is used for screen updates and other simulator-specific
> > actions. That is why we added new clock which is recorded and
> > replayed when needed.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  include/qemu/timer.h |    7 +++++++
> >  qemu-timer.c         |    2 ++
> >  replay/replay.h      |    4 +++-
> >  3 files changed, 12 insertions(+), 1 deletions(-)
> >
> > diff --git a/include/qemu/timer.h b/include/qemu/timer.h
> > index 7b43331..df27157 100644
> > --- a/include/qemu/timer.h
> > +++ b/include/qemu/timer.h
> > @@ -37,12 +37,19 @@
> >   * is suspended, and it will reflect system time changes the host may
> >   * undergo (e.g. due to NTP). The host clock has the same precision as
> >   * the virtual clock.
> > + *
> > + * @QEMU_CLOCK_VIRTUAL_RT: realtime clock used for icount warp
> > + *
> > + * This clock runs as a realtime clock, but is used for icount warp
> > + * and thus should be traced with record/replay to make warp function
> > + * behave deterministically.
> >   */
> 
> I think it should also stop/restart across "stop" and "cont" commands,
> similar to QEMU_CLOCK_VIRTUAL.  This is as simple as changing
> get_clock() to cpu_get_clock().

Not so easy :)
cpu_get_clock() checks vm_clock_seqlock which is locked in icount_warp_rt().
And after locking it requests the value of QEMU_CLOCK_VIRTUAL_RT:

    seqlock_write_lock(&timers_state.vm_clock_seqlock);
    if (runstate_is_running()) {
        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 22/31] timer: introduce new QEMU_CLOCK_VIRTUAL_RT clock

[Paolo Bonzini]

On 28/11/2014 12:28, Pavel Dovgaluk wrote:
> Not so easy :)
> cpu_get_clock() checks vm_clock_seqlock which is locked in icount_warp_rt().
> And after locking it requests the value of QEMU_CLOCK_VIRTUAL_RT:
> 
>     seqlock_write_lock(&timers_state.vm_clock_seqlock);
>     if (runstate_is_running()) {
>         int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL_RT);

Not so hard :D  You can use cpu_get_clock_locked() there.

In fact, cpu_get_clock_locked() is already used below in the "if", so we
can reuse "clock" instead of the other variable "cur_time".

Paolo

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Paolo Bonzini]

On 26/11/2014 11:39, Pavel Dovgalyuk wrote:
> +int64_t cpu_get_instructions_counter(void)
> +{
> +    /* This function calls are synchnonized to timer changes,
> +       calling cpu_get_instructions_counter_locked without lock is safe */
> +    int64_t icount = timers_state.qemu_icount;
> +    CPUState *cpu = current_cpu;
> +
> +    if (cpu) {
> +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> +    }
> +    return icount;

Why do you need to do this if !cpu_can_do_io(cpu)?

Perhaps a better name for the functions is

- cpu_get_instructions_counter_locked -> cpu_get_icount_raw

- cpu_get_instructions_counter -> cpu_get_icount_raw_nocheck

This makes it clear that cpu_get_instructions_counter should raise
questions to a reviewer.

Paolo

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
> On 26/11/2014 11:39, Pavel Dovgalyuk wrote:
> > +int64_t cpu_get_instructions_counter(void)
> > +{
> > +    /* This function calls are synchnonized to timer changes,
> > +       calling cpu_get_instructions_counter_locked without lock is safe */
> > +    int64_t icount = timers_state.qemu_icount;
> > +    CPUState *cpu = current_cpu;
> > +
> > +    if (cpu) {
> > +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> > +    }
> > +    return icount;
> 
> Why do you need to do this if !cpu_can_do_io(cpu)?

We save number of executed instruction when saving interrupt or exception event.
It leads to the call of cpu_get_instructions_counter() from cpu_exec function
(through several replay functions). It is correct (because no block is executing
at that moment) but is different to prior usage of icount requests.

> Perhaps a better name for the functions is
> 
> - cpu_get_instructions_counter_locked -> cpu_get_icount_raw
> 
> - cpu_get_instructions_counter -> cpu_get_icount_raw_nocheck

Ok, I'll rename these functions.


Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Paolo Bonzini]

On 04/12/2014 12:02, Pavel Dovgaluk wrote:
>> > Why do you need to do this if !cpu_can_do_io(cpu)?
> We save number of executed instruction when saving interrupt or exception event.
> It leads to the call of cpu_get_instructions_counter() from cpu_exec function
> (through several replay functions). It is correct (because no block is executing
> at that moment) but is different to prior usage of icount requests.

Why is !cpu_can_do_io(cpu) if no block is executing?

I'm not saying the function is wrong, just that it warrants a more
thorough review.

Paolo

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 04/12/2014 12:02, Pavel Dovgaluk wrote:
> >> > Why do you need to do this if !cpu_can_do_io(cpu)?
> > We save number of executed instruction when saving interrupt or exception event.
> > It leads to the call of cpu_get_instructions_counter() from cpu_exec function
> > (through several replay functions). It is correct (because no block is executing
> > at that moment) but is different to prior usage of icount requests.
> 
> Why is !cpu_can_do_io(cpu) if no block is executing?

Because it returns cpu->can_do_io which is equal to zero at that moment.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Paolo Bonzini]

On 05/12/2014 06:34, Pavel Dovgaluk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 04/12/2014 12:02, Pavel Dovgaluk wrote:
>>>>> Why do you need to do this if !cpu_can_do_io(cpu)?
>>> We save number of executed instruction when saving interrupt or exception event.
>>> It leads to the call of cpu_get_instructions_counter() from cpu_exec function
>>> (through several replay functions). It is correct (because no block is executing
>>> at that moment) but is different to prior usage of icount requests.
>>
>> Why is !cpu_can_do_io(cpu) if no block is executing?
> 
> Because it returns cpu->can_do_io which is equal to zero at that moment.

And why is can_do_io zero? :)  Is the fix to move the place where
can_do_io becomes nonzero?

Paolo

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 05/12/2014 06:34, Pavel Dovgaluk wrote:
> >> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> >> On 04/12/2014 12:02, Pavel Dovgaluk wrote:
> >>>>> Why do you need to do this if !cpu_can_do_io(cpu)?
> >>> We save number of executed instruction when saving interrupt or exception event.
> >>> It leads to the call of cpu_get_instructions_counter() from cpu_exec function
> >>> (through several replay functions). It is correct (because no block is executing
> >>> at that moment) but is different to prior usage of icount requests.
> >>
> >> Why is !cpu_can_do_io(cpu) if no block is executing?
> >
> > Because it returns cpu->can_do_io which is equal to zero at that moment.
> 
> And why is can_do_io zero? :)  Is the fix to move the place where
> can_do_io becomes nonzero?

can_do_io is set by gen_io_start function.
As I understand, it is used to protect determinism in icount mode,
because it allows non-deterministic (port io, raising interrupt)
operations only at the end of the translation blocks.
When someone tries to use MMIO in the middle of TB, that TB is
recompiled to place this instruction at the end of the block.

Do you mean that we can set can_do_io before execution of the block
and reset it at the beginning of the execution?

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Paolo Bonzini]

On 05/12/2014 11:55, Pavel Dovgaluk wrote:
>> > 
>> > And why is can_do_io zero? :)  Is the fix to move the place where
>> > can_do_io becomes nonzero?
> can_do_io is set by gen_io_start function.
> As I understand, it is used to protect determinism in icount mode,
> because it allows non-deterministic (port io, raising interrupt)
> operations only at the end of the translation blocks.
> When someone tries to use MMIO in the middle of TB, that TB is
> recompiled to place this instruction at the end of the block.
> 
> Do you mean that we can set can_do_io before execution of the block
> and reset it at the beginning of the execution?

Yes, we could try setting it after execution of the block and clearing
it afterwards.  Peter knows that part of icount better though (I know
mostly the timer/warping parts).

Paolo

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 05/12/2014 11:55, Pavel Dovgaluk wrote:
> >> >
> >> > And why is can_do_io zero? :)  Is the fix to move the place where
> >> > can_do_io becomes nonzero?
> > can_do_io is set by gen_io_start function.
> > As I understand, it is used to protect determinism in icount mode,
> > because it allows non-deterministic (port io, raising interrupt)
> > operations only at the end of the translation blocks.
> > When someone tries to use MMIO in the middle of TB, that TB is
> > recompiled to place this instruction at the end of the block.
> >
> > Do you mean that we can set can_do_io before execution of the block
> > and reset it at the beginning of the execution?
> 
> Yes, we could try setting it after execution of the block and clearing
> it afterwards.  Peter knows that part of icount better though (I know
> mostly the timer/warping parts).

Ok, how about these changes?

diff --git a/cpu-exec.c b/cpu-exec.c
index f52f292..88675ca 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -168,7 +168,9 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
     }
 #endif /* DEBUG_DISAS */
 
+    cpu->can_do_io = 0;
     next_tb = tcg_qemu_tb_exec(env, tb_ptr);
+    cpu->can_do_io = 1;
     trace_exec_tb_exit((void *) (next_tb & ~TB_EXIT_MASK),
                        next_tb & TB_EXIT_MASK);
 
@@ -548,6 +550,7 @@ int cpu_exec(CPUArchState *env)
             cpu = current_cpu;
             env = cpu->env_ptr;
             cc = CPU_GET_CLASS(cpu);
+            cpu->can_do_io = 1;
 #ifdef TARGET_I386
             x86_cpu = X86_CPU(cpu);
 #endif
diff --git a/cpus.c b/cpus.c
index 0c33458..7a45a51 100644
--- a/cpus.c
+++ b/cpus.c
@@ -934,6 +934,7 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
     qemu_mutex_lock(&qemu_global_mutex);
     qemu_thread_get_self(cpu->thread);
     cpu->thread_id = qemu_get_thread_id();
+    cpu->can_do_io = 1;
     current_cpu = cpu;
 
     r = kvm_init_vcpu(cpu);
@@ -974,6 +975,7 @@ static void *qemu_dummy_cpu_thread_fn(void *arg)
     qemu_mutex_lock_iothread();
     qemu_thread_get_self(cpu->thread);
     cpu->thread_id = qemu_get_thread_id();
+    cpu->can_do_io = 1;
 
     sigemptyset(&waitset);
     sigaddset(&waitset, SIG_IPI);
@@ -1016,6 +1018,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
     CPU_FOREACH(cpu) {
         cpu->thread_id = qemu_get_thread_id();
         cpu->created = true;
+        cpu->can_do_io = 1;
     }
     qemu_cond_signal(&qemu_cpu_cond);

Re: [Qemu-devel] [RFC PATCH v5 07/31] icount: implement icount requesting

[Paolo Bonzini]

On 05/12/2014 13:59, Pavel Dovgaluk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 05/12/2014 11:55, Pavel Dovgaluk wrote:
>>>>>
>>>>> And why is can_do_io zero? :)  Is the fix to move the place where
>>>>> can_do_io becomes nonzero?
>>> can_do_io is set by gen_io_start function.
>>> As I understand, it is used to protect determinism in icount mode,
>>> because it allows non-deterministic (port io, raising interrupt)
>>> operations only at the end of the translation blocks.
>>> When someone tries to use MMIO in the middle of TB, that TB is
>>> recompiled to place this instruction at the end of the block.
>>>
>>> Do you mean that we can set can_do_io before execution of the block
>>> and reset it at the beginning of the execution?
>>
>> Yes, we could try setting it after execution of the block and clearing
>> it afterwards.  Peter knows that part of icount better though (I know
>> mostly the timer/warping parts).
> 
> Ok, how about these changes?
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index f52f292..88675ca 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -168,7 +168,9 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
>      }
>  #endif /* DEBUG_DISAS */
>  
> +    cpu->can_do_io = 0;
>      next_tb = tcg_qemu_tb_exec(env, tb_ptr);
> +    cpu->can_do_io = 1;
>      trace_exec_tb_exit((void *) (next_tb & ~TB_EXIT_MASK),
>                         next_tb & TB_EXIT_MASK);
>  
> @@ -548,6 +550,7 @@ int cpu_exec(CPUArchState *env)
>              cpu = current_cpu;
>              env = cpu->env_ptr;
>              cc = CPU_GET_CLASS(cpu);
> +            cpu->can_do_io = 1;
>  #ifdef TARGET_I386
>              x86_cpu = X86_CPU(cpu);
>  #endif
> diff --git a/cpus.c b/cpus.c
> index 0c33458..7a45a51 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -934,6 +934,7 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
>      qemu_mutex_lock(&qemu_global_mutex);
>      qemu_thread_get_self(cpu->thread);
>      cpu->thread_id = qemu_get_thread_id();
> +    cpu->can_do_io = 1;
>      current_cpu = cpu;
>  
>      r = kvm_init_vcpu(cpu);
> @@ -974,6 +975,7 @@ static void *qemu_dummy_cpu_thread_fn(void *arg)
>      qemu_mutex_lock_iothread();
>      qemu_thread_get_self(cpu->thread);
>      cpu->thread_id = qemu_get_thread_id();
> +    cpu->can_do_io = 1;
>  
>      sigemptyset(&waitset);
>      sigaddset(&waitset, SIG_IPI);
> @@ -1016,6 +1018,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
>      CPU_FOREACH(cpu) {
>          cpu->thread_id = qemu_get_thread_id();
>          cpu->created = true;
> +        cpu->can_do_io = 1;
>      }
>      qemu_cond_signal(&qemu_cpu_cond);

Yes, this would work too, thanks for trying it!

Paolo