[PATCH v3 0/9] Many improvements to HVF memory-related codes

[PATCH v3 0/9] Many improvements to HVF memory-related codes

[Yan-Jie Wang]

changes in v3:
* Fix last two patches which contain mistakes:
 - hvf: only consider directly writeable memory regions for
    dirty-tracking
 - hvf: remove the need to lookup memory slots when clearing dirty-bits

changes in v2:
* Rebase to the current master.
* Correct a mistake in "hvf: simplify data structures and codes of
  memory related functions" patch
* add two patches for HVF memory listener. The changes are
 - only consider directly writeable memory regions for dirty-tracking
 - in `hvf_log_clear`, use provided `section` (MemoryRegionSection)
   from the caller to determine the pages that need to write-protected
   instead of calling hvf_set_dirty_tracking to write-protect the memory
   slots that contains the pages whose dirty-bits are cleared.

----------

I recently bought a Mac with M1 Pro chip, and use QEMU to setup a Linux
virtual machine.  QEMU crashed when I started a VM with HVF accelerator
enabled and with the device, bochs-display, added.

After digging into the source code, I found that dirty-tracking in HVF
did not work properly, which made QEMU crashed. Therefore I made this
series of patches to fix the problem.

Followings are the summary of the changes that these patches make:
 1. Move HVF memory-related functions and codes into a new file
    hvf-mem.c
 2. Simplify the logics of adding and removing memory regions in HVF
    memory listener
 3. Fix HVF dirty-tracking logics for both Intel and Apple Silicon Macs
 4. Use GTree and dynamically-allocated structures to store HVF memory
    slots instead of fixed-size arrays. This makes memory slots more
    scalable. It is inspired by the recent changes in Linux kernel
    (v5.17) that use red-black trees instead of arrays to store
    in-kernel KVM memory slots.
 5. Add a lock to protect the data structures of HVF memory slots

Patches have been tested on Apple Silicon Macs and Intel Macs.

Yan-Jie Wang (9):
  hvf: move memory related functions from hvf-accel-ops.c to hvf-mem.c
  hvf: simplify data structures and codes of memory related functions
  hvf: use correct data types for addresses in memory related functions
  hvf: rename struct hvf_slot to HVFSlot
  hvf: fix memory dirty-tracking
  hvf: add a lock for memory related functions
  hvf: use GTree to store memory slots instead of fixed-size array
  hvf: only consider directly writeable memory regions for
    dirty-tracking
  hvf: remove the need to lookup memory slots when clearing dirty-bits

 accel/hvf/hvf-accel-ops.c | 221 +-----------------------
 accel/hvf/hvf-mem.c       | 343 ++++++++++++++++++++++++++++++++++++++
 accel/hvf/meson.build     |   1 +
 include/sysemu/hvf_int.h  |  18 +-
 target/arm/hvf/hvf.c      |   5 +
 target/i386/hvf/hvf.c     |  25 +--
 6 files changed, 359 insertions(+), 254 deletions(-)
 create mode 100644 accel/hvf/hvf-mem.c

-- 
2.32.0 (Apple Git-132)

[PATCH v3 1/9] hvf: move memory related functions from hvf-accel-ops.c to hvf-mem.c

[Yan-Jie Wang]

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-accel-ops.c | 220 +--------------------------------
 accel/hvf/hvf-mem.c       | 252 ++++++++++++++++++++++++++++++++++++++
 accel/hvf/meson.build     |   1 +
 include/sysemu/hvf_int.h  |   2 +
 4 files changed, 256 insertions(+), 219 deletions(-)
 create mode 100644 accel/hvf/hvf-mem.c

diff --git a/accel/hvf/hvf-accel-ops.c b/accel/hvf/hvf-accel-ops.c
index 54457c76c2..50a563bfe0 100644
--- a/accel/hvf/hvf-accel-ops.c
+++ b/accel/hvf/hvf-accel-ops.c
@@ -48,7 +48,6 @@
  */
 
 #include "qemu/osdep.h"
-#include "qemu/error-report.h"
 #include "qemu/main-loop.h"
 #include "exec/address-spaces.h"
 #include "exec/exec-all.h"
@@ -64,143 +63,6 @@ HVFState *hvf_state;
 #define HV_VM_DEFAULT NULL
 #endif
 
-/* Memory slots */
-
-hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
-{
-    hvf_slot *slot;
-    int x;
-    for (x = 0; x < hvf_state->num_slots; ++x) {
-        slot = &hvf_state->slots[x];
-        if (slot->size && start < (slot->start + slot->size) &&
-            (start + size) > slot->start) {
-            return slot;
-        }
-    }
-    return NULL;
-}
-
-struct mac_slot {
-    int present;
-    uint64_t size;
-    uint64_t gpa_start;
-    uint64_t gva;
-};
-
-struct mac_slot mac_slots[32];
-
-static int do_hvf_set_memory(hvf_slot *slot, hv_memory_flags_t flags)
-{
-    struct mac_slot *macslot;
-    hv_return_t ret;
-
-    macslot = &mac_slots[slot->slot_id];
-
-    if (macslot->present) {
-        if (macslot->size != slot->size) {
-            macslot->present = 0;
-            ret = hv_vm_unmap(macslot->gpa_start, macslot->size);
-            assert_hvf_ok(ret);
-        }
-    }
-
-    if (!slot->size) {
-        return 0;
-    }
-
-    macslot->present = 1;
-    macslot->gpa_start = slot->start;
-    macslot->size = slot->size;
-    ret = hv_vm_map(slot->mem, slot->start, slot->size, flags);
-    assert_hvf_ok(ret);
-    return 0;
-}
-
-static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
-{
-    hvf_slot *mem;
-    MemoryRegion *area = section->mr;
-    bool writeable = !area->readonly && !area->rom_device;
-    hv_memory_flags_t flags;
-    uint64_t page_size = qemu_real_host_page_size;
-
-    if (!memory_region_is_ram(area)) {
-        if (writeable) {
-            return;
-        } else if (!memory_region_is_romd(area)) {
-            /*
-             * If the memory device is not in romd_mode, then we actually want
-             * to remove the hvf memory slot so all accesses will trap.
-             */
-             add = false;
-        }
-    }
-
-    if (!QEMU_IS_ALIGNED(int128_get64(section->size), page_size) ||
-        !QEMU_IS_ALIGNED(section->offset_within_address_space, page_size)) {
-        /* Not page aligned, so we can not map as RAM */
-        add = false;
-    }
-
-    mem = hvf_find_overlap_slot(
-            section->offset_within_address_space,
-            int128_get64(section->size));
-
-    if (mem && add) {
-        if (mem->size == int128_get64(section->size) &&
-            mem->start == section->offset_within_address_space &&
-            mem->mem == (memory_region_get_ram_ptr(area) +
-            section->offset_within_region)) {
-            return; /* Same region was attempted to register, go away. */
-        }
-    }
-
-    /* Region needs to be reset. set the size to 0 and remap it. */
-    if (mem) {
-        mem->size = 0;
-        if (do_hvf_set_memory(mem, 0)) {
-            error_report("Failed to reset overlapping slot");
-            abort();
-        }
-    }
-
-    if (!add) {
-        return;
-    }
-
-    if (area->readonly ||
-        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
-        flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
-    } else {
-        flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
-    }
-
-    /* Now make a new slot. */
-    int x;
-
-    for (x = 0; x < hvf_state->num_slots; ++x) {
-        mem = &hvf_state->slots[x];
-        if (!mem->size) {
-            break;
-        }
-    }
-
-    if (x == hvf_state->num_slots) {
-        error_report("No free slots");
-        abort();
-    }
-
-    mem->size = int128_get64(section->size);
-    mem->mem = memory_region_get_ram_ptr(area) + section->offset_within_region;
-    mem->start = section->offset_within_address_space;
-    mem->region = area;
-
-    if (do_hvf_set_memory(mem, flags)) {
-        error_report("Error registering new memory slot");
-        abort();
-    }
-}
-
 static void do_hvf_cpu_synchronize_state(CPUState *cpu, run_on_cpu_data arg)
 {
     if (!cpu->vcpu_dirty) {
@@ -238,79 +100,6 @@ static void hvf_cpu_synchronize_pre_loadvm(CPUState *cpu)
     run_on_cpu(cpu, do_hvf_cpu_synchronize_set_dirty, RUN_ON_CPU_NULL);
 }
 
-static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
-{
-    hvf_slot *slot;
-
-    slot = hvf_find_overlap_slot(
-            section->offset_within_address_space,
-            int128_get64(section->size));
-
-    /* protect region against writes; begin tracking it */
-    if (on) {
-        slot->flags |= HVF_SLOT_LOG;
-        hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_EXEC);
-    /* stop tracking region*/
-    } else {
-        slot->flags &= ~HVF_SLOT_LOG;
-        hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
-                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
-    }
-}
-
-static void hvf_log_start(MemoryListener *listener,
-                          MemoryRegionSection *section, int old, int new)
-{
-    if (old != 0) {
-        return;
-    }
-
-    hvf_set_dirty_tracking(section, 1);
-}
-
-static void hvf_log_stop(MemoryListener *listener,
-                         MemoryRegionSection *section, int old, int new)
-{
-    if (new != 0) {
-        return;
-    }
-
-    hvf_set_dirty_tracking(section, 0);
-}
-
-static void hvf_log_sync(MemoryListener *listener,
-                         MemoryRegionSection *section)
-{
-    /*
-     * sync of dirty pages is handled elsewhere; just make sure we keep
-     * tracking the region.
-     */
-    hvf_set_dirty_tracking(section, 1);
-}
-
-static void hvf_region_add(MemoryListener *listener,
-                           MemoryRegionSection *section)
-{
-    hvf_set_phys_mem(section, true);
-}
-
-static void hvf_region_del(MemoryListener *listener,
-                           MemoryRegionSection *section)
-{
-    hvf_set_phys_mem(section, false);
-}
-
-static MemoryListener hvf_memory_listener = {
-    .name = "hvf",
-    .priority = 10,
-    .region_add = hvf_region_add,
-    .region_del = hvf_region_del,
-    .log_start = hvf_log_start,
-    .log_stop = hvf_log_stop,
-    .log_sync = hvf_log_sync,
-};
-
 static void dummy_signal(int sig)
 {
 }
@@ -319,7 +108,6 @@ bool hvf_allowed;
 
 static int hvf_accel_init(MachineState *ms)
 {
-    int x;
     hv_return_t ret;
     HVFState *s;
 
@@ -328,14 +116,8 @@ static int hvf_accel_init(MachineState *ms)
 
     s = g_new0(HVFState, 1);
 
-    s->num_slots = ARRAY_SIZE(s->slots);
-    for (x = 0; x < s->num_slots; ++x) {
-        s->slots[x].size = 0;
-        s->slots[x].slot_id = x;
-    }
-
     hvf_state = s;
-    memory_listener_register(&hvf_memory_listener, &address_space_memory);
+    hvf_init_memslots();
 
     return hvf_arch_init();
 }
diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
new file mode 100644
index 0000000000..3712731ed9
--- /dev/null
+++ b/accel/hvf/hvf-mem.c
@@ -0,0 +1,252 @@
+/*
+ * Copyright 2008 IBM Corporation
+ *           2008 Red Hat, Inc.
+ * Copyright 2011 Intel Corporation
+ * Copyright 2016 Veertu, Inc.
+ * Copyright 2017 The Android Open Source Project
+ *
+ * QEMU Hypervisor.framework support
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/error-report.h"
+#include "exec/address-spaces.h"
+#include "sysemu/hvf.h"
+#include "sysemu/hvf_int.h"
+
+/* Memory slots */
+
+hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
+{
+    hvf_slot *slot;
+    int x;
+    for (x = 0; x < hvf_state->num_slots; ++x) {
+        slot = &hvf_state->slots[x];
+        if (slot->size && start < (slot->start + slot->size) &&
+            (start + size) > slot->start) {
+            return slot;
+        }
+    }
+    return NULL;
+}
+
+struct mac_slot {
+    int present;
+    uint64_t size;
+    uint64_t gpa_start;
+    uint64_t gva;
+};
+
+struct mac_slot mac_slots[32];
+
+static int do_hvf_set_memory(hvf_slot *slot, hv_memory_flags_t flags)
+{
+    struct mac_slot *macslot;
+    hv_return_t ret;
+
+    macslot = &mac_slots[slot->slot_id];
+
+    if (macslot->present) {
+        if (macslot->size != slot->size) {
+            macslot->present = 0;
+            ret = hv_vm_unmap(macslot->gpa_start, macslot->size);
+            assert_hvf_ok(ret);
+        }
+    }
+
+    if (!slot->size) {
+        return 0;
+    }
+
+    macslot->present = 1;
+    macslot->gpa_start = slot->start;
+    macslot->size = slot->size;
+    ret = hv_vm_map(slot->mem, slot->start, slot->size, flags);
+    assert_hvf_ok(ret);
+    return 0;
+}
+
+static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
+{
+    hvf_slot *mem;
+    MemoryRegion *area = section->mr;
+    bool writeable = !area->readonly && !area->rom_device;
+    hv_memory_flags_t flags;
+    uint64_t page_size = qemu_real_host_page_size;
+
+    if (!memory_region_is_ram(area)) {
+        if (writeable) {
+            return;
+        } else if (!memory_region_is_romd(area)) {
+            /*
+             * If the memory device is not in romd_mode, then we actually want
+             * to remove the hvf memory slot so all accesses will trap.
+             */
+             add = false;
+        }
+    }
+
+    if (!QEMU_IS_ALIGNED(int128_get64(section->size), page_size) ||
+        !QEMU_IS_ALIGNED(section->offset_within_address_space, page_size)) {
+        /* Not page aligned, so we can not map as RAM */
+        add = false;
+    }
+
+    mem = hvf_find_overlap_slot(
+            section->offset_within_address_space,
+            int128_get64(section->size));
+
+    if (mem && add) {
+        if (mem->size == int128_get64(section->size) &&
+            mem->start == section->offset_within_address_space &&
+            mem->mem == (memory_region_get_ram_ptr(area) +
+            section->offset_within_region)) {
+            return; /* Same region was attempted to register, go away. */
+        }
+    }
+
+    /* Region needs to be reset. set the size to 0 and remap it. */
+    if (mem) {
+        mem->size = 0;
+        if (do_hvf_set_memory(mem, 0)) {
+            error_report("Failed to reset overlapping slot");
+            abort();
+        }
+    }
+
+    if (!add) {
+        return;
+    }
+
+    if (area->readonly ||
+        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
+        flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
+    } else {
+        flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
+    }
+
+    /* Now make a new slot. */
+    int x;
+
+    for (x = 0; x < hvf_state->num_slots; ++x) {
+        mem = &hvf_state->slots[x];
+        if (!mem->size) {
+            break;
+        }
+    }
+
+    if (x == hvf_state->num_slots) {
+        error_report("No free slots");
+        abort();
+    }
+
+    mem->size = int128_get64(section->size);
+    mem->mem = memory_region_get_ram_ptr(area) + section->offset_within_region;
+    mem->start = section->offset_within_address_space;
+    mem->region = area;
+
+    if (do_hvf_set_memory(mem, flags)) {
+        error_report("Error registering new memory slot");
+        abort();
+    }
+}
+
+
+static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
+{
+    hvf_slot *slot;
+
+    slot = hvf_find_overlap_slot(
+            section->offset_within_address_space,
+            int128_get64(section->size));
+
+    /* protect region against writes; begin tracking it */
+    if (on) {
+        slot->flags |= HVF_SLOT_LOG;
+        hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
+                      HV_MEMORY_READ | HV_MEMORY_EXEC);
+    /* stop tracking region*/
+    } else {
+        slot->flags &= ~HVF_SLOT_LOG;
+        hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
+                      HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
+    }
+}
+
+static void hvf_log_start(MemoryListener *listener,
+                          MemoryRegionSection *section, int old, int new)
+{
+    if (old != 0) {
+        return;
+    }
+
+    hvf_set_dirty_tracking(section, 1);
+}
+
+static void hvf_log_stop(MemoryListener *listener,
+                         MemoryRegionSection *section, int old, int new)
+{
+    if (new != 0) {
+        return;
+    }
+
+    hvf_set_dirty_tracking(section, 0);
+}
+
+static void hvf_log_sync(MemoryListener *listener,
+                         MemoryRegionSection *section)
+{
+    /*
+     * sync of dirty pages is handled elsewhere; just make sure we keep
+     * tracking the region.
+     */
+    hvf_set_dirty_tracking(section, 1);
+}
+
+static void hvf_region_add(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    hvf_set_phys_mem(section, true);
+}
+
+static void hvf_region_del(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    hvf_set_phys_mem(section, false);
+}
+
+static MemoryListener hvf_memory_listener = {
+    .name = "hvf",
+    .priority = 10,
+    .region_add = hvf_region_add,
+    .region_del = hvf_region_del,
+    .log_start = hvf_log_start,
+    .log_stop = hvf_log_stop,
+    .log_sync = hvf_log_sync,
+};
+
+void hvf_init_memslots(void)
+{
+    int x;
+    HVFState *s = hvf_state;
+
+    s->num_slots = ARRAY_SIZE(s->slots);
+    for (x = 0; x < s->num_slots; ++x) {
+        s->slots[x].size = 0;
+        s->slots[x].slot_id = x;
+    }
+
+    memory_listener_register(&hvf_memory_listener, &address_space_memory);
+}
diff --git a/accel/hvf/meson.build b/accel/hvf/meson.build
index fc52cb7843..7e7a2034f1 100644
--- a/accel/hvf/meson.build
+++ b/accel/hvf/meson.build
@@ -2,6 +2,7 @@ hvf_ss = ss.source_set()
 hvf_ss.add(files(
   'hvf-all.c',
   'hvf-accel-ops.c',
+  'hvf-mem.c',
 ))
 
 specific_ss.add_all(when: 'CONFIG_HVF', if_true: hvf_ss)
diff --git a/include/sysemu/hvf_int.h b/include/sysemu/hvf_int.h
index 6545f7cd61..cef20d750d 100644
--- a/include/sysemu/hvf_int.h
+++ b/include/sysemu/hvf_int.h
@@ -65,4 +65,6 @@ int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
 void hvf_kick_vcpu_thread(CPUState *cpu);
 
+void hvf_init_memslots(void);
+
 #endif
-- 
2.32.0 (Apple Git-132)

[PATCH v3 2/9] hvf: simplify data structures and codes of memory related functions

[Yan-Jie Wang]

* Remove mac_slot and use hvf_slot only. The function of the two structures
  are similar.

* Refactor function hvf_set_phys_mem():
 - Remove unnecessary checks because any modified memory sections
   will be removed first (region_del called first) before being added.
   Therefore, new sections do not overlap with existing sections.
 - Try to align memory sections first before giving up sections that are not
   aligned to host page size.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-accel-ops.c |   1 -
 accel/hvf/hvf-mem.c       | 211 +++++++++++++++++++-------------------
 include/sysemu/hvf_int.h  |   8 +-
 3 files changed, 107 insertions(+), 113 deletions(-)

diff --git a/accel/hvf/hvf-accel-ops.c b/accel/hvf/hvf-accel-ops.c
index 50a563bfe0..c77f142f2b 100644
--- a/accel/hvf/hvf-accel-ops.c
+++ b/accel/hvf/hvf-accel-ops.c
@@ -49,7 +49,6 @@
 
 #include "qemu/osdep.h"
 #include "qemu/main-loop.h"
-#include "exec/address-spaces.h"
 #include "exec/exec-all.h"
 #include "sysemu/cpus.h"
 #include "sysemu/hvf.h"
diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 3712731ed9..32452696b6 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -28,12 +28,16 @@
 
 /* Memory slots */
 
+#define HVF_NUM_SLOTS 32
+
+static hvf_slot memslots[HVF_NUM_SLOTS];
+
 hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
 {
     hvf_slot *slot;
     int x;
-    for (x = 0; x < hvf_state->num_slots; ++x) {
-        slot = &hvf_state->slots[x];
+    for (x = 0; x < HVF_NUM_SLOTS; ++x) {
+        slot = &memslots[x];
         if (slot->size && start < (slot->start + slot->size) &&
             (start + size) > slot->start) {
             return slot;
@@ -42,128 +46,130 @@ hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
     return NULL;
 }
 
-struct mac_slot {
-    int present;
-    uint64_t size;
-    uint64_t gpa_start;
-    uint64_t gva;
-};
-
-struct mac_slot mac_slots[32];
-
-static int do_hvf_set_memory(hvf_slot *slot, hv_memory_flags_t flags)
+static hvf_slot *hvf_find_free_slot(void)
 {
-    struct mac_slot *macslot;
-    hv_return_t ret;
-
-    macslot = &mac_slots[slot->slot_id];
-
-    if (macslot->present) {
-        if (macslot->size != slot->size) {
-            macslot->present = 0;
-            ret = hv_vm_unmap(macslot->gpa_start, macslot->size);
-            assert_hvf_ok(ret);
+    hvf_slot *slot;
+    int x;
+    for (x = 0; x < HVF_NUM_SLOTS; x++) {
+        slot = &memslots[x];
+        if (!slot->size) {
+            return slot;
         }
     }
 
-    if (!slot->size) {
-        return 0;
-    }
-
-    macslot->present = 1;
-    macslot->gpa_start = slot->start;
-    macslot->size = slot->size;
-    ret = hv_vm_map(slot->mem, slot->start, slot->size, flags);
-    assert_hvf_ok(ret);
-    return 0;
+    return NULL;
+}
+
+/*
+ * Hypervisor.framework requires that the host virtual address,
+ * the guest physical address and the size of memory regions are aligned
+ * to the host page size.
+ *
+ * The function here tries to align the guest physical address and the size.
+ *
+ * The return value is the aligned size.
+ * The aligned guest physical address will be written to `start'.
+ * The delta between the aligned address and the original address will be
+ * written to `delta'.
+ */
+static hwaddr hvf_align_section(MemoryRegionSection *section,
+                              hwaddr *start, hwaddr *delta)
+{
+    hwaddr unaligned, _start, size, _delta;
+
+    unaligned = section->offset_within_address_space;
+    size = int128_get64(section->size);
+    _start = ROUND_UP(unaligned, qemu_real_host_page_size);
+    _delta = _start - unaligned;
+    size = (size - _delta) & qemu_real_host_page_mask;
+
+    *start = _start;
+    *delta = _delta;
+
+    return size;
 }
 
 static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
 {
-    hvf_slot *mem;
+    hvf_slot *slot;
+    hwaddr start, size, offset, delta;
+    uint8_t *host_addr;
     MemoryRegion *area = section->mr;
-    bool writeable = !area->readonly && !area->rom_device;
+    bool readonly, dirty_tracking;
     hv_memory_flags_t flags;
-    uint64_t page_size = qemu_real_host_page_size;
+    hv_return_t ret;
 
-    if (!memory_region_is_ram(area)) {
-        if (writeable) {
+    if (add && !memory_region_is_ram(area) && !memory_region_is_romd(area)) {
+        /*
+         * If the memory region is not RAM and is in ROMD mode,
+         * do not map it to the guest.
+         */
+        return;
+    }
+
+    size = hvf_align_section(section, &start, &delta);
+
+    if (!size) {
+        /* The size is 0 after aligned. Do not map the region */
+        return;
+    }
+
+    if (add) {
+        /* add memory region */
+        offset = section->offset_within_region + delta;
+        host_addr = memory_region_get_ram_ptr(area) + offset;
+
+        if (!QEMU_PTR_IS_ALIGNED(host_addr, qemu_real_host_page_size)) {
+            /* The host virtual address is not aligned. It cannot be mapped */
             return;
-        } else if (!memory_region_is_romd(area)) {
-            /*
-             * If the memory device is not in romd_mode, then we actually want
-             * to remove the hvf memory slot so all accesses will trap.
-             */
-             add = false;
         }
-    }
 
-    if (!QEMU_IS_ALIGNED(int128_get64(section->size), page_size) ||
-        !QEMU_IS_ALIGNED(section->offset_within_address_space, page_size)) {
-        /* Not page aligned, so we can not map as RAM */
-        add = false;
-    }
+        dirty_tracking = !!memory_region_get_dirty_log_mask(area);
+        readonly = memory_region_is_rom(area) || memory_region_is_romd(area);
 
-    mem = hvf_find_overlap_slot(
-            section->offset_within_address_space,
-            int128_get64(section->size));
-
-    if (mem && add) {
-        if (mem->size == int128_get64(section->size) &&
-            mem->start == section->offset_within_address_space &&
-            mem->mem == (memory_region_get_ram_ptr(area) +
-            section->offset_within_region)) {
-            return; /* Same region was attempted to register, go away. */
-        }
-    }
-
-    /* Region needs to be reset. set the size to 0 and remap it. */
-    if (mem) {
-        mem->size = 0;
-        if (do_hvf_set_memory(mem, 0)) {
-            error_report("Failed to reset overlapping slot");
+        /* setup a slot */
+        slot = hvf_find_free_slot();
+        if (!slot) {
+            error_report("No free slots");
             abort();
         }
-    }
 
-    if (!add) {
-        return;
-    }
+        slot->start = start;
+        slot->size = size;
+        slot->offset = offset;
+        slot->flags = 0;
+        slot->region = area;
 
-    if (area->readonly ||
-        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
-        flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
+        if (readonly) {
+            slot->flags |= HVF_SLOT_READONLY;
+        }
+
+        if (dirty_tracking) {
+            slot->flags |= HVF_SLOT_LOG;
+        }
+
+        /* set Hypervisor.framework memory mapping flags */
+        if (readonly || dirty_tracking) {
+            flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
+        } else {
+            flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
+        }
+
+        ret = hv_vm_map(host_addr, start, size, flags);
+        assert_hvf_ok(ret);
     } else {
-        flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
-    }
+        /* remove memory region */
+        slot = hvf_find_overlap_slot(start, size);
 
-    /* Now make a new slot. */
-    int x;
+        if (slot) {
+            ret = hv_vm_unmap(start, size);
+            assert_hvf_ok(ret);
 
-    for (x = 0; x < hvf_state->num_slots; ++x) {
-        mem = &hvf_state->slots[x];
-        if (!mem->size) {
-            break;
+            slot->size = 0;
         }
     }
-
-    if (x == hvf_state->num_slots) {
-        error_report("No free slots");
-        abort();
-    }
-
-    mem->size = int128_get64(section->size);
-    mem->mem = memory_region_get_ram_ptr(area) + section->offset_within_region;
-    mem->start = section->offset_within_address_space;
-    mem->region = area;
-
-    if (do_hvf_set_memory(mem, flags)) {
-        error_report("Error registering new memory slot");
-        abort();
-    }
 }
 
-
 static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
 {
     hvf_slot *slot;
@@ -239,14 +245,5 @@ static MemoryListener hvf_memory_listener = {
 
 void hvf_init_memslots(void)
 {
-    int x;
-    HVFState *s = hvf_state;
-
-    s->num_slots = ARRAY_SIZE(s->slots);
-    for (x = 0; x < s->num_slots; ++x) {
-        s->slots[x].size = 0;
-        s->slots[x].slot_id = x;
-    }
-
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
 }
diff --git a/include/sysemu/hvf_int.h b/include/sysemu/hvf_int.h
index cef20d750d..8ee31a16ac 100644
--- a/include/sysemu/hvf_int.h
+++ b/include/sysemu/hvf_int.h
@@ -19,12 +19,12 @@
 
 /* hvf_slot flags */
 #define HVF_SLOT_LOG (1 << 0)
+#define HVF_SLOT_READONLY (1 << 1)
 
 typedef struct hvf_slot {
     uint64_t start;
-    uint64_t size;
-    uint8_t *mem;
-    int slot_id;
+    uint64_t size;  /* 0 if the slot is free */
+    uint64_t offset;  /* offset within memory region */
     uint32_t flags;
     MemoryRegion *region;
 } hvf_slot;
@@ -40,8 +40,6 @@ typedef struct hvf_vcpu_caps {
 
 struct HVFState {
     AccelState parent;
-    hvf_slot slots[32];
-    int num_slots;
 
     hvf_vcpu_caps *hvf_caps;
     uint64_t vtimer_offset;
-- 
2.32.0 (Apple Git-132)

[PATCH v3 3/9] hvf: use correct data types for addresses in memory related functions

[Yan-Jie Wang]

Follow the QEMU coding style. Use hwaddr for guest physical address.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c      | 2 +-
 include/sysemu/hvf_int.h | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 32452696b6..6b82be3220 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -32,7 +32,7 @@
 
 static hvf_slot memslots[HVF_NUM_SLOTS];
 
-hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
+hvf_slot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
 {
     hvf_slot *slot;
     int x;
diff --git a/include/sysemu/hvf_int.h b/include/sysemu/hvf_int.h
index 8ee31a16ac..2c4a97debe 100644
--- a/include/sysemu/hvf_int.h
+++ b/include/sysemu/hvf_int.h
@@ -22,9 +22,9 @@
 #define HVF_SLOT_READONLY (1 << 1)
 
 typedef struct hvf_slot {
-    uint64_t start;
-    uint64_t size;  /* 0 if the slot is free */
-    uint64_t offset;  /* offset within memory region */
+    hwaddr start;
+    hwaddr size;  /* 0 if the slot is free */
+    hwaddr offset;  /* offset within memory region */
     uint32_t flags;
     MemoryRegion *region;
 } hvf_slot;
@@ -58,7 +58,7 @@ int hvf_arch_init(void);
 int hvf_arch_init_vcpu(CPUState *cpu);
 void hvf_arch_vcpu_destroy(CPUState *cpu);
 int hvf_vcpu_exec(CPUState *);
-hvf_slot *hvf_find_overlap_slot(uint64_t, uint64_t);
+hvf_slot *hvf_find_overlap_slot(hwaddr, hwaddr);
 int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
 void hvf_kick_vcpu_thread(CPUState *cpu);
-- 
2.32.0 (Apple Git-132)

[PATCH v3 4/9] hvf: rename struct hvf_slot to HVFSlot

[Yan-Jie Wang]

Follow the QEMU coding style. Structured type names are in CamelCase.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c      | 14 +++++++-------
 include/sysemu/hvf_int.h |  8 ++++----
 target/i386/hvf/hvf.c    |  4 ++--
 3 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 6b82be3220..b8e9f30e4c 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -30,11 +30,11 @@
 
 #define HVF_NUM_SLOTS 32
 
-static hvf_slot memslots[HVF_NUM_SLOTS];
+static HVFSlot memslots[HVF_NUM_SLOTS];
 
-hvf_slot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
+HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
 {
-    hvf_slot *slot;
+    HVFSlot *slot;
     int x;
     for (x = 0; x < HVF_NUM_SLOTS; ++x) {
         slot = &memslots[x];
@@ -46,9 +46,9 @@ hvf_slot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
     return NULL;
 }
 
-static hvf_slot *hvf_find_free_slot(void)
+static HVFSlot *hvf_find_free_slot(void)
 {
-    hvf_slot *slot;
+    HVFSlot *slot;
     int x;
     for (x = 0; x < HVF_NUM_SLOTS; x++) {
         slot = &memslots[x];
@@ -91,7 +91,7 @@ static hwaddr hvf_align_section(MemoryRegionSection *section,
 
 static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
 {
-    hvf_slot *slot;
+    HVFSlot *slot;
     hwaddr start, size, offset, delta;
     uint8_t *host_addr;
     MemoryRegion *area = section->mr;
@@ -172,7 +172,7 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
 
 static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
 {
-    hvf_slot *slot;
+    HVFSlot *slot;
 
     slot = hvf_find_overlap_slot(
             section->offset_within_address_space,
diff --git a/include/sysemu/hvf_int.h b/include/sysemu/hvf_int.h
index 2c4a97debe..0aafbc9357 100644
--- a/include/sysemu/hvf_int.h
+++ b/include/sysemu/hvf_int.h
@@ -17,17 +17,17 @@
 #include <Hypervisor/hv.h>
 #endif
 
-/* hvf_slot flags */
+/* HVFSlot flags */
 #define HVF_SLOT_LOG (1 << 0)
 #define HVF_SLOT_READONLY (1 << 1)
 
-typedef struct hvf_slot {
+typedef struct HVFSlot {
     hwaddr start;
     hwaddr size;  /* 0 if the slot is free */
     hwaddr offset;  /* offset within memory region */
     uint32_t flags;
     MemoryRegion *region;
-} hvf_slot;
+} HVFSlot;
 
 typedef struct hvf_vcpu_caps {
     uint64_t vmx_cap_pinbased;
@@ -58,7 +58,7 @@ int hvf_arch_init(void);
 int hvf_arch_init_vcpu(CPUState *cpu);
 void hvf_arch_vcpu_destroy(CPUState *cpu);
 int hvf_vcpu_exec(CPUState *);
-hvf_slot *hvf_find_overlap_slot(hwaddr, hwaddr);
+HVFSlot *hvf_find_overlap_slot(hwaddr, hwaddr);
 int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
 void hvf_kick_vcpu_thread(CPUState *cpu);
diff --git a/target/i386/hvf/hvf.c b/target/i386/hvf/hvf.c
index 4ba6e82fab..2ddb4fc825 100644
--- a/target/i386/hvf/hvf.c
+++ b/target/i386/hvf/hvf.c
@@ -113,7 +113,7 @@ void hvf_handle_io(CPUArchState *env, uint16_t port, void *buffer,
     }
 }
 
-static bool ept_emulation_fault(hvf_slot *slot, uint64_t gpa, uint64_t ept_qual)
+static bool ept_emulation_fault(HVFSlot *slot, uint64_t gpa, uint64_t ept_qual)
 {
     int read, write;
 
@@ -469,7 +469,7 @@ int hvf_vcpu_exec(CPUState *cpu)
         /* Need to check if MMIO or unmapped fault */
         case EXIT_REASON_EPT_FAULT:
         {
-            hvf_slot *slot;
+            HVFSlot *slot;
             uint64_t gpa = rvmcs(cpu->hvf->fd, VMCS_GUEST_PHYSICAL_ADDRESS);
 
             if (((idtvec_info & VMCS_IDT_VEC_VALID) == 0) &&
-- 
2.32.0 (Apple Git-132)

[PATCH v3 5/9] hvf: fix memory dirty-tracking

[Yan-Jie Wang]

Dirty-tracking in HVF is not properly implemented.

On Intel Macs, Ubuntu ISO boot menu does not show properly.

On Apple Silicon, using bochs-display may cause the guest crashes because
the guest may uses load/store instructions on framebuffer which causes
vmexits and the exception register does not contain enough information
(ESR_EL2.ISV = 0) for QEMU to emulate the memory operation.

The strategy to log the dirty pages is to write-protect the memory regions
that are being dirty-tracked.

When the guest is trapped to the host because of memory write, check whether
the address being written is being dirty-tracked.

If it is being dirty-tracked, restore the write permission of the page and
mark the accessed page dirty, and resume the guest without increasing
program counter, and then the same instruction will be execute again.

This patch fixes the problem and make the dirty-tracking work properly.

Buglink: https://bugs.launchpad.net/qemu/+bug/1827005
Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c      | 62 ++++++++++++++++++++++++++++++++++++----
 include/sysemu/hvf_int.h | 14 +--------
 target/arm/hvf/hvf.c     |  5 ++++
 target/i386/hvf/hvf.c    | 25 ++++------------
 4 files changed, 68 insertions(+), 38 deletions(-)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index b8e9f30e4c..896e718374 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -30,9 +30,21 @@
 
 #define HVF_NUM_SLOTS 32
 
+/* HVFSlot flags */
+#define HVF_SLOT_LOG (1 << 0)
+#define HVF_SLOT_READONLY (1 << 1)
+
+typedef struct HVFSlot {
+    hwaddr start;
+    hwaddr size;  /* 0 if the slot is free */
+    hwaddr offset;  /* offset within memory region */
+    uint32_t flags;
+    MemoryRegion *region;
+} HVFSlot;
+
 static HVFSlot memslots[HVF_NUM_SLOTS];
 
-HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
+static HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
 {
     HVFSlot *slot;
     int x;
@@ -194,7 +206,7 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
 static void hvf_log_start(MemoryListener *listener,
                           MemoryRegionSection *section, int old, int new)
 {
-    if (old != 0) {
+    if (old == new) {
         return;
     }
 
@@ -211,12 +223,12 @@ static void hvf_log_stop(MemoryListener *listener,
     hvf_set_dirty_tracking(section, 0);
 }
 
-static void hvf_log_sync(MemoryListener *listener,
+static void hvf_log_clear(MemoryListener *listener,
                          MemoryRegionSection *section)
 {
     /*
-     * sync of dirty pages is handled elsewhere; just make sure we keep
-     * tracking the region.
+     * The dirty bits are being cleared.
+     * Make the section write-protected again.
      */
     hvf_set_dirty_tracking(section, 1);
 }
@@ -240,9 +252,47 @@ static MemoryListener hvf_memory_listener = {
     .region_del = hvf_region_del,
     .log_start = hvf_log_start,
     .log_stop = hvf_log_stop,
-    .log_sync = hvf_log_sync,
+    .log_clear = hvf_log_clear,
 };
 
+
+/*
+ * The function is called when the guest is accessing memory causing vmexit.
+ * Check whether the guest can access the memory directly and
+ * also mark the accessed page being written dirty
+ * if the page is being dirty-tracked.
+ *
+ * Return true if the access is within the mapped region,
+ * otherwise return false.
+ */
+bool hvf_access_memory(hwaddr address, bool write)
+{
+    HVFSlot *slot;
+    hv_return_t ret;
+    hwaddr start, size;
+
+    slot = hvf_find_overlap_slot(address, 1);
+
+    if (!slot || (write && slot->flags & HVF_SLOT_READONLY)) {
+        /* MMIO or unmapped area, return false */
+        return false;
+    }
+
+    if (write && (slot->flags & HVF_SLOT_LOG)) {
+        /* The slot is being dirty-tracked. Mark the accessed page dirty. */
+        start = address & qemu_real_host_page_mask;
+        size = qemu_real_host_page_size;
+
+        memory_region_set_dirty(slot->region,
+                                start - slot->start + slot->offset, size);
+        ret = hv_vm_protect(start, size,
+                    HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
+        assert_hvf_ok(ret);
+    }
+
+    return true;
+}
+
 void hvf_init_memslots(void)
 {
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
diff --git a/include/sysemu/hvf_int.h b/include/sysemu/hvf_int.h
index 0aafbc9357..16e5faf0ff 100644
--- a/include/sysemu/hvf_int.h
+++ b/include/sysemu/hvf_int.h
@@ -17,18 +17,6 @@
 #include <Hypervisor/hv.h>
 #endif
 
-/* HVFSlot flags */
-#define HVF_SLOT_LOG (1 << 0)
-#define HVF_SLOT_READONLY (1 << 1)
-
-typedef struct HVFSlot {
-    hwaddr start;
-    hwaddr size;  /* 0 if the slot is free */
-    hwaddr offset;  /* offset within memory region */
-    uint32_t flags;
-    MemoryRegion *region;
-} HVFSlot;
-
 typedef struct hvf_vcpu_caps {
     uint64_t vmx_cap_pinbased;
     uint64_t vmx_cap_procbased;
@@ -58,11 +46,11 @@ int hvf_arch_init(void);
 int hvf_arch_init_vcpu(CPUState *cpu);
 void hvf_arch_vcpu_destroy(CPUState *cpu);
 int hvf_vcpu_exec(CPUState *);
-HVFSlot *hvf_find_overlap_slot(hwaddr, hwaddr);
 int hvf_put_registers(CPUState *);
 int hvf_get_registers(CPUState *);
 void hvf_kick_vcpu_thread(CPUState *cpu);
 
+bool hvf_access_memory(hwaddr address, bool write);
 void hvf_init_memslots(void);
 
 #endif
diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
index 4d4ddab348..398ad50a29 100644
--- a/target/arm/hvf/hvf.c
+++ b/target/arm/hvf/hvf.c
@@ -1202,6 +1202,11 @@ int hvf_vcpu_exec(CPUState *cpu)
             break;
         }
 
+        if (iswrite &&
+            hvf_access_memory(hvf_exit->exception.physical_address, 1)) {
+            break;
+        }
+
         assert(isv);
 
         if (iswrite) {
diff --git a/target/i386/hvf/hvf.c b/target/i386/hvf/hvf.c
index 2ddb4fc825..c4c544dc54 100644
--- a/target/i386/hvf/hvf.c
+++ b/target/i386/hvf/hvf.c
@@ -113,7 +113,7 @@ void hvf_handle_io(CPUArchState *env, uint16_t port, void *buffer,
     }
 }
 
-static bool ept_emulation_fault(HVFSlot *slot, uint64_t gpa, uint64_t ept_qual)
+static bool ept_emulation_fault(uint64_t gpa, uint64_t ept_qual)
 {
     int read, write;
 
@@ -129,14 +129,6 @@ static bool ept_emulation_fault(HVFSlot *slot, uint64_t gpa, uint64_t ept_qual)
         return false;
     }
 
-    if (write && slot) {
-        if (slot->flags & HVF_SLOT_LOG) {
-            memory_region_set_dirty(slot->region, gpa - slot->start, 1);
-            hv_vm_protect((hv_gpaddr_t)slot->start, (size_t)slot->size,
-                          HV_MEMORY_READ | HV_MEMORY_WRITE);
-        }
-    }
-
     /*
      * The EPT violation must have been caused by accessing a
      * guest-physical address that is a translation of a guest-linear
@@ -147,14 +139,11 @@ static bool ept_emulation_fault(HVFSlot *slot, uint64_t gpa, uint64_t ept_qual)
         return false;
     }
 
-    if (!slot) {
-        return true;
+    if (hvf_access_memory(gpa, write)) {
+        return false;
     }
-    if (!memory_region_is_ram(slot->region) &&
-        !(read && memory_region_is_romd(slot->region))) {
-        return true;
-    }
-    return false;
+
+    return true;
 }
 
 void hvf_arch_vcpu_destroy(CPUState *cpu)
@@ -469,7 +458,6 @@ int hvf_vcpu_exec(CPUState *cpu)
         /* Need to check if MMIO or unmapped fault */
         case EXIT_REASON_EPT_FAULT:
         {
-            HVFSlot *slot;
             uint64_t gpa = rvmcs(cpu->hvf->fd, VMCS_GUEST_PHYSICAL_ADDRESS);
 
             if (((idtvec_info & VMCS_IDT_VEC_VALID) == 0) &&
@@ -477,9 +465,8 @@ int hvf_vcpu_exec(CPUState *cpu)
                 vmx_set_nmi_blocking(cpu);
             }
 
-            slot = hvf_find_overlap_slot(gpa, 1);
             /* mmio */
-            if (ept_emulation_fault(slot, gpa, exit_qual)) {
+            if (ept_emulation_fault(gpa, exit_qual)) {
                 struct x86_decode decode;
 
                 load_regs(cpu);
-- 
2.32.0 (Apple Git-132)

[PATCH v3 6/9] hvf: add a lock for memory related functions

[Yan-Jie Wang]

We follow how KVM accel does in its memory listener (kvm-all.c) and add
a lock for the memory related functions.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 896e718374..081029ba98 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -43,6 +43,7 @@ typedef struct HVFSlot {
 } HVFSlot;
 
 static HVFSlot memslots[HVF_NUM_SLOTS];
+static QemuMutex memlock;
 
 static HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
 {
@@ -140,6 +141,8 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
         readonly = memory_region_is_rom(area) || memory_region_is_romd(area);
 
         /* setup a slot */
+        qemu_mutex_lock(&memlock);
+
         slot = hvf_find_free_slot();
         if (!slot) {
             error_report("No free slots");
@@ -169,8 +172,12 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
 
         ret = hv_vm_map(host_addr, start, size, flags);
         assert_hvf_ok(ret);
+
+        qemu_mutex_unlock(&memlock);
     } else {
         /* remove memory region */
+        qemu_mutex_lock(&memlock);
+
         slot = hvf_find_overlap_slot(start, size);
 
         if (slot) {
@@ -179,6 +186,8 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
 
             slot->size = 0;
         }
+
+        qemu_mutex_unlock(&memlock);
     }
 }
 
@@ -186,6 +195,8 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
 {
     HVFSlot *slot;
 
+    qemu_mutex_lock(&memlock);
+
     slot = hvf_find_overlap_slot(
             section->offset_within_address_space,
             int128_get64(section->size));
@@ -201,6 +212,8 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
         hv_vm_protect((uintptr_t)slot->start, (size_t)slot->size,
                       HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);
     }
+
+    qemu_mutex_unlock(&memlock);
 }
 
 static void hvf_log_start(MemoryListener *listener,
@@ -271,10 +284,13 @@ bool hvf_access_memory(hwaddr address, bool write)
     hv_return_t ret;
     hwaddr start, size;
 
+    qemu_mutex_lock(&memlock);
+
     slot = hvf_find_overlap_slot(address, 1);
 
     if (!slot || (write && slot->flags & HVF_SLOT_READONLY)) {
         /* MMIO or unmapped area, return false */
+        qemu_mutex_unlock(&memlock);
         return false;
     }
 
@@ -290,10 +306,12 @@ bool hvf_access_memory(hwaddr address, bool write)
         assert_hvf_ok(ret);
     }
 
+    qemu_mutex_unlock(&memlock);
     return true;
 }
 
 void hvf_init_memslots(void)
 {
+    qemu_mutex_init(&memlock);
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
 }
-- 
2.32.0 (Apple Git-132)

[PATCH v3 7/9] hvf: use GTree to store memory slots instead of fixed-size array

[Yan-Jie Wang]

Currently, there are only 32 memory slots in the fixed size array.
It is not scalable. Instead of using fixed size array, use GTree
(from glib library) and dynamically-allocated structures to store
memory slots.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c | 63 +++++++++++++++++++++++----------------------
 1 file changed, 32 insertions(+), 31 deletions(-)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 081029ba98..2f70ceb307 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -28,8 +28,6 @@
 
 /* Memory slots */
 
-#define HVF_NUM_SLOTS 32
-
 /* HVFSlot flags */
 #define HVF_SLOT_LOG (1 << 0)
 #define HVF_SLOT_READONLY (1 << 1)
@@ -42,35 +40,24 @@ typedef struct HVFSlot {
     MemoryRegion *region;
 } HVFSlot;
 
-static HVFSlot memslots[HVF_NUM_SLOTS];
+static GTree *memslots;
 static QemuMutex memlock;
 
 static HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
 {
-    HVFSlot *slot;
-    int x;
-    for (x = 0; x < HVF_NUM_SLOTS; ++x) {
-        slot = &memslots[x];
-        if (slot->size && start < (slot->start + slot->size) &&
-            (start + size) > slot->start) {
-            return slot;
-        }
-    }
-    return NULL;
+    HVFSlot key = {.start = start, .size = 1};
+    return g_tree_lookup(memslots, &key);
 }
 
-static HVFSlot *hvf_find_free_slot(void)
+static void hvf_insert_slot(HVFSlot *slot)
 {
-    HVFSlot *slot;
-    int x;
-    for (x = 0; x < HVF_NUM_SLOTS; x++) {
-        slot = &memslots[x];
-        if (!slot->size) {
-            return slot;
-        }
-    }
+    g_tree_insert(memslots, slot, slot);
+}
 
-    return NULL;
+static bool hvf_remove_slot(hwaddr start)
+{
+    HVFSlot key = {.start = start, .size = 1};
+    return g_tree_remove(memslots, &key);
 }
 
 /*
@@ -141,9 +128,7 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
         readonly = memory_region_is_rom(area) || memory_region_is_romd(area);
 
         /* setup a slot */
-        qemu_mutex_lock(&memlock);
-
-        slot = hvf_find_free_slot();
+        slot = g_new0(HVFSlot, 1);
         if (!slot) {
             error_report("No free slots");
             abort();
@@ -170,6 +155,10 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
             flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
         }
 
+        qemu_mutex_lock(&memlock);
+
+        hvf_insert_slot(slot);
+
         ret = hv_vm_map(host_addr, start, size, flags);
         assert_hvf_ok(ret);
 
@@ -178,13 +167,9 @@ static void hvf_set_phys_mem(MemoryRegionSection *section, bool add)
         /* remove memory region */
         qemu_mutex_lock(&memlock);
 
-        slot = hvf_find_overlap_slot(start, size);
-
-        if (slot) {
+        if (hvf_remove_slot(start)) {
             ret = hv_vm_unmap(start, size);
             assert_hvf_ok(ret);
-
-            slot->size = 0;
         }
 
         qemu_mutex_unlock(&memlock);
@@ -310,8 +295,24 @@ bool hvf_access_memory(hwaddr address, bool write)
     return true;
 }
 
+/* compare function for GTree */
+static gint _hvf_slot_compare(gconstpointer a, gconstpointer b, gpointer data)
+{
+    const HVFSlot *m1 = (const HVFSlot *)a;
+    const HVFSlot *m2 = (const HVFSlot *)b;
+
+    if (m2->start >= m1->start + m1->size) {
+        return -1;
+    } else if (m1->start >= m2->start + m2->size) {
+        return 1;
+    }
+
+    return 0;
+}
+
 void hvf_init_memslots(void)
 {
     qemu_mutex_init(&memlock);
+    memslots = g_tree_new_full(_hvf_slot_compare, NULL, g_free, NULL);
     memory_listener_register(&hvf_memory_listener, &address_space_memory);
 }
-- 
2.32.0 (Apple Git-132)

[PATCH v3 8/9] hvf: only consider directly writeable memory regions for dirty-tracking

[Yan-Jie Wang]

It is no need to dirty-track MMIO regions or other readonly regions.

Before we start or stop to dirty-track a memory region, check the type of
the memory region. The region must be a writeable ram to be dirty-tracked.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 2f70ceb307..60ece20eb4 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -180,6 +180,12 @@ static void hvf_set_dirty_tracking(MemoryRegionSection *section, bool on)
 {
     HVFSlot *slot;
 
+    if (!memory_region_is_ram(section->mr) ||
+        memory_region_is_rom(section->mr)) {
+        /* do not consider memory regions which are not directly writeable */
+        return;
+    }
+
     qemu_mutex_lock(&memlock);
 
     slot = hvf_find_overlap_slot(
-- 
2.32.0 (Apple Git-132)

[PATCH v3 9/9] hvf: remove the need to lookup memory slots when clearing dirty-bits

[Yan-Jie Wang]

Originally, when log_clear gets called, log_clear calls
hvf_set_dirty_tracking to write-protect memory slots whose dirty-bits
are cleared.

Calling hvf_set_dirty_tracking means that memory slots will be
look up and the lock for memory slots will be held during the call.

We can use the parameter `section` passed by the caller to determine the
pages that need to be write-protected. Compared to the original method,
this saves time.

Moreover, this makes only pages whose dirty-bits
are cleared write-protected instead of making the whole memory slot
write-protected.

Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
---
 accel/hvf/hvf-mem.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
index 60ece20eb4..47d23faec8 100644
--- a/accel/hvf/hvf-mem.c
+++ b/accel/hvf/hvf-mem.c
@@ -84,7 +84,10 @@ static hwaddr hvf_align_section(MemoryRegionSection *section,
     size = (size - _delta) & qemu_real_host_page_mask;
 
     *start = _start;
-    *delta = _delta;
+
+    if (delta) {
+        *delta = _delta;
+    }
 
     return size;
 }
@@ -230,11 +233,27 @@ static void hvf_log_stop(MemoryListener *listener,
 static void hvf_log_clear(MemoryListener *listener,
                          MemoryRegionSection *section)
 {
+    hwaddr start, size;
+
+    if (!memory_region_is_ram(section->mr) ||
+        memory_region_is_rom(section->mr)) {
+        /* do not consider memory regions which are not directly writeable */
+        return;
+    }
+
+    if (!memory_region_get_dirty_log_mask(section->mr)) {
+        /* the region is not being dirty-tracked */
+        return;
+    }
+
     /*
      * The dirty bits are being cleared.
      * Make the section write-protected again.
      */
-    hvf_set_dirty_tracking(section, 1);
+    size = hvf_align_section(section, &start, NULL);
+    if (size) {
+        hv_vm_protect(start, size, HV_MEMORY_READ | HV_MEMORY_EXEC);
+    }
 }
 
 static void hvf_region_add(MemoryListener *listener,
-- 
2.32.0 (Apple Git-132)

Re: [PATCH v3 1/9] hvf: move memory related functions from hvf-accel-ops.c to hvf-mem.c

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---
>  accel/hvf/hvf-accel-ops.c | 220 +--------------------------------
>  accel/hvf/hvf-mem.c       | 252 ++++++++++++++++++++++++++++++++++++++
>  accel/hvf/meson.build     |   1 +
>  include/sysemu/hvf_int.h  |   2 +
>  4 files changed, 256 insertions(+), 219 deletions(-)
>  create mode 100644 accel/hvf/hvf-mem.c

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [PATCH v3 2/9] hvf: simplify data structures and codes of memory related functions

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> * Remove mac_slot and use hvf_slot only. The function of the two structures
>   are similar.
>
> * Refactor function hvf_set_phys_mem():
>  - Remove unnecessary checks because any modified memory sections
>    will be removed first (region_del called first) before being added.
>    Therefore, new sections do not overlap with existing sections.
>  - Try to align memory sections first before giving up sections that are not
>    aligned to host page size.
>
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---
>  accel/hvf/hvf-accel-ops.c |   1 -
>  accel/hvf/hvf-mem.c       | 211 +++++++++++++++++++-------------------
>  include/sysemu/hvf_int.h  |   8 +-
>  3 files changed, 107 insertions(+), 113 deletions(-)
>
> diff --git a/accel/hvf/hvf-accel-ops.c b/accel/hvf/hvf-accel-ops.c
> index 50a563bfe0..c77f142f2b 100644
> --- a/accel/hvf/hvf-accel-ops.c
> +++ b/accel/hvf/hvf-accel-ops.c
> @@ -49,7 +49,6 @@
>
>  #include "qemu/osdep.h"
>  #include "qemu/main-loop.h"
> -#include "exec/address-spaces.h"
>  #include "exec/exec-all.h"
>  #include "sysemu/cpus.h"
>  #include "sysemu/hvf.h"

I think this deletion of the #include line should have been in patch 1?

> diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
> index 3712731ed9..32452696b6 100644
> --- a/accel/hvf/hvf-mem.c
> +++ b/accel/hvf/hvf-mem.c
> @@ -28,12 +28,16 @@
>
>  /* Memory slots */
>
> +#define HVF_NUM_SLOTS 32
> +
> +static hvf_slot memslots[HVF_NUM_SLOTS];
> +
>  hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
>  {
>      hvf_slot *slot;
>      int x;
> -    for (x = 0; x < hvf_state->num_slots; ++x) {
> -        slot = &hvf_state->slots[x];
> +    for (x = 0; x < HVF_NUM_SLOTS; ++x) {
> +        slot = &memslots[x];
>          if (slot->size && start < (slot->start + slot->size) &&
>              (start + size) > slot->start) {
>              return slot;
> @@ -42,128 +46,130 @@ hvf_slot *hvf_find_overlap_slot(uint64_t start, uint64_t size)
>      return NULL;
>  }
>
> -struct mac_slot {
> -    int present;
> -    uint64_t size;
> -    uint64_t gpa_start;
> -    uint64_t gva;
> -};
> -
> -struct mac_slot mac_slots[32];
> -
> -static int do_hvf_set_memory(hvf_slot *slot, hv_memory_flags_t flags)
> +static hvf_slot *hvf_find_free_slot(void)
>  {
> -    struct mac_slot *macslot;
> -    hv_return_t ret;
> -
> -    macslot = &mac_slots[slot->slot_id];
> -
> -    if (macslot->present) {
> -        if (macslot->size != slot->size) {
> -            macslot->present = 0;
> -            ret = hv_vm_unmap(macslot->gpa_start, macslot->size);
> -            assert_hvf_ok(ret);
> +    hvf_slot *slot;
> +    int x;
> +    for (x = 0; x < HVF_NUM_SLOTS; x++) {
> +        slot = &memslots[x];
> +        if (!slot->size) {
> +            return slot;
>          }
>      }
>
> -    if (!slot->size) {
> -        return 0;
> -    }
> -
> -    macslot->present = 1;
> -    macslot->gpa_start = slot->start;
> -    macslot->size = slot->size;
> -    ret = hv_vm_map(slot->mem, slot->start, slot->size, flags);
> -    assert_hvf_ok(ret);
> -    return 0;
> +    return NULL;
> +}
> +
> +/*
> + * Hypervisor.framework requires that the host virtual address,
> + * the guest physical address and the size of memory regions are aligned
> + * to the host page size.
> + *
> + * The function here tries to align the guest physical address and the size.
> + *
> + * The return value is the aligned size.
> + * The aligned guest physical address will be written to `start'.
> + * The delta between the aligned address and the original address will be
> + * written to `delta'.
> + */
> +static hwaddr hvf_align_section(MemoryRegionSection *section,
> +                              hwaddr *start, hwaddr *delta)

Indentation is slightly wrong here.

> +{
> +    hwaddr unaligned, _start, size, _delta;

Don't use variable names with leading underscores, please.

> +
> +    unaligned = section->offset_within_address_space;
> +    size = int128_get64(section->size);
> +    _start = ROUND_UP(unaligned, qemu_real_host_page_size);
> +    _delta = _start - unaligned;
> +    size = (size - _delta) & qemu_real_host_page_mask;
> +
> +    *start = _start;
> +    *delta = _delta;
> +
> +    return size;
>  }

> -    if (area->readonly ||
> -        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
> -        flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
> +        if (readonly) {
> +            slot->flags |= HVF_SLOT_READONLY;
> +        }
> +
> +        if (dirty_tracking) {
> +            slot->flags |= HVF_SLOT_LOG;
> +        }
> +
> +        /* set Hypervisor.framework memory mapping flags */
> +        if (readonly || dirty_tracking) {
> +            flags = HV_MEMORY_READ | HV_MEMORY_EXEC;
> +        } else {
> +            flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
> +        }
> +
> +        ret = hv_vm_map(host_addr, start, size, flags);
> +        assert_hvf_ok(ret);
>      } else {
> -        flags = HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC;
> -    }
> +        /* remove memory region */
> +        slot = hvf_find_overlap_slot(start, size);
>
> -    /* Now make a new slot. */
> -    int x;
> +        if (slot) {
> +            ret = hv_vm_unmap(start, size);
> +            assert_hvf_ok(ret);
>

This 'remove memory region' logic doesn't look quite right. In the old
code, we unmap the entirety of the memory range covered by the old
slot (this happens in do_hvf_set_memory() where it calls hv_vm_unmap()
using the gpa_start and size in the macslot we're about to reuse).
In the new code, we only unmap memory covered by the start/size
we've just calculated, but we then mark the whole slot as unused.
Shouldn't we be unmapping (slot->start, slot->size) here ?

Maybe it's actually the case that we can guarantee start == slot->start
and size == slot->size ? But in that case "find an overlapping slot"
seems like the wrong operation and actually what we're doing is
finding the exact matching slot.

The rest of the logic here looks OK, I think.

-- PMM

Re: [PATCH v3 3/9] hvf: use correct data types for addresses in memory related functions

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> Follow the QEMU coding style. Use hwaddr for guest physical address.
>
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---
>  accel/hvf/hvf-mem.c      | 2 +-
>  include/sysemu/hvf_int.h | 8 ++++----
>  2 files changed, 5 insertions(+), 5 deletions(-)

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [PATCH v3 4/9] hvf: rename struct hvf_slot to HVFSlot

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> Follow the QEMU coding style. Structured type names are in CamelCase.
>
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [PATCH v3 6/9] hvf: add a lock for memory related functions

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> We follow how KVM accel does in its memory listener (kvm-all.c) and add
> a lock for the memory related functions.

Could you outline the race condition or conflicting access that
adding this mutex is fixing, please ?

thanks
-- PMM

Re: [PATCH v3 7/9] hvf: use GTree to store memory slots instead of fixed-size array

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> Currently, there are only 32 memory slots in the fixed size array.
> It is not scalable. Instead of using fixed size array, use GTree
> (from glib library) and dynamically-allocated structures to store
> memory slots.
>
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---
>  accel/hvf/hvf-mem.c | 63 +++++++++++++++++++++++----------------------
>  1 file changed, 32 insertions(+), 31 deletions(-)
>
> diff --git a/accel/hvf/hvf-mem.c b/accel/hvf/hvf-mem.c
> index 081029ba98..2f70ceb307 100644
> --- a/accel/hvf/hvf-mem.c
> +++ b/accel/hvf/hvf-mem.c
> @@ -28,8 +28,6 @@
>
>  /* Memory slots */
>
> -#define HVF_NUM_SLOTS 32
> -
>  /* HVFSlot flags */
>  #define HVF_SLOT_LOG (1 << 0)
>  #define HVF_SLOT_READONLY (1 << 1)
> @@ -42,35 +40,24 @@ typedef struct HVFSlot {
>      MemoryRegion *region;
>  } HVFSlot;
>
> -static HVFSlot memslots[HVF_NUM_SLOTS];
> +static GTree *memslots;
>  static QemuMutex memlock;
>
>  static HVFSlot *hvf_find_overlap_slot(hwaddr start, hwaddr size)
>  {
> -    HVFSlot *slot;
> -    int x;
> -    for (x = 0; x < HVF_NUM_SLOTS; ++x) {
> -        slot = &memslots[x];
> -        if (slot->size && start < (slot->start + slot->size) &&
> -            (start + size) > slot->start) {
> -            return slot;
> -        }
> -    }
> -    return NULL;
> +    HVFSlot key = {.start = start, .size = 1};

Doesn't using a size of 1 mean that this function no longer
finds an overlapping slot which starts somewhere above
our 'start' address ?

> +    return g_tree_lookup(memslots, &key);
>  }
>

-- PMM

Re: [PATCH v3 5/9] hvf: fix memory dirty-tracking

[Peter Maydell]

On Wed, 2 Mar 2022 at 13:04, Yan-Jie Wang <ubzeme@gmail.com> wrote:
>
> Dirty-tracking in HVF is not properly implemented.
>
> On Intel Macs, Ubuntu ISO boot menu does not show properly.
>
> On Apple Silicon, using bochs-display may cause the guest crashes because
> the guest may uses load/store instructions on framebuffer which causes
> vmexits and the exception register does not contain enough information
> (ESR_EL2.ISV = 0) for QEMU to emulate the memory operation.
>
> The strategy to log the dirty pages is to write-protect the memory regions
> that are being dirty-tracked.
>
> When the guest is trapped to the host because of memory write, check whether
> the address being written is being dirty-tracked.
>
> If it is being dirty-tracked, restore the write permission of the page and
> mark the accessed page dirty, and resume the guest without increasing
> program counter, and then the same instruction will be execute again.
>
> This patch fixes the problem and make the dirty-tracking work properly.
>
> Buglink: https://bugs.launchpad.net/qemu/+bug/1827005
> Signed-off-by: Yan-Jie Wang <ubzeme@gmail.com>
> ---
>  accel/hvf/hvf-mem.c      | 62 ++++++++++++++++++++++++++++++++++++----
>  include/sysemu/hvf_int.h | 14 +--------
>  target/arm/hvf/hvf.c     |  5 ++++
>  target/i386/hvf/hvf.c    | 25 ++++------------
>  4 files changed, 68 insertions(+), 38 deletions(-)
>


> +/*
> + * The function is called when the guest is accessing memory causing vmexit.
> + * Check whether the guest can access the memory directly and
> + * also mark the accessed page being written dirty
> + * if the page is being dirty-tracked.
> + *
> + * Return true if the access is within the mapped region,
> + * otherwise return false.
> + */
> +bool hvf_access_memory(hwaddr address, bool write)
> +{
> +    HVFSlot *slot;
> +    hv_return_t ret;
> +    hwaddr start, size;
> +
> +    slot = hvf_find_overlap_slot(address, 1);

What happens if the guest does an unaligned 4 byte access
such that byte 1 is in one slot and bytes 2-4 are in a
different slot, or not covered by a slot at all ?

> diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
> index 4d4ddab348..398ad50a29 100644
> --- a/target/arm/hvf/hvf.c
> +++ b/target/arm/hvf/hvf.c
> @@ -1202,6 +1202,11 @@ int hvf_vcpu_exec(CPUState *cpu)
>              break;
>          }
>
> +        if (iswrite &&
> +            hvf_access_memory(hvf_exit->exception.physical_address, 1)) {

hvf_access_memory() can return true even if it has not changed the
protection flags on the memory, in which case we'll go into an
infinite loop of taking the fault again.

> +            break;
> +        }
> +
>          assert(isv);
>
>          if (iswrite)

thanks
-- PMM