[PATCH V3 00/10] Add architecture agnostic code to support vCPU Hotplug

[PATCH V3 00/10] Add architecture agnostic code to support vCPU Hotplug

[Salil Mehta via]

Virtual CPU hotplug support is being added across various architectures[1][3].
This series adds various code bits common across all architectures:

1. vCPU creation and Parking code refactor [Patch 1]
2. Update ACPI GED framework to support vCPU Hotplug [Patch 4,6,7]
3. ACPI CPUs AML code change [Patch 5]
4. Helper functions to support unrealization of CPU objects [Patch 8,9]
5. Misc [Patch 2,3,10]


Repository:

[*] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v2.common.v3


Revision History:

RFC V2 -> RFC V3
1. Addressed Jonathan Cameron's comments
   - Fixed 'vcpu-id' type wrongly changed from 'unsigned long' to 'integer'
   - Removed unnecessary use of variable 'vcpu_id' in kvm_park_vcpu
   - Updated [Patch V2 03/10] commit-log with details of ACPI_CPU_SCAN_METHOD macro
   - Updated [Patch V2 05/10] commit-log with conditional event handler method details
   - Added Reviewed-by tags for patches {2,3,4,6,7}
2. Addressed Gavin Shan's comments
   - Remove unnecessary use of variable 'vcpu_id' in kvm_par_vcpu
   - Fixed return value in kvm_get_vcpu from -1 to -ENOENT
   - Reset the value of 'gdb_num_g_regs' in gdb_unregister_coprocessor_all
   - Fixed the kvm_{create,park}_vcpu prototypes docs
   - Added Reviewed-by tags for patches {2,3,4,5,6,7,9,10}
3. Addressed one earlier missed comment by Alex Bennée in RFC V1
   - Added traces instead of DPRINTF in the newly added and some existing functions

RFC V1 -> RFC V2
1. Addressed Alex Bennée's comments
   - Refactored the kvm_create_vcpu logic to get rid of goto
   - Added the docs for kvm_{create,park}_vcpu prototypes
   - Splitted the gdbstub and AddressSpace destruction change into separate patches
   - Added Reviewed-by tags for patches {2,10}

References:

[1] https://lore.kernel.org/qemu-devel/20230926100436.28284-1-salil.mehta@huawei.com/
[2] https://lore.kernel.org/all/20230913163823.7880-1-james.morse@arm.com/
[3] https://lore.kernel.org/qemu-devel/cover.1695697701.git.lixianglai@loongson.cn/


Jean-Philippe Brucker (1):
  target/arm/kvm: Write CPU state back to KVM on reset

Salil Mehta (9):
  accel/kvm: Extract common KVM vCPU {creation,parking} code
  hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file
  hw/acpi: Add ACPI CPU hotplug init stub
  hw/acpi: Init GED framework with cpu hotplug events
  hw/acpi: Update CPUs AML with cpu-(ctrl)dev change
  hw/acpi: Update GED _EVT method AML with CPU scan
  hw/acpi: Update ACPI GED framework to support vCPU Hotplug
  physmem: Add helper function to destroy CPU AddressSpace
  gdbstub: Add helper function to unregister GDB register space

 accel/kvm/kvm-all.c                    | 64 ++++++++++++++++++++------
 accel/kvm/trace-events                 |  4 ++
 gdbstub/gdbstub.c                      | 15 ++++++
 hw/acpi/acpi-cpu-hotplug-stub.c        |  6 +++
 hw/acpi/cpu.c                          | 27 +++++++----
 hw/acpi/generic_event_device.c         | 22 +++++++++
 hw/i386/acpi-build.c                   |  2 +-
 include/exec/cpu-common.h              |  8 ++++
 include/exec/gdbstub.h                 |  5 ++
 include/hw/acpi/cpu.h                  |  5 +-
 include/hw/acpi/cpu_hotplug.h          |  4 ++
 include/hw/acpi/generic_event_device.h |  5 ++
 include/hw/core/cpu.h                  |  1 +
 include/sysemu/kvm.h                   | 14 ++++++
 softmmu/physmem.c                      | 25 ++++++++++
 target/arm/kvm.c                       |  8 +++-
 16 files changed, 187 insertions(+), 28 deletions(-)

-- 
2.34.1

[PATCH V3 00/10] Add architecture agnostic code to support vCPU Hotplug

[Salil Mehta]

Virtual CPU hotplug support is being added across various architectures[1][3].
This series adds various code bits common across all architectures:

1. vCPU creation and Parking code refactor [Patch 1]
2. Update ACPI GED framework to support vCPU Hotplug [Patch 4,6,7]
3. ACPI CPUs AML code change [Patch 5]
4. Helper functions to support unrealization of CPU objects [Patch 8,9]
5. Misc [Patch 2,3,10]


Repository:

[*] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v2.common.v3


Revision History:

RFC V2 -> RFC V3
1. Addressed Jonathan Cameron's comments
   - Fixed 'vcpu-id' type wrongly changed from 'unsigned long' to 'integer'
   - Removed unnecessary use of variable 'vcpu_id' in kvm_park_vcpu
   - Updated [Patch V2 03/10] commit-log with details of ACPI_CPU_SCAN_METHOD macro
   - Updated [Patch V2 05/10] commit-log with conditional event handler method details
   - Added Reviewed-by tags for patches {2,3,4,6,7}
2. Addressed Gavin Shan's comments
   - Remove unnecessary use of variable 'vcpu_id' in kvm_par_vcpu
   - Fixed return value in kvm_get_vcpu from -1 to -ENOENT
   - Reset the value of 'gdb_num_g_regs' in gdb_unregister_coprocessor_all
   - Fixed the kvm_{create,park}_vcpu prototypes docs
   - Added Reviewed-by tags for patches {2,3,4,5,6,7,9,10}
3. Addressed one earlier missed comment by Alex Bennée in RFC V1
   - Added traces instead of DPRINTF in the newly added and some existing functions

RFC V1 -> RFC V2
1. Addressed Alex Bennée's comments
   - Refactored the kvm_create_vcpu logic to get rid of goto
   - Added the docs for kvm_{create,park}_vcpu prototypes
   - Splitted the gdbstub and AddressSpace destruction change into separate patches
   - Added Reviewed-by tags for patches {2,10}

References:

[1] https://lore.kernel.org/qemu-devel/20230926100436.28284-1-salil.mehta@huawei.com/
[2] https://lore.kernel.org/all/20230913163823.7880-1-james.morse@arm.com/
[3] https://lore.kernel.org/qemu-devel/cover.1695697701.git.lixianglai@loongson.cn/


Jean-Philippe Brucker (1):
  target/arm/kvm: Write CPU state back to KVM on reset

Salil Mehta (9):
  accel/kvm: Extract common KVM vCPU {creation,parking} code
  hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file
  hw/acpi: Add ACPI CPU hotplug init stub
  hw/acpi: Init GED framework with cpu hotplug events
  hw/acpi: Update CPUs AML with cpu-(ctrl)dev change
  hw/acpi: Update GED _EVT method AML with CPU scan
  hw/acpi: Update ACPI GED framework to support vCPU Hotplug
  physmem: Add helper function to destroy CPU AddressSpace
  gdbstub: Add helper function to unregister GDB register space

 accel/kvm/kvm-all.c                    | 64 ++++++++++++++++++++------
 accel/kvm/trace-events                 |  4 ++
 gdbstub/gdbstub.c                      | 15 ++++++
 hw/acpi/acpi-cpu-hotplug-stub.c        |  6 +++
 hw/acpi/cpu.c                          | 27 +++++++----
 hw/acpi/generic_event_device.c         | 22 +++++++++
 hw/i386/acpi-build.c                   |  2 +-
 include/exec/cpu-common.h              |  8 ++++
 include/exec/gdbstub.h                 |  5 ++
 include/hw/acpi/cpu.h                  |  5 +-
 include/hw/acpi/cpu_hotplug.h          |  4 ++
 include/hw/acpi/generic_event_device.h |  5 ++
 include/hw/core/cpu.h                  |  1 +
 include/sysemu/kvm.h                   | 14 ++++++
 softmmu/physmem.c                      | 25 ++++++++++
 target/arm/kvm.c                       |  8 +++-
 16 files changed, 187 insertions(+), 28 deletions(-)

-- 
2.34.1

[PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation, parking} code

[Salil Mehta via]

KVM vCPU creation is done once during the initialization of the VM when Qemu
thread is spawned. This is common to all the architectures.

Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
corresponding KVM vCPU object in the Host KVM is not destroyed and its
representative KVM vCPU object/context in Qemu is parked.

Refactor common logic so that some APIs could be reused by vCPU Hotplug code.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++----------
 accel/kvm/trace-events |  4 +++
 include/sysemu/kvm.h   | 14 +++++++++
 3 files changed, 67 insertions(+), 15 deletions(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index ff1578bb32..0dcaa15276 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -137,6 +137,7 @@ static QemuMutex kml_slots_lock;
 #define kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
 
 static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
+static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
 
 static inline void kvm_resample_fd_remove(int gsi)
 {
@@ -320,14 +321,53 @@ err:
     return ret;
 }
 
+void kvm_park_vcpu(CPUState *cpu)
+{
+    struct KVMParkedVcpu *vcpu;
+
+    trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+
+    vcpu = g_malloc0(sizeof(*vcpu));
+    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
+    vcpu->kvm_fd = cpu->kvm_fd;
+    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+}
+
+int kvm_create_vcpu(CPUState *cpu)
+{
+    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
+    KVMState *s = kvm_state;
+    int kvm_fd;
+
+    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+
+    /* check if the KVM vCPU already exist but is parked */
+    kvm_fd = kvm_get_vcpu(s, vcpu_id);
+    if (kvm_fd < 0) {
+        /* vCPU not parked: create a new KVM vCPU */
+        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
+        if (kvm_fd < 0) {
+            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu", vcpu_id);
+            return kvm_fd;
+        }
+    }
+
+    cpu->kvm_fd = kvm_fd;
+    cpu->kvm_state = s;
+    cpu->vcpu_dirty = true;
+    cpu->dirty_pages = 0;
+    cpu->throttle_us_per_full = 0;
+
+    return 0;
+}
+
 static int do_kvm_destroy_vcpu(CPUState *cpu)
 {
     KVMState *s = kvm_state;
     long mmap_size;
-    struct KVMParkedVcpu *vcpu = NULL;
     int ret = 0;
 
-    DPRINTF("kvm_destroy_vcpu\n");
+    trace_kvm_destroy_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
 
     ret = kvm_arch_destroy_vcpu(cpu);
     if (ret < 0) {
@@ -353,10 +393,7 @@ static int do_kvm_destroy_vcpu(CPUState *cpu)
         }
     }
 
-    vcpu = g_malloc0(sizeof(*vcpu));
-    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
-    vcpu->kvm_fd = cpu->kvm_fd;
-    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+    kvm_park_vcpu(cpu);
 err:
     return ret;
 }
@@ -377,6 +414,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
         if (cpu->vcpu_id == vcpu_id) {
             int kvm_fd;
 
+            trace_kvm_get_vcpu(vcpu_id);
+
             QLIST_REMOVE(cpu, node);
             kvm_fd = cpu->kvm_fd;
             g_free(cpu);
@@ -384,7 +423,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
         }
     }
 
-    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
+    return -ENOENT;
 }
 
 int kvm_init_vcpu(CPUState *cpu, Error **errp)
@@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
 
     trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
 
-    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
+    ret = kvm_create_vcpu(cpu);
     if (ret < 0) {
-        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
+        error_setg_errno(errp, -ret,
+                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
                          kvm_arch_vcpu_id(cpu));
         goto err;
     }
 
-    cpu->kvm_fd = ret;
-    cpu->kvm_state = s;
-    cpu->vcpu_dirty = true;
-    cpu->dirty_pages = 0;
-    cpu->throttle_us_per_full = 0;
-
     mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
     if (mmap_size < 0) {
         ret = mmap_size;
diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
index 399aaeb0ec..08e2dc253f 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
 kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
 kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
 kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
+kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
 kvm_irqchip_commit_routes(void) ""
 kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
 kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index ee9025f8e9..57bd8f8fd6 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -464,6 +464,20 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
 
 int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
                                        hwaddr *phys_addr);
+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+/**
+ * kvm_park_vcpu - Gets a parked KVM vCPU if it exists
+ * @cpu: QOM CPUState object for which Qemu KVM vCPU context has to be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);
 
 #endif /* NEED_CPU_H */
 
-- 
2.34.1

[PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation, parking} code

[Salil Mehta]

KVM vCPU creation is done once during the initialization of the VM when Qemu
thread is spawned. This is common to all the architectures.

Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
corresponding KVM vCPU object in the Host KVM is not destroyed and its
representative KVM vCPU object/context in Qemu is parked.

Refactor common logic so that some APIs could be reused by vCPU Hotplug code.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++----------
 accel/kvm/trace-events |  4 +++
 include/sysemu/kvm.h   | 14 +++++++++
 3 files changed, 67 insertions(+), 15 deletions(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index ff1578bb32..0dcaa15276 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -137,6 +137,7 @@ static QemuMutex kml_slots_lock;
 #define kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
 
 static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
+static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
 
 static inline void kvm_resample_fd_remove(int gsi)
 {
@@ -320,14 +321,53 @@ err:
     return ret;
 }
 
+void kvm_park_vcpu(CPUState *cpu)
+{
+    struct KVMParkedVcpu *vcpu;
+
+    trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+
+    vcpu = g_malloc0(sizeof(*vcpu));
+    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
+    vcpu->kvm_fd = cpu->kvm_fd;
+    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+}
+
+int kvm_create_vcpu(CPUState *cpu)
+{
+    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
+    KVMState *s = kvm_state;
+    int kvm_fd;
+
+    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+
+    /* check if the KVM vCPU already exist but is parked */
+    kvm_fd = kvm_get_vcpu(s, vcpu_id);
+    if (kvm_fd < 0) {
+        /* vCPU not parked: create a new KVM vCPU */
+        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
+        if (kvm_fd < 0) {
+            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu", vcpu_id);
+            return kvm_fd;
+        }
+    }
+
+    cpu->kvm_fd = kvm_fd;
+    cpu->kvm_state = s;
+    cpu->vcpu_dirty = true;
+    cpu->dirty_pages = 0;
+    cpu->throttle_us_per_full = 0;
+
+    return 0;
+}
+
 static int do_kvm_destroy_vcpu(CPUState *cpu)
 {
     KVMState *s = kvm_state;
     long mmap_size;
-    struct KVMParkedVcpu *vcpu = NULL;
     int ret = 0;
 
-    DPRINTF("kvm_destroy_vcpu\n");
+    trace_kvm_destroy_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
 
     ret = kvm_arch_destroy_vcpu(cpu);
     if (ret < 0) {
@@ -353,10 +393,7 @@ static int do_kvm_destroy_vcpu(CPUState *cpu)
         }
     }
 
-    vcpu = g_malloc0(sizeof(*vcpu));
-    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
-    vcpu->kvm_fd = cpu->kvm_fd;
-    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+    kvm_park_vcpu(cpu);
 err:
     return ret;
 }
@@ -377,6 +414,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
         if (cpu->vcpu_id == vcpu_id) {
             int kvm_fd;
 
+            trace_kvm_get_vcpu(vcpu_id);
+
             QLIST_REMOVE(cpu, node);
             kvm_fd = cpu->kvm_fd;
             g_free(cpu);
@@ -384,7 +423,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
         }
     }
 
-    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
+    return -ENOENT;
 }
 
 int kvm_init_vcpu(CPUState *cpu, Error **errp)
@@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
 
     trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
 
-    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
+    ret = kvm_create_vcpu(cpu);
     if (ret < 0) {
-        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
+        error_setg_errno(errp, -ret,
+                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
                          kvm_arch_vcpu_id(cpu));
         goto err;
     }
 
-    cpu->kvm_fd = ret;
-    cpu->kvm_state = s;
-    cpu->vcpu_dirty = true;
-    cpu->dirty_pages = 0;
-    cpu->throttle_us_per_full = 0;
-
     mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
     if (mmap_size < 0) {
         ret = mmap_size;
diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
index 399aaeb0ec..08e2dc253f 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
 kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
 kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
 kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
+kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
 kvm_irqchip_commit_routes(void) ""
 kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
 kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index ee9025f8e9..57bd8f8fd6 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -464,6 +464,20 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
 
 int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
                                        hwaddr *phys_addr);
+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+/**
+ * kvm_park_vcpu - Gets a parked KVM vCPU if it exists
+ * @cpu: QOM CPUState object for which Qemu KVM vCPU context has to be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);
 
 #endif /* NEED_CPU_H */
 
-- 
2.34.1

[PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file

[Salil Mehta via]

CPU ctrl-dev MMIO region length could be used in ACPI GED and various other
architecture specific places. Move ACPI_CPU_HOTPLUG_REG_LEN macro to more
appropriate common header file.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c                 | 2 +-
 include/hw/acpi/cpu_hotplug.h | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 19c154d78f..45defdc0e2 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -1,12 +1,12 @@
 #include "qemu/osdep.h"
 #include "migration/vmstate.h"
 #include "hw/acpi/cpu.h"
+#include "hw/acpi/cpu_hotplug.h"
 #include "qapi/error.h"
 #include "qapi/qapi-events-acpi.h"
 #include "trace.h"
 #include "sysemu/numa.h"
 
-#define ACPI_CPU_HOTPLUG_REG_LEN 12
 #define ACPI_CPU_SELECTOR_OFFSET_WR 0
 #define ACPI_CPU_FLAGS_OFFSET_RW 4
 #define ACPI_CPU_CMD_OFFSET_WR 5
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 3b932abbbb..48b291e45e 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -19,6 +19,8 @@
 #include "hw/hotplug.h"
 #include "hw/acpi/cpu.h"
 
+#define ACPI_CPU_HOTPLUG_REG_LEN 12
+
 typedef struct AcpiCpuHotplug {
     Object *device;
     MemoryRegion io;
-- 
2.34.1

[PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file

[Salil Mehta]

CPU ctrl-dev MMIO region length could be used in ACPI GED and various other
architecture specific places. Move ACPI_CPU_HOTPLUG_REG_LEN macro to more
appropriate common header file.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c                 | 2 +-
 include/hw/acpi/cpu_hotplug.h | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 19c154d78f..45defdc0e2 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -1,12 +1,12 @@
 #include "qemu/osdep.h"
 #include "migration/vmstate.h"
 #include "hw/acpi/cpu.h"
+#include "hw/acpi/cpu_hotplug.h"
 #include "qapi/error.h"
 #include "qapi/qapi-events-acpi.h"
 #include "trace.h"
 #include "sysemu/numa.h"
 
-#define ACPI_CPU_HOTPLUG_REG_LEN 12
 #define ACPI_CPU_SELECTOR_OFFSET_WR 0
 #define ACPI_CPU_FLAGS_OFFSET_RW 4
 #define ACPI_CPU_CMD_OFFSET_WR 5
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 3b932abbbb..48b291e45e 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -19,6 +19,8 @@
 #include "hw/hotplug.h"
 #include "hw/acpi/cpu.h"
 
+#define ACPI_CPU_HOTPLUG_REG_LEN 12
+
 typedef struct AcpiCpuHotplug {
     Object *device;
     MemoryRegion io;
-- 
2.34.1

[PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta via]

ACPI CPU hotplug related initialization should only happen if ACPI_CPU_HOTPLUG
support has been enabled for particular architecture. Add cpu_hotplug_hw_init()
stub to avoid compilation break.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-stub.c
index 3fc4b14c26..c6c61bb9cd 100644
--- a/hw/acpi/acpi-cpu-hotplug-stub.c
+++ b/hw/acpi/acpi-cpu-hotplug-stub.c
@@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
     return;
 }
 
+void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
+                         CPUHotplugState *state, hwaddr base_addr)
+{
+    return;
+}
+
 void acpi_cpu_ospm_status(CPUHotplugState *cpu_st, ACPIOSTInfoList ***list)
 {
     return;
-- 
2.34.1

[PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta]

ACPI CPU hotplug related initialization should only happen if ACPI_CPU_HOTPLUG
support has been enabled for particular architecture. Add cpu_hotplug_hw_init()
stub to avoid compilation break.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-stub.c
index 3fc4b14c26..c6c61bb9cd 100644
--- a/hw/acpi/acpi-cpu-hotplug-stub.c
+++ b/hw/acpi/acpi-cpu-hotplug-stub.c
@@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
     return;
 }
 
+void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
+                         CPUHotplugState *state, hwaddr base_addr)
+{
+    return;
+}
+
 void acpi_cpu_ospm_status(CPUHotplugState *cpu_st, ACPIOSTInfoList ***list)
 {
     return;
-- 
2.34.1

[PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta via]

ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI events
when OSPM/guest receives an interrupt listed in the _CRS object of GED. OSPM
then maps or demultiplexes the event by evaluating _EVT method.

This change adds the support of cpu hotplug event initialization in the
existing GED framework.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/generic_event_device.c         | 8 ++++++++
 include/hw/acpi/generic_event_device.h | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index a3d31631fe..d2fa1d0e4a 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
     ACPI_GED_MEM_HOTPLUG_EVT,
     ACPI_GED_PWR_DOWN_EVT,
     ACPI_GED_NVDIMM_HOTPLUG_EVT,
+    ACPI_GED_CPU_HOTPLUG_EVT,
 };
 
 /*
@@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
     memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
                           TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
     sysbus_init_mmio(sbd, &ged_st->regs);
+
+    s->cpuhp.device = OBJECT(s);
+    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp container",
+                       ACPI_CPU_HOTPLUG_REG_LEN);
+    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
+    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
+                        &s->cpuhp_state, 0);
 }
 
 static void acpi_ged_class_init(ObjectClass *class, void *data)
diff --git a/include/hw/acpi/generic_event_device.h b/include/hw/acpi/generic_event_device.h
index d831bbd889..d0a5a43abf 100644
--- a/include/hw/acpi/generic_event_device.h
+++ b/include/hw/acpi/generic_event_device.h
@@ -60,6 +60,7 @@
 #define HW_ACPI_GENERIC_EVENT_DEVICE_H
 
 #include "hw/sysbus.h"
+#include "hw/acpi/cpu_hotplug.h"
 #include "hw/acpi/memory_hotplug.h"
 #include "hw/acpi/ghes.h"
 #include "qom/object.h"
@@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
 #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
 #define ACPI_GED_PWR_DOWN_EVT      0x2
 #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
+#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
 
 typedef struct GEDState {
     MemoryRegion evt;
@@ -108,6 +110,9 @@ struct AcpiGedState {
     SysBusDevice parent_obj;
     MemHotplugState memhp_state;
     MemoryRegion container_memhp;
+    CPUHotplugState cpuhp_state;
+    MemoryRegion container_cpuhp;
+    AcpiCpuHotplug cpuhp;
     GEDState ged_state;
     uint32_t ged_event_bitmap;
     qemu_irq irq;
-- 
2.34.1

[PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta]

ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI events
when OSPM/guest receives an interrupt listed in the _CRS object of GED. OSPM
then maps or demultiplexes the event by evaluating _EVT method.

This change adds the support of cpu hotplug event initialization in the
existing GED framework.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/generic_event_device.c         | 8 ++++++++
 include/hw/acpi/generic_event_device.h | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index a3d31631fe..d2fa1d0e4a 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
     ACPI_GED_MEM_HOTPLUG_EVT,
     ACPI_GED_PWR_DOWN_EVT,
     ACPI_GED_NVDIMM_HOTPLUG_EVT,
+    ACPI_GED_CPU_HOTPLUG_EVT,
 };
 
 /*
@@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
     memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
                           TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
     sysbus_init_mmio(sbd, &ged_st->regs);
+
+    s->cpuhp.device = OBJECT(s);
+    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp container",
+                       ACPI_CPU_HOTPLUG_REG_LEN);
+    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
+    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
+                        &s->cpuhp_state, 0);
 }
 
 static void acpi_ged_class_init(ObjectClass *class, void *data)
diff --git a/include/hw/acpi/generic_event_device.h b/include/hw/acpi/generic_event_device.h
index d831bbd889..d0a5a43abf 100644
--- a/include/hw/acpi/generic_event_device.h
+++ b/include/hw/acpi/generic_event_device.h
@@ -60,6 +60,7 @@
 #define HW_ACPI_GENERIC_EVENT_DEVICE_H
 
 #include "hw/sysbus.h"
+#include "hw/acpi/cpu_hotplug.h"
 #include "hw/acpi/memory_hotplug.h"
 #include "hw/acpi/ghes.h"
 #include "qom/object.h"
@@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
 #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
 #define ACPI_GED_PWR_DOWN_EVT      0x2
 #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
+#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
 
 typedef struct GEDState {
     MemoryRegion evt;
@@ -108,6 +110,9 @@ struct AcpiGedState {
     SysBusDevice parent_obj;
     MemHotplugState memhp_state;
     MemoryRegion container_memhp;
+    CPUHotplugState cpuhp_state;
+    MemoryRegion container_cpuhp;
+    AcpiCpuHotplug cpuhp;
     GEDState ged_state;
     uint32_t ged_event_bitmap;
     qemu_irq irq;
-- 
2.34.1

[PATCH V3 05/10] hw/acpi: Update CPUs AML with cpu-(ctrl)dev change

[Salil Mehta via]

CPUs Control device(\\_SB.PCI0) register interface for the x86 arch is based on
PCI and is IO port based and hence existing CPUs AML code assumes _CRS objects
would evaluate to a system resource which describes IO Port address. But on ARM
arch CPUs control device(\\_SB.PRES) register interface is memory-mapped hence
_CRS object should evaluate to system resource which describes memory-mapped
base address. Update build CPUs AML function to accept both IO/MEMORY region
spaces and accordingly update the _CRS object.

Legacy CPU Hotplug uses Generic ACPI GPE Block Bit 2 (GPE.2) event handler to
notify OSPM about any CPU hot(un)plug events. GED framework uses new register
interface for cpu-(ctrl)dev. Make AML for GPE.2 event handler conditional.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c         | 23 ++++++++++++++++-------
 hw/i386/acpi-build.c  |  2 +-
 include/hw/acpi/cpu.h |  5 +++--
 3 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 45defdc0e2..66a71660ec 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -338,9 +338,10 @@ const VMStateDescription vmstate_cpu_hotplug = {
 #define CPU_FW_EJECT_EVENT "CEJF"
 
 void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
-                    hwaddr io_base,
+                    hwaddr base_addr,
                     const char *res_root,
-                    const char *event_handler_method)
+                    const char *event_handler_method,
+                    AmlRegionSpace rs)
 {
     Aml *ifctx;
     Aml *field;
@@ -367,13 +368,19 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
         aml_append(cpu_ctrl_dev, aml_mutex(CPU_LOCK, 0));
 
         crs = aml_resource_template();
-        aml_append(crs, aml_io(AML_DECODE16, io_base, io_base, 1,
+        if (rs == AML_SYSTEM_IO) {
+            aml_append(crs, aml_io(AML_DECODE16, base_addr, base_addr, 1,
                                ACPI_CPU_HOTPLUG_REG_LEN));
+        } else {
+            aml_append(crs, aml_memory32_fixed(base_addr,
+                               ACPI_CPU_HOTPLUG_REG_LEN, AML_READ_WRITE));
+        }
+
         aml_append(cpu_ctrl_dev, aml_name_decl("_CRS", crs));
 
         /* declare CPU hotplug MMIO region with related access fields */
         aml_append(cpu_ctrl_dev,
-            aml_operation_region("PRST", AML_SYSTEM_IO, aml_int(io_base),
+            aml_operation_region("PRST", rs, aml_int(base_addr),
                                  ACPI_CPU_HOTPLUG_REG_LEN));
 
         field = aml_field("PRST", AML_BYTE_ACC, AML_NOLOCK,
@@ -699,9 +706,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
     aml_append(sb_scope, cpus_dev);
     aml_append(table, sb_scope);
 
-    method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED);
-    aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD));
-    aml_append(table, method);
+    if (event_handler_method) {
+        method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED);
+        aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD));
+        aml_append(table, method);
+    }
 
     g_free(cphp_res_path);
 }
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 4d2d40bab5..611d3d044d 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1550,7 +1550,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
             .fw_unplugs_cpu = pm->smi_on_cpu_unplug,
         };
         build_cpus_aml(dsdt, machine, opts, pm->cpu_hp_io_base,
-                       "\\_SB.PCI0", "\\_GPE._E02");
+                       "\\_SB.PCI0", "\\_GPE._E02", AML_SYSTEM_IO);
     }
 
     if (pcms->memhp_io_base && nr_mem) {
diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
index 999caaf510..b87ebfdf4b 100644
--- a/include/hw/acpi/cpu.h
+++ b/include/hw/acpi/cpu.h
@@ -56,9 +56,10 @@ typedef struct CPUHotplugFeatures {
 } CPUHotplugFeatures;
 
 void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
-                    hwaddr io_base,
+                    hwaddr base_addr,
                     const char *res_root,
-                    const char *event_handler_method);
+                    const char *event_handler_method,
+                    AmlRegionSpace rs);
 
 void acpi_cpu_ospm_status(CPUHotplugState *cpu_st, ACPIOSTInfoList ***list);
 
-- 
2.34.1

[PATCH V3 05/10] hw/acpi: Update CPUs AML with cpu-(ctrl)dev change

[Salil Mehta]

CPUs Control device(\\_SB.PCI0) register interface for the x86 arch is based on
PCI and is IO port based and hence existing CPUs AML code assumes _CRS objects
would evaluate to a system resource which describes IO Port address. But on ARM
arch CPUs control device(\\_SB.PRES) register interface is memory-mapped hence
_CRS object should evaluate to system resource which describes memory-mapped
base address. Update build CPUs AML function to accept both IO/MEMORY region
spaces and accordingly update the _CRS object.

Legacy CPU Hotplug uses Generic ACPI GPE Block Bit 2 (GPE.2) event handler to
notify OSPM about any CPU hot(un)plug events. GED framework uses new register
interface for cpu-(ctrl)dev. Make AML for GPE.2 event handler conditional.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c         | 23 ++++++++++++++++-------
 hw/i386/acpi-build.c  |  2 +-
 include/hw/acpi/cpu.h |  5 +++--
 3 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 45defdc0e2..66a71660ec 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -338,9 +338,10 @@ const VMStateDescription vmstate_cpu_hotplug = {
 #define CPU_FW_EJECT_EVENT "CEJF"
 
 void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
-                    hwaddr io_base,
+                    hwaddr base_addr,
                     const char *res_root,
-                    const char *event_handler_method)
+                    const char *event_handler_method,
+                    AmlRegionSpace rs)
 {
     Aml *ifctx;
     Aml *field;
@@ -367,13 +368,19 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
         aml_append(cpu_ctrl_dev, aml_mutex(CPU_LOCK, 0));
 
         crs = aml_resource_template();
-        aml_append(crs, aml_io(AML_DECODE16, io_base, io_base, 1,
+        if (rs == AML_SYSTEM_IO) {
+            aml_append(crs, aml_io(AML_DECODE16, base_addr, base_addr, 1,
                                ACPI_CPU_HOTPLUG_REG_LEN));
+        } else {
+            aml_append(crs, aml_memory32_fixed(base_addr,
+                               ACPI_CPU_HOTPLUG_REG_LEN, AML_READ_WRITE));
+        }
+
         aml_append(cpu_ctrl_dev, aml_name_decl("_CRS", crs));
 
         /* declare CPU hotplug MMIO region with related access fields */
         aml_append(cpu_ctrl_dev,
-            aml_operation_region("PRST", AML_SYSTEM_IO, aml_int(io_base),
+            aml_operation_region("PRST", rs, aml_int(base_addr),
                                  ACPI_CPU_HOTPLUG_REG_LEN));
 
         field = aml_field("PRST", AML_BYTE_ACC, AML_NOLOCK,
@@ -699,9 +706,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
     aml_append(sb_scope, cpus_dev);
     aml_append(table, sb_scope);
 
-    method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED);
-    aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD));
-    aml_append(table, method);
+    if (event_handler_method) {
+        method = aml_method(event_handler_method, 0, AML_NOTSERIALIZED);
+        aml_append(method, aml_call0("\\_SB.CPUS." CPU_SCAN_METHOD));
+        aml_append(table, method);
+    }
 
     g_free(cphp_res_path);
 }
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 4d2d40bab5..611d3d044d 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1550,7 +1550,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
             .fw_unplugs_cpu = pm->smi_on_cpu_unplug,
         };
         build_cpus_aml(dsdt, machine, opts, pm->cpu_hp_io_base,
-                       "\\_SB.PCI0", "\\_GPE._E02");
+                       "\\_SB.PCI0", "\\_GPE._E02", AML_SYSTEM_IO);
     }
 
     if (pcms->memhp_io_base && nr_mem) {
diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
index 999caaf510..b87ebfdf4b 100644
--- a/include/hw/acpi/cpu.h
+++ b/include/hw/acpi/cpu.h
@@ -56,9 +56,10 @@ typedef struct CPUHotplugFeatures {
 } CPUHotplugFeatures;
 
 void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
-                    hwaddr io_base,
+                    hwaddr base_addr,
                     const char *res_root,
-                    const char *event_handler_method);
+                    const char *event_handler_method,
+                    AmlRegionSpace rs);
 
 void acpi_cpu_ospm_status(CPUHotplugState *cpu_st, ACPIOSTInfoList ***list);
 
-- 
2.34.1

[PATCH V3 06/10] hw/acpi: Update GED _EVT method AML with CPU scan

[Salil Mehta via]

OSPM evaluates _EVT method to map the event. The CPU hotplug event eventually
results in start of the CPU scan. Scan figures out the CPU and the kind of
event(plug/unplug) and notifies it back to the guest. Update the GED AML _EVT
method with the call to \\_SB.CPUS.CSCN

Also, macro CPU_SCAN_METHOD might be referred in other places like during GED
intialization so it makes sense to have its definition placed in some common
header file like cpu_hotplug.h. But doing this can cause compilation break
because of the conflicting macro definitions present in cpu.c and cpu_hotplug.c
and because both these files get compiled due to historic reasons of x86 world
i.e. decision to use legacy(GPE.2)/modern(GED) CPU hotplug interface happens
during runtime [1]. To mitigate above, for now, declare a new common macro
ACPI_CPU_SCAN_METHOD for CPU scan method instead.
(This needs a separate discussion later on for clean-up)

Reference:
[1] https://lore.kernel.org/qemu-devel/1463496205-251412-24-git-send-email-imammedo@redhat.com/

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c                  | 2 +-
 hw/acpi/generic_event_device.c | 4 ++++
 include/hw/acpi/cpu_hotplug.h  | 2 ++
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 66a71660ec..d7d7b5b8d2 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -322,7 +322,7 @@ const VMStateDescription vmstate_cpu_hotplug = {
 #define CPUHP_RES_DEVICE  "PRES"
 #define CPU_LOCK          "CPLK"
 #define CPU_STS_METHOD    "CSTA"
-#define CPU_SCAN_METHOD   "CSCN"
+#define CPU_SCAN_METHOD   ACPI_CPU_SCAN_METHOD
 #define CPU_NOTIFY_METHOD "CTFY"
 #define CPU_EJECT_METHOD  "CEJ0"
 #define CPU_OST_METHOD    "COST"
diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index d2fa1d0e4a..62d504d231 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -108,6 +108,10 @@ void build_ged_aml(Aml *table, const char *name, HotplugHandler *hotplug_dev,
                 aml_append(if_ctx, aml_call0(MEMORY_DEVICES_CONTAINER "."
                                              MEMORY_SLOT_SCAN_METHOD));
                 break;
+            case ACPI_GED_CPU_HOTPLUG_EVT:
+                aml_append(if_ctx, aml_call0(ACPI_CPU_CONTAINER "."
+                                             ACPI_CPU_SCAN_METHOD));
+                break;
             case ACPI_GED_PWR_DOWN_EVT:
                 aml_append(if_ctx,
                            aml_notify(aml_name(ACPI_POWER_BUTTON_DEVICE),
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 48b291e45e..ef631750b4 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -20,6 +20,8 @@
 #include "hw/acpi/cpu.h"
 
 #define ACPI_CPU_HOTPLUG_REG_LEN 12
+#define ACPI_CPU_SCAN_METHOD "CSCN"
+#define ACPI_CPU_CONTAINER "\\_SB.CPUS"
 
 typedef struct AcpiCpuHotplug {
     Object *device;
-- 
2.34.1

[PATCH V3 06/10] hw/acpi: Update GED _EVT method AML with CPU scan

[Salil Mehta]

OSPM evaluates _EVT method to map the event. The CPU hotplug event eventually
results in start of the CPU scan. Scan figures out the CPU and the kind of
event(plug/unplug) and notifies it back to the guest. Update the GED AML _EVT
method with the call to \\_SB.CPUS.CSCN

Also, macro CPU_SCAN_METHOD might be referred in other places like during GED
intialization so it makes sense to have its definition placed in some common
header file like cpu_hotplug.h. But doing this can cause compilation break
because of the conflicting macro definitions present in cpu.c and cpu_hotplug.c
and because both these files get compiled due to historic reasons of x86 world
i.e. decision to use legacy(GPE.2)/modern(GED) CPU hotplug interface happens
during runtime [1]. To mitigate above, for now, declare a new common macro
ACPI_CPU_SCAN_METHOD for CPU scan method instead.
(This needs a separate discussion later on for clean-up)

Reference:
[1] https://lore.kernel.org/qemu-devel/1463496205-251412-24-git-send-email-imammedo@redhat.com/

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/cpu.c                  | 2 +-
 hw/acpi/generic_event_device.c | 4 ++++
 include/hw/acpi/cpu_hotplug.h  | 2 ++
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 66a71660ec..d7d7b5b8d2 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -322,7 +322,7 @@ const VMStateDescription vmstate_cpu_hotplug = {
 #define CPUHP_RES_DEVICE  "PRES"
 #define CPU_LOCK          "CPLK"
 #define CPU_STS_METHOD    "CSTA"
-#define CPU_SCAN_METHOD   "CSCN"
+#define CPU_SCAN_METHOD   ACPI_CPU_SCAN_METHOD
 #define CPU_NOTIFY_METHOD "CTFY"
 #define CPU_EJECT_METHOD  "CEJ0"
 #define CPU_OST_METHOD    "COST"
diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index d2fa1d0e4a..62d504d231 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -108,6 +108,10 @@ void build_ged_aml(Aml *table, const char *name, HotplugHandler *hotplug_dev,
                 aml_append(if_ctx, aml_call0(MEMORY_DEVICES_CONTAINER "."
                                              MEMORY_SLOT_SCAN_METHOD));
                 break;
+            case ACPI_GED_CPU_HOTPLUG_EVT:
+                aml_append(if_ctx, aml_call0(ACPI_CPU_CONTAINER "."
+                                             ACPI_CPU_SCAN_METHOD));
+                break;
             case ACPI_GED_PWR_DOWN_EVT:
                 aml_append(if_ctx,
                            aml_notify(aml_name(ACPI_POWER_BUTTON_DEVICE),
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 48b291e45e..ef631750b4 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -20,6 +20,8 @@
 #include "hw/acpi/cpu.h"
 
 #define ACPI_CPU_HOTPLUG_REG_LEN 12
+#define ACPI_CPU_SCAN_METHOD "CSCN"
+#define ACPI_CPU_CONTAINER "\\_SB.CPUS"
 
 typedef struct AcpiCpuHotplug {
     Object *device;
-- 
2.34.1

[PATCH V3 07/10] hw/acpi: Update ACPI GED framework to support vCPU Hotplug

[Salil Mehta via]

ACPI GED shall be used to convey to the guest kernel about any CPU hot-(un)plug
events. Therefore, existing ACPI GED framework inside QEMU needs to be enhanced
to support CPU hotplug state and events.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/generic_event_device.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index 62d504d231..0d5f0140e5 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -12,6 +12,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/acpi/acpi.h"
+#include "hw/acpi/cpu.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/irq.h"
 #include "hw/mem/pc-dimm.h"
@@ -239,6 +240,8 @@ static void acpi_ged_device_plug_cb(HotplugHandler *hotplug_dev,
         } else {
             acpi_memory_plug_cb(hotplug_dev, &s->memhp_state, dev, errp);
         }
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_plug_cb(hotplug_dev, &s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "virt: device plug request for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -253,6 +256,8 @@ static void acpi_ged_unplug_request_cb(HotplugHandler *hotplug_dev,
     if ((object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) &&
                        !(object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM)))) {
         acpi_memory_unplug_request_cb(hotplug_dev, &s->memhp_state, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_request_cb(hotplug_dev, &s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "acpi: device unplug request for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -266,6 +271,8 @@ static void acpi_ged_unplug_cb(HotplugHandler *hotplug_dev,
 
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         acpi_memory_unplug_cb(&s->memhp_state, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_cb(&s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "acpi: device unplug for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -277,6 +284,7 @@ static void acpi_ged_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list)
     AcpiGedState *s = ACPI_GED(adev);
 
     acpi_memory_ospm_status(&s->memhp_state, list);
+    acpi_cpu_ospm_status(&s->cpuhp_state, list);
 }
 
 static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev)
@@ -291,6 +299,8 @@ static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev)
         sel = ACPI_GED_PWR_DOWN_EVT;
     } else if (ev & ACPI_NVDIMM_HOTPLUG_STATUS) {
         sel = ACPI_GED_NVDIMM_HOTPLUG_EVT;
+    } else if (ev & ACPI_CPU_HOTPLUG_STATUS) {
+        sel = ACPI_GED_CPU_HOTPLUG_EVT;
     } else {
         /* Unknown event. Return without generating interrupt. */
         warn_report("GED: Unsupported event %d. No irq injected", ev);
-- 
2.34.1

[PATCH V3 07/10] hw/acpi: Update ACPI GED framework to support vCPU Hotplug

[Salil Mehta]

ACPI GED shall be used to convey to the guest kernel about any CPU hot-(un)plug
events. Therefore, existing ACPI GED framework inside QEMU needs to be enhanced
to support CPU hotplug state and events.

Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 hw/acpi/generic_event_device.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
index 62d504d231..0d5f0140e5 100644
--- a/hw/acpi/generic_event_device.c
+++ b/hw/acpi/generic_event_device.c
@@ -12,6 +12,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/acpi/acpi.h"
+#include "hw/acpi/cpu.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/irq.h"
 #include "hw/mem/pc-dimm.h"
@@ -239,6 +240,8 @@ static void acpi_ged_device_plug_cb(HotplugHandler *hotplug_dev,
         } else {
             acpi_memory_plug_cb(hotplug_dev, &s->memhp_state, dev, errp);
         }
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_plug_cb(hotplug_dev, &s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "virt: device plug request for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -253,6 +256,8 @@ static void acpi_ged_unplug_request_cb(HotplugHandler *hotplug_dev,
     if ((object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) &&
                        !(object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM)))) {
         acpi_memory_unplug_request_cb(hotplug_dev, &s->memhp_state, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_request_cb(hotplug_dev, &s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "acpi: device unplug request for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -266,6 +271,8 @@ static void acpi_ged_unplug_cb(HotplugHandler *hotplug_dev,
 
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         acpi_memory_unplug_cb(&s->memhp_state, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_cb(&s->cpuhp_state, dev, errp);
     } else {
         error_setg(errp, "acpi: device unplug for unsupported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
@@ -277,6 +284,7 @@ static void acpi_ged_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list)
     AcpiGedState *s = ACPI_GED(adev);
 
     acpi_memory_ospm_status(&s->memhp_state, list);
+    acpi_cpu_ospm_status(&s->cpuhp_state, list);
 }
 
 static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev)
@@ -291,6 +299,8 @@ static void acpi_ged_send_event(AcpiDeviceIf *adev, AcpiEventStatusBits ev)
         sel = ACPI_GED_PWR_DOWN_EVT;
     } else if (ev & ACPI_NVDIMM_HOTPLUG_STATUS) {
         sel = ACPI_GED_NVDIMM_HOTPLUG_EVT;
+    } else if (ev & ACPI_CPU_HOTPLUG_STATUS) {
+        sel = ACPI_GED_CPU_HOTPLUG_EVT;
     } else {
         /* Unknown event. Return without generating interrupt. */
         warn_report("GED: Unsupported event %d. No irq injected", ev);
-- 
2.34.1

[PATCH V3 08/10] physmem: Add helper function to destroy CPU AddressSpace

[Salil Mehta via]

Virtual CPU Hot-unplug leads to unrealization of a CPU object. This also
involves destruction of the CPU AddressSpace. Add common function to help
destroy the CPU AddressSpace.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 include/exec/cpu-common.h |  8 ++++++++
 include/hw/core/cpu.h     |  1 +
 softmmu/physmem.c         | 25 +++++++++++++++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index 41788c0bdd..eb56a228a2 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -120,6 +120,14 @@ size_t qemu_ram_pagesize_largest(void);
  */
 void cpu_address_space_init(CPUState *cpu, int asidx,
                             const char *prefix, MemoryRegion *mr);
+/**
+ * cpu_address_space_destroy:
+ * @cpu: CPU for which address space needs to be destroyed
+ * @asidx: integer index of this address space
+ *
+ * Note that with KVM only one address space is supported.
+ */
+void cpu_address_space_destroy(CPUState *cpu, int asidx);
 
 void cpu_physical_memory_rw(hwaddr addr, void *buf,
                             hwaddr len, bool is_write);
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 648b5b3586..65d2ae4581 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -355,6 +355,7 @@ struct CPUState {
     QSIMPLEQ_HEAD(, qemu_work_item) work_list;
 
     CPUAddressSpace *cpu_ases;
+    int cpu_ases_count;
     int num_ases;
     AddressSpace *as;
     MemoryRegion *memory;
diff --git a/softmmu/physmem.c b/softmmu/physmem.c
index 4f6ca653b3..4dfa0ca66f 100644
--- a/softmmu/physmem.c
+++ b/softmmu/physmem.c
@@ -761,6 +761,7 @@ void cpu_address_space_init(CPUState *cpu, int asidx,
 
     if (!cpu->cpu_ases) {
         cpu->cpu_ases = g_new0(CPUAddressSpace, cpu->num_ases);
+        cpu->cpu_ases_count = cpu->num_ases;
     }
 
     newas = &cpu->cpu_ases[asidx];
@@ -774,6 +775,30 @@ void cpu_address_space_init(CPUState *cpu, int asidx,
     }
 }
 
+void cpu_address_space_destroy(CPUState *cpu, int asidx)
+{
+    CPUAddressSpace *cpuas;
+
+    assert(asidx < cpu->num_ases);
+    assert(asidx == 0 || !kvm_enabled());
+    assert(cpu->cpu_ases);
+
+    cpuas = &cpu->cpu_ases[asidx];
+    if (tcg_enabled()) {
+        memory_listener_unregister(&cpuas->tcg_as_listener);
+    }
+
+    address_space_destroy(cpuas->as);
+    g_free_rcu(cpuas->as, rcu);
+
+    if (cpu->cpu_ases_count == 1) {
+        g_free(cpu->cpu_ases);
+        cpu->cpu_ases = NULL;
+    }
+
+    cpu->cpu_ases_count--;
+}
+
 AddressSpace *cpu_get_address_space(CPUState *cpu, int asidx)
 {
     /* Return the AddressSpace corresponding to the specified index */
-- 
2.34.1

[PATCH V3 08/10] physmem: Add helper function to destroy CPU AddressSpace

[Salil Mehta]

Virtual CPU Hot-unplug leads to unrealization of a CPU object. This also
involves destruction of the CPU AddressSpace. Add common function to help
destroy the CPU AddressSpace.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 include/exec/cpu-common.h |  8 ++++++++
 include/hw/core/cpu.h     |  1 +
 softmmu/physmem.c         | 25 +++++++++++++++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index 41788c0bdd..eb56a228a2 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -120,6 +120,14 @@ size_t qemu_ram_pagesize_largest(void);
  */
 void cpu_address_space_init(CPUState *cpu, int asidx,
                             const char *prefix, MemoryRegion *mr);
+/**
+ * cpu_address_space_destroy:
+ * @cpu: CPU for which address space needs to be destroyed
+ * @asidx: integer index of this address space
+ *
+ * Note that with KVM only one address space is supported.
+ */
+void cpu_address_space_destroy(CPUState *cpu, int asidx);
 
 void cpu_physical_memory_rw(hwaddr addr, void *buf,
                             hwaddr len, bool is_write);
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 648b5b3586..65d2ae4581 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -355,6 +355,7 @@ struct CPUState {
     QSIMPLEQ_HEAD(, qemu_work_item) work_list;
 
     CPUAddressSpace *cpu_ases;
+    int cpu_ases_count;
     int num_ases;
     AddressSpace *as;
     MemoryRegion *memory;
diff --git a/softmmu/physmem.c b/softmmu/physmem.c
index 4f6ca653b3..4dfa0ca66f 100644
--- a/softmmu/physmem.c
+++ b/softmmu/physmem.c
@@ -761,6 +761,7 @@ void cpu_address_space_init(CPUState *cpu, int asidx,
 
     if (!cpu->cpu_ases) {
         cpu->cpu_ases = g_new0(CPUAddressSpace, cpu->num_ases);
+        cpu->cpu_ases_count = cpu->num_ases;
     }
 
     newas = &cpu->cpu_ases[asidx];
@@ -774,6 +775,30 @@ void cpu_address_space_init(CPUState *cpu, int asidx,
     }
 }
 
+void cpu_address_space_destroy(CPUState *cpu, int asidx)
+{
+    CPUAddressSpace *cpuas;
+
+    assert(asidx < cpu->num_ases);
+    assert(asidx == 0 || !kvm_enabled());
+    assert(cpu->cpu_ases);
+
+    cpuas = &cpu->cpu_ases[asidx];
+    if (tcg_enabled()) {
+        memory_listener_unregister(&cpuas->tcg_as_listener);
+    }
+
+    address_space_destroy(cpuas->as);
+    g_free_rcu(cpuas->as, rcu);
+
+    if (cpu->cpu_ases_count == 1) {
+        g_free(cpu->cpu_ases);
+        cpu->cpu_ases = NULL;
+    }
+
+    cpu->cpu_ases_count--;
+}
+
 AddressSpace *cpu_get_address_space(CPUState *cpu, int asidx)
 {
     /* Return the AddressSpace corresponding to the specified index */
-- 
2.34.1

[PATCH V3 09/10] gdbstub: Add helper function to unregister GDB register space

[Salil Mehta via]

Add common function to help unregister the GDB Register Space. This shall be
done in context to the CPU unrealization.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 gdbstub/gdbstub.c      | 15 +++++++++++++++
 include/exec/gdbstub.h |  5 +++++
 2 files changed, 20 insertions(+)

diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
index 349d348c7b..97b89e2d00 100644
--- a/gdbstub/gdbstub.c
+++ b/gdbstub/gdbstub.c
@@ -491,6 +491,21 @@ void gdb_register_coprocessor(CPUState *cpu,
     }
 }
 
+void gdb_unregister_coprocessor_all(CPUState *cpu)
+{
+    GDBRegisterState *s, *p;
+
+    p = cpu->gdb_regs;
+    while (p) {
+        s = p;
+        p = p->next;
+        /* s->xml is static const char so isn't freed */
+        g_free(s);
+    }
+    cpu->gdb_regs = NULL;
+    cpu->gdb_num_g_regs = 0;
+}
+
 static void gdb_process_breakpoint_remove_all(GDBProcess *p)
 {
     CPUState *cpu = gdb_get_first_cpu_in_process(p);
diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
index 16a139043f..7d1368d377 100644
--- a/include/exec/gdbstub.h
+++ b/include/exec/gdbstub.h
@@ -27,6 +27,11 @@ typedef int (*gdb_set_reg_cb)(CPUArchState *env, uint8_t *buf, int reg);
 void gdb_register_coprocessor(CPUState *cpu,
                               gdb_get_reg_cb get_reg, gdb_set_reg_cb set_reg,
                               int num_regs, const char *xml, int g_pos);
+/**
+ * gdb_unregister_coprocessor_all() - unregisters supplemental set of registers
+ * @cpu - the CPU associated with registers
+ */
+void gdb_unregister_coprocessor_all(CPUState *cpu);
 
 /**
  * gdbserver_start: start the gdb server
-- 
2.34.1

[PATCH V3 09/10] gdbstub: Add helper function to unregister GDB register space

[Salil Mehta]

Add common function to help unregister the GDB Register Space. This shall be
done in context to the CPU unrealization.

Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
 gdbstub/gdbstub.c      | 15 +++++++++++++++
 include/exec/gdbstub.h |  5 +++++
 2 files changed, 20 insertions(+)

diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
index 349d348c7b..97b89e2d00 100644
--- a/gdbstub/gdbstub.c
+++ b/gdbstub/gdbstub.c
@@ -491,6 +491,21 @@ void gdb_register_coprocessor(CPUState *cpu,
     }
 }
 
+void gdb_unregister_coprocessor_all(CPUState *cpu)
+{
+    GDBRegisterState *s, *p;
+
+    p = cpu->gdb_regs;
+    while (p) {
+        s = p;
+        p = p->next;
+        /* s->xml is static const char so isn't freed */
+        g_free(s);
+    }
+    cpu->gdb_regs = NULL;
+    cpu->gdb_num_g_regs = 0;
+}
+
 static void gdb_process_breakpoint_remove_all(GDBProcess *p)
 {
     CPUState *cpu = gdb_get_first_cpu_in_process(p);
diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
index 16a139043f..7d1368d377 100644
--- a/include/exec/gdbstub.h
+++ b/include/exec/gdbstub.h
@@ -27,6 +27,11 @@ typedef int (*gdb_set_reg_cb)(CPUArchState *env, uint8_t *buf, int reg);
 void gdb_register_coprocessor(CPUState *cpu,
                               gdb_get_reg_cb get_reg, gdb_set_reg_cb set_reg,
                               int num_regs, const char *xml, int g_pos);
+/**
+ * gdb_unregister_coprocessor_all() - unregisters supplemental set of registers
+ * @cpu - the CPU associated with registers
+ */
+void gdb_unregister_coprocessor_all(CPUState *cpu);
 
 /**
  * gdbserver_start: start the gdb server
-- 
2.34.1

[PATCH V3 10/10] target/arm/kvm: Write CPU state back to KVM on reset

[Salil Mehta via]

From: Jean-Philippe Brucker <jean-philippe@linaro.org>

When a KVM vCPU is reset following a PSCI CPU_ON call, its power state
is not synchronized with KVM at the moment. Because the vCPU is not
marked dirty, we miss the call to kvm_arch_put_registers() that writes
to KVM's MP_STATE. Force mp_state synchronization.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 target/arm/kvm.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index b66b936a95..8cb70b9e7c 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -642,11 +642,12 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu)
 void kvm_arm_reset_vcpu(ARMCPU *cpu)
 {
     int ret;
+    CPUState *cs = CPU(cpu);
 
     /* Re-init VCPU so that all registers are set to
      * their respective reset values.
      */
-    ret = kvm_arm_vcpu_init(CPU(cpu));
+    ret = kvm_arm_vcpu_init(cs);
     if (ret < 0) {
         fprintf(stderr, "kvm_arm_vcpu_init failed: %s\n", strerror(-ret));
         abort();
@@ -663,6 +664,11 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu)
      * for the same reason we do so in kvm_arch_get_registers().
      */
     write_list_to_cpustate(cpu);
+    /*
+     * Ensure we call kvm_arch_put_registers(). The vCPU isn't marked dirty if
+     * it was parked in KVM and is now booting from a PSCI CPU_ON call.
+     */
+    cs->vcpu_dirty = true;
 }
 
 /*
-- 
2.34.1

[PATCH V3 10/10] target/arm/kvm: Write CPU state back to KVM on reset

[Salil Mehta]

From: Jean-Philippe Brucker <jean-philippe@linaro.org>

When a KVM vCPU is reset following a PSCI CPU_ON call, its power state
is not synchronized with KVM at the moment. Because the vCPU is not
marked dirty, we miss the call to kvm_arch_put_registers() that writes
to KVM's MP_STATE. Force mp_state synchronization.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
---
 target/arm/kvm.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index b66b936a95..8cb70b9e7c 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -642,11 +642,12 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu)
 void kvm_arm_reset_vcpu(ARMCPU *cpu)
 {
     int ret;
+    CPUState *cs = CPU(cpu);
 
     /* Re-init VCPU so that all registers are set to
      * their respective reset values.
      */
-    ret = kvm_arm_vcpu_init(CPU(cpu));
+    ret = kvm_arm_vcpu_init(cs);
     if (ret < 0) {
         fprintf(stderr, "kvm_arm_vcpu_init failed: %s\n", strerror(-ret));
         abort();
@@ -663,6 +664,11 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu)
      * for the same reason we do so in kvm_arch_get_registers().
      */
     write_list_to_cpustate(cpu);
+    /*
+     * Ensure we call kvm_arch_put_registers(). The vCPU isn't marked dirty if
+     * it was parked in KVM and is now booting from a PSCI CPU_ON call.
+     */
+    cs->vcpu_dirty = true;
 }
 
 /*
-- 
2.34.1

Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[David Hildenbrand]

On 09.10.23 13:28, Salil Mehta wrote:
> KVM vCPU creation is done once during the initialization of the VM when Qemu
> thread is spawned. This is common to all the architectures.
> 
> Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
> corresponding KVM vCPU object in the Host KVM is not destroyed and its
> representative KVM vCPU object/context in Qemu is parked.
> 
> Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>

[...]

>   
>   int kvm_init_vcpu(CPUState *cpu, Error **errp)
> @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
>   
>       trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
>   
> -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
> +    ret = kvm_create_vcpu(cpu);
>       if (ret < 0) {
> -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
> +        error_setg_errno(errp, -ret,
> +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",

Unrelated change.

>                            kvm_arch_vcpu_id(cpu));
>           goto err;
>       }
>   
> -    cpu->kvm_fd = ret;
> -    cpu->kvm_state = s;
> -    cpu->vcpu_dirty = true;
> -    cpu->dirty_pages = 0;
> -    cpu->throttle_us_per_full = 0;
> -
>       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
>       if (mmap_size < 0) {
>           ret = mmap_size;
> diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
> index 399aaeb0ec..08e2dc253f 100644
> --- a/accel/kvm/trace-events
> +++ b/accel/kvm/trace-events
> @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
>   kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
>   kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
>   kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
> +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
> +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
> +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
> +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"

It's a bit confusing that there is now

1) create (create new or return parked)
2) destroy (cleanup + park)
3) park (park only)

Why would one use 2) instead of 3) or the other way around? But I 
suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal 
helper ...

>   kvm_irqchip_commit_routes(void) ""
>   kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
>   kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index ee9025f8e9..57bd8f8fd6 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -464,6 +464,20 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
>   
>   int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
>                                          hwaddr *phys_addr);
> +/**
> + * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
> + * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
> + *
> + * @returns: 0 when success, errno (<0) when failed.
> + */
> +int kvm_create_vcpu(CPUState *cpu);
> +/**
> + * kvm_park_vcpu - Gets a parked KVM vCPU if it exists


^ I suspect that description is wrong.

-- 
Cheers,

David / dhildenb

Re: [PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file

[David Hildenbrand]

On 09.10.23 13:28, Salil Mehta wrote:
> CPU ctrl-dev MMIO region length could be used in ACPI GED and various other
> architecture specific places. Move ACPI_CPU_HOTPLUG_REG_LEN macro to more
> appropriate common header file.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> ---
>   hw/acpi/cpu.c                 | 2 +-
>   include/hw/acpi/cpu_hotplug.h | 2 ++
>   2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 19c154d78f..45defdc0e2 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -1,12 +1,12 @@
>   #include "qemu/osdep.h"
>   #include "migration/vmstate.h"
>   #include "hw/acpi/cpu.h"
> +#include "hw/acpi/cpu_hotplug.h"
>   #include "qapi/error.h"
>   #include "qapi/qapi-events-acpi.h"
>   #include "trace.h"
>   #include "sysemu/numa.h"
>   
> -#define ACPI_CPU_HOTPLUG_REG_LEN 12
>   #define ACPI_CPU_SELECTOR_OFFSET_WR 0
>   #define ACPI_CPU_FLAGS_OFFSET_RW 4
>   #define ACPI_CPU_CMD_OFFSET_WR 5
> diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
> index 3b932abbbb..48b291e45e 100644
> --- a/include/hw/acpi/cpu_hotplug.h
> +++ b/include/hw/acpi/cpu_hotplug.h
> @@ -19,6 +19,8 @@
>   #include "hw/hotplug.h"
>   #include "hw/acpi/cpu.h"
>   
> +#define ACPI_CPU_HOTPLUG_REG_LEN 12
> +
>   typedef struct AcpiCpuHotplug {
>       Object *device;
>       MemoryRegion io;

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[David Hildenbrand]

On 09.10.23 13:28, Salil Mehta wrote:
> ACPI CPU hotplug related initialization should only happen if ACPI_CPU_HOTPLUG
> support has been enabled for particular architecture. Add cpu_hotplug_hw_init()
> stub to avoid compilation break.
> 
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> ---
>   hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-stub.c
> index 3fc4b14c26..c6c61bb9cd 100644
> --- a/hw/acpi/acpi-cpu-hotplug-stub.c
> +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
> @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
>       return;
>   }
>   
> +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> +                         CPUHotplugState *state, hwaddr base_addr)
> +{
> +    return;
> +}

While at it, can we prefix that function with acpi?

-- 
Cheers,

David / dhildenb

Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[David Hildenbrand]

On 09.10.23 13:28, Salil Mehta wrote:
> ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI events
> when OSPM/guest receives an interrupt listed in the _CRS object of GED. OSPM
> then maps or demultiplexes the event by evaluating _EVT method.
> 
> This change adds the support of cpu hotplug event initialization in the
> existing GED framework.
> 
> Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
> Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> ---
>   hw/acpi/generic_event_device.c         | 8 ++++++++
>   include/hw/acpi/generic_event_device.h | 5 +++++
>   2 files changed, 13 insertions(+)
> 
> diff --git a/hw/acpi/generic_event_device.c b/hw/acpi/generic_event_device.c
> index a3d31631fe..d2fa1d0e4a 100644
> --- a/hw/acpi/generic_event_device.c
> +++ b/hw/acpi/generic_event_device.c
> @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
>       ACPI_GED_MEM_HOTPLUG_EVT,
>       ACPI_GED_PWR_DOWN_EVT,
>       ACPI_GED_NVDIMM_HOTPLUG_EVT,
> +    ACPI_GED_CPU_HOTPLUG_EVT,
>   };
>   
>   /*
> @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
>       memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
>                             TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
>       sysbus_init_mmio(sbd, &ged_st->regs);
> +
> +    s->cpuhp.device = OBJECT(s);
> +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp container",
> +                       ACPI_CPU_HOTPLUG_REG_LEN);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
> +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
> +                        &s->cpuhp_state, 0);
>   }
>   
>   static void acpi_ged_class_init(ObjectClass *class, void *data)
> diff --git a/include/hw/acpi/generic_event_device.h b/include/hw/acpi/generic_event_device.h
> index d831bbd889..d0a5a43abf 100644
> --- a/include/hw/acpi/generic_event_device.h
> +++ b/include/hw/acpi/generic_event_device.h
> @@ -60,6 +60,7 @@
>   #define HW_ACPI_GENERIC_EVENT_DEVICE_H
>   
>   #include "hw/sysbus.h"
> +#include "hw/acpi/cpu_hotplug.h"
>   #include "hw/acpi/memory_hotplug.h"
>   #include "hw/acpi/ghes.h"
>   #include "qom/object.h"
> @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
>   #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
>   #define ACPI_GED_PWR_DOWN_EVT      0x2
>   #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
> +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
>   
>   typedef struct GEDState {
>       MemoryRegion evt;
> @@ -108,6 +110,9 @@ struct AcpiGedState {
>       SysBusDevice parent_obj;
>       MemHotplugState memhp_state;
>       MemoryRegion container_memhp;
> +    CPUHotplugState cpuhp_state;
> +    MemoryRegion container_cpuhp;
> +    AcpiCpuHotplug cpuhp;

Am I wrong or is that member completely unused/uninitialized?

-- 
Cheers,

David / dhildenb

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta via]

Hi David,
Thanks for the review.

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:21 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > KVM vCPU creation is done once during the initialization of the VM when Qemu
> > thread is spawned. This is common to all the architectures.
> >
> > Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
> > corresponding KVM vCPU object in the Host KVM is not destroyed and its
> > representative KVM vCPU object/context in Qemu is parked.
> >
> > Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> 
> [...]
> 
> >
> >   int kvm_init_vcpu(CPUState *cpu, Error **errp)
> > @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >
> >       trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
> >
> > -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
> > +    ret = kvm_create_vcpu(cpu);
> >       if (ret < 0) {
> > -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
> > +        error_setg_errno(errp, -ret,
> > +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
> 
> Unrelated change.


It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
in the string.


> >                            kvm_arch_vcpu_id(cpu));
> >           goto err;
> >       }
> >
> > -    cpu->kvm_fd = ret;
> > -    cpu->kvm_state = s;
> > -    cpu->vcpu_dirty = true;
> > -    cpu->dirty_pages = 0;
> > -    cpu->throttle_us_per_full = 0;
> > -
> >       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
> >       if (mmap_size < 0) {
> >           ret = mmap_size;
> > diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
> > index 399aaeb0ec..08e2dc253f 100644
> > --- a/accel/kvm/trace-events
> > +++ b/accel/kvm/trace-events
> > @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
> >   kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
> >   kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
> >   kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
> > +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
> > +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
> > +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
> > +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
> 
> It's a bit confusing that there is now
> 
> 1) create (create new or return parked)
> 2) destroy (cleanup + park)
> 3) park (park only)
> 
> Why would one use 2) instead of 3) or the other way around? But I
> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
> helper ...

kvm_destroy_vcpu is more than just parking:

1. Arch destroy vcpu
2. Unmap cpu->kvm_run
3. Parking logic

To support virtual CPU Hotplug on ARM platforms we pre-create all
the KVM vCPUs but their corresponding Qemu threads are not spawned
(and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
parked in the list. Hence, only step-3 is required.

https://lore.kernel.org/qemu-devel/b9dd8569-e95d-2085-9965-08686ce6666d@redhat.com/

When a virtual CPU is plugged. QOM CPU object is realized and
corresponding thread is spawned. kvm_init_vcpu ends up in unaprking
the KVM vCPU, mapping of cpu->kvm_run and kvm_arch_init_vcpu.

When a virtul CPU is un-plugged, reverse of step-1, 2 and 3 is
required during un-realization of QOM CPU object. We do not destroy
vCPU inside the KVM.



> >   kvm_irqchip_commit_routes(void) ""
> >   kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
> >   kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index ee9025f8e9..57bd8f8fd6 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -464,6 +464,20 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
> >
> >   int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> >                                          hwaddr *phys_addr);
> > +/**
> > + * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
> > + * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
> > + *
> > + * @returns: 0 when success, errno (<0) when failed.
> > + */
> > +int kvm_create_vcpu(CPUState *cpu);
> > +/**
> > + * kvm_park_vcpu - Gets a parked KVM vCPU if it exists
> 
> 
> ^ I suspect that description is wrong.

Good catch. I think manual merge error while copying the change.
Will fix.

Thanks
Salil.

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta]

Hi David,
Thanks for the review.

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:21 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > KVM vCPU creation is done once during the initialization of the VM when Qemu
> > thread is spawned. This is common to all the architectures.
> >
> > Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
> > corresponding KVM vCPU object in the Host KVM is not destroyed and its
> > representative KVM vCPU object/context in Qemu is parked.
> >
> > Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> 
> [...]
> 
> >
> >   int kvm_init_vcpu(CPUState *cpu, Error **errp)
> > @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >
> >       trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
> >
> > -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
> > +    ret = kvm_create_vcpu(cpu);
> >       if (ret < 0) {
> > -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
> > +        error_setg_errno(errp, -ret,
> > +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
> 
> Unrelated change.


It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
in the string.


> >                            kvm_arch_vcpu_id(cpu));
> >           goto err;
> >       }
> >
> > -    cpu->kvm_fd = ret;
> > -    cpu->kvm_state = s;
> > -    cpu->vcpu_dirty = true;
> > -    cpu->dirty_pages = 0;
> > -    cpu->throttle_us_per_full = 0;
> > -
> >       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
> >       if (mmap_size < 0) {
> >           ret = mmap_size;
> > diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
> > index 399aaeb0ec..08e2dc253f 100644
> > --- a/accel/kvm/trace-events
> > +++ b/accel/kvm/trace-events
> > @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
> >   kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
> >   kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
> >   kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
> > +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
> > +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
> > +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
> > +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
> 
> It's a bit confusing that there is now
> 
> 1) create (create new or return parked)
> 2) destroy (cleanup + park)
> 3) park (park only)
> 
> Why would one use 2) instead of 3) or the other way around? But I
> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
> helper ...

kvm_destroy_vcpu is more than just parking:

1. Arch destroy vcpu
2. Unmap cpu->kvm_run
3. Parking logic

To support virtual CPU Hotplug on ARM platforms we pre-create all
the KVM vCPUs but their corresponding Qemu threads are not spawned
(and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
parked in the list. Hence, only step-3 is required.

https://lore.kernel.org/qemu-devel/b9dd8569-e95d-2085-9965-08686ce6666d@redhat.com/

When a virtual CPU is plugged. QOM CPU object is realized and
corresponding thread is spawned. kvm_init_vcpu ends up in unaprking
the KVM vCPU, mapping of cpu->kvm_run and kvm_arch_init_vcpu.

When a virtul CPU is un-plugged, reverse of step-1, 2 and 3 is
required during un-realization of QOM CPU object. We do not destroy
vCPU inside the KVM.



> >   kvm_irqchip_commit_routes(void) ""
> >   kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
> >   kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index ee9025f8e9..57bd8f8fd6 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -464,6 +464,20 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
> >
> >   int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> >                                          hwaddr *phys_addr);
> > +/**
> > + * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
> > + * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
> > + *
> > + * @returns: 0 when success, errno (<0) when failed.
> > + */
> > +int kvm_create_vcpu(CPUState *cpu);
> > +/**
> > + * kvm_park_vcpu - Gets a parked KVM vCPU if it exists
> 
> 
> ^ I suspect that description is wrong.

Good catch. I think manual merge error while copying the change.
Will fix.

Thanks
Salil.

RE: [PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:22 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len
> macro to common header file
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > CPU ctrl-dev MMIO region length could be used in ACPI GED and various
> other
> > architecture specific places. Move ACPI_CPU_HOTPLUG_REG_LEN macro to more
> > appropriate common header file.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/cpu.c                 | 2 +-
> >   include/hw/acpi/cpu_hotplug.h | 2 ++
> >   2 files changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 19c154d78f..45defdc0e2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -1,12 +1,12 @@
> >   #include "qemu/osdep.h"
> >   #include "migration/vmstate.h"
> >   #include "hw/acpi/cpu.h"
> > +#include "hw/acpi/cpu_hotplug.h"
> >   #include "qapi/error.h"
> >   #include "qapi/qapi-events-acpi.h"
> >   #include "trace.h"
> >   #include "sysemu/numa.h"
> >
> > -#define ACPI_CPU_HOTPLUG_REG_LEN 12
> >   #define ACPI_CPU_SELECTOR_OFFSET_WR 0
> >   #define ACPI_CPU_FLAGS_OFFSET_RW 4
> >   #define ACPI_CPU_CMD_OFFSET_WR 5
> > diff --git a/include/hw/acpi/cpu_hotplug.h
> b/include/hw/acpi/cpu_hotplug.h
> > index 3b932abbbb..48b291e45e 100644
> > --- a/include/hw/acpi/cpu_hotplug.h
> > +++ b/include/hw/acpi/cpu_hotplug.h
> > @@ -19,6 +19,8 @@
> >   #include "hw/hotplug.h"
> >   #include "hw/acpi/cpu.h"
> >
> > +#define ACPI_CPU_HOTPLUG_REG_LEN 12
> > +
> >   typedef struct AcpiCpuHotplug {
> >       Object *device;
> >       MemoryRegion io;
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>

Thanks
Salil.

RE: [PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:22 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len
> macro to common header file
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > CPU ctrl-dev MMIO region length could be used in ACPI GED and various
> other
> > architecture specific places. Move ACPI_CPU_HOTPLUG_REG_LEN macro to more
> > appropriate common header file.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/cpu.c                 | 2 +-
> >   include/hw/acpi/cpu_hotplug.h | 2 ++
> >   2 files changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> > index 19c154d78f..45defdc0e2 100644
> > --- a/hw/acpi/cpu.c
> > +++ b/hw/acpi/cpu.c
> > @@ -1,12 +1,12 @@
> >   #include "qemu/osdep.h"
> >   #include "migration/vmstate.h"
> >   #include "hw/acpi/cpu.h"
> > +#include "hw/acpi/cpu_hotplug.h"
> >   #include "qapi/error.h"
> >   #include "qapi/qapi-events-acpi.h"
> >   #include "trace.h"
> >   #include "sysemu/numa.h"
> >
> > -#define ACPI_CPU_HOTPLUG_REG_LEN 12
> >   #define ACPI_CPU_SELECTOR_OFFSET_WR 0
> >   #define ACPI_CPU_FLAGS_OFFSET_RW 4
> >   #define ACPI_CPU_CMD_OFFSET_WR 5
> > diff --git a/include/hw/acpi/cpu_hotplug.h
> b/include/hw/acpi/cpu_hotplug.h
> > index 3b932abbbb..48b291e45e 100644
> > --- a/include/hw/acpi/cpu_hotplug.h
> > +++ b/include/hw/acpi/cpu_hotplug.h
> > @@ -19,6 +19,8 @@
> >   #include "hw/hotplug.h"
> >   #include "hw/acpi/cpu.h"
> >
> > +#define ACPI_CPU_HOTPLUG_REG_LEN 12
> > +
> >   typedef struct AcpiCpuHotplug {
> >       Object *device;
> >       MemoryRegion io;
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>

Thanks
Salil.

RE: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:23 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > ACPI CPU hotplug related initialization should only happen if
> ACPI_CPU_HOTPLUG
> > support has been enabled for particular architecture. Add
> cpu_hotplug_hw_init()
> > stub to avoid compilation break.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
> >   1 file changed, 6 insertions(+)
> >
> > diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-
> stub.c
> > index 3fc4b14c26..c6c61bb9cd 100644
> > --- a/hw/acpi/acpi-cpu-hotplug-stub.c
> > +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
> > @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion
> *parent, Object *owner,
> >       return;
> >   }
> >
> > +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> > +                         CPUHotplugState *state, hwaddr base_addr)
> > +{
> > +    return;
> > +}
> 
> While at it, can we prefix that function with acpi?

I can do that but it has to be done at other places as well
such as in hw/acpi/cpu_hotplug.c <acpi_switch_to_modern_cphp()>

Hope this is not an extraneous change to this patch-set?

Thanks
Salil.

RE: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:23 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > ACPI CPU hotplug related initialization should only happen if
> ACPI_CPU_HOTPLUG
> > support has been enabled for particular architecture. Add
> cpu_hotplug_hw_init()
> > stub to avoid compilation break.
> >
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
> >   1 file changed, 6 insertions(+)
> >
> > diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-
> stub.c
> > index 3fc4b14c26..c6c61bb9cd 100644
> > --- a/hw/acpi/acpi-cpu-hotplug-stub.c
> > +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
> > @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion
> *parent, Object *owner,
> >       return;
> >   }
> >
> > +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> > +                         CPUHotplugState *state, hwaddr base_addr)
> > +{
> > +    return;
> > +}
> 
> While at it, can we prefix that function with acpi?

I can do that but it has to be done at other places as well
such as in hw/acpi/cpu_hotplug.c <acpi_switch_to_modern_cphp()>

Hope this is not an extraneous change to this patch-set?

Thanks
Salil.

Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[David Hildenbrand]

On 09.10.23 15:49, Salil Mehta wrote:
>> From: David Hildenbrand <david@redhat.com>
>> Sent: Monday, October 9, 2023 1:23 PM
>> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
>> arm@nongnu.org
>> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
>> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>> peter.maydell@linaro.org; richard.henderson@linaro.org;
>> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
>> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
>> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
>> alex.bennee@linaro.org; linux@armlinux.org.uk;
>> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
>> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
>> Subject: Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub
>>
>> On 09.10.23 13:28, Salil Mehta wrote:
>>> ACPI CPU hotplug related initialization should only happen if
>> ACPI_CPU_HOTPLUG
>>> support has been enabled for particular architecture. Add
>> cpu_hotplug_hw_init()
>>> stub to avoid compilation break.
>>>
>>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> Reviewed-by: Gavin Shan <gshan@redhat.com>
>>> ---
>>>    hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
>>>    1 file changed, 6 insertions(+)
>>>
>>> diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-hotplug-
>> stub.c
>>> index 3fc4b14c26..c6c61bb9cd 100644
>>> --- a/hw/acpi/acpi-cpu-hotplug-stub.c
>>> +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
>>> @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion
>> *parent, Object *owner,
>>>        return;
>>>    }
>>>
>>> +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
>>> +                         CPUHotplugState *state, hwaddr base_addr)
>>> +{
>>> +    return;
>>> +}
>>
>> While at it, can we prefix that function with acpi?
> 
> I can do that but it has to be done at other places as well
> such as in hw/acpi/cpu_hotplug.c <acpi_switch_to_modern_cphp()>
> 

$ git grep cpu_hotplug_hw_init
hw/acpi/cpu.c:void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
hw/acpi/cpu_hotplug.c:    cpu_hotplug_hw_init(parent, gpe_cpu->device, cpuhp_state, io_port);
include/hw/acpi/cpu.h:void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,

Might want to do that as a separate patch, agreed.

-- 
Cheers,

David / dhildenb

Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[David Hildenbrand]

On 09.10.23 15:42, Salil Mehta wrote:
> Hi David,
> Thanks for the review.
> 
>> From: David Hildenbrand <david@redhat.com>
>> Sent: Monday, October 9, 2023 1:21 PM
>> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
>> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
>> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>> peter.maydell@linaro.org; richard.henderson@linaro.org;
>> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
>> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
>> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
>> alex.bennee@linaro.org; linux@armlinux.org.uk;
>> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
>> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
>> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
>> {creation,parking} code
>>
>> On 09.10.23 13:28, Salil Mehta wrote:
>>> KVM vCPU creation is done once during the initialization of the VM when Qemu
>>> thread is spawned. This is common to all the architectures.
>>>
>>> Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
>>> corresponding KVM vCPU object in the Host KVM is not destroyed and its
>>> representative KVM vCPU object/context in Qemu is parked.
>>>
>>> Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
>>>
>>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>>
>> [...]
>>
>>>
>>>    int kvm_init_vcpu(CPUState *cpu, Error **errp)
>>> @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
>>>
>>>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
>>>
>>> -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
>>> +    ret = kvm_create_vcpu(cpu);
>>>        if (ret < 0) {
>>> -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
>>> +        error_setg_errno(errp, -ret,
>>> +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
>>
>> Unrelated change.
> 
> 
> It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
> in the string.

Indeed, I did :)

> 
> 
>>>                             kvm_arch_vcpu_id(cpu));
>>>            goto err;
>>>        }
>>>
>>> -    cpu->kvm_fd = ret;
>>> -    cpu->kvm_state = s;
>>> -    cpu->vcpu_dirty = true;
>>> -    cpu->dirty_pages = 0;
>>> -    cpu->throttle_us_per_full = 0;
>>> -
>>>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
>>>        if (mmap_size < 0) {
>>>            ret = mmap_size;
>>> diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
>>> index 399aaeb0ec..08e2dc253f 100644
>>> --- a/accel/kvm/trace-events
>>> +++ b/accel/kvm/trace-events
>>> @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
>>>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
>>>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
>>>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
>>> +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
>>> +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
>>> +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
>>> +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
>>
>> It's a bit confusing that there is now
>>
>> 1) create (create new or return parked)
>> 2) destroy (cleanup + park)
>> 3) park (park only)
>>
>> Why would one use 2) instead of 3) or the other way around? But I
>> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
>> helper ...
> 
> kvm_destroy_vcpu is more than just parking:
> 
> 1. Arch destroy vcpu
> 2. Unmap cpu->kvm_run
> 3. Parking logic
> 
> To support virtual CPU Hotplug on ARM platforms we pre-create all
> the KVM vCPUs but their corresponding Qemu threads are not spawned
> (and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
> parked in the list. Hence, only step-3 is required.

IIUC, your current flow is going to be

1) Create
2) Park
3) Create [which ends up reusing the parked VCPU]
4) Destroy [when unplugging the CPU]

If that's the case, that API really is suboptimal.

What speaks against an API that models 1) and 2) in a single step

kvm_precreate_vcpu
kvm_create_vcpu
kvm_destroy_vcpu

One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't 
been pre-created.

Or did I get it all wrong? :)

-- 
Cheers,

David / dhildenb

RE: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:27 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug
> events
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI
> events
> > when OSPM/guest receives an interrupt listed in the _CRS object of GED.
> OSPM
> > then maps or demultiplexes the event by evaluating _EVT method.
> >
> > This change adds the support of cpu hotplug event initialization in the
> > existing GED framework.
> >
> > Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
> > Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/generic_event_device.c         | 8 ++++++++
> >   include/hw/acpi/generic_event_device.h | 5 +++++
> >   2 files changed, 13 insertions(+)
> >
> > diff --git a/hw/acpi/generic_event_device.c
> b/hw/acpi/generic_event_device.c
> > index a3d31631fe..d2fa1d0e4a 100644
> > --- a/hw/acpi/generic_event_device.c
> > +++ b/hw/acpi/generic_event_device.c
> > @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
> >       ACPI_GED_MEM_HOTPLUG_EVT,
> >       ACPI_GED_PWR_DOWN_EVT,
> >       ACPI_GED_NVDIMM_HOTPLUG_EVT,
> > +    ACPI_GED_CPU_HOTPLUG_EVT,
> >   };
> >
> >   /*
> > @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
> >       memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
> >                             TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
> >       sysbus_init_mmio(sbd, &ged_st->regs);
> > +
> > +    s->cpuhp.device = OBJECT(s);
> > +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp
> container",
> > +                       ACPI_CPU_HOTPLUG_REG_LEN);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
> > +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
> > +                        &s->cpuhp_state, 0);
> >   }
> >
> >   static void acpi_ged_class_init(ObjectClass *class, void *data)
> > diff --git a/include/hw/acpi/generic_event_device.h
> b/include/hw/acpi/generic_event_device.h
> > index d831bbd889..d0a5a43abf 100644
> > --- a/include/hw/acpi/generic_event_device.h
> > +++ b/include/hw/acpi/generic_event_device.h
> > @@ -60,6 +60,7 @@
> >   #define HW_ACPI_GENERIC_EVENT_DEVICE_H
> >
> >   #include "hw/sysbus.h"
> > +#include "hw/acpi/cpu_hotplug.h"
> >   #include "hw/acpi/memory_hotplug.h"
> >   #include "hw/acpi/ghes.h"
> >   #include "qom/object.h"
> > @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
> >   #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
> >   #define ACPI_GED_PWR_DOWN_EVT      0x2
> >   #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
> > +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
> >
> >   typedef struct GEDState {
> >       MemoryRegion evt;
> > @@ -108,6 +110,9 @@ struct AcpiGedState {
> >       SysBusDevice parent_obj;
> >       MemHotplugState memhp_state;
> >       MemoryRegion container_memhp;
> > +    CPUHotplugState cpuhp_state;
> > +    MemoryRegion container_cpuhp;
> > +    AcpiCpuHotplug cpuhp;
> 
> Am I wrong or is that member completely unused/uninitialized?

No it is not. Please check below change in acpi_ged_initfn()

+    s->cpuhp.device = OBJECT(s);


Thanks
Salil.

RE: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 1:27 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug
> events
> 
> On 09.10.23 13:28, Salil Mehta wrote:
> > ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI
> events
> > when OSPM/guest receives an interrupt listed in the _CRS object of GED.
> OSPM
> > then maps or demultiplexes the event by evaluating _EVT method.
> >
> > This change adds the support of cpu hotplug event initialization in the
> > existing GED framework.
> >
> > Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
> > Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
> > Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Reviewed-by: Gavin Shan <gshan@redhat.com>
> > ---
> >   hw/acpi/generic_event_device.c         | 8 ++++++++
> >   include/hw/acpi/generic_event_device.h | 5 +++++
> >   2 files changed, 13 insertions(+)
> >
> > diff --git a/hw/acpi/generic_event_device.c
> b/hw/acpi/generic_event_device.c
> > index a3d31631fe..d2fa1d0e4a 100644
> > --- a/hw/acpi/generic_event_device.c
> > +++ b/hw/acpi/generic_event_device.c
> > @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
> >       ACPI_GED_MEM_HOTPLUG_EVT,
> >       ACPI_GED_PWR_DOWN_EVT,
> >       ACPI_GED_NVDIMM_HOTPLUG_EVT,
> > +    ACPI_GED_CPU_HOTPLUG_EVT,
> >   };
> >
> >   /*
> > @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
> >       memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
> >                             TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
> >       sysbus_init_mmio(sbd, &ged_st->regs);
> > +
> > +    s->cpuhp.device = OBJECT(s);
> > +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp
> container",
> > +                       ACPI_CPU_HOTPLUG_REG_LEN);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
> > +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
> > +                        &s->cpuhp_state, 0);
> >   }
> >
> >   static void acpi_ged_class_init(ObjectClass *class, void *data)
> > diff --git a/include/hw/acpi/generic_event_device.h
> b/include/hw/acpi/generic_event_device.h
> > index d831bbd889..d0a5a43abf 100644
> > --- a/include/hw/acpi/generic_event_device.h
> > +++ b/include/hw/acpi/generic_event_device.h
> > @@ -60,6 +60,7 @@
> >   #define HW_ACPI_GENERIC_EVENT_DEVICE_H
> >
> >   #include "hw/sysbus.h"
> > +#include "hw/acpi/cpu_hotplug.h"
> >   #include "hw/acpi/memory_hotplug.h"
> >   #include "hw/acpi/ghes.h"
> >   #include "qom/object.h"
> > @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
> >   #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
> >   #define ACPI_GED_PWR_DOWN_EVT      0x2
> >   #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
> > +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
> >
> >   typedef struct GEDState {
> >       MemoryRegion evt;
> > @@ -108,6 +110,9 @@ struct AcpiGedState {
> >       SysBusDevice parent_obj;
> >       MemHotplugState memhp_state;
> >       MemoryRegion container_memhp;
> > +    CPUHotplugState cpuhp_state;
> > +    MemoryRegion container_cpuhp;
> > +    AcpiCpuHotplug cpuhp;
> 
> Am I wrong or is that member completely unused/uninitialized?

No it is not. Please check below change in acpi_ged_initfn()

+    s->cpuhp.device = OBJECT(s);


Thanks
Salil.

Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[David Hildenbrand]

On 09.10.23 16:12, Salil Mehta wrote:
>> From: David Hildenbrand <david@redhat.com>
>> Sent: Monday, October 9, 2023 1:27 PM
>> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
>> arm@nongnu.org
>> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
>> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>> peter.maydell@linaro.org; richard.henderson@linaro.org;
>> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
>> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
>> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
>> alex.bennee@linaro.org; linux@armlinux.org.uk;
>> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
>> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
>> Subject: Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug
>> events
>>
>> On 09.10.23 13:28, Salil Mehta wrote:
>>> ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI
>> events
>>> when OSPM/guest receives an interrupt listed in the _CRS object of GED.
>> OSPM
>>> then maps or demultiplexes the event by evaluating _EVT method.
>>>
>>> This change adds the support of cpu hotplug event initialization in the
>>> existing GED framework.
>>>
>>> Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
>>> Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
>>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> Reviewed-by: Gavin Shan <gshan@redhat.com>
>>> ---
>>>    hw/acpi/generic_event_device.c         | 8 ++++++++
>>>    include/hw/acpi/generic_event_device.h | 5 +++++
>>>    2 files changed, 13 insertions(+)
>>>
>>> diff --git a/hw/acpi/generic_event_device.c
>> b/hw/acpi/generic_event_device.c
>>> index a3d31631fe..d2fa1d0e4a 100644
>>> --- a/hw/acpi/generic_event_device.c
>>> +++ b/hw/acpi/generic_event_device.c
>>> @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
>>>        ACPI_GED_MEM_HOTPLUG_EVT,
>>>        ACPI_GED_PWR_DOWN_EVT,
>>>        ACPI_GED_NVDIMM_HOTPLUG_EVT,
>>> +    ACPI_GED_CPU_HOTPLUG_EVT,
>>>    };
>>>
>>>    /*
>>> @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
>>>        memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
>>>                              TYPE_ACPI_GED "-regs", ACPI_GED_REG_COUNT);
>>>        sysbus_init_mmio(sbd, &ged_st->regs);
>>> +
>>> +    s->cpuhp.device = OBJECT(s);
>>> +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp
>> container",
>>> +                       ACPI_CPU_HOTPLUG_REG_LEN);
>>> +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
>>> +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
>>> +                        &s->cpuhp_state, 0);
>>>    }
>>>
>>>    static void acpi_ged_class_init(ObjectClass *class, void *data)
>>> diff --git a/include/hw/acpi/generic_event_device.h
>> b/include/hw/acpi/generic_event_device.h
>>> index d831bbd889..d0a5a43abf 100644
>>> --- a/include/hw/acpi/generic_event_device.h
>>> +++ b/include/hw/acpi/generic_event_device.h
>>> @@ -60,6 +60,7 @@
>>>    #define HW_ACPI_GENERIC_EVENT_DEVICE_H
>>>
>>>    #include "hw/sysbus.h"
>>> +#include "hw/acpi/cpu_hotplug.h"
>>>    #include "hw/acpi/memory_hotplug.h"
>>>    #include "hw/acpi/ghes.h"
>>>    #include "qom/object.h"
>>> @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
>>>    #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
>>>    #define ACPI_GED_PWR_DOWN_EVT      0x2
>>>    #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
>>> +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
>>>
>>>    typedef struct GEDState {
>>>        MemoryRegion evt;
>>> @@ -108,6 +110,9 @@ struct AcpiGedState {
>>>        SysBusDevice parent_obj;
>>>        MemHotplugState memhp_state;
>>>        MemoryRegion container_memhp;
>>> +    CPUHotplugState cpuhp_state;
>>> +    MemoryRegion container_cpuhp;
>>> +    AcpiCpuHotplug cpuhp;
>>
>> Am I wrong or is that member completely unused/uninitialized?
> 
> No it is not. Please check below change in acpi_ged_initfn()
> 
> +    s->cpuhp.device = OBJECT(s);

Not the best of my mondays, sorry for that.

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 3:11 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> On 09.10.23 15:42, Salil Mehta wrote:
> > Hi David,
> > Thanks for the review.
> >
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:21 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> >> {creation,parking} code
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> KVM vCPU creation is done once during the initialization of the VM when Qemu
> >>> thread is spawned. This is common to all the architectures.
> >>>
> >>> Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
> >>> corresponding KVM vCPU object in the Host KVM is not destroyed and its
> >>> representative KVM vCPU object/context in Qemu is parked.
> >>>
> >>> Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
> >>>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>
> >> [...]
> >>
> >>>
> >>>    int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >>> @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >>>
> >>>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
> >>>
> >>> -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
> >>> +    ret = kvm_create_vcpu(cpu);
> >>>        if (ret < 0) {
> >>> -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
> >>> +        error_setg_errno(errp, -ret,
> >>> +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
> >>
> >> Unrelated change.
> >
> >
> > It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
> > in the string.
> 
> Indeed, I did :)
> 
> >
> >
> >>>                             kvm_arch_vcpu_id(cpu));
> >>>            goto err;
> >>>        }
> >>>
> >>> -    cpu->kvm_fd = ret;
> >>> -    cpu->kvm_state = s;
> >>> -    cpu->vcpu_dirty = true;
> >>> -    cpu->dirty_pages = 0;
> >>> -    cpu->throttle_us_per_full = 0;
> >>> -
> >>>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
> >>>        if (mmap_size < 0) {
> >>>            ret = mmap_size;
> >>> diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
> >>> index 399aaeb0ec..08e2dc253f 100644
> >>> --- a/accel/kvm/trace-events
> >>> +++ b/accel/kvm/trace-events
> >>> @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
> >>>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
> >>>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
> >>>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
> >>> +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
> >>> +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
> >>> +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
> >>> +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
> >>
> >> It's a bit confusing that there is now
> >>
> >> 1) create (create new or return parked)
> >> 2) destroy (cleanup + park)
> >> 3) park (park only)
> >>
> >> Why would one use 2) instead of 3) or the other way around? But I
> >> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
> >> helper ...
> >
> > kvm_destroy_vcpu is more than just parking:
> >
> > 1. Arch destroy vcpu
> > 2. Unmap cpu->kvm_run
> > 3. Parking logic
> >
> > To support virtual CPU Hotplug on ARM platforms we pre-create all
> > the KVM vCPUs but their corresponding Qemu threads are not spawned
> > (and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
> > parked in the list. Hence, only step-3 is required.
> 
> IIUC, your current flow is going to be
> 
> 1) Create
> 2) Park
> 3) Create [which ends up reusing the parked VCPU]
> 4) Destroy [when unplugging the CPU]


In the ARM specific code, Yes.

 
> If that's the case, that API really is suboptimal.
> 
> What speaks against an API that models 1) and 2) in a single step


API is generic and is part of architecture agnostic code.


> 
> kvm_precreate_vcpu

pre-creation is very much specific to ARM right now. I am not sure
if it is right to have an API with this name in the code which is
common to other architectures.


> kvm_create_vcpu
> kvm_destroy_vcpu
> 
> One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't
> been pre-created.

Right now, we abort the CPU initialization process if this happens. I
am planning to change abort() into 'fatal_error' in RFC V3 though.



> 
> Or did I get it all wrong? :)

I won't say that it is just another point of view which is absolutely
fine. But I would like to stick to current APIs.


Thanks
Salil.

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 3:11 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> On 09.10.23 15:42, Salil Mehta wrote:
> > Hi David,
> > Thanks for the review.
> >
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:21 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> >> {creation,parking} code
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> KVM vCPU creation is done once during the initialization of the VM when Qemu
> >>> thread is spawned. This is common to all the architectures.
> >>>
> >>> Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
> >>> corresponding KVM vCPU object in the Host KVM is not destroyed and its
> >>> representative KVM vCPU object/context in Qemu is parked.
> >>>
> >>> Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
> >>>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>
> >> [...]
> >>
> >>>
> >>>    int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >>> @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
> >>>
> >>>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
> >>>
> >>> -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
> >>> +    ret = kvm_create_vcpu(cpu);
> >>>        if (ret < 0) {
> >>> -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
> >>> +        error_setg_errno(errp, -ret,
> >>> +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
> >>
> >> Unrelated change.
> >
> >
> > It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
> > in the string.
> 
> Indeed, I did :)
> 
> >
> >
> >>>                             kvm_arch_vcpu_id(cpu));
> >>>            goto err;
> >>>        }
> >>>
> >>> -    cpu->kvm_fd = ret;
> >>> -    cpu->kvm_state = s;
> >>> -    cpu->vcpu_dirty = true;
> >>> -    cpu->dirty_pages = 0;
> >>> -    cpu->throttle_us_per_full = 0;
> >>> -
> >>>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
> >>>        if (mmap_size < 0) {
> >>>            ret = mmap_size;
> >>> diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
> >>> index 399aaeb0ec..08e2dc253f 100644
> >>> --- a/accel/kvm/trace-events
> >>> +++ b/accel/kvm/trace-events
> >>> @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
> >>>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
> >>>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
> >>>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
> >>> +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
> >>> +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
> >>> +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
> >>> +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
> >>
> >> It's a bit confusing that there is now
> >>
> >> 1) create (create new or return parked)
> >> 2) destroy (cleanup + park)
> >> 3) park (park only)
> >>
> >> Why would one use 2) instead of 3) or the other way around? But I
> >> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
> >> helper ...
> >
> > kvm_destroy_vcpu is more than just parking:
> >
> > 1. Arch destroy vcpu
> > 2. Unmap cpu->kvm_run
> > 3. Parking logic
> >
> > To support virtual CPU Hotplug on ARM platforms we pre-create all
> > the KVM vCPUs but their corresponding Qemu threads are not spawned
> > (and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
> > parked in the list. Hence, only step-3 is required.
> 
> IIUC, your current flow is going to be
> 
> 1) Create
> 2) Park
> 3) Create [which ends up reusing the parked VCPU]
> 4) Destroy [when unplugging the CPU]


In the ARM specific code, Yes.

 
> If that's the case, that API really is suboptimal.
> 
> What speaks against an API that models 1) and 2) in a single step


API is generic and is part of architecture agnostic code.


> 
> kvm_precreate_vcpu

pre-creation is very much specific to ARM right now. I am not sure
if it is right to have an API with this name in the code which is
common to other architectures.


> kvm_create_vcpu
> kvm_destroy_vcpu
> 
> One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't
> been pre-created.

Right now, we abort the CPU initialization process if this happens. I
am planning to change abort() into 'fatal_error' in RFC V3 though.



> 
> Or did I get it all wrong? :)

I won't say that it is just another point of view which is absolutely
fine. But I would like to stick to current APIs.


Thanks
Salil.

Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[David Hildenbrand]

> 
>>
>> kvm_precreate_vcpu
> 
> pre-creation is very much specific to ARM right now. I am not sure
> if it is right to have an API with this name in the code which is
> common to other architectures.

I don't like exposing the concept of "parking" CPUs externally, which is 
so far handled completely internally.

[...]

> 
> 
>> kvm_create_vcpu
>> kvm_destroy_vcpu
>>
>> One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't
>> been pre-created.
> 
> Right now, we abort the CPU initialization process if this happens. I
> am planning to change abort() into 'fatal_error' in RFC V3 though.
> 
> 
> 
>>
>> Or did I get it all wrong? :)
> 
> I won't say that it is just another point of view which is absolutely
> fine. But I would like to stick to current APIs.

No really strong opinion, I wouldn't do it that way. I'll let others 
chime in if they have an opinion.

-- 
Cheers,

David / dhildenb

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 4:21 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> >
> >>
> >> kvm_precreate_vcpu
> >
> > pre-creation is very much specific to ARM right now. I am not sure
> > if it is right to have an API with this name in the code which is
> > common to other architectures.
> 
> I don't like exposing the concept of "parking" CPUs externally, which is
> so far handled completely internally.


I understand your point of view. There is a subtle difference in the
way parking logic has been used till now, say in x86 world and how it
is being used in the ARM in the RFC patches being proposed. AFAICS, in
x86 world we have a liberty to delay the creation of the vCPUs in KVM
for the first time but once they are created cannot be destroyed in the
KVM so are (un)parked for subsequent use during hot(un)plug.

Because of the ARM CPU architecture limitations and that of GIC, we
are not allowed to do this. Hence, we have to pre-create all the
KVM vCPUs and size VGIC during initialization. Since some of the
KVM vCPUs wont have any QOM CPU objects because they are
'yet-to-be-plugged' so need to be parked. Hence, we require that
common parking logic.



> 
> [...]
> 
> >
> >
> >> kvm_create_vcpu
> >> kvm_destroy_vcpu
> >>
> >> One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't
> >> been pre-created.
> >
> > Right now, we abort the CPU initialization process if this happens. I
> > am planning to change abort() into 'fatal_error' in RFC V3 though.
> >
> >
> >
> >>
> >> Or did I get it all wrong? :)
> >
> > I won't say that it is just another point of view which is absolutely
> > fine. But I would like to stick to current APIs.
> 
> No really strong opinion, I wouldn't do it that way. I'll let others
> chime in if they have an opinion.

Ok, Thanks.

Salil.

RE: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 4:21 PM
> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> arm@nongnu.org
> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> peter.maydell@linaro.org; richard.henderson@linaro.org;
> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> alex.bennee@linaro.org; linux@armlinux.org.uk;
> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
> {creation,parking} code
> 
> >
> >>
> >> kvm_precreate_vcpu
> >
> > pre-creation is very much specific to ARM right now. I am not sure
> > if it is right to have an API with this name in the code which is
> > common to other architectures.
> 
> I don't like exposing the concept of "parking" CPUs externally, which is
> so far handled completely internally.


I understand your point of view. There is a subtle difference in the
way parking logic has been used till now, say in x86 world and how it
is being used in the ARM in the RFC patches being proposed. AFAICS, in
x86 world we have a liberty to delay the creation of the vCPUs in KVM
for the first time but once they are created cannot be destroyed in the
KVM so are (un)parked for subsequent use during hot(un)plug.

Because of the ARM CPU architecture limitations and that of GIC, we
are not allowed to do this. Hence, we have to pre-create all the
KVM vCPUs and size VGIC during initialization. Since some of the
KVM vCPUs wont have any QOM CPU objects because they are
'yet-to-be-plugged' so need to be parked. Hence, we require that
common parking logic.



> 
> [...]
> 
> >
> >
> >> kvm_create_vcpu
> >> kvm_destroy_vcpu
> >>
> >> One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't
> >> been pre-created.
> >
> > Right now, we abort the CPU initialization process if this happens. I
> > am planning to change abort() into 'fatal_error' in RFC V3 though.
> >
> >
> >
> >>
> >> Or did I get it all wrong? :)
> >
> > I won't say that it is just another point of view which is absolutely
> > fine. But I would like to stick to current APIs.
> 
> No really strong opinion, I wouldn't do it that way. I'll let others
> chime in if they have an opinion.

Ok, Thanks.

Salil.

RE: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 3:14 PM
> 
> On 09.10.23 16:12, Salil Mehta wrote:
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:27 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug
> >> events
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI
> >> events
> >>> when OSPM/guest receives an interrupt listed in the _CRS object of GED.
> >> OSPM
> >>> then maps or demultiplexes the event by evaluating _EVT method.
> >>>
> >>> This change adds the support of cpu hotplug event initialization in the
> >>> existing GED framework.
> >>>
> >>> Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
> >>> Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> >>> Reviewed-by: Gavin Shan <gshan@redhat.com>
> >>> ---
> >>>    hw/acpi/generic_event_device.c         | 8 ++++++++
> >>>    include/hw/acpi/generic_event_device.h | 5 +++++
> >>>    2 files changed, 13 insertions(+)
> >>>
> >>> diff --git a/hw/acpi/generic_event_device.c
> >> b/hw/acpi/generic_event_device.c
> >>> index a3d31631fe..d2fa1d0e4a 100644
> >>> --- a/hw/acpi/generic_event_device.c
> >>> +++ b/hw/acpi/generic_event_device.c
> >>> @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
> >>>        ACPI_GED_MEM_HOTPLUG_EVT,
> >>>        ACPI_GED_PWR_DOWN_EVT,
> >>>        ACPI_GED_NVDIMM_HOTPLUG_EVT,
> >>> +    ACPI_GED_CPU_HOTPLUG_EVT,
> >>>    };
> >>>
> >>>    /*
> >>> @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
> >>>        memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
> >>>                              TYPE_ACPI_GED "-regs",
> ACPI_GED_REG_COUNT);
> >>>        sysbus_init_mmio(sbd, &ged_st->regs);
> >>> +
> >>> +    s->cpuhp.device = OBJECT(s);
> >>> +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp
> >> container",
> >>> +                       ACPI_CPU_HOTPLUG_REG_LEN);
> >>> +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
> >>> +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
> >>> +                        &s->cpuhp_state, 0);
> >>>    }
> >>>
> >>>    static void acpi_ged_class_init(ObjectClass *class, void *data)
> >>> diff --git a/include/hw/acpi/generic_event_device.h
> >> b/include/hw/acpi/generic_event_device.h
> >>> index d831bbd889..d0a5a43abf 100644
> >>> --- a/include/hw/acpi/generic_event_device.h
> >>> +++ b/include/hw/acpi/generic_event_device.h
> >>> @@ -60,6 +60,7 @@
> >>>    #define HW_ACPI_GENERIC_EVENT_DEVICE_H
> >>>
> >>>    #include "hw/sysbus.h"
> >>> +#include "hw/acpi/cpu_hotplug.h"
> >>>    #include "hw/acpi/memory_hotplug.h"
> >>>    #include "hw/acpi/ghes.h"
> >>>    #include "qom/object.h"
> >>> @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
> >>>    #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
> >>>    #define ACPI_GED_PWR_DOWN_EVT      0x2
> >>>    #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
> >>> +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
> >>>
> >>>    typedef struct GEDState {
> >>>        MemoryRegion evt;
> >>> @@ -108,6 +110,9 @@ struct AcpiGedState {
> >>>        SysBusDevice parent_obj;
> >>>        MemHotplugState memhp_state;
> >>>        MemoryRegion container_memhp;
> >>> +    CPUHotplugState cpuhp_state;
> >>> +    MemoryRegion container_cpuhp;
> >>> +    AcpiCpuHotplug cpuhp;
> >>
> >> Am I wrong or is that member completely unused/uninitialized?
> >
> > No it is not. Please check below change in acpi_ged_initfn()
> >
> > +    s->cpuhp.device = OBJECT(s);
> 
> Not the best of my mondays, sorry for that.
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>

No issues at all. Thanks for taking time to review this.

Best regards
Salil.

RE: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug events

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 3:14 PM
> 
> On 09.10.23 16:12, Salil Mehta wrote:
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:27 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 04/10] hw/acpi: Init GED framework with cpu hotplug
> >> events
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> ACPI GED(as described in the ACPI 6.2 spec) can be used to generate ACPI
> >> events
> >>> when OSPM/guest receives an interrupt listed in the _CRS object of GED.
> >> OSPM
> >>> then maps or demultiplexes the event by evaluating _EVT method.
> >>>
> >>> This change adds the support of cpu hotplug event initialization in the
> >>> existing GED framework.
> >>>
> >>> Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
> >>> Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> >>> Reviewed-by: Gavin Shan <gshan@redhat.com>
> >>> ---
> >>>    hw/acpi/generic_event_device.c         | 8 ++++++++
> >>>    include/hw/acpi/generic_event_device.h | 5 +++++
> >>>    2 files changed, 13 insertions(+)
> >>>
> >>> diff --git a/hw/acpi/generic_event_device.c
> >> b/hw/acpi/generic_event_device.c
> >>> index a3d31631fe..d2fa1d0e4a 100644
> >>> --- a/hw/acpi/generic_event_device.c
> >>> +++ b/hw/acpi/generic_event_device.c
> >>> @@ -25,6 +25,7 @@ static const uint32_t ged_supported_events[] = {
> >>>        ACPI_GED_MEM_HOTPLUG_EVT,
> >>>        ACPI_GED_PWR_DOWN_EVT,
> >>>        ACPI_GED_NVDIMM_HOTPLUG_EVT,
> >>> +    ACPI_GED_CPU_HOTPLUG_EVT,
> >>>    };
> >>>
> >>>    /*
> >>> @@ -400,6 +401,13 @@ static void acpi_ged_initfn(Object *obj)
> >>>        memory_region_init_io(&ged_st->regs, obj, &ged_regs_ops, ged_st,
> >>>                              TYPE_ACPI_GED "-regs",
> ACPI_GED_REG_COUNT);
> >>>        sysbus_init_mmio(sbd, &ged_st->regs);
> >>> +
> >>> +    s->cpuhp.device = OBJECT(s);
> >>> +    memory_region_init(&s->container_cpuhp, OBJECT(dev), "cpuhp
> >> container",
> >>> +                       ACPI_CPU_HOTPLUG_REG_LEN);
> >>> +    sysbus_init_mmio(SYS_BUS_DEVICE(dev), &s->container_cpuhp);
> >>> +    cpu_hotplug_hw_init(&s->container_cpuhp, OBJECT(dev),
> >>> +                        &s->cpuhp_state, 0);
> >>>    }
> >>>
> >>>    static void acpi_ged_class_init(ObjectClass *class, void *data)
> >>> diff --git a/include/hw/acpi/generic_event_device.h
> >> b/include/hw/acpi/generic_event_device.h
> >>> index d831bbd889..d0a5a43abf 100644
> >>> --- a/include/hw/acpi/generic_event_device.h
> >>> +++ b/include/hw/acpi/generic_event_device.h
> >>> @@ -60,6 +60,7 @@
> >>>    #define HW_ACPI_GENERIC_EVENT_DEVICE_H
> >>>
> >>>    #include "hw/sysbus.h"
> >>> +#include "hw/acpi/cpu_hotplug.h"
> >>>    #include "hw/acpi/memory_hotplug.h"
> >>>    #include "hw/acpi/ghes.h"
> >>>    #include "qom/object.h"
> >>> @@ -97,6 +98,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(AcpiGedState, ACPI_GED)
> >>>    #define ACPI_GED_MEM_HOTPLUG_EVT   0x1
> >>>    #define ACPI_GED_PWR_DOWN_EVT      0x2
> >>>    #define ACPI_GED_NVDIMM_HOTPLUG_EVT 0x4
> >>> +#define ACPI_GED_CPU_HOTPLUG_EVT    0x8
> >>>
> >>>    typedef struct GEDState {
> >>>        MemoryRegion evt;
> >>> @@ -108,6 +110,9 @@ struct AcpiGedState {
> >>>        SysBusDevice parent_obj;
> >>>        MemHotplugState memhp_state;
> >>>        MemoryRegion container_memhp;
> >>> +    CPUHotplugState cpuhp_state;
> >>> +    MemoryRegion container_cpuhp;
> >>> +    AcpiCpuHotplug cpuhp;
> >>
> >> Am I wrong or is that member completely unused/uninitialized?
> >
> > No it is not. Please check below change in acpi_ged_initfn()
> >
> > +    s->cpuhp.device = OBJECT(s);
> 
> Not the best of my mondays, sorry for that.
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>

No issues at all. Thanks for taking time to review this.

Best regards
Salil.

RE: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta via]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 2:55 PM
> 
> On 09.10.23 15:49, Salil Mehta wrote:
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:23 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> >> arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> ACPI CPU hotplug related initialization should only happen if
> >> ACPI_CPU_HOTPLUG
> >>> support has been enabled for particular architecture. Add
> >> cpu_hotplug_hw_init()
> >>> stub to avoid compilation break.
> >>>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> >>> Reviewed-by: Gavin Shan <gshan@redhat.com>
> >>> ---
> >>>    hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
> >>>    1 file changed, 6 insertions(+)
> >>>
> >>> diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-
> hotplug-
> >> stub.c
> >>> index 3fc4b14c26..c6c61bb9cd 100644
> >>> --- a/hw/acpi/acpi-cpu-hotplug-stub.c
> >>> +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
> >>> @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion
> >> *parent, Object *owner,
> >>>        return;
> >>>    }
> >>>
> >>> +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> >>> +                         CPUHotplugState *state, hwaddr base_addr)
> >>> +{
> >>> +    return;
> >>> +}
> >>
> >> While at it, can we prefix that function with acpi?
> >
> > I can do that but it has to be done at other places as well
> > such as in hw/acpi/cpu_hotplug.c <acpi_switch_to_modern_cphp()>
> >
> 
> $ git grep cpu_hotplug_hw_init
> hw/acpi/cpu.c:void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> hw/acpi/cpu_hotplug.c:    cpu_hotplug_hw_init(parent, gpe_cpu->device,
> cpuhp_state, io_port);
> include/hw/acpi/cpu.h:void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> 
> Might want to do that as a separate patch, agreed.

Sure, thanks
Salil.

RE: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub

[Salil Mehta]

> From: David Hildenbrand <david@redhat.com>
> Sent: Monday, October 9, 2023 2:55 PM
> 
> On 09.10.23 15:49, Salil Mehta wrote:
> >> From: David Hildenbrand <david@redhat.com>
> >> Sent: Monday, October 9, 2023 1:23 PM
> >> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-
> >> arm@nongnu.org
> >> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
> >> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
> >> peter.maydell@linaro.org; richard.henderson@linaro.org;
> >> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
> >> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
> >> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
> >> alex.bennee@linaro.org; linux@armlinux.org.uk;
> >> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
> >> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
> >> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
> >> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
> >> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
> >> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm
> <linuxarm@huawei.com>
> >> Subject: Re: [PATCH V3 03/10] hw/acpi: Add ACPI CPU hotplug init stub
> >>
> >> On 09.10.23 13:28, Salil Mehta wrote:
> >>> ACPI CPU hotplug related initialization should only happen if
> >> ACPI_CPU_HOTPLUG
> >>> support has been enabled for particular architecture. Add
> >> cpu_hotplug_hw_init()
> >>> stub to avoid compilation break.
> >>>
> >>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
> >>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> >>> Reviewed-by: Gavin Shan <gshan@redhat.com>
> >>> ---
> >>>    hw/acpi/acpi-cpu-hotplug-stub.c | 6 ++++++
> >>>    1 file changed, 6 insertions(+)
> >>>
> >>> diff --git a/hw/acpi/acpi-cpu-hotplug-stub.c b/hw/acpi/acpi-cpu-
> hotplug-
> >> stub.c
> >>> index 3fc4b14c26..c6c61bb9cd 100644
> >>> --- a/hw/acpi/acpi-cpu-hotplug-stub.c
> >>> +++ b/hw/acpi/acpi-cpu-hotplug-stub.c
> >>> @@ -19,6 +19,12 @@ void legacy_acpi_cpu_hotplug_init(MemoryRegion
> >> *parent, Object *owner,
> >>>        return;
> >>>    }
> >>>
> >>> +void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> >>> +                         CPUHotplugState *state, hwaddr base_addr)
> >>> +{
> >>> +    return;
> >>> +}
> >>
> >> While at it, can we prefix that function with acpi?
> >
> > I can do that but it has to be done at other places as well
> > such as in hw/acpi/cpu_hotplug.c <acpi_switch_to_modern_cphp()>
> >
> 
> $ git grep cpu_hotplug_hw_init
> hw/acpi/cpu.c:void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
> hw/acpi/cpu_hotplug.c:    cpu_hotplug_hw_init(parent, gpe_cpu->device,
> cpuhp_state, io_port);
> include/hw/acpi/cpu.h:void cpu_hotplug_hw_init(MemoryRegion *as, Object
> *owner,
> 
> Might want to do that as a separate patch, agreed.

Sure, thanks
Salil.