[Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Zhu Guihua]

This series is based on chen fan's previous i386 cpu hot remove patchset:
https://lists.nongnu.org/archive/html/qemu-devel/2013-12/msg04266.html

Via implementing ACPI standard methods _EJ0 in ACPI table, after Guest
OS remove one vCPU online, the fireware will store removed bitmap to
QEMU, then QEMU could know to notify the assigned vCPU of exiting.
Meanwhile, intruduce the QOM command 'device_del' to remove vCPU from
QEMU itself.

The whole work is based on the new hot plug/unplug framework, ,the unplug request
callback does the pre-check and send the request, unplug callback does the
removal handling.

This series depends on tangchen's common hot plug/unplug enhance patchset.
[PATCH v2 0/5] Common unplug and unplug request cb for memory and CPU hot-unplug
https://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg03929.html

The is the second half of the previous series:                                      
[RFC V2 00/10] cpu: add device_add foo-x86_64-cpu and i386 cpu hot remove support
https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg04779.html             

If you want to test the series, you need to apply the 'device_add foo-x86_64-cpu'
patchset first:
[PATCH v4 00/10] cpu: add device_add foo-x86_64-cpu support
https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg02584.html

---
Changelog since v2:
 -drop ICC bus impl
 -fix delete cpu exceed 32 issue
 -fix bug about deleting the last cpu

Changelog since v1:
 -rebase on the latest version.
 -delete patch i386/cpu: add instance finalize callback, and put it into patchset
  [PATCH v3 0/6] cpu: add device_add foo-x86_64-cpu support.

Changelog since RFC:
 -splited the i386 cpu hot remove into single thread.
 -replaced apic_no with apic_id, so does the related stuff to make it
  work with arbitrary CPU hotadd.
 -add the icc_device_unrealize callback to handle apic unrealize.
 -rework on the new hot plug/unplug platform.
---

Chen Fan (2):
  x86: add x86_cpu_unrealizefn() for cpu apic remove
  cpu hotplug: implement function cpu_status_write() for vcpu ejection

Gu Zheng (3):
  acpi/cpu: add cpu hot unplug request callback function
  acpi, pc: add cpu hot unplug callback support
  cpus: reclaim allocated vCPU objects

Zhu Guihua (2):
  acpi, pc: add cpu hot unplug request callback support
  acpi/cpu: add cpu hot unplug callback function

 cpus.c                            | 44 ++++++++++++++++++++
 hw/acpi/cpu_hotplug.c             | 87 ++++++++++++++++++++++++++++++++++++---
 hw/acpi/ich9.c                    | 17 ++++++--
 hw/acpi/piix4.c                   | 12 +++++-
 hw/core/qdev.c                    |  2 +-
 hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++-
 hw/i386/kvm/apic.c                |  5 +++
 hw/i386/pc.c                      | 68 ++++++++++++++++++++++++++++--
 hw/intc/apic.c                    |  9 ++++
 hw/intc/apic_common.c             | 21 ++++++----
 include/hw/acpi/cpu_hotplug.h     |  8 ++++
 include/hw/i386/apic_internal.h   |  1 +
 include/hw/qdev-core.h            |  1 +
 include/qom/cpu.h                 |  9 ++++
 include/sysemu/kvm.h              |  1 +
 kvm-all.c                         | 57 ++++++++++++++++++++++++-
 target-i386/cpu.c                 | 46 +++++++++++++++++++++
 17 files changed, 377 insertions(+), 27 deletions(-)

-- 
1.9.3

[Qemu-devel] [PATCH v3 1/7] x86: add x86_cpu_unrealizefn() for cpu apic remove

[Zhu Guihua]

From: Chen Fan <chen.fan.fnst@cn.fujitsu.com>

Implement x86_cpu_unrealizefn() for corresponding x86_cpu_realizefn(),
which is mostly used to clean the apic related allocation and vmstates
at here.

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
---
 hw/i386/kvm/apic.c              |  5 +++++
 hw/intc/apic.c                  |  9 ++++++++
 hw/intc/apic_common.c           | 21 ++++++++++++-------
 include/hw/i386/apic_internal.h |  1 +
 target-i386/cpu.c               | 46 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 74 insertions(+), 8 deletions(-)

diff --git a/hw/i386/kvm/apic.c b/hw/i386/kvm/apic.c
index 4b7ce57..45abfe9 100644
--- a/hw/i386/kvm/apic.c
+++ b/hw/i386/kvm/apic.c
@@ -195,11 +195,16 @@ static void kvm_apic_realize(DeviceState *dev, Error **errp)
     }
 }
 
+static void kvm_apic_unrealize(DeviceState *dev, Error **errp)
+{
+}
+
 static void kvm_apic_class_init(ObjectClass *klass, void *data)
 {
     APICCommonClass *k = APIC_COMMON_CLASS(klass);
 
     k->realize = kvm_apic_realize;
+    k->unrealize = kvm_apic_unrealize;
     k->reset = kvm_apic_reset;
     k->set_base = kvm_apic_set_base;
     k->set_tpr = kvm_apic_set_tpr;
diff --git a/hw/intc/apic.c b/hw/intc/apic.c
index e48974f..f260cdf 100644
--- a/hw/intc/apic.c
+++ b/hw/intc/apic.c
@@ -895,11 +895,20 @@ static void apic_realize(DeviceState *dev, Error **errp)
     msi_supported = true;
 }
 
+static void apic_unrealize(DeviceState *dev, Error **errp)
+{
+    APICCommonState *s = APIC_COMMON(dev);
+
+    timer_free(s->timer);
+    local_apics[s->idx] = NULL;
+}
+
 static void apic_class_init(ObjectClass *klass, void *data)
 {
     APICCommonClass *k = APIC_COMMON_CLASS(klass);
 
     k->realize = apic_realize;
+    k->unrealize = apic_unrealize;
     k->set_base = apic_set_base;
     k->set_tpr = apic_set_tpr;
     k->get_tpr = apic_get_tpr;
diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index 52b933d..657ffa0 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -302,17 +302,9 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     APICCommonState *s = APIC_COMMON(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
-    static int apic_no;
     PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
     static bool mmio_registered;
 
-    if (apic_no >= MAX_APICS) {
-        error_setg(errp, "%s initialization failed.",
-                   object_get_typename(OBJECT(dev)));
-        return;
-    }
-    s->idx = apic_no++;
-
     info = APIC_COMMON_GET_CLASS(s);
     info->realize(dev, errp);
 
@@ -335,6 +327,18 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
 
 }
 
+static void apic_common_unrealize(DeviceState *dev, Error **errp)
+{
+    APICCommonState *s = APIC_COMMON(dev);
+    APICCommonClass *info = APIC_COMMON_GET_CLASS(s);
+
+    info->unrealize(dev, errp);
+
+    if (apic_report_tpr_access && info->enable_tpr_reporting) {
+        info->enable_tpr_reporting(s, false);
+    }
+}
+
 static int apic_pre_load(void *opaque)
 {
     APICCommonState *s = APIC_COMMON(opaque);
@@ -442,6 +446,7 @@ static void apic_common_class_init(ObjectClass *klass, void *data)
     dc->reset = apic_reset_common;
     dc->props = apic_properties_common;
     dc->realize = apic_common_realize;
+    dc->unrealize = apic_common_unrealize;
     /*
      * Reason: APIC and CPU need to be wired up by
      * x86_cpu_apic_create()
diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
index 578a616..3432a78 100644
--- a/include/hw/i386/apic_internal.h
+++ b/include/hw/i386/apic_internal.h
@@ -81,6 +81,7 @@ typedef struct APICCommonClass
     DeviceClass parent_class;
 
     DeviceRealize realize;
+    DeviceUnrealize unrealize;
     void (*set_base)(APICCommonState *s, uint64_t val);
     void (*set_tpr)(APICCommonState *s, uint8_t val);
     uint8_t (*get_tpr)(APICCommonState *s);
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index fc0e4a6..84c88fe 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2775,6 +2775,8 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 
     /* TODO: convert to link<> */
     apic = APIC_COMMON(cpu->apic_state);
+    g_assert(apic_id < MAX_APICS);
+    apic->idx = apic_id;
     apic->cpu = cpu;
 }
 
@@ -2798,10 +2800,32 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
         return;
     }
 }
+
+static void x86_cpu_apic_unrealize(X86CPU *cpu, Error **errp)
+{
+    Error *local_err = NULL;
+
+    if (cpu->apic_state == NULL) {
+        return;
+    }
+
+    object_property_set_bool(OBJECT(cpu->apic_state),
+                             false, "realized", &local_err);
+    if (local_err != NULL) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    vmstate_unregister(NULL, &vmstate_apic_common, cpu->apic_state);
+    object_unparent(OBJECT(cpu->apic_state));
+}
 #else
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
 {
 }
+static void x86_cpu_apic_unrealize(X86CPU *cpu, Error **errp)
+{
+}
 #endif
 
 
@@ -2884,6 +2908,27 @@ out:
     }
 }
 
+static void x86_cpu_unrealizefn(DeviceState *dev, Error **errp)
+{
+    X86CPU *cpu = X86_CPU(dev);
+    CPUClass *cc = CPU_GET_CLASS(dev);
+    Error *local_err = NULL;
+
+    if (qdev_get_vmsd(DEVICE(cpu)) == NULL) {
+        vmstate_unregister(NULL, &vmstate_cpu_common, cpu);
+    }
+
+    if (cc->vmsd != NULL) {
+        vmstate_unregister(NULL, cc->vmsd, cpu);
+    }
+
+    x86_cpu_apic_unrealize(cpu, &local_err);
+    if (local_err != NULL) {
+        error_propagate(errp, local_err);
+        return;
+    }
+}
+
 /* Enables contiguous-apic-ID mode, for compatibility */
 static bool compat_apic_id_mode;
 
@@ -3059,6 +3104,7 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 
     xcc->parent_realize = dc->realize;
     dc->realize = x86_cpu_realizefn;
+    dc->unrealize = x86_cpu_unrealizefn;
     dc->props = x86_cpu_properties;
 
     xcc->parent_reset = cc->reset;
-- 
1.9.3

[Qemu-devel] [PATCH v3 2/7] acpi/cpu: add cpu hot unplug request callback function

[Zhu Guihua]

From: Gu Zheng <guz.fnst@cn.fujitsu.com>

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
---
 hw/acpi/cpu_hotplug.c         | 37 ++++++++++++++++++++++++++++++++-----
 include/hw/acpi/cpu_hotplug.h |  4 ++++
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/hw/acpi/cpu_hotplug.c b/hw/acpi/cpu_hotplug.c
index 8e4ed6e..42109cd 100644
--- a/hw/acpi/cpu_hotplug.c
+++ b/hw/acpi/cpu_hotplug.c
@@ -12,6 +12,11 @@
 #include "hw/hw.h"
 #include "hw/acpi/cpu_hotplug.h"
 
+typedef enum STS_OPT {
+    SET,
+    CLEAR,
+} STS_OPT;
+
 static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned int size)
 {
     AcpiCpuHotplug *cpus = opaque;
@@ -36,8 +41,8 @@ static const MemoryRegionOps AcpiCpuHotplug_ops = {
     },
 };
 
-static void acpi_set_cpu_present_bit(AcpiCpuHotplug *g, CPUState *cpu,
-                                     Error **errp)
+static void acpi_update_cpu_present_bit(AcpiCpuHotplug *g, CPUState *cpu,
+                                        STS_OPT opt, Error **errp)
 {
     CPUClass *k = CPU_GET_CLASS(cpu);
     int64_t cpu_id;
@@ -48,13 +53,23 @@ static void acpi_set_cpu_present_bit(AcpiCpuHotplug *g, CPUState *cpu,
         return;
     }
 
-    g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
+    switch (opt) {
+    case SET:
+        g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
+        break;
+    case CLEAR:
+        g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
+        break;
+    default:
+        g_assert(0);
+        break;
+    }
 }
 
 void acpi_cpu_plug_cb(ACPIREGS *ar, qemu_irq irq,
                       AcpiCpuHotplug *g, DeviceState *dev, Error **errp)
 {
-    acpi_set_cpu_present_bit(g, CPU(dev), errp);
+    acpi_update_cpu_present_bit(g, CPU(dev), SET, errp);
     if (*errp != NULL) {
         return;
     }
@@ -65,13 +80,25 @@ void acpi_cpu_plug_cb(ACPIREGS *ar, qemu_irq irq,
     }
 }
 
+void acpi_cpu_unplug_request_cb(ACPIREGS *ar, qemu_irq irq,
+                                AcpiCpuHotplug *g, DeviceState *dev,
+                                Error **errp)
+{
+    acpi_update_cpu_present_bit(g, CPU(dev), CLEAR, errp);
+    if (*errp != NULL) {
+        return;
+    }
+
+    acpi_send_gpe_event(ar, irq, ACPI_CPU_HOTPLUG_STATUS);
+}
+
 void acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
                            AcpiCpuHotplug *gpe_cpu, uint16_t base)
 {
     CPUState *cpu;
 
     CPU_FOREACH(cpu) {
-        acpi_set_cpu_present_bit(gpe_cpu, cpu, &error_abort);
+        acpi_update_cpu_present_bit(gpe_cpu, cpu, SET, &error_abort);
     }
     memory_region_init_io(&gpe_cpu->io, owner, &AcpiCpuHotplug_ops,
                           gpe_cpu, "acpi-cpu-hotplug", ACPI_GPE_PROC_LEN);
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index f6d358d..8b15a3d 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -23,6 +23,10 @@ typedef struct AcpiCpuHotplug {
 void acpi_cpu_plug_cb(ACPIREGS *ar, qemu_irq irq,
                       AcpiCpuHotplug *g, DeviceState *dev, Error **errp);
 
+void acpi_cpu_unplug_request_cb(ACPIREGS *ar, qemu_irq irq,
+                                AcpiCpuHotplug *g, DeviceState *dev,
+                                Error **errp);
+
 void acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
                            AcpiCpuHotplug *gpe_cpu, uint16_t base);
 #endif
-- 
1.9.3

[Qemu-devel] [PATCH v3 3/7] acpi, pc: add cpu hot unplug request callback support

[Zhu Guihua]

Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
---
 hw/acpi/ich9.c  |  9 +++++++--
 hw/acpi/piix4.c |  2 ++
 hw/i386/pc.c    | 42 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/hw/acpi/ich9.c b/hw/acpi/ich9.c
index 5352e19..7600e88 100644
--- a/hw/acpi/ich9.c
+++ b/hw/acpi/ich9.c
@@ -400,8 +400,13 @@ void ich9_pm_device_plug_cb(ICH9LPCPMRegs *pm, DeviceState *dev, Error **errp)
 void ich9_pm_device_unplug_request_cb(ICH9LPCPMRegs *pm, DeviceState *dev,
                                       Error **errp)
 {
-    error_setg(errp, "acpi: device unplug request for not supported device"
-               " type: %s", object_get_typename(OBJECT(dev)));
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_request_cb(&pm->acpi_regs, pm->irq,
+                                   &pm->gpe_cpu, dev, errp);
+    } else {
+        error_setg(errp, "acpi: device unplug request for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
 }
 
 void ich9_pm_device_unplug_cb(ICH9LPCPMRegs *pm, DeviceState *dev,
diff --git a/hw/acpi/piix4.c b/hw/acpi/piix4.c
index 14d40a0..23595b6 100644
--- a/hw/acpi/piix4.c
+++ b/hw/acpi/piix4.c
@@ -364,6 +364,8 @@ static void piix4_device_unplug_request_cb(HotplugHandler *hotplug_dev,
     if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
         acpi_pcihp_device_unplug_cb(&s->ar, s->irq, &s->acpi_pci_hotplug, dev,
                                     errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_request_cb(&s->ar, s->irq, &s->gpe_cpu, dev, errp);
     } else {
         error_setg(errp, "acpi: device unplug request for not supported device"
                    " type: %s", object_get_typename(OBJECT(dev)));
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index c1ecf08..80dac20 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1653,6 +1653,40 @@ out:
     error_propagate(errp, local_err);
 }
 
+static void pc_cpu_unplug_request(HotplugHandler *hotplug_dev,
+                                  DeviceState *dev, Error **errp)
+{
+    HotplugHandlerClass *hhc;
+    Error *local_err = NULL;
+    PCMachineState *pcms;
+    CPUState *cpu;
+    int current_cpus = 0;
+
+    CPU_FOREACH(cpu) {
+        current_cpus++;
+    }
+
+    if (current_cpus == 1) {
+        error_setg(&local_err,
+                   "remove the last cpu is not allowed");
+        goto out;
+    }
+
+    pcms = PC_MACHINE(hotplug_dev);
+    if (!pcms->acpi_dev) {
+        if (dev->hotplugged) {
+            error_setg(&local_err,
+                       "cpu hot unplug is not enabled: missing acpi device");
+        }
+        goto out;
+    }
+
+    hhc = HOTPLUG_HANDLER_GET_CLASS(pcms->acpi_dev);
+    hhc->unplug_request(HOTPLUG_HANDLER(pcms->acpi_dev), dev, &local_err);
+out:
+    error_propagate(errp, local_err);
+}
+
 static void pc_machine_device_plug_cb(HotplugHandler *hotplug_dev,
                                       DeviceState *dev, Error **errp)
 {
@@ -1666,8 +1700,12 @@ static void pc_machine_device_plug_cb(HotplugHandler *hotplug_dev,
 static void pc_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev,
                                                 DeviceState *dev, Error **errp)
 {
-    error_setg(errp, "acpi: device unplug request for not supported device"
-               " type: %s", object_get_typename(OBJECT(dev)));
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        pc_cpu_unplug_request(hotplug_dev, dev, errp);
+    } else {
+        error_setg(errp, "acpi: device unplug request for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
 }
 
 static void pc_machine_device_unplug_cb(HotplugHandler *hotplug_dev,
-- 
1.9.3

[Qemu-devel] [PATCH v3 4/7] acpi/cpu: add cpu hot unplug callback function

[Zhu Guihua]

Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
---
 cpus.c                        | 7 +++++++
 hw/acpi/cpu_hotplug.c         | 8 ++++++++
 include/hw/acpi/cpu_hotplug.h | 3 +++
 include/qom/cpu.h             | 9 +++++++++
 4 files changed, 27 insertions(+)

diff --git a/cpus.c b/cpus.c
index 67d10a7..d6e5a5f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1188,6 +1188,13 @@ void resume_all_vcpus(void)
     }
 }
 
+void cpu_remove(CPUState *cpu)
+{
+    cpu->stop = true;
+    cpu->exit = true;
+    qemu_cpu_kick(cpu);
+}
+
 /* For temporary buffers for forming a name */
 #define VCPU_THREAD_NAME_SIZE 16
 
diff --git a/hw/acpi/cpu_hotplug.c b/hw/acpi/cpu_hotplug.c
index 42109cd..c47909c 100644
--- a/hw/acpi/cpu_hotplug.c
+++ b/hw/acpi/cpu_hotplug.c
@@ -92,6 +92,14 @@ void acpi_cpu_unplug_request_cb(ACPIREGS *ar, qemu_irq irq,
     acpi_send_gpe_event(ar, irq, ACPI_CPU_HOTPLUG_STATUS);
 }
 
+void acpi_cpu_unplug_cb(ACPIREGS *ar, qemu_irq irq, AcpiCpuHotplug *g,
+                        DeviceState *dev, Error **errp)
+{
+    CPUState *cpu = CPU(dev);
+
+    cpu_remove(cpu);
+}
+
 void acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
                            AcpiCpuHotplug *gpe_cpu, uint16_t base)
 {
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 8b15a3d..0f84adb 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -27,6 +27,9 @@ void acpi_cpu_unplug_request_cb(ACPIREGS *ar, qemu_irq irq,
                                 AcpiCpuHotplug *g, DeviceState *dev,
                                 Error **errp);
 
+void acpi_cpu_unplug_cb(ACPIREGS *ar, qemu_irq irq,
+                        AcpiCpuHotplug *g, DeviceState *dev, Error **errp);
+
 void acpi_cpu_hotplug_init(MemoryRegion *parent, Object *owner,
                            AcpiCpuHotplug *gpe_cpu, uint16_t base);
 #endif
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index a7d601b..e5e0f2c 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -250,6 +250,7 @@ struct CPUState {
     bool created;
     bool stop;
     bool stopped;
+    bool exit;
     volatile sig_atomic_t exit_request;
     uint32_t interrupt_request;
     int singlestep_enabled;
@@ -614,6 +615,14 @@ void cpu_exit(CPUState *cpu);
 void cpu_resume(CPUState *cpu);
 
 /**
+ * cpu_remove:
+ * @cpu: The vCPU to remove.
+ *
+ * Requests the CPU @cpu to be removed.
+ */
+void cpu_remove(CPUState *cpu);
+
+/**
  * qemu_init_vcpu:
  * @cpu: The vCPU to initialize.
  *
-- 
1.9.3

[Qemu-devel] [PATCH v3 5/7] acpi, pc: add cpu hot unplug callback support

[Zhu Guihua]

From: Gu Zheng <guz.fnst@cn.fujitsu.com>

Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
---
 hw/acpi/ich9.c  |  8 ++++++--
 hw/acpi/piix4.c | 10 ++++++++--
 hw/i386/pc.c    | 26 ++++++++++++++++++++++++--
 3 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/hw/acpi/ich9.c b/hw/acpi/ich9.c
index 7600e88..f095999 100644
--- a/hw/acpi/ich9.c
+++ b/hw/acpi/ich9.c
@@ -412,8 +412,12 @@ void ich9_pm_device_unplug_request_cb(ICH9LPCPMRegs *pm, DeviceState *dev,
 void ich9_pm_device_unplug_cb(ICH9LPCPMRegs *pm, DeviceState *dev,
                               Error **errp)
 {
-    error_setg(errp, "acpi: device unplug for not supported device"
-               " type: %s", object_get_typename(OBJECT(dev)));
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_cb(&pm->acpi_regs, pm->irq, &pm->gpe_cpu, dev, errp);
+    } else {
+        error_setg(errp, "acpi: device unplug for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
 }
 
 void ich9_pm_ospm_status(AcpiDeviceIf *adev, ACPIOSTInfoList ***list)
diff --git a/hw/acpi/piix4.c b/hw/acpi/piix4.c
index 23595b6..4ca9153 100644
--- a/hw/acpi/piix4.c
+++ b/hw/acpi/piix4.c
@@ -375,8 +375,14 @@ static void piix4_device_unplug_request_cb(HotplugHandler *hotplug_dev,
 static void piix4_device_unplug_cb(HotplugHandler *hotplug_dev,
                                    DeviceState *dev, Error **errp)
 {
-    error_setg(errp, "acpi: device unplug for not supported device"
-               " type: %s", object_get_typename(OBJECT(dev)));
+    PIIX4PMState *s = PIIX4_PM(hotplug_dev);
+
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        acpi_cpu_unplug_cb(&s->ar, s->irq, &s->gpe_cpu, dev, errp);
+    } else {
+        error_setg(errp, "acpi: device unplug for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
 }
 
 static void piix4_update_bus_hotplug(PCIBus *pci_bus, void *opaque)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 80dac20..d0d289b 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1687,6 +1687,24 @@ out:
     error_propagate(errp, local_err);
 }
 
+static void pc_cpu_unplug(HotplugHandler *hotplug_dev,
+                          DeviceState *dev, Error **errp)
+{
+    HotplugHandlerClass *hhc;
+    Error *local_err = NULL;
+    PCMachineState *pcms = PC_MACHINE(hotplug_dev);
+
+    hhc = HOTPLUG_HANDLER_GET_CLASS(pcms->acpi_dev);
+    hhc->unplug(HOTPLUG_HANDLER(pcms->acpi_dev), dev, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /* decrement the number of CPUs */
+    rtc_set_memory(pcms->rtc, 0x5f, rtc_get_memory(pcms->rtc, 0x5f) - 1);
+}
+
 static void pc_machine_device_plug_cb(HotplugHandler *hotplug_dev,
                                       DeviceState *dev, Error **errp)
 {
@@ -1711,8 +1729,12 @@ static void pc_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev,
 static void pc_machine_device_unplug_cb(HotplugHandler *hotplug_dev,
                                         DeviceState *dev, Error **errp)
 {
-    error_setg(errp, "acpi: device unplug for not supported device"
-               " type: %s", object_get_typename(OBJECT(dev)));
+    if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) {
+        pc_cpu_unplug(hotplug_dev, dev, errp);
+    } else {
+        error_setg(errp, "acpi: device unplug for not supported device"
+                   " type: %s", object_get_typename(OBJECT(dev)));
+    }
 }
 
 static HotplugHandler *pc_get_hotpug_handler(MachineState *machine,
-- 
1.9.3

[Qemu-devel] [PATCH v3 6/7] cpu hotplug: implement function cpu_status_write() for vcpu ejection

[Zhu Guihua]

From: Chen Fan <chen.fan.fnst@cn.fujitsu.com>

When OS ejected a vcpu (like: echo 1 > /sys/bus/acpi/devices/LNXCPUXX/eject),
it would call acpi EJ0 method, the firmware need to write the new cpumap, QEMU
would know which vcpu need to be ejected.

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
---
 hw/acpi/cpu_hotplug.c             | 42 ++++++++++++++++++++++++++++++++++++++-
 hw/core/qdev.c                    |  2 +-
 hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++++++++++-
 include/hw/acpi/cpu_hotplug.h     |  1 +
 include/hw/qdev-core.h            |  1 +
 5 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/hw/acpi/cpu_hotplug.c b/hw/acpi/cpu_hotplug.c
index c47909c..1874ee0 100644
--- a/hw/acpi/cpu_hotplug.c
+++ b/hw/acpi/cpu_hotplug.c
@@ -17,6 +17,26 @@ typedef enum STS_OPT {
     CLEAR,
 } STS_OPT;
 
+static void acpi_eject_vcpu(AcpiCpuHotplug *cpus_status, int64_t cpu_id)
+{
+    CPUState *cpu;
+
+    CPU_FOREACH(cpu) {
+        CPUClass *cc = CPU_GET_CLASS(cpu);
+        int64_t id = cc->get_arch_id(cpu);
+        HotplugHandler *hotplug_ctrl;
+
+        if (cpu_id == id) {
+            cpus_status->old_sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
+            cpus_status->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
+
+            hotplug_ctrl = qdev_get_hotplug_handler(DEVICE(cpu));
+            hotplug_handler_unplug(hotplug_ctrl, DEVICE(cpu), NULL);
+            break;
+        }
+    }
+}
+
 static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned int size)
 {
     AcpiCpuHotplug *cpus = opaque;
@@ -28,7 +48,26 @@ static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned int size)
 static void cpu_status_write(void *opaque, hwaddr addr, uint64_t data,
                              unsigned int size)
 {
-    /* TODO: implement VCPU removal on guest signal that CPU can be removed */
+    AcpiCpuHotplug *cpus = opaque;
+    uint8_t val;
+    int i;
+    int64_t cpu_id = -1;
+
+    val = cpus->old_sts[addr] ^ data;
+
+    if (val == 0) {
+        return;
+    }
+
+    for (i = 0; i < 8; i++) {
+        if (val & 1 << i) {
+            cpu_id = 8 * addr + i;
+        }
+    }
+
+    if (cpu_id != -1) {
+        acpi_eject_vcpu(cpus, cpu_id);
+    }
 }
 
 static const MemoryRegionOps AcpiCpuHotplug_ops = {
@@ -56,6 +95,7 @@ static void acpi_update_cpu_present_bit(AcpiCpuHotplug *g, CPUState *cpu,
     switch (opt) {
     case SET:
         g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
+        g->old_sts[cpu_id / 8] |= (1 << (cpu_id % 8));
         break;
     case CLEAR:
         g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index 2eacac0..2f3d1df 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -273,7 +273,7 @@ void qdev_set_legacy_instance_id(DeviceState *dev, int alias_id,
     dev->alias_required_for_version = required_for_version;
 }
 
-static HotplugHandler *qdev_get_hotplug_handler(DeviceState *dev)
+HotplugHandler *qdev_get_hotplug_handler(DeviceState *dev)
 {
     HotplugHandler *hotplug_ctrl = NULL;
 
diff --git a/hw/i386/acpi-dsdt-cpu-hotplug.dsl b/hw/i386/acpi-dsdt-cpu-hotplug.dsl
index 268d870..e03517f 100644
--- a/hw/i386/acpi-dsdt-cpu-hotplug.dsl
+++ b/hw/i386/acpi-dsdt-cpu-hotplug.dsl
@@ -50,7 +50,21 @@ Scope(\_SB) {
     }
     Method(CPEJ, 2, NotSerialized) {
         // _EJ0 method - eject callback
-        Sleep(200)
+        Store(Zero, Index(CPON, ToInteger(Arg0)))
+        Store(PRS, Local5)
+        // Local0 = the index of cpu bitmap
+        Store(ShiftRight(ToInteger(Arg0), 3), Local0)
+        // Local1 = the subobject of PRS with dereferece
+        Store(DerefOf(Index(PRS, Local0)), Local1)
+        // Local2 = the bit index in cpu bitmap
+        And(ToInteger(Arg0), 0x7, Local2)
+        Store(One, Local3)
+        ShiftLeft(Local3, Local2, Local3)
+        Not(Local3, Local3)
+        // discard the bit index in cpu bitmap
+        And(Local1, Local3, Local1)
+        Store(Local1, Index(Local5, Local0))
+        Store(Local5, PRS)
     }
 
 #define CPU_STATUS_LEN ACPI_GPE_PROC_LEN
diff --git a/include/hw/acpi/cpu_hotplug.h b/include/hw/acpi/cpu_hotplug.h
index 0f84adb..abbb29e 100644
--- a/include/hw/acpi/cpu_hotplug.h
+++ b/include/hw/acpi/cpu_hotplug.h
@@ -18,6 +18,7 @@
 typedef struct AcpiCpuHotplug {
     MemoryRegion io;
     uint8_t sts[ACPI_GPE_PROC_LEN];
+    uint8_t old_sts[ACPI_GPE_PROC_LEN];
 } AcpiCpuHotplug;
 
 void acpi_cpu_plug_cb(ACPIREGS *ar, qemu_irq irq,
diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 15a226f..de42e78 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -269,6 +269,7 @@ void qdev_set_legacy_instance_id(DeviceState *dev, int alias_id,
 void qdev_unplug(DeviceState *dev, Error **errp);
 void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
                                   DeviceState *dev, Error **errp);
+HotplugHandler *qdev_get_hotplug_handler(DeviceState *dev);
 void qdev_machine_creation_done(void);
 bool qdev_machine_modified(void);
 
-- 
1.9.3

[Qemu-devel] [PATCH v3 7/7] cpus: reclaim allocated vCPU objects

[Zhu Guihua]

From: Gu Zheng <guz.fnst@cn.fujitsu.com>

After ACPI get a signal to eject a vCPU, the vCPU must be
removed from CPU list，before the vCPU really removed,  then
release the all related vCPU objects.

In order to deal well with the kvm vcpus (which can not be removed without any
protection), we do not close KVM vcpu fd, just record and mark it as stopped
into a list, so that we can reuse it for the appending cpu hot-add request if
possible. It is also the approach that kvm guys suggested:
https://www.mail-archive.com/kvm@vger.kernel.org/msg102839.html

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
---
 cpus.c               | 37 ++++++++++++++++++++++++++++++++++
 include/sysemu/kvm.h |  1 +
 kvm-all.c            | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 94 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index d6e5a5f..a11941f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -854,6 +854,24 @@ void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
     qemu_cpu_kick(cpu);
 }
 
+static void qemu_kvm_destroy_vcpu(CPUState *cpu)
+{
+    CPU_REMOVE(cpu);
+
+    if (kvm_destroy_vcpu(cpu) < 0) {
+        error_report("kvm_destroy_vcpu failed.\n");
+        exit(EXIT_FAILURE);
+    }
+
+    object_unparent(OBJECT(cpu));
+}
+
+static void qemu_tcg_destroy_vcpu(CPUState *cpu)
+{
+    CPU_REMOVE(cpu);
+    object_unparent(OBJECT(cpu));
+}
+
 static void flush_queued_work(CPUState *cpu)
 {
     struct qemu_work_item *wi;
@@ -946,6 +964,11 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
             }
         }
         qemu_kvm_wait_io_event(cpu);
+        if (cpu->exit && !cpu_can_run(cpu)) {
+            qemu_kvm_destroy_vcpu(cpu);
+            qemu_mutex_unlock(&qemu_global_mutex);
+            return NULL;
+        }
     }
 
     return NULL;
@@ -999,6 +1022,7 @@ static void tcg_exec_all(void);
 static void *qemu_tcg_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
+    CPUState *remove_cpu = NULL;
 
     qemu_tcg_init_cpu_signals();
     qemu_thread_get_self(cpu->thread);
@@ -1032,6 +1056,16 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
             }
         }
         qemu_tcg_wait_io_event();
+        CPU_FOREACH(cpu) {
+            if (cpu->exit && !cpu_can_run(cpu)) {
+                remove_cpu = cpu;
+                break;
+            }
+        }
+        if (remove_cpu) {
+            qemu_tcg_destroy_vcpu(remove_cpu);
+            remove_cpu = NULL;
+        }
     }
 
     return NULL;
@@ -1389,6 +1423,9 @@ static void tcg_exec_all(void)
                 break;
             }
         } else if (cpu->stop || cpu->stopped) {
+            if (cpu->exit) {
+                next_cpu = CPU_NEXT(cpu);
+            }
             break;
         }
     }
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 30cb84d..560caef 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -188,6 +188,7 @@ int kvm_has_intx_set_mask(void);
 
 int kvm_init_vcpu(CPUState *cpu);
 int kvm_cpu_exec(CPUState *cpu);
+int kvm_destroy_vcpu(CPUState *cpu);
 
 #ifdef NEED_CPU_H
 
diff --git a/kvm-all.c b/kvm-all.c
index 05a79c2..46e7853 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -71,6 +71,12 @@ typedef struct KVMSlot
 
 typedef struct kvm_dirty_log KVMDirtyLog;
 
+struct KVMParkedVcpu {
+    unsigned long vcpu_id;
+    int kvm_fd;
+    QLIST_ENTRY(KVMParkedVcpu) node;
+};
+
 struct KVMState
 {
     AccelState parent_obj;
@@ -107,6 +113,7 @@ struct KVMState
     QTAILQ_HEAD(msi_hashtab, KVMMSIRoute) msi_hashtab[KVM_MSI_HASHTAB_SIZE];
     bool direct_msi;
 #endif
+    QLIST_HEAD(, KVMParkedVcpu) kvm_parked_vcpus;
 };
 
 #define TYPE_KVM_ACCEL ACCEL_CLASS_NAME("kvm")
@@ -247,6 +254,53 @@ static int kvm_set_user_memory_region(KVMState *s, KVMSlot *slot)
     return kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
 }
 
+int kvm_destroy_vcpu(CPUState *cpu)
+{
+    KVMState *s = kvm_state;
+    long mmap_size;
+    struct KVMParkedVcpu *vcpu = NULL;
+    int ret = 0;
+
+    DPRINTF("kvm_destroy_vcpu\n");
+
+    mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
+    if (mmap_size < 0) {
+        ret = mmap_size;
+        DPRINTF("KVM_GET_VCPU_MMAP_SIZE failed\n");
+        goto err;
+    }
+
+    ret = munmap(cpu->kvm_run, mmap_size);
+    if (ret < 0) {
+        goto err;
+    }
+
+    vcpu = g_malloc0(sizeof(*vcpu));
+    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
+    vcpu->kvm_fd = cpu->kvm_fd;
+    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+err:
+    return ret;
+}
+
+static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
+{
+    struct KVMParkedVcpu *cpu;
+
+    QLIST_FOREACH(cpu, &s->kvm_parked_vcpus, node) {
+        if (cpu->vcpu_id == vcpu_id) {
+            int kvm_fd;
+
+            QLIST_REMOVE(cpu, node);
+            kvm_fd = cpu->kvm_fd;
+            g_free(cpu);
+            return kvm_fd;
+        }
+    }
+
+    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
+}
+
 int kvm_init_vcpu(CPUState *cpu)
 {
     KVMState *s = kvm_state;
@@ -255,7 +309,7 @@ int kvm_init_vcpu(CPUState *cpu)
 
     DPRINTF("kvm_init_vcpu\n");
 
-    ret = kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)kvm_arch_vcpu_id(cpu));
+    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
     if (ret < 0) {
         DPRINTF("kvm_create_vcpu failed\n");
         goto err;
@@ -1448,6 +1502,7 @@ static int kvm_init(MachineState *ms)
 #ifdef KVM_CAP_SET_GUEST_DEBUG
     QTAILQ_INIT(&s->kvm_sw_breakpoints);
 #endif
+    QLIST_INIT(&s->kvm_parked_vcpus);
     s->vmfd = -1;
     s->fd = qemu_open("/dev/kvm", O_RDWR);
     if (s->fd == -1) {
-- 
1.9.3

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Gu Zheng]

Hi Igor, Andreas,

Could you please help to review this version?

Thanks,
Gu
On 02/13/2015 06:40 PM, Zhu Guihua wrote:

> This series is based on chen fan's previous i386 cpu hot remove patchset:
> https://lists.nongnu.org/archive/html/qemu-devel/2013-12/msg04266.html
> 
> Via implementing ACPI standard methods _EJ0 in ACPI table, after Guest
> OS remove one vCPU online, the fireware will store removed bitmap to
> QEMU, then QEMU could know to notify the assigned vCPU of exiting.
> Meanwhile, intruduce the QOM command 'device_del' to remove vCPU from
> QEMU itself.
> 
> The whole work is based on the new hot plug/unplug framework, ,the unplug request
> callback does the pre-check and send the request, unplug callback does the
> removal handling.
> 
> This series depends on tangchen's common hot plug/unplug enhance patchset.
> [PATCH v2 0/5] Common unplug and unplug request cb for memory and CPU hot-unplug
> https://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg03929.html
> 
> The is the second half of the previous series:                                      
> [RFC V2 00/10] cpu: add device_add foo-x86_64-cpu and i386 cpu hot remove support
> https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg04779.html             
> 
> If you want to test the series, you need to apply the 'device_add foo-x86_64-cpu'
> patchset first:
> [PATCH v4 00/10] cpu: add device_add foo-x86_64-cpu support
> https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg02584.html
> 
> ---
> Changelog since v2:
>  -drop ICC bus impl
>  -fix delete cpu exceed 32 issue
>  -fix bug about deleting the last cpu
> 
> Changelog since v1:
>  -rebase on the latest version.
>  -delete patch i386/cpu: add instance finalize callback, and put it into patchset
>   [PATCH v3 0/6] cpu: add device_add foo-x86_64-cpu support.
> 
> Changelog since RFC:
>  -splited the i386 cpu hot remove into single thread.
>  -replaced apic_no with apic_id, so does the related stuff to make it
>   work with arbitrary CPU hotadd.
>  -add the icc_device_unrealize callback to handle apic unrealize.
>  -rework on the new hot plug/unplug platform.
> ---
> 
> Chen Fan (2):
>   x86: add x86_cpu_unrealizefn() for cpu apic remove
>   cpu hotplug: implement function cpu_status_write() for vcpu ejection
> 
> Gu Zheng (3):
>   acpi/cpu: add cpu hot unplug request callback function
>   acpi, pc: add cpu hot unplug callback support
>   cpus: reclaim allocated vCPU objects
> 
> Zhu Guihua (2):
>   acpi, pc: add cpu hot unplug request callback support
>   acpi/cpu: add cpu hot unplug callback function
> 
>  cpus.c                            | 44 ++++++++++++++++++++
>  hw/acpi/cpu_hotplug.c             | 87 ++++++++++++++++++++++++++++++++++++---
>  hw/acpi/ich9.c                    | 17 ++++++--
>  hw/acpi/piix4.c                   | 12 +++++-
>  hw/core/qdev.c                    |  2 +-
>  hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++-
>  hw/i386/kvm/apic.c                |  5 +++
>  hw/i386/pc.c                      | 68 ++++++++++++++++++++++++++++--
>  hw/intc/apic.c                    |  9 ++++
>  hw/intc/apic_common.c             | 21 ++++++----
>  include/hw/acpi/cpu_hotplug.h     |  8 ++++
>  include/hw/i386/apic_internal.h   |  1 +
>  include/hw/qdev-core.h            |  1 +
>  include/qom/cpu.h                 |  9 ++++
>  include/sysemu/kvm.h              |  1 +
>  kvm-all.c                         | 57 ++++++++++++++++++++++++-
>  target-i386/cpu.c                 | 46 +++++++++++++++++++++
>  17 files changed, 377 insertions(+), 27 deletions(-)
> 

Re: [Qemu-devel] [PATCH v3 7/7] cpus: reclaim allocated vCPU objects

[Bharata B Rao]

On Fri, Feb 13, 2015 at 4:10 PM, Zhu Guihua <zhugh.fnst@cn.fujitsu.com> wrote:
> From: Gu Zheng <guz.fnst@cn.fujitsu.com>
>
> After ACPI get a signal to eject a vCPU, the vCPU must be
> removed from CPU list，before the vCPU really removed,  then
> release the all related vCPU objects.
>
> In order to deal well with the kvm vcpus (which can not be removed without any
> protection), we do not close KVM vcpu fd, just record and mark it as stopped
> into a list, so that we can reuse it for the appending cpu hot-add request if
> possible. It is also the approach that kvm guys suggested:
> https://www.mail-archive.com/kvm@vger.kernel.org/msg102839.html
>
> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
> Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
> ---
>  cpus.c               | 37 ++++++++++++++++++++++++++++++++++
>  include/sysemu/kvm.h |  1 +
>  kvm-all.c            | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 94 insertions(+), 1 deletion(-)
>
> diff --git a/cpus.c b/cpus.c
> index d6e5a5f..a11941f 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -854,6 +854,24 @@ void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
>      qemu_cpu_kick(cpu);
>  }
>
> +static void qemu_kvm_destroy_vcpu(CPUState *cpu)
> +{
> +    CPU_REMOVE(cpu);
> +
> +    if (kvm_destroy_vcpu(cpu) < 0) {
> +        error_report("kvm_destroy_vcpu failed.\n");
> +        exit(EXIT_FAILURE);
> +    }
> +
> +    object_unparent(OBJECT(cpu));
> +}
> +
> +static void qemu_tcg_destroy_vcpu(CPUState *cpu)
> +{
> +    CPU_REMOVE(cpu);
> +    object_unparent(OBJECT(cpu));
> +}
> +
>  static void flush_queued_work(CPUState *cpu)
>  {
>      struct qemu_work_item *wi;
> @@ -946,6 +964,11 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
>              }
>          }
>          qemu_kvm_wait_io_event(cpu);
> +        if (cpu->exit && !cpu_can_run(cpu)) {
> +            qemu_kvm_destroy_vcpu(cpu);
> +            qemu_mutex_unlock(&qemu_global_mutex);
> +            return NULL;
> +        }
>      }
>
>      return NULL;
> @@ -999,6 +1022,7 @@ static void tcg_exec_all(void);
>  static void *qemu_tcg_cpu_thread_fn(void *arg)
>  {
>      CPUState *cpu = arg;
> +    CPUState *remove_cpu = NULL;
>
>      qemu_tcg_init_cpu_signals();
>      qemu_thread_get_self(cpu->thread);
> @@ -1032,6 +1056,16 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
>              }
>          }
>          qemu_tcg_wait_io_event();
> +        CPU_FOREACH(cpu) {
> +            if (cpu->exit && !cpu_can_run(cpu)) {
> +                remove_cpu = cpu;
> +                break;
> +            }
> +        }
> +        if (remove_cpu) {
> +            qemu_tcg_destroy_vcpu(remove_cpu);
> +            remove_cpu = NULL;
> +        }
>      }
>
>      return NULL;
> @@ -1389,6 +1423,9 @@ static void tcg_exec_all(void)
>                  break;
>              }
>          } else if (cpu->stop || cpu->stopped) {
> +            if (cpu->exit) {
> +                next_cpu = CPU_NEXT(cpu);
> +            }
>              break;
>          }
>      }
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 30cb84d..560caef 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -188,6 +188,7 @@ int kvm_has_intx_set_mask(void);
>
>  int kvm_init_vcpu(CPUState *cpu);
>  int kvm_cpu_exec(CPUState *cpu);
> +int kvm_destroy_vcpu(CPUState *cpu);
>
>  #ifdef NEED_CPU_H
>
> diff --git a/kvm-all.c b/kvm-all.c
> index 05a79c2..46e7853 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -71,6 +71,12 @@ typedef struct KVMSlot
>
>  typedef struct kvm_dirty_log KVMDirtyLog;
>
> +struct KVMParkedVcpu {
> +    unsigned long vcpu_id;
> +    int kvm_fd;
> +    QLIST_ENTRY(KVMParkedVcpu) node;
> +};
> +
>  struct KVMState
>  {
>      AccelState parent_obj;
> @@ -107,6 +113,7 @@ struct KVMState
>      QTAILQ_HEAD(msi_hashtab, KVMMSIRoute) msi_hashtab[KVM_MSI_HASHTAB_SIZE];
>      bool direct_msi;
>  #endif
> +    QLIST_HEAD(, KVMParkedVcpu) kvm_parked_vcpus;
>  };
>
>  #define TYPE_KVM_ACCEL ACCEL_CLASS_NAME("kvm")
> @@ -247,6 +254,53 @@ static int kvm_set_user_memory_region(KVMState *s, KVMSlot *slot)
>      return kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
>  }
>
> +int kvm_destroy_vcpu(CPUState *cpu)

You need to provide a stub for this in kvm-stub.c without which I see
that aarch64-softmmu won't compile.

Regards,
Bharata.

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 3767 bytes --]

Hello Zhu,

Are you still working on this feature? Could you provide a rebased
version of this series?

Regards,

On Fri, Feb 13, 2015 at 06=40=15PM +0800, Zhu Guihua wrote:
> This series is based on chen fan's previous i386 cpu hot remove patchset:
> https://lists.nongnu.org/archive/html/qemu-devel/2013-12/msg04266.html
> 
> Via implementing ACPI standard methods _EJ0 in ACPI table, after Guest
> OS remove one vCPU online, the fireware will store removed bitmap to
> QEMU, then QEMU could know to notify the assigned vCPU of exiting.
> Meanwhile, intruduce the QOM command 'device_del' to remove vCPU from
> QEMU itself.
> 
> The whole work is based on the new hot plug/unplug framework, ,the unplug request
> callback does the pre-check and send the request, unplug callback does the
> removal handling.
> 
> This series depends on tangchen's common hot plug/unplug enhance patchset.
> [PATCH v2 0/5] Common unplug and unplug request cb for memory and CPU hot-unplug
> https://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg03929.html
> 
> The is the second half of the previous series:                                      
> [RFC V2 00/10] cpu: add device_add foo-x86_64-cpu and i386 cpu hot remove support
> https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg04779.html             
> 
> If you want to test the series, you need to apply the 'device_add foo-x86_64-cpu'
> patchset first:
> [PATCH v4 00/10] cpu: add device_add foo-x86_64-cpu support
> https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg02584.html
> 
> ---
> Changelog since v2:
>  -drop ICC bus impl
>  -fix delete cpu exceed 32 issue
>  -fix bug about deleting the last cpu
> 
> Changelog since v1:
>  -rebase on the latest version.
>  -delete patch i386/cpu: add instance finalize callback, and put it into patchset
>   [PATCH v3 0/6] cpu: add device_add foo-x86_64-cpu support.
> 
> Changelog since RFC:
>  -splited the i386 cpu hot remove into single thread.
>  -replaced apic_no with apic_id, so does the related stuff to make it
>   work with arbitrary CPU hotadd.
>  -add the icc_device_unrealize callback to handle apic unrealize.
>  -rework on the new hot plug/unplug platform.
> ---
> 
> Chen Fan (2):
>   x86: add x86_cpu_unrealizefn() for cpu apic remove
>   cpu hotplug: implement function cpu_status_write() for vcpu ejection
> 
> Gu Zheng (3):
>   acpi/cpu: add cpu hot unplug request callback function
>   acpi, pc: add cpu hot unplug callback support
>   cpus: reclaim allocated vCPU objects
> 
> Zhu Guihua (2):
>   acpi, pc: add cpu hot unplug request callback support
>   acpi/cpu: add cpu hot unplug callback function
> 
>  cpus.c                            | 44 ++++++++++++++++++++
>  hw/acpi/cpu_hotplug.c             | 87 ++++++++++++++++++++++++++++++++++++---
>  hw/acpi/ich9.c                    | 17 ++++++--
>  hw/acpi/piix4.c                   | 12 +++++-
>  hw/core/qdev.c                    |  2 +-
>  hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++-
>  hw/i386/kvm/apic.c                |  5 +++
>  hw/i386/pc.c                      | 68 ++++++++++++++++++++++++++++--
>  hw/intc/apic.c                    |  9 ++++
>  hw/intc/apic_common.c             | 21 ++++++----
>  include/hw/acpi/cpu_hotplug.h     |  8 ++++
>  include/hw/i386/apic_internal.h   |  1 +
>  include/hw/qdev-core.h            |  1 +
>  include/qom/cpu.h                 |  9 ++++
>  include/sysemu/kvm.h              |  1 +
>  kvm-all.c                         | 57 ++++++++++++++++++++++++-
>  target-i386/cpu.c                 | 46 +++++++++++++++++++++
>  17 files changed, 377 insertions(+), 27 deletions(-)
> 
> -- 
> 1.9.3
> 
> 

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Zhu Guihua]

Hi,
On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> Hello Zhu,
>
> Are you still working on this feature? Could you provide a rebased
> version of this series?

Sorry for late reply.

Yes, we are still working on this feature.

I have updated my github, you can get the rebased version from it.

https://github.com/zhuguihua/qemu.git  cpu-hotplug

Thanks,
Zhu

> Regards,
>
> On Fri, Feb 13, 2015 at 06=40=15PM +0800, Zhu Guihua wrote:
>> This series is based on chen fan's previous i386 cpu hot remove patchset:
>> https://lists.nongnu.org/archive/html/qemu-devel/2013-12/msg04266.html
>>
>> Via implementing ACPI standard methods _EJ0 in ACPI table, after Guest
>> OS remove one vCPU online, the fireware will store removed bitmap to
>> QEMU, then QEMU could know to notify the assigned vCPU of exiting.
>> Meanwhile, intruduce the QOM command 'device_del' to remove vCPU from
>> QEMU itself.
>>
>> The whole work is based on the new hot plug/unplug framework, ,the unplug request
>> callback does the pre-check and send the request, unplug callback does the
>> removal handling.
>>
>> This series depends on tangchen's common hot plug/unplug enhance patchset.
>> [PATCH v2 0/5] Common unplug and unplug request cb for memory and CPU hot-unplug
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg03929.html
>>
>> The is the second half of the previous series:
>> [RFC V2 00/10] cpu: add device_add foo-x86_64-cpu and i386 cpu hot remove support
>> https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg04779.html
>>
>> If you want to test the series, you need to apply the 'device_add foo-x86_64-cpu'
>> patchset first:
>> [PATCH v4 00/10] cpu: add device_add foo-x86_64-cpu support
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg02584.html
>>
>> ---
>> Changelog since v2:
>>   -drop ICC bus impl
>>   -fix delete cpu exceed 32 issue
>>   -fix bug about deleting the last cpu
>>
>> Changelog since v1:
>>   -rebase on the latest version.
>>   -delete patch i386/cpu: add instance finalize callback, and put it into patchset
>>    [PATCH v3 0/6] cpu: add device_add foo-x86_64-cpu support.
>>
>> Changelog since RFC:
>>   -splited the i386 cpu hot remove into single thread.
>>   -replaced apic_no with apic_id, so does the related stuff to make it
>>    work with arbitrary CPU hotadd.
>>   -add the icc_device_unrealize callback to handle apic unrealize.
>>   -rework on the new hot plug/unplug platform.
>> ---
>>
>> Chen Fan (2):
>>    x86: add x86_cpu_unrealizefn() for cpu apic remove
>>    cpu hotplug: implement function cpu_status_write() for vcpu ejection
>>
>> Gu Zheng (3):
>>    acpi/cpu: add cpu hot unplug request callback function
>>    acpi, pc: add cpu hot unplug callback support
>>    cpus: reclaim allocated vCPU objects
>>
>> Zhu Guihua (2):
>>    acpi, pc: add cpu hot unplug request callback support
>>    acpi/cpu: add cpu hot unplug callback function
>>
>>   cpus.c                            | 44 ++++++++++++++++++++
>>   hw/acpi/cpu_hotplug.c             | 87 ++++++++++++++++++++++++++++++++++++---
>>   hw/acpi/ich9.c                    | 17 ++++++--
>>   hw/acpi/piix4.c                   | 12 +++++-
>>   hw/core/qdev.c                    |  2 +-
>>   hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++-
>>   hw/i386/kvm/apic.c                |  5 +++
>>   hw/i386/pc.c                      | 68 ++++++++++++++++++++++++++++--
>>   hw/intc/apic.c                    |  9 ++++
>>   hw/intc/apic_common.c             | 21 ++++++----
>>   include/hw/acpi/cpu_hotplug.h     |  8 ++++
>>   include/hw/i386/apic_internal.h   |  1 +
>>   include/hw/qdev-core.h            |  1 +
>>   include/qom/cpu.h                 |  9 ++++
>>   include/sysemu/kvm.h              |  1 +
>>   kvm-all.c                         | 57 ++++++++++++++++++++++++-
>>   target-i386/cpu.c                 | 46 +++++++++++++++++++++
>>   17 files changed, 377 insertions(+), 27 deletions(-)
>>
>> -- 
>> 1.9.3
>>
>>

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 4418 bytes --]

On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> Hi,
> On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> >Hello Zhu,
> >
> >Are you still working on this feature? Could you provide a rebased
> >version of this series?
> 
> Sorry for late reply.
> 
> Yes, we are still working on this feature.
> 
> I have updated my github, you can get the rebased version from it.
> 
> https://github.com/zhuguihua/qemu.git  cpu-hotplug

I'll review and test. I'll also provide my comments on the github as
well. Thanks for the update, Zhu! :)

Regards,

> 
> Thanks,
> Zhu
> 
> >Regards,
> >
> >On Fri, Feb 13, 2015 at 06=40=15PM +0800, Zhu Guihua wrote:
> >>This series is based on chen fan's previous i386 cpu hot remove patchset:
> >>https://lists.nongnu.org/archive/html/qemu-devel/2013-12/msg04266.html
> >>
> >>Via implementing ACPI standard methods _EJ0 in ACPI table, after Guest
> >>OS remove one vCPU online, the fireware will store removed bitmap to
> >>QEMU, then QEMU could know to notify the assigned vCPU of exiting.
> >>Meanwhile, intruduce the QOM command 'device_del' to remove vCPU from
> >>QEMU itself.
> >>
> >>The whole work is based on the new hot plug/unplug framework, ,the unplug request
> >>callback does the pre-check and send the request, unplug callback does the
> >>removal handling.
> >>
> >>This series depends on tangchen's common hot plug/unplug enhance patchset.
> >>[PATCH v2 0/5] Common unplug and unplug request cb for memory and CPU hot-unplug
> >>https://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg03929.html
> >>
> >>The is the second half of the previous series:
> >>[RFC V2 00/10] cpu: add device_add foo-x86_64-cpu and i386 cpu hot remove support
> >>https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg04779.html
> >>
> >>If you want to test the series, you need to apply the 'device_add foo-x86_64-cpu'
> >>patchset first:
> >>[PATCH v4 00/10] cpu: add device_add foo-x86_64-cpu support
> >>https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg02584.html
> >>
> >>---
> >>Changelog since v2:
> >>  -drop ICC bus impl
> >>  -fix delete cpu exceed 32 issue
> >>  -fix bug about deleting the last cpu
> >>
> >>Changelog since v1:
> >>  -rebase on the latest version.
> >>  -delete patch i386/cpu: add instance finalize callback, and put it into patchset
> >>   [PATCH v3 0/6] cpu: add device_add foo-x86_64-cpu support.
> >>
> >>Changelog since RFC:
> >>  -splited the i386 cpu hot remove into single thread.
> >>  -replaced apic_no with apic_id, so does the related stuff to make it
> >>   work with arbitrary CPU hotadd.
> >>  -add the icc_device_unrealize callback to handle apic unrealize.
> >>  -rework on the new hot plug/unplug platform.
> >>---
> >>
> >>Chen Fan (2):
> >>   x86: add x86_cpu_unrealizefn() for cpu apic remove
> >>   cpu hotplug: implement function cpu_status_write() for vcpu ejection
> >>
> >>Gu Zheng (3):
> >>   acpi/cpu: add cpu hot unplug request callback function
> >>   acpi, pc: add cpu hot unplug callback support
> >>   cpus: reclaim allocated vCPU objects
> >>
> >>Zhu Guihua (2):
> >>   acpi, pc: add cpu hot unplug request callback support
> >>   acpi/cpu: add cpu hot unplug callback function
> >>
> >>  cpus.c                            | 44 ++++++++++++++++++++
> >>  hw/acpi/cpu_hotplug.c             | 87 ++++++++++++++++++++++++++++++++++++---
> >>  hw/acpi/ich9.c                    | 17 ++++++--
> >>  hw/acpi/piix4.c                   | 12 +++++-
> >>  hw/core/qdev.c                    |  2 +-
> >>  hw/i386/acpi-dsdt-cpu-hotplug.dsl | 16 ++++++-
> >>  hw/i386/kvm/apic.c                |  5 +++
> >>  hw/i386/pc.c                      | 68 ++++++++++++++++++++++++++++--
> >>  hw/intc/apic.c                    |  9 ++++
> >>  hw/intc/apic_common.c             | 21 ++++++----
> >>  include/hw/acpi/cpu_hotplug.h     |  8 ++++
> >>  include/hw/i386/apic_internal.h   |  1 +
> >>  include/hw/qdev-core.h            |  1 +
> >>  include/qom/cpu.h                 |  9 ++++
> >>  include/sysemu/kvm.h              |  1 +
> >>  kvm-all.c                         | 57 ++++++++++++++++++++++++-
> >>  target-i386/cpu.c                 | 46 +++++++++++++++++++++
> >>  17 files changed, 377 insertions(+), 27 deletions(-)
> >>
> >>-- 
> >>1.9.3
> >>
> >>
> 
> 

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> Hi,
> On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> >Hello Zhu,
> >
> >Are you still working on this feature? Could you provide a rebased
> >version of this series?
> 
> Sorry for late reply.
> 
> Yes, we are still working on this feature.
> 
> I have updated my github, you can get the rebased version from it.
> 
> https://github.com/zhuguihua/qemu.git  cpu-hotplug

Hi, thanks a lot for rebasing it. I didn't have time to test it, though.
I appologise.

Regarding your branch, your plan is to stick with device_del and
object_del interface?

Best regards,

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Zhu Guihua]

On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
> On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
>> Hi,
>> On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
>>> Hello Zhu,
>>>
>>> Are you still working on this feature? Could you provide a rebased
>>> version of this series?
>> Sorry for late reply.
>>
>> Yes, we are still working on this feature.
>>
>> I have updated my github, you can get the rebased version from it.
>>
>> https://github.com/zhuguihua/qemu.git  cpu-hotplug
> Hi, thanks a lot for rebasing it. I didn't have time to test it, though.
> I appologise.
>
> Regarding your branch, your plan is to stick with device_del and
> object_del interface?

Yeah, we will only stick with device_del interface.
There is no need to use object_del interface for cpu hot remove.

Thanks,
Zhu

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

On Mon, Jul 13, 2015 at 09=45=57AM +0800, Zhu Guihua wrote:
> 
> On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
> >On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> >>Hi,
> >>On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> >>>Hello Zhu,
> >>>
> >>>Are you still working on this feature? Could you provide a rebased
> >>>version of this series?
> >>Sorry for late reply.
> >>
> >>Yes, we are still working on this feature.
> >>
> >>I have updated my github, you can get the rebased version from it.
> >>
> >>https://github.com/zhuguihua/qemu.git  cpu-hotplug
> >Hi, thanks a lot for rebasing it. I didn't have time to test it, though.
> >I appologise.
> >
> >Regarding your branch, your plan is to stick with device_del and
> >object_del interface?
> 
> Yeah, we will only stick with device_del interface.
> There is no need to use object_del interface for cpu hot remove.
> 
> Thanks,
> Zhu
> 

This is almost a year old email, but I decided to git it a try anyway.
Are you still working on this? I'm trying your patches and it seems I
had a simple issue: EJ0 is sent, guest ack's and removes the CPU, but it
get's stuck somewhere between the BIOS and Qemu. CPU is successfully
removed from the guest, but it still appears on QOM:

    (qemu) info cpus
    * CPU #0: pc=0xffffffff8104f596 (halted) thread_id=22208
      CPU #1: pc=0xffffffff8104f596 (halted) thread_id=22209
      CPU #2: pc=0xffffffff810429c8 (halted) thread_id=22281
    (qemu) info qom-tree 
    /machine (pc-i440fx-2.1-machine)
      /fw_cfg (fw_cfg_io)
        /fwcfg[0] (qemu:memory-region)
    
    [...]
    
        /cpu3 (SandyBridge-x86_64-cpu)
    
    [...]

Do you have any clues?

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Igor Mammedov]

On Wed, 22 Jun 2016 14:12:49 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> On Mon, Jul 13, 2015 at 09=45=57AM +0800, Zhu Guihua wrote:
> > 
> > On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
> > >On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> > >>Hi,
> > >>On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> > >>>Hello Zhu,
> > >>>
> > >>>Are you still working on this feature? Could you provide a
> > >>>rebased version of this series?
> > >>Sorry for late reply.
> > >>
> > >>Yes, we are still working on this feature.
> > >>
> > >>I have updated my github, you can get the rebased version from it.
> > >>
> > >>https://github.com/zhuguihua/qemu.git  cpu-hotplug
> > >Hi, thanks a lot for rebasing it. I didn't have time to test it,
> > >though. I appologise.
> > >
> > >Regarding your branch, your plan is to stick with device_del and
> > >object_del interface?
> > 
> > Yeah, we will only stick with device_del interface.
> > There is no need to use object_del interface for cpu hot remove.
> > 
> > Thanks,
> > Zhu
> > 
> 
> This is almost a year old email, but I decided to git it a try anyway.
> Are you still working on this? I'm trying your patches and it seems I
> had a simple issue: EJ0 is sent, guest ack's and removes the CPU, but
> it get's stuck somewhere between the BIOS and Qemu. CPU is
> successfully removed from the guest, but it still appears on QOM:
I'm working on new version and I plan to post patches this week.

> 
>     (qemu) info cpus
>     * CPU #0: pc=0xffffffff8104f596 (halted) thread_id=22208
>       CPU #1: pc=0xffffffff8104f596 (halted) thread_id=22209
>       CPU #2: pc=0xffffffff810429c8 (halted) thread_id=22281
>     (qemu) info qom-tree 
>     /machine (pc-i440fx-2.1-machine)
>       /fw_cfg (fw_cfg_io)
>         /fwcfg[0] (qemu:memory-region)
>     
>     [...]
>     
>         /cpu3 (SandyBridge-x86_64-cpu)
>     
>     [...]
> 
> Do you have any clues?
> 

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Eduardo Otubo]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On Wed, Jun 22, 2016 at 03=24=14PM +0200, Igor Mammedov wrote:
> On Wed, 22 Jun 2016 14:12:49 +0200
> Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> 
> > On Mon, Jul 13, 2015 at 09=45=57AM +0800, Zhu Guihua wrote:
> > > 
> > > On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
> > > >On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> > > >>Hi,
> > > >>On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> > > >>>Hello Zhu,
> > > >>>
> > > >>>Are you still working on this feature? Could you provide a
> > > >>>rebased version of this series?
> > > >>Sorry for late reply.
> > > >>
> > > >>Yes, we are still working on this feature.
> > > >>
> > > >>I have updated my github, you can get the rebased version from it.
> > > >>
> > > >>https://github.com/zhuguihua/qemu.git  cpu-hotplug
> > > >Hi, thanks a lot for rebasing it. I didn't have time to test it,
> > > >though. I appologise.
> > > >
> > > >Regarding your branch, your plan is to stick with device_del and
> > > >object_del interface?
> > > 
> > > Yeah, we will only stick with device_del interface.
> > > There is no need to use object_del interface for cpu hot remove.
> > > 
> > > Thanks,
> > > Zhu
> > > 
> > 
> > This is almost a year old email, but I decided to git it a try anyway.
> > Are you still working on this? I'm trying your patches and it seems I
> > had a simple issue: EJ0 is sent, guest ack's and removes the CPU, but
> > it get's stuck somewhere between the BIOS and Qemu. CPU is
> > successfully removed from the guest, but it still appears on QOM:
> I'm working on new version and I plan to post patches this week.
> 

I saw your RFCs. Will your new patches include QMP/HMP interface?
Looking forward to test.

regards,

-- 
Eduardo Otubo
ProfitBricks GmbH

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Igor Mammedov]

On Wed, 22 Jun 2016 15:27:17 +0200
Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:

> On Wed, Jun 22, 2016 at 03=24=14PM +0200, Igor Mammedov wrote:
> > On Wed, 22 Jun 2016 14:12:49 +0200
> > Eduardo Otubo <eduardo.otubo@profitbricks.com> wrote:
> > 
> > > On Mon, Jul 13, 2015 at 09=45=57AM +0800, Zhu Guihua wrote:
> > > > 
> > > > On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
> > > > >On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
> > > > >>Hi,
> > > > >>On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
> > > > >>>Hello Zhu,
> > > > >>>
> > > > >>>Are you still working on this feature? Could you provide a
> > > > >>>rebased version of this series?
> > > > >>Sorry for late reply.
> > > > >>
> > > > >>Yes, we are still working on this feature.
> > > > >>
> > > > >>I have updated my github, you can get the rebased version
> > > > >>from it.
> > > > >>
> > > > >>https://github.com/zhuguihua/qemu.git  cpu-hotplug
> > > > >Hi, thanks a lot for rebasing it. I didn't have time to test
> > > > >it, though. I appologise.
> > > > >
> > > > >Regarding your branch, your plan is to stick with device_del
> > > > >and object_del interface?
> > > > 
> > > > Yeah, we will only stick with device_del interface.
> > > > There is no need to use object_del interface for cpu hot remove.
> > > > 
> > > > Thanks,
> > > > Zhu
> > > > 
> > > 
> > > This is almost a year old email, but I decided to git it a try
> > > anyway. Are you still working on this? I'm trying your patches
> > > and it seems I had a simple issue: EJ0 is sent, guest ack's and
> > > removes the CPU, but it get's stuck somewhere between the BIOS
> > > and Qemu. CPU is successfully removed from the guest, but it
> > > still appears on QOM:
> > I'm working on new version and I plan to post patches this week.
> > 
> 
> I saw your RFCs. Will your new patches include QMP/HMP interface?
interface is already in master with spapr cpu hotplug support.
my patches will provide x86 implementation of it.

> Looking forward to test.
> 
> regards,
> 

Re: [Qemu-devel] [PATCH v3 0/7] cpu: add i386 cpu hot remove support

[Dou Liyang]

Hi,

Thanks to test.

在 2016年06月22日 20:12, Eduardo Otubo 写道:
> On Mon, Jul 13, 2015 at 09=45=57AM +0800, Zhu Guihua wrote:
>> On 07/09/2015 10:25 PM, Eduardo Otubo wrote:
>>> On Fri, Jun 26, 2015 at 11=37=43AM +0800, Zhu Guihua wrote:
>>>> Hi,
>>>> On 06/24/2015 09:28 PM, Eduardo Otubo wrote:
>>>>> Hello Zhu,
>>>>>
>>>>> Are you still working on this feature? Could you provide a rebased
>>>>> version of this series?
>>>> Sorry for late reply.
>>>>
>>>> Yes, we are still working on this feature.
>>>>
>>>> I have updated my github, you can get the rebased version from it.
>>>>
>>>> https://github.com/zhuguihua/qemu.git  cpu-hotplug
>>> Hi, thanks a lot for rebasing it. I didn't have time to test it, though.
>>> I appologise.
>>>
>>> Regarding your branch, your plan is to stick with device_del and
>>> object_del interface?
>> Yeah, we will only stick with device_del interface.
>> There is no need to use object_del interface for cpu hot remove.
>>
>> Thanks,
>> Zhu
>>
> This is almost a year old email, but I decided to git it a try anyway.
> Are you still working on this?
Yes, we are still working on this feature.

but, as we discuss almost one year age and the Abstract CPU core type is 
defined recently, this patch's QMP/HMP interface will be replace.

we are  following Igor's patches to see what we can do for this feature.
> I'm trying your patches and it seems I
> had a simple issue: EJ0 is sent, guest ack's and removes the CPU, but it
> get's stuck somewhere between the BIOS and Qemu. CPU is successfully
> removed from the guest, but it still appears on QOM:
>
>      (qemu) info cpus
>      * CPU #0: pc=0xffffffff8104f596 (halted) thread_id=22208
>        CPU #1: pc=0xffffffff8104f596 (halted) thread_id=22209
>        CPU #2: pc=0xffffffff810429c8 (halted) thread_id=22281
>      (qemu) info qom-tree
>      /machine (pc-i440fx-2.1-machine)
>        /fw_cfg (fw_cfg_io)
>          /fwcfg[0] (qemu:memory-region)
>      
>      [...]
>      
>          /cpu3 (SandyBridge-x86_64-cpu)
>      
>      [...]
>
> Do you have any clues?
we guess Qom may not be updated during the remove event. we will concern it.

Thanks,
Dou.
>