[Qemu-devel] [PATCH v4 0/5] Allow hotplug of s390 CPUs

[Qemu-devel] [PATCH v4 0/5] Allow hotplug of s390 CPUs

[Matthew Rosato]

Changes from v3-v4:
* Drop patches related to device_add / device_del
* Add patch to enable cpu_add for s390x.

**************

As discussed in the KVM call, we will go ahead with cpu_add for 
s390x to get cpu hotplug functionality in s390x now, until
architectures that require a more robust hotplug interface
settle on a design.

To configure a guest with 2 CPUs online at 
boot and 4 maximum:

qemu -smp 2,maxcpus=4

Or, when using libvirt:
  <domain>
    ...
    <vcpu current="2">4</vcpu>
    ...
  </domain> 


To subsequently hotplug a CPU:

Issue 'cpu-add <id>' from qemu monitor, or use virsh setvcpus --count <n> 
<domain>, where <n> is the total number of desired guest CPUs.

At this point, the guest must bring the CPU online for use -- This can be 
achieved via "echo 1 > /sys/devices/system/cpu/cpuX/online" or via a management 
tool like cpuplugd.

This patch set is based on work previously done by Jason Herne.


Matthew Rosato (5):
  s390x/cpu: Cleanup init in preparation for hotplug
  s390x/cpu: Set initial CPU state in common routine
  s390x/cpu: Move some CPU initialization into realize
  s390x/cpu: Add functions to register CPU state
  s390x/cpu: Allow hotplug of CPUs

 hw/s390x/s390-virtio-ccw.c |  3 ++-
 hw/s390x/s390-virtio.c     | 55 +++++++++++++++++++++++++++-----------
 hw/s390x/s390-virtio.h     |  2 +-
 target-s390x/cpu.c         | 66 +++++++++++++++++++++++++++++++++++++++++++---
 target-s390x/cpu.h         |  2 ++
 5 files changed, 107 insertions(+), 21 deletions(-)

-- 
1.9.1

[Qemu-devel] [PATCH v4 1/5] s390x/cpu: Cleanup init in preparation for hotplug

[Matthew Rosato]

Ensure a valid cpu_model is set upfront by setting the
default value directly into the MachineState when none is
specified.  This is needed to ensure hotplugged CPUs share
the same cpu_model.

Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
---
 hw/s390x/s390-virtio-ccw.c | 2 +-
 hw/s390x/s390-virtio.c     | 8 ++++----
 hw/s390x/s390-virtio.h     | 2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 89f5d0d..b05ed8b 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -136,7 +136,7 @@ static void ccw_init(MachineState *machine)
     virtio_ccw_register_hcalls();
 
     /* init CPUs */
-    s390_init_cpus(machine->cpu_model);
+    s390_init_cpus(machine);
 
     if (kvm_enabled()) {
         kvm_s390_enable_css_support(s390_cpu_addr2state(0));
diff --git a/hw/s390x/s390-virtio.c b/hw/s390x/s390-virtio.c
index c320878..b576811 100644
--- a/hw/s390x/s390-virtio.c
+++ b/hw/s390x/s390-virtio.c
@@ -95,12 +95,12 @@ void s390_init_ipl_dev(const char *kernel_filename,
     qdev_init_nofail(dev);
 }
 
-void s390_init_cpus(const char *cpu_model)
+void s390_init_cpus(MachineState *machine)
 {
     int i;
 
-    if (cpu_model == NULL) {
-        cpu_model = "host";
+    if (machine->cpu_model == NULL) {
+        machine->cpu_model = "host";
     }
 
     ipi_states = g_malloc(sizeof(S390CPU *) * smp_cpus);
@@ -109,7 +109,7 @@ void s390_init_cpus(const char *cpu_model)
         S390CPU *cpu;
         CPUState *cs;
 
-        cpu = cpu_s390x_init(cpu_model);
+        cpu = cpu_s390x_init(machine->cpu_model);
         cs = CPU(cpu);
 
         ipi_states[i] = cpu;
diff --git a/hw/s390x/s390-virtio.h b/hw/s390x/s390-virtio.h
index eebce8e..ffd014c 100644
--- a/hw/s390x/s390-virtio.h
+++ b/hw/s390x/s390-virtio.h
@@ -19,7 +19,7 @@
 typedef int (*s390_virtio_fn)(const uint64_t *args);
 void s390_register_virtio_hypercall(uint64_t code, s390_virtio_fn fn);
 
-void s390_init_cpus(const char *cpu_model);
+void s390_init_cpus(MachineState *machine);
 void s390_init_ipl_dev(const char *kernel_filename,
                        const char *kernel_cmdline,
                        const char *initrd_filename,
-- 
1.9.1

[Qemu-devel] [PATCH v4 2/5] s390x/cpu: Set initial CPU state in common routine

[Matthew Rosato]

Both initial and hotplugged CPUs need to set the same initial
state.

Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
---
 hw/s390x/s390-virtio.c | 4 ----
 target-s390x/cpu.c     | 2 ++
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/hw/s390x/s390-virtio.c b/hw/s390x/s390-virtio.c
index b576811..b3707f4 100644
--- a/hw/s390x/s390-virtio.c
+++ b/hw/s390x/s390-virtio.c
@@ -107,14 +107,10 @@ void s390_init_cpus(MachineState *machine)
 
     for (i = 0; i < smp_cpus; i++) {
         S390CPU *cpu;
-        CPUState *cs;
 
         cpu = cpu_s390x_init(machine->cpu_model);
-        cs = CPU(cpu);
 
         ipi_states[i] = cpu;
-        cs->halted = 1;
-        cs->exception_index = EXCP_HLT;
     }
 }
 
diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
index 73a910d..603c2a1 100644
--- a/target-s390x/cpu.c
+++ b/target-s390x/cpu.c
@@ -219,6 +219,8 @@ static void s390_cpu_initfn(Object *obj)
 #endif
 
     cs->env_ptr = env;
+    cs->halted = 1;
+    cs->exception_index = EXCP_HLT;
     cpu_exec_init(cs, &error_abort);
 #if !defined(CONFIG_USER_ONLY)
     qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
-- 
1.9.1

[Qemu-devel] [PATCH v4 3/5] s390x/cpu: Move some CPU initialization into realize

[Matthew Rosato]

In preparation for hotplug, defer some CPU initialization
until the device is actually being realized.

Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
---
 target-s390x/cpu.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
index 603c2a1..5f6411f 100644
--- a/target-s390x/cpu.c
+++ b/target-s390x/cpu.c
@@ -37,6 +37,9 @@
 #define CR0_RESET       0xE0UL
 #define CR14_RESET      0xC2000000UL;
 
+/* Older kernels require CPUs to be added sequentially by id */
+static int next_cpu_id;
+
 /* generate CPU information for cpu -? */
 void s390_cpu_list(FILE *f, fprintf_function cpu_fprintf)
 {
@@ -195,7 +198,13 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
 {
     CPUState *cs = CPU(dev);
     S390CPUClass *scc = S390_CPU_GET_CLASS(dev);
+    S390CPU *cpu = S390_CPU(dev);
+    CPUS390XState *env = &cpu->env;
 
+#if !defined(CONFIG_USER_ONLY)
+    qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
+#endif
+    env->cpu_num = next_cpu_id++;
     s390_cpu_gdb_init(cs);
     qemu_init_vcpu(cs);
 #if !defined(CONFIG_USER_ONLY)
@@ -213,7 +222,6 @@ static void s390_cpu_initfn(Object *obj)
     S390CPU *cpu = S390_CPU(obj);
     CPUS390XState *env = &cpu->env;
     static bool inited;
-    static int cpu_num = 0;
 #if !defined(CONFIG_USER_ONLY)
     struct tm tm;
 #endif
@@ -223,7 +231,6 @@ static void s390_cpu_initfn(Object *obj)
     cs->exception_index = EXCP_HLT;
     cpu_exec_init(cs, &error_abort);
 #if !defined(CONFIG_USER_ONLY)
-    qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
     qemu_get_timedate(&tm, 0);
     env->tod_offset = TOD_UNIX_EPOCH +
                       (time2tod(mktimegm(&tm)) * 1000000000ULL);
@@ -232,7 +239,6 @@ static void s390_cpu_initfn(Object *obj)
     env->cpu_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, s390x_cpu_timer, cpu);
     s390_cpu_set_state(CPU_STATE_STOPPED, cpu);
 #endif
-    env->cpu_num = cpu_num++;
 
     if (tcg_enabled() && !inited) {
         inited = true;
-- 
1.9.1

[Qemu-devel] [PATCH v4 4/5] s390x/cpu: Add functions to register CPU state

[Matthew Rosato]

Introduce s390_register_cpustate, which will set the
machine/cpu[n] link with the current CPU state.  Additionally,
maintain an array of state pointers indexed by CPU id for fast lookup
during interrupt handling.

Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
---
 hw/s390x/s390-virtio.c | 45 ++++++++++++++++++++++++++++++++++++---------
 target-s390x/cpu.c     | 14 +++++++++++++-
 target-s390x/cpu.h     |  1 +
 3 files changed, 50 insertions(+), 10 deletions(-)

diff --git a/hw/s390x/s390-virtio.c b/hw/s390x/s390-virtio.c
index b3707f4..6a1e3f6 100644
--- a/hw/s390x/s390-virtio.c
+++ b/hw/s390x/s390-virtio.c
@@ -60,15 +60,35 @@
 #define S390_TOD_CLOCK_VALUE_MISSING    0x00
 #define S390_TOD_CLOCK_VALUE_PRESENT    0x01
 
-static S390CPU **ipi_states;
+static S390CPU **cpu_states;
 
 S390CPU *s390_cpu_addr2state(uint16_t cpu_addr)
 {
-    if (cpu_addr >= smp_cpus) {
+    if (cpu_addr >= max_cpus) {
         return NULL;
     }
 
-    return ipi_states[cpu_addr];
+    /* Fast lookup via CPU ID */
+    return cpu_states[cpu_addr];
+}
+
+int s390_register_cpustate(uint16_t cpu_addr, S390CPU *state)
+{
+    gchar *name;
+    int r = 0;
+
+    name = g_strdup_printf("cpu[%i]", cpu_addr);
+    if (object_property_get_link(qdev_get_machine(), name, NULL)) {
+        r = -EEXIST;
+        goto out;
+    }
+
+    object_property_set_link(qdev_get_machine(), OBJECT(state), name,\
+                             &error_abort);
+
+out:
+    g_free(name);
+    return r;
 }
 
 void s390_init_ipl_dev(const char *kernel_filename,
@@ -98,19 +118,26 @@ void s390_init_ipl_dev(const char *kernel_filename,
 void s390_init_cpus(MachineState *machine)
 {
     int i;
+    gchar *name;
 
     if (machine->cpu_model == NULL) {
         machine->cpu_model = "host";
     }
 
-    ipi_states = g_malloc(sizeof(S390CPU *) * smp_cpus);
-
-    for (i = 0; i < smp_cpus; i++) {
-        S390CPU *cpu;
+    cpu_states = g_malloc0(sizeof(S390CPU *) * max_cpus);
 
-        cpu = cpu_s390x_init(machine->cpu_model);
+    for (i = 0; i < max_cpus; i++) {
+        name = g_strdup_printf("cpu[%i]", i);
+        object_property_add_link(qdev_get_machine(), name, TYPE_S390_CPU,
+                                 (Object **) &cpu_states[i],
+                                 object_property_allow_set_link,
+                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
+                                 &error_abort);
+        g_free(name);
+    }
 
-        ipi_states[i] = cpu;
+    for (i = 0; i < smp_cpus; i++) {
+        cpu_s390x_init(machine->cpu_model);
     }
 }
 
diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
index 5f6411f..620b2e3 100644
--- a/target-s390x/cpu.c
+++ b/target-s390x/cpu.c
@@ -32,6 +32,7 @@
 #include "trace.h"
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/arch_init.h"
+#include "sysemu/sysemu.h"
 #endif
 
 #define CR0_RESET       0xE0UL
@@ -202,9 +203,20 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
     CPUS390XState *env = &cpu->env;
 
 #if !defined(CONFIG_USER_ONLY)
+    if (s390_register_cpustate(next_cpu_id, cpu) < 0) {
+        error_setg(errp, "Cannot have more than %d CPUs", max_cpus);
+        return;
+    }
     qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
-#endif
+    env->cpu_num = next_cpu_id;
+    while (next_cpu_id < max_cpus - 1) {
+        if (!cpu_exists(++next_cpu_id)) {
+            break;
+        }
+    }
+#else
     env->cpu_num = next_cpu_id++;
+#endif
     s390_cpu_gdb_init(cs);
     qemu_init_vcpu(cs);
 #if !defined(CONFIG_USER_ONLY)
diff --git a/target-s390x/cpu.h b/target-s390x/cpu.h
index 06ca60b..429bf90 100644
--- a/target-s390x/cpu.h
+++ b/target-s390x/cpu.h
@@ -531,6 +531,7 @@ static inline int s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
 }
 
 S390CPU *s390_cpu_addr2state(uint16_t cpu_addr);
+int s390_register_cpustate(uint16_t cpu_addr, S390CPU *state);
 unsigned int s390_cpu_halt(S390CPU *cpu);
 void s390_cpu_unhalt(S390CPU *cpu);
 unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
-- 
1.9.1

[Qemu-devel] [PATCH v4 5/5] s390x/cpu: Allow hotplug of CPUs

[Matthew Rosato]

Implement cpu hotplug routine and add the machine hook.

Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
---
 hw/s390x/s390-virtio-ccw.c |  1 +
 target-s390x/cpu.c         | 40 ++++++++++++++++++++++++++++++++++++++++
 target-s390x/cpu.h         |  1 +
 3 files changed, 42 insertions(+)

diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index b05ed8b..a15904d 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -163,6 +163,7 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
 
     mc->init = ccw_init;
     mc->reset = s390_machine_reset;
+    mc->hot_add_cpu = s390_hot_add_cpu;
     mc->block_default_type = IF_VIRTIO;
     mc->no_cdrom = 1;
     mc->no_floppy = 1;
diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
index 620b2e3..b4c59c9 100644
--- a/target-s390x/cpu.c
+++ b/target-s390x/cpu.c
@@ -33,6 +33,8 @@
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/arch_init.h"
 #include "sysemu/sysemu.h"
+#include "hw/boards.h"
+#include "hw/s390x/sclp.h"
 #endif
 
 #define CR0_RESET       0xE0UL
@@ -226,6 +228,12 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
 #endif
 
     scc->parent_realize(dev, errp);
+
+#if !defined(CONFIG_USER_ONLY)
+    if (dev->hotplugged) {
+        raise_irq_cpu_hotplug();
+    }
+#endif
 }
 
 static void s390_cpu_initfn(Object *obj)
@@ -269,6 +277,38 @@ static void s390_cpu_finalize(Object *obj)
 }
 
 #if !defined(CONFIG_USER_ONLY)
+void s390_hot_add_cpu(const int64_t id, Error **errp)
+{
+    MachineState *machine = MACHINE(qdev_get_machine());
+
+    if (id < 0) {
+        error_setg(errp, "Invalid CPU id: %" PRIi64, id);
+        return;
+    }
+
+    if (cpu_exists(id)) {
+        error_setg(errp, "Unable to add CPU: %" PRIi64
+                   ", it already exists", id);
+        return;
+    }
+
+    if (id >= max_cpus) {
+        error_setg(errp, "Unable to add CPU: %" PRIi64
+                   ", max allowed: %d", id, max_cpus - 1);
+        return;
+    }
+
+    if (id != next_cpu_id) {
+        error_setg(errp, "Unable to add CPU: %" PRIi64
+                   ", The next available id is %d", id, next_cpu_id);
+        return;
+    }
+
+    cpu_s390x_init(machine->cpu_model);
+}
+#endif
+
+#if !defined(CONFIG_USER_ONLY)
 static bool disabled_wait(CPUState *cpu)
 {
     return cpu->halted && !(S390_CPU(cpu)->env.psw.mask &
diff --git a/target-s390x/cpu.h b/target-s390x/cpu.h
index 429bf90..3752932 100644
--- a/target-s390x/cpu.h
+++ b/target-s390x/cpu.h
@@ -542,6 +542,7 @@ static inline uint8_t s390_cpu_get_state(S390CPU *cpu)
 
 void gtod_save(QEMUFile *f, void *opaque);
 int gtod_load(QEMUFile *f, void *opaque, int version_id);
+void s390_hot_add_cpu(const int64_t id, Error **errp);
 
 /* service interrupts are floating therefore we must not pass an cpustate */
 void s390_sclp_extint(uint32_t parm);
-- 
1.9.1

Re: [Qemu-devel] [PATCH v4 4/5] s390x/cpu: Add functions to register CPU state

[David Hildenbrand]

> Introduce s390_register_cpustate, which will set the
> machine/cpu[n] link with the current CPU state.  Additionally,
> maintain an array of state pointers indexed by CPU id for fast lookup
> during interrupt handling.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>

Acked-by: David Hildenbrand <dahi@linux.vnet.ibm.com>

David

Re: [Qemu-devel] [PATCH v4 5/5] s390x/cpu: Allow hotplug of CPUs

[David Hildenbrand]

> Implement cpu hotplug routine and add the machine hook.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
>  hw/s390x/s390-virtio-ccw.c |  1 +
>  target-s390x/cpu.c         | 40 ++++++++++++++++++++++++++++++++++++++++
>  target-s390x/cpu.h         |  1 +
>  3 files changed, 42 insertions(+)
> 
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index b05ed8b..a15904d 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -163,6 +163,7 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
>  
>      mc->init = ccw_init;
>      mc->reset = s390_machine_reset;
> +    mc->hot_add_cpu = s390_hot_add_cpu;
>      mc->block_default_type = IF_VIRTIO;
>      mc->no_cdrom = 1;
>      mc->no_floppy = 1;
> diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
> index 620b2e3..b4c59c9 100644
> --- a/target-s390x/cpu.c
> +++ b/target-s390x/cpu.c
> @@ -33,6 +33,8 @@
>  #ifndef CONFIG_USER_ONLY
>  #include "sysemu/arch_init.h"
>  #include "sysemu/sysemu.h"
> +#include "hw/boards.h"
> +#include "hw/s390x/sclp.h"
>  #endif
>  
>  #define CR0_RESET       0xE0UL
> @@ -226,6 +228,12 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
>  #endif
>  
>      scc->parent_realize(dev, errp);
> +
> +#if !defined(CONFIG_USER_ONLY)
> +    if (dev->hotplugged) {
> +        raise_irq_cpu_hotplug();
> +    }
> +#endif
>  }
>  
>  static void s390_cpu_initfn(Object *obj)
> @@ -269,6 +277,38 @@ static void s390_cpu_finalize(Object *obj)
>  }
>  
>  #if !defined(CONFIG_USER_ONLY)
> +void s390_hot_add_cpu(const int64_t id, Error **errp)
> +{
> +    MachineState *machine = MACHINE(qdev_get_machine());
> +
> +    if (id < 0) {
> +        error_setg(errp, "Invalid CPU id: %" PRIi64, id);
> +        return;
> +    }
> +
> +    if (cpu_exists(id)) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", it already exists", id);
> +        return;
> +    }
> +
> +    if (id >= max_cpus) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", max allowed: %d", id, max_cpus - 1);
> +        return;
> +    }
> +
> +    if (id != next_cpu_id) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", The next available id is %d", id, next_cpu_id);
> +        return;
> +    }
> +
> +    cpu_s390x_init(machine->cpu_model);
> +}
> +#endif
> +
> +#if !defined(CONFIG_USER_ONLY)

This #if block could be combined with the previous one. But it's not worth
blocking this series so

Acked-by: David Hildenbrand <dahi@linux.vnet.ibm.com>

As we have a clean interface now (cpu-add id=x) most of this series is internal
stuff, so I think this is ready to be picked up :) .

David

Re: [Qemu-devel] [PATCH v4 4/5] s390x/cpu: Add functions to register CPU state

[Igor Mammedov]

On Wed, 17 Feb 2016 15:12:34 -0500
Matthew Rosato <mjrosato@linux.vnet.ibm.com> wrote:

> Introduce s390_register_cpustate, which will set the
> machine/cpu[n] link with the current CPU state.  Additionally,
> maintain an array of state pointers indexed by CPU id for fast lookup
> during interrupt handling.
> 
cosmetic nit,
you call qdev_get_machine() several time withing single function
or when function already has machine argument. You probably shouldn't
do it or do it once and use return value throughout function.

> Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
>  hw/s390x/s390-virtio.c | 45 ++++++++++++++++++++++++++++++++++++---------
>  target-s390x/cpu.c     | 14 +++++++++++++-
>  target-s390x/cpu.h     |  1 +
>  3 files changed, 50 insertions(+), 10 deletions(-)
> 
> diff --git a/hw/s390x/s390-virtio.c b/hw/s390x/s390-virtio.c
> index b3707f4..6a1e3f6 100644
> --- a/hw/s390x/s390-virtio.c
> +++ b/hw/s390x/s390-virtio.c
> @@ -60,15 +60,35 @@
>  #define S390_TOD_CLOCK_VALUE_MISSING    0x00
>  #define S390_TOD_CLOCK_VALUE_PRESENT    0x01
>  
> -static S390CPU **ipi_states;
> +static S390CPU **cpu_states;
>  
>  S390CPU *s390_cpu_addr2state(uint16_t cpu_addr)
>  {
> -    if (cpu_addr >= smp_cpus) {
> +    if (cpu_addr >= max_cpus) {
>          return NULL;
>      }
>  
> -    return ipi_states[cpu_addr];
> +    /* Fast lookup via CPU ID */
> +    return cpu_states[cpu_addr];
> +}
> +
> +int s390_register_cpustate(uint16_t cpu_addr, S390CPU *state)
isn't it the board responsibility to link CPU somewhere?
I'd suggest to use for it generic device hotplug hooks
see how get_hotplug_handler() and pc_machine_device_plug_cb() are used,
it's executed after CPU's realize is successfully completed.

> +{
> +    gchar *name;
> +    int r = 0;
> +
> +    name = g_strdup_printf("cpu[%i]", cpu_addr);
> +    if (object_property_get_link(qdev_get_machine(), name, NULL)) {
> +        r = -EEXIST;
> +        goto out;
> +    }
> +
> +    object_property_set_link(qdev_get_machine(), OBJECT(state), name,\
> +                             &error_abort);
> +
> +out:
> +    g_free(name);
> +    return r;
>  }
>  
>  void s390_init_ipl_dev(const char *kernel_filename,
> @@ -98,19 +118,26 @@ void s390_init_ipl_dev(const char *kernel_filename,
>  void s390_init_cpus(MachineState *machine)
>  {
>      int i;
> +    gchar *name;
>  
>      if (machine->cpu_model == NULL) {
>          machine->cpu_model = "host";
>      }
>  
> -    ipi_states = g_malloc(sizeof(S390CPU *) * smp_cpus);
> -
> -    for (i = 0; i < smp_cpus; i++) {
> -        S390CPU *cpu;
> +    cpu_states = g_malloc0(sizeof(S390CPU *) * max_cpus);
>  
> -        cpu = cpu_s390x_init(machine->cpu_model);
> +    for (i = 0; i < max_cpus; i++) {
> +        name = g_strdup_printf("cpu[%i]", i);
> +        object_property_add_link(qdev_get_machine(), name, TYPE_S390_CPU,
> +                                 (Object **) &cpu_states[i],
> +                                 object_property_allow_set_link,
> +                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
> +                                 &error_abort);
> +        g_free(name);
> +    }
>  
> -        ipi_states[i] = cpu;
> +    for (i = 0; i < smp_cpus; i++) {
> +        cpu_s390x_init(machine->cpu_model);
>      }
>  }
>  
> diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
> index 5f6411f..620b2e3 100644
> --- a/target-s390x/cpu.c
> +++ b/target-s390x/cpu.c
> @@ -32,6 +32,7 @@
>  #include "trace.h"
>  #ifndef CONFIG_USER_ONLY
>  #include "sysemu/arch_init.h"
> +#include "sysemu/sysemu.h"
>  #endif
>  
>  #define CR0_RESET       0xE0UL
> @@ -202,9 +203,20 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
>      CPUS390XState *env = &cpu->env;
>  
>  #if !defined(CONFIG_USER_ONLY)
> +    if (s390_register_cpustate(next_cpu_id, cpu) < 0) {
> +        error_setg(errp, "Cannot have more than %d CPUs", max_cpus);
> +        return;
> +    }
>      qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
> -#endif
> +    env->cpu_num = next_cpu_id;
> +    while (next_cpu_id < max_cpus - 1) {
> +        if (!cpu_exists(++next_cpu_id)) {
> +            break;
> +        }
> +    }
maybe it's possible to reuse cpu_get_free_index() here?


> +#else
>      env->cpu_num = next_cpu_id++;
> +#endif
>      s390_cpu_gdb_init(cs);
>      qemu_init_vcpu(cs);
>  #if !defined(CONFIG_USER_ONLY)
> diff --git a/target-s390x/cpu.h b/target-s390x/cpu.h
> index 06ca60b..429bf90 100644
> --- a/target-s390x/cpu.h
> +++ b/target-s390x/cpu.h
> @@ -531,6 +531,7 @@ static inline int s390_set_clock(uint8_t *tod_high, uint64_t *tod_low)
>  }
>  
>  S390CPU *s390_cpu_addr2state(uint16_t cpu_addr);
> +int s390_register_cpustate(uint16_t cpu_addr, S390CPU *state);
>  unsigned int s390_cpu_halt(S390CPU *cpu);
>  void s390_cpu_unhalt(S390CPU *cpu);
>  unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);

Re: [Qemu-devel] [PATCH v4 5/5] s390x/cpu: Allow hotplug of CPUs

[Igor Mammedov]

On Wed, 17 Feb 2016 15:12:35 -0500
Matthew Rosato <mjrosato@linux.vnet.ibm.com> wrote:

> Implement cpu hotplug routine and add the machine hook.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
>  hw/s390x/s390-virtio-ccw.c |  1 +
>  target-s390x/cpu.c         | 40 ++++++++++++++++++++++++++++++++++++++++
>  target-s390x/cpu.h         |  1 +
>  3 files changed, 42 insertions(+)
> 
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index b05ed8b..a15904d 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -163,6 +163,7 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
>  
>      mc->init = ccw_init;
>      mc->reset = s390_machine_reset;
> +    mc->hot_add_cpu = s390_hot_add_cpu;
>      mc->block_default_type = IF_VIRTIO;
>      mc->no_cdrom = 1;
>      mc->no_floppy = 1;
> diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
> index 620b2e3..b4c59c9 100644
> --- a/target-s390x/cpu.c
> +++ b/target-s390x/cpu.c
> @@ -33,6 +33,8 @@
>  #ifndef CONFIG_USER_ONLY
>  #include "sysemu/arch_init.h"
>  #include "sysemu/sysemu.h"
> +#include "hw/boards.h"
> +#include "hw/s390x/sclp.h"
>  #endif
>  
>  #define CR0_RESET       0xE0UL
> @@ -226,6 +228,12 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
>  #endif
>  
>      scc->parent_realize(dev, errp);
> +
> +#if !defined(CONFIG_USER_ONLY)
> +    if (dev->hotplugged) {
> +        raise_irq_cpu_hotplug();
> +    }
> +#endif
>  }
>  
>  static void s390_cpu_initfn(Object *obj)
> @@ -269,6 +277,38 @@ static void s390_cpu_finalize(Object *obj)
>  }
>  
>  #if !defined(CONFIG_USER_ONLY)
> +void s390_hot_add_cpu(const int64_t id, Error **errp)
> +{
to make it future-proof wrt migration it could be better to
enforce here that 'id' grows in +1 steps so user
won't be able create cpus with gaps.


> +    MachineState *machine = MACHINE(qdev_get_machine());
> +
> +    if (id < 0) {
> +        error_setg(errp, "Invalid CPU id: %" PRIi64, id);
> +        return;
> +    }
> +
> +    if (cpu_exists(id)) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", it already exists", id);
> +        return;
> +    }
> +
> +    if (id >= max_cpus) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", max allowed: %d", id, max_cpus - 1);
> +        return;
> +    }
> +
> +    if (id != next_cpu_id) {
> +        error_setg(errp, "Unable to add CPU: %" PRIi64
> +                   ", The next available id is %d", id, next_cpu_id);
> +        return;
> +    }
> +
> +    cpu_s390x_init(machine->cpu_model);
> +}
> +#endif
> +
> +#if !defined(CONFIG_USER_ONLY)
>  static bool disabled_wait(CPUState *cpu)
>  {
>      return cpu->halted && !(S390_CPU(cpu)->env.psw.mask &
> diff --git a/target-s390x/cpu.h b/target-s390x/cpu.h
> index 429bf90..3752932 100644
> --- a/target-s390x/cpu.h
> +++ b/target-s390x/cpu.h
> @@ -542,6 +542,7 @@ static inline uint8_t s390_cpu_get_state(S390CPU *cpu)
>  
>  void gtod_save(QEMUFile *f, void *opaque);
>  int gtod_load(QEMUFile *f, void *opaque, int version_id);
> +void s390_hot_add_cpu(const int64_t id, Error **errp);
>  
>  /* service interrupts are floating therefore we must not pass an cpustate */
>  void s390_sclp_extint(uint32_t parm);

Re: [Qemu-devel] [PATCH v4 5/5] s390x/cpu: Allow hotplug of CPUs

[David Hildenbrand]

> >  #if !defined(CONFIG_USER_ONLY)
> > +void s390_hot_add_cpu(const int64_t id, Error **errp)
> > +{  
> to make it future-proof wrt migration it could be better to
> enforce here that 'id' grows in +1 steps so user
> won't be able create cpus with gaps.

That should be already covered by:

if (id != next_cpu_id)
...

or am I missing something?

David

Re: [Qemu-devel] [PATCH v4 5/5] s390x/cpu: Allow hotplug of CPUs

[Igor Mammedov]

On Thu, 18 Feb 2016 10:46:52 +0100
David Hildenbrand <dahi@linux.vnet.ibm.com> wrote:

> > >  #if !defined(CONFIG_USER_ONLY)
> > > +void s390_hot_add_cpu(const int64_t id, Error **errp)
> > > +{    
> > to make it future-proof wrt migration it could be better to
> > enforce here that 'id' grows in +1 steps so user
> > won't be able create cpus with gaps.  
> 
> That should be already covered by:
> 
> if (id != next_cpu_id)
> ...
> 
> or am I missing something?
yep, that should do the job, but it might be better to introduce
next_cpu_id in this patch so it would be clear were it comes from
and how it's used while dropping it completely from previous one.

essentially next_cpu_id is (cpu_index + 1) so you don't even need
to create a global var, just get the last CPU and to math,
something like that:
include/qom/cpu.h:
+#define last_cpu QTAILQ_LAST(&cpus)
 
if (id != (last_cpu->cpu_index + 1)) { .... }

> 
> David
> 

Re: [Qemu-devel] [PATCH v4 4/5] s390x/cpu: Add functions to register CPU state

[Matthew Rosato]

On 02/18/2016 04:36 AM, Igor Mammedov wrote:
> On Wed, 17 Feb 2016 15:12:34 -0500
> Matthew Rosato <mjrosato@linux.vnet.ibm.com> wrote:
> 
>> Introduce s390_register_cpustate, which will set the
>> machine/cpu[n] link with the current CPU state.  Additionally,
>> maintain an array of state pointers indexed by CPU id for fast lookup
>> during interrupt handling.
>>
> cosmetic nit,
> you call qdev_get_machine() several time withing single function
> or when function already has machine argument. You probably shouldn't
> do it or do it once and use return value throughout function.
> 

Oops, will fix that.

>> Signed-off-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
>> ---
>>  hw/s390x/s390-virtio.c | 45 ++++++++++++++++++++++++++++++++++++---------
>>  target-s390x/cpu.c     | 14 +++++++++++++-
>>  target-s390x/cpu.h     |  1 +
>>  3 files changed, 50 insertions(+), 10 deletions(-)
>>
>> diff --git a/hw/s390x/s390-virtio.c b/hw/s390x/s390-virtio.c
>> index b3707f4..6a1e3f6 100644
>> --- a/hw/s390x/s390-virtio.c
>> +++ b/hw/s390x/s390-virtio.c
>> @@ -60,15 +60,35 @@
>>  #define S390_TOD_CLOCK_VALUE_MISSING    0x00
>>  #define S390_TOD_CLOCK_VALUE_PRESENT    0x01
>>  
>> -static S390CPU **ipi_states;
>> +static S390CPU **cpu_states;
>>  
>>  S390CPU *s390_cpu_addr2state(uint16_t cpu_addr)
>>  {
>> -    if (cpu_addr >= smp_cpus) {
>> +    if (cpu_addr >= max_cpus) {
>>          return NULL;
>>      }
>>  
>> -    return ipi_states[cpu_addr];
>> +    /* Fast lookup via CPU ID */
>> +    return cpu_states[cpu_addr];
>> +}
>> +
>> +int s390_register_cpustate(uint16_t cpu_addr, S390CPU *state)
> isn't it the board responsibility to link CPU somewhere?
> I'd suggest to use for it generic device hotplug hooks
> see how get_hotplug_handler() and pc_machine_device_plug_cb() are used,
> it's executed after CPU's realize is successfully completed.
> 

OK, I'll have a look at this.

>> +{
>> +    gchar *name;
>> +    int r = 0;
>> +
>> +    name = g_strdup_printf("cpu[%i]", cpu_addr);
>> +    if (object_property_get_link(qdev_get_machine(), name, NULL)) {
>> +        r = -EEXIST;
>> +        goto out;
>> +    }
>> +
>> +    object_property_set_link(qdev_get_machine(), OBJECT(state), name,\
>> +                             &error_abort);
>> +
>> +out:
>> +    g_free(name);
>> +    return r;
>>  }
>>  
>>  void s390_init_ipl_dev(const char *kernel_filename,
>> @@ -98,19 +118,26 @@ void s390_init_ipl_dev(const char *kernel_filename,
>>  void s390_init_cpus(MachineState *machine)
>>  {
>>      int i;
>> +    gchar *name;
>>  
>>      if (machine->cpu_model == NULL) {
>>          machine->cpu_model = "host";
>>      }
>>  
>> -    ipi_states = g_malloc(sizeof(S390CPU *) * smp_cpus);
>> -
>> -    for (i = 0; i < smp_cpus; i++) {
>> -        S390CPU *cpu;
>> +    cpu_states = g_malloc0(sizeof(S390CPU *) * max_cpus);
>>  
>> -        cpu = cpu_s390x_init(machine->cpu_model);
>> +    for (i = 0; i < max_cpus; i++) {
>> +        name = g_strdup_printf("cpu[%i]", i);
>> +        object_property_add_link(qdev_get_machine(), name, TYPE_S390_CPU,
>> +                                 (Object **) &cpu_states[i],
>> +                                 object_property_allow_set_link,
>> +                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
>> +                                 &error_abort);
>> +        g_free(name);
>> +    }
>>  
>> -        ipi_states[i] = cpu;
>> +    for (i = 0; i < smp_cpus; i++) {
>> +        cpu_s390x_init(machine->cpu_model);
>>      }
>>  }
>>  
>> diff --git a/target-s390x/cpu.c b/target-s390x/cpu.c
>> index 5f6411f..620b2e3 100644
>> --- a/target-s390x/cpu.c
>> +++ b/target-s390x/cpu.c
>> @@ -32,6 +32,7 @@
>>  #include "trace.h"
>>  #ifndef CONFIG_USER_ONLY
>>  #include "sysemu/arch_init.h"
>> +#include "sysemu/sysemu.h"
>>  #endif
>>  
>>  #define CR0_RESET       0xE0UL
>> @@ -202,9 +203,20 @@ static void s390_cpu_realizefn(DeviceState *dev, Error **errp)
>>      CPUS390XState *env = &cpu->env;
>>  
>>  #if !defined(CONFIG_USER_ONLY)
>> +    if (s390_register_cpustate(next_cpu_id, cpu) < 0) {
>> +        error_setg(errp, "Cannot have more than %d CPUs", max_cpus);
>> +        return;
>> +    }
>>      qemu_register_reset(s390_cpu_machine_reset_cb, cpu);
>> -#endif
>> +    env->cpu_num = next_cpu_id;
>> +    while (next_cpu_id < max_cpus - 1) {
>> +        if (!cpu_exists(++next_cpu_id)) {
>> +            break;
>> +        }
>> +    }
> maybe it's possible to reuse cpu_get_free_index() here?
> 

Hmm, but based on your other comment, there's no need to track
next_cpu_id in this fashion afterall.  To bring these 2 points together,
how about the following:

1) In Patch 4 (this patch), drop next_cpu_id.  Since we don't allow
unplug and only do sequential cpu adds, rely on env->cpu_num =
cs->cpu_index here in realizefn.

2) For hotplug in patch 5, check against QTAILQ_LAST(&cpus, CPUTailQ) as
you suggest.

I tried this out quick, seems to work fine.

Matt