[Qemu-devel] [PATCH 00/22 v2] target-i386: CPU hot-add with cpu-add QMP command

[Qemu-devel] [PATCH 00/22 v2] target-i386: CPU hot-add with cpu-add QMP command

[Igor Mammedov]

Implements alternative way for hot-adding CPU using cpu-add QMP command,
wich could be useful until it would be possible to add CPUs via device_add.

All patches except the last are also applicable to device_add aprroach.

To hot-add CPU use following command from qmp-shell:
 cpu-add id=[apic id]

git tree for testing: https://github.com/imammedo/qemu/tree/cpu_add.v2

v1->v2:
  * generalize cpu sync to KVM, resume and hot-plug notification and
    invoke them form CPUClass, to make available to all targets.
  * introduce cpu_exists() and CPUClass.get_firmware_id() and use
    the last one in acpi_piix to make code target independent.
  * move IOAPIC to ICC bus, it was suggested and easy to convert.
  * leave kvmvapic as SysBusDevice, it doesn't affect hot-plug and
    created only once for all APIC instances. I haven't found yet
    good/clean enough way to convert it to ICCDevice. May be follow-up
    though.
  * split one big ICC patch into several, one per converted device
  * add cpu_hot_add hook to machine and implement it for target-i386,
    instead of adding stabs. Could be used by other targets to
    implement cpu-add.
  * pre-allocate links<CPU> for all possible CPUs and make them available
    at /machine/icc-bridge/cpu[0..N] QOM path, so users could find out
    possible/free CPU IDs to use in cpu-add command.

CC: pbonzini@redhat.com
CC: afaerber@suse.de
CC: ehabkost@redhat.com

Igor Mammedov (22):
  target-i386: consolidate error propagation in x86_cpu_realizefn()
  target-i386: split APIC creation from initialization in
    x86_cpu_realizefn()
  target-i386: split out CPU creation and features parsing into
    cpu_x86_create()
  cpu: Pass CPUState to *cpu_synchronize_post*()
  cpu: call cpu_synchronize_post_init() from CPUClass.realize() if
    hotplugged
  cpu: introduce CPUClass.resume() method
  target-i386: kvmvapic: replace FROM_SYSBUS() with QOM type cast
  target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast
  introduce CPU hot-plug notifier
  rtc: update rtc_cmos on CPU hot-plug
  cpu: introduce get_firmware_id() method and override it for
    target-i386
  cpu: add helper cpu_exists(), to check if CPU with specified id exists
  acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest
  target-i386: introduce apic-id property
  introduce ICC bus/device/bridge
  target-i386: cpu: attach ICC bus to CPU on its creation
  target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE
  target-i386: move APIC to ICC bus
  target-i386: move IOAPIC to ICC bus
  qdev: set device's parent before calling realize() down inheritance
    chain.
  target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N]
    links
  add cpu-add qmp command and implement CPU hot-add for target-i386

 Makefile.target         |   6 ++
 cpus.c                  |  19 ++++--
 hw/acpi_piix4.c         | 114 ++++++++++++++++++++++++++++++++-
 hw/apic.c               |   2 +-
 hw/apic_common.c        |  17 +++--
 hw/apic_internal.h      |   8 +--
 hw/boards.h             |   3 +
 hw/i386/Makefile.objs   |   2 +-
 hw/i386/kvmvapic.c      |   8 ++-
 hw/i386/pc.c            |  85 +++++++++++++++++++-----
 hw/i386/pc_piix.c       |   9 ++-
 hw/i386/pc_q35.c        |   9 ++-
 hw/icc_bus.c            | 167 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/icc_bus.h            |  55 ++++++++++++++++
 hw/ioapic_common.c      |  17 +++--
 hw/ioapic_internal.h    |   6 +-
 hw/kvm/apic.c           |   2 +-
 hw/kvm/ioapic.c         |   2 +-
 hw/mc146818rtc.c        |  12 ++++
 hw/pc.h                 |   2 +-
 hw/qdev.c               |   8 +--
 hw/xen_apic.c           |   2 +-
 include/exec/memory.h   |  19 ++++++
 include/qom/cpu.h       |  14 ++++
 include/sysemu/kvm.h    |  30 +++++----
 include/sysemu/sysemu.h |   3 +
 kvm-all.c               |   9 +--
 kvm-stub.c              |   9 ++-
 qapi-schema.json        |  11 ++++
 qmp-commands.hx         |  23 +++++++
 qmp.c                   |  10 +++
 qom/cpu.c               |  48 ++++++++++++++
 stubs/do_cpu_hot_add.c  |   7 ++
 target-i386/cpu.c       | 138 ++++++++++++++++++++++++++++++---------
 target-i386/cpu.h       |   2 +
 vl.c                    |   7 +-
 36 files changed, 775 insertions(+), 110 deletions(-)
 create mode 100644 hw/icc_bus.c
 create mode 100644 hw/icc_bus.h
 create mode 100644 stubs/do_cpu_hot_add.c

-- 
1.8.1.4

[Qemu-devel] [PATCH 01/22] target-i386: consolidate error propagation in x86_cpu_realizefn()

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
 target-i386/cpu.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 69c3570..8b348d0 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2099,9 +2099,7 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
     X86CPU *cpu = X86_CPU(dev);
     X86CPUClass *xcc = X86_CPU_GET_CLASS(dev);
     CPUX86State *env = &cpu->env;
-#ifndef CONFIG_USER_ONLY
     Error *local_err = NULL;
-#endif
 
     if (env->cpuid_7_0_ebx_features && env->cpuid_level < 7) {
         env->cpuid_level = 7;
@@ -2131,8 +2129,9 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
     } else {
         if (check_cpuid && kvm_check_features_against_host(cpu)
             && enforce_cpuid) {
-            error_setg(errp, "Host's CPU doesn't support requested features");
-            return;
+            error_setg(&local_err,
+                       "Host's CPU doesn't support requested features");
+            goto out;
         }
 #ifdef CONFIG_KVM
         filter_features_for_kvm(cpu);
@@ -2145,8 +2144,7 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
     if (cpu->env.cpuid_features & CPUID_APIC || smp_cpus > 1) {
         x86_cpu_apic_init(cpu, &local_err);
         if (local_err != NULL) {
-            error_propagate(errp, local_err);
-            return;
+            goto out;
         }
     }
 #endif
@@ -2155,7 +2153,12 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
     qemu_init_vcpu(&cpu->env);
     cpu_reset(CPU(cpu));
 
-    xcc->parent_realize(dev, errp);
+    xcc->parent_realize(dev, &local_err);
+out:
+    if (local_err != NULL) {
+        error_propagate(errp, local_err);
+        return;
+    }
 }
 
 /* Enables contiguous-apic-ID mode, for compatibility */
-- 
1.8.1.4

[Qemu-devel] [PATCH 02/22] target-i386: split APIC creation from initialization in x86_cpu_realizefn()

[Igor Mammedov]

When APIC is hotplugged during CPU hotplug, device_set_realized()
calls device_reset() on it. And if QEMU runs in KVM mode, following
call chain will fail:
    apic_reset_common()
        -> kvm_apic_vapic_base_update()
            -> kvm_vcpu_ioctl(cpu->kvm_fd,...)
due to cpu->kvm_fd not being initialized yet.

cpu->kvm_fd is initialized during qemu_init_vcpu() call but x86_cpu_apic_init()
can't be moved after it because kvm_init_vcpu() -> kvm_arch_reset_vcpu()
relies on APIC to determine if CPU is BSP for setting initial env->mp_state.

So split APIC device creation from its initialization and realize APIC
after CPU is created, when it's safe to call APIC's reset method.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * s/x86_cpu_apic_init()/x86_cpu_apic_realize()/
---
 target-i386/cpu.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 8b348d0..41f0f47 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2050,9 +2050,8 @@ static void mce_init(X86CPU *cpu)
 }
 
 #ifndef CONFIG_USER_ONLY
-static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
+static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 {
-    static int apic_mapped;
     CPUX86State *env = &cpu->env;
     APICCommonState *apic;
     const char *apic_type = "apic";
@@ -2075,6 +2074,16 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
     /* TODO: convert to link<> */
     apic = APIC_COMMON(env->apic_state);
     apic->cpu = cpu;
+}
+
+static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
+{
+    CPUX86State *env = &cpu->env;
+    static int apic_mapped;
+
+    if (env->apic_state == NULL) {
+        return;
+    }
 
     if (qdev_init(env->apic_state)) {
         error_setg(errp, "APIC device '%s' could not be initialized",
@@ -2092,6 +2101,10 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
         apic_mapped = 1;
     }
 }
+#else
+static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
+{
+}
 #endif
 
 static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
@@ -2142,7 +2155,7 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
     qemu_register_reset(x86_cpu_machine_reset_cb, cpu);
 
     if (cpu->env.cpuid_features & CPUID_APIC || smp_cpus > 1) {
-        x86_cpu_apic_init(cpu, &local_err);
+        x86_cpu_apic_create(cpu, &local_err);
         if (local_err != NULL) {
             goto out;
         }
@@ -2151,6 +2164,11 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
 
     mce_init(cpu);
     qemu_init_vcpu(&cpu->env);
+
+    x86_cpu_apic_realize(cpu, &local_err);
+    if (local_err != NULL) {
+        goto out;
+    }
     cpu_reset(CPU(cpu));
 
     xcc->parent_realize(dev, &local_err);
-- 
1.8.1.4

[Qemu-devel] [PATCH 03/22] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Igor Mammedov]

Move CPU creation and features parsing into a separate cpu_x86_create()
function, so that board would be able to set board specific CPU
properties before CPU is realized.

Keep cpu_x86_init() for compatibility with the code that uses cpu_init()
and doesn't need to modify CPU properties.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 target-i386/cpu.c | 28 +++++++++++++++++++---------
 target-i386/cpu.h |  1 +
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 41f0f47..269a681 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1563,17 +1563,16 @@ static void cpu_x86_register(X86CPU *cpu, const char *name, Error **errp)
     object_property_set_str(OBJECT(cpu), def->model_id, "model-id", errp);
 }
 
-X86CPU *cpu_x86_init(const char *cpu_model)
+X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
 {
     X86CPU *cpu = NULL;
     CPUX86State *env;
     gchar **model_pieces;
     char *name, *features;
-    Error *error = NULL;
 
     model_pieces = g_strsplit(cpu_model, ",", 2);
     if (!model_pieces[0]) {
-        error_setg(&error, "Invalid/empty CPU model name");
+        error_setg(errp, "Invalid/empty CPU model name");
         goto out;
     }
     name = model_pieces[0];
@@ -1583,23 +1582,34 @@ X86CPU *cpu_x86_init(const char *cpu_model)
     env = &cpu->env;
     env->cpu_model_str = cpu_model;
 
-    cpu_x86_register(cpu, name, &error);
-    if (error) {
+    cpu_x86_register(cpu, name, errp);
+    if (error_is_set(errp)) {
         goto out;
     }
 
-    cpu_x86_parse_featurestr(cpu, features, &error);
-    if (error) {
+    cpu_x86_parse_featurestr(cpu, features, errp);
+    if (error_is_set(errp)) {
         goto out;
     }
 
-    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
+out:
+    g_strfreev(model_pieces);
+    return cpu;
+}
+
+X86CPU *cpu_x86_init(const char *cpu_model)
+{
+    Error *error = NULL;
+    X86CPU *cpu;
+
+    cpu = cpu_x86_create(cpu_model, &error);
     if (error) {
         goto out;
     }
 
+    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
+
 out:
-    g_strfreev(model_pieces);
     if (error) {
         fprintf(stderr, "%s\n", error_get_pretty(error));
         error_free(error);
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index 069a2e2..b98efd2 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -896,6 +896,7 @@ typedef struct CPUX86State {
 #include "cpu-qom.h"
 
 X86CPU *cpu_x86_init(const char *cpu_model);
+X86CPU *cpu_x86_create(const char *cpu_model, Error **errp);
 int cpu_x86_exec(CPUX86State *s);
 void x86_cpu_list(FILE *f, fprintf_function cpu_fprintf);
 void x86_cpudef_setup(void);
-- 
1.8.1.4

[Qemu-devel] [PATCH 04/22] cpu: Pass CPUState to *cpu_synchronize_post*()

[Igor Mammedov]

... so it could be called from without requiring CPUArchState

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 cpus.c               |  4 ++--
 include/sysemu/kvm.h | 12 ++++++------
 kvm-all.c            |  8 ++------
 kvm-stub.c           |  4 ++--
 4 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/cpus.c b/cpus.c
index e919dd7..9b9a32f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -419,7 +419,7 @@ void cpu_synchronize_all_post_reset(void)
     CPUArchState *cpu;
 
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
-        cpu_synchronize_post_reset(cpu);
+        cpu_synchronize_post_reset(ENV_GET_CPU(cpu));
     }
 }
 
@@ -428,7 +428,7 @@ void cpu_synchronize_all_post_init(void)
     CPUArchState *cpu;
 
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
-        cpu_synchronize_post_init(cpu);
+        cpu_synchronize_post_init(ENV_GET_CPU(cpu));
     }
 }
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index f2d97b5..495e6f8 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -250,8 +250,8 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
 uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
                                       uint32_t index, int reg);
 void kvm_cpu_synchronize_state(CPUArchState *env);
-void kvm_cpu_synchronize_post_reset(CPUArchState *env);
-void kvm_cpu_synchronize_post_init(CPUArchState *env);
+void kvm_cpu_synchronize_post_reset(CPUState *cpu);
+void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -262,17 +262,17 @@ static inline void cpu_synchronize_state(CPUArchState *env)
     }
 }
 
-static inline void cpu_synchronize_post_reset(CPUArchState *env)
+static inline void cpu_synchronize_post_reset(CPUState *cpu)
 {
     if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_reset(env);
+        kvm_cpu_synchronize_post_reset(cpu);
     }
 }
 
-static inline void cpu_synchronize_post_init(CPUArchState *env)
+static inline void cpu_synchronize_post_init(CPUState *cpu)
 {
     if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_init(env);
+        kvm_cpu_synchronize_post_init(cpu);
     }
 }
 
diff --git a/kvm-all.c b/kvm-all.c
index 9b433d3..fc4e17c 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1510,18 +1510,14 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
     }
 }
 
-void kvm_cpu_synchronize_post_reset(CPUArchState *env)
+void kvm_cpu_synchronize_post_reset(CPUState *cpu)
 {
-    CPUState *cpu = ENV_GET_CPU(env);
-
     kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE);
     cpu->kvm_vcpu_dirty = false;
 }
 
-void kvm_cpu_synchronize_post_init(CPUArchState *env)
+void kvm_cpu_synchronize_post_init(CPUState *cpu)
 {
-    CPUState *cpu = ENV_GET_CPU(env);
-
     kvm_arch_put_registers(cpu, KVM_PUT_FULL_STATE);
     cpu->kvm_vcpu_dirty = false;
 }
diff --git a/kvm-stub.c b/kvm-stub.c
index 760aadc..82875dd 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -42,11 +42,11 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
 {
 }
 
-void kvm_cpu_synchronize_post_reset(CPUArchState *env)
+void kvm_cpu_synchronize_post_reset(CPUState *env)
 {
 }
 
-void kvm_cpu_synchronize_post_init(CPUArchState *env)
+void kvm_cpu_synchronize_post_init(CPUState *cpu)
 {
 }
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 05/22] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Igor Mammedov]

... to synchronize CPU state to KVM

* in addition link kvm-stub.o to *-user target and fix related compiling issues.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 Makefile.target      |  6 ++++++
 include/sysemu/kvm.h | 22 ++++++++++++----------
 kvm-all.c            |  1 +
 kvm-stub.c           |  5 +++++
 qom/cpu.c            |  4 ++++
 vl.c                 |  1 -
 6 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/Makefile.target b/Makefile.target
index 2bd6d14..0c2c24a 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -76,6 +76,12 @@ obj-y += disas.o
 obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
 
 #########################################################
+# user emulator target
+ifdef CONFIG_USER_ONLY
+obj-y += kvm-stub.o
+endif #CONFIG_USER_ONLY
+
+#########################################################
 # Linux user emulator target
 
 ifdef CONFIG_LINUX_USER
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 495e6f8..8534e44 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
 #if !defined(CONFIG_USER_ONLY)
 void *kvm_vmalloc(ram_addr_t size);
 void *kvm_arch_vmalloc(ram_addr_t size);
-void kvm_setup_guest_memory(void *start, size_t size);
+#endif
 
+void kvm_setup_guest_memory(void *start, size_t size);
 void kvm_flush_coalesced_mmio_buffer(void);
-#endif
 
 int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
                           target_ulong len, int type);
@@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
 uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
                                       uint32_t index, int reg);
 void kvm_cpu_synchronize_state(CPUArchState *env);
-void kvm_cpu_synchronize_post_reset(CPUState *cpu);
-void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
     }
 }
 
+#if !defined(CONFIG_USER_ONLY)
+int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
+                                       hwaddr *phys_addr);
+#endif
+
+#endif
+
+void kvm_cpu_synchronize_post_reset(CPUState *cpu);
+void kvm_cpu_synchronize_post_init(CPUState *cpu);
+
 static inline void cpu_synchronize_post_reset(CPUState *cpu)
 {
     if (kvm_enabled()) {
@@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 }
 
 
-#if !defined(CONFIG_USER_ONLY)
-int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
-                                       hwaddr *phys_addr);
-#endif
-
-#endif
 int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
                            uint32_t size);
 
diff --git a/kvm-all.c b/kvm-all.c
index fc4e17c..1d17128 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
 bool kvm_irqfds_allowed;
 bool kvm_msi_via_irqfd_allowed;
 bool kvm_gsi_routing_allowed;
+bool kvm_allowed;
 
 static const KVMCapabilityInfo kvm_required_capabilites[] = {
     KVM_CAP_INFO(USER_MEMORY),
diff --git a/kvm-stub.c b/kvm-stub.c
index 82875dd..f3e9438 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -12,7 +12,9 @@
 
 #include "qemu-common.h"
 #include "hw/hw.h"
+#ifndef CONFIG_USER_ONLY
 #include "hw/pci/msi.h"
+#endif
 #include "cpu.h"
 #include "exec/gdbstub.h"
 #include "sysemu/kvm.h"
@@ -23,6 +25,7 @@ bool kvm_async_interrupts_allowed;
 bool kvm_irqfds_allowed;
 bool kvm_msi_via_irqfd_allowed;
 bool kvm_gsi_routing_allowed;
+bool kvm_allowed;
 
 int kvm_init_vcpu(CPUState *cpu)
 {
@@ -122,6 +125,7 @@ int kvm_on_sigbus(int code, void *addr)
     return 1;
 }
 
+#ifndef CONFIG_USER_ONLY
 int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
 {
     return -ENOSYS;
@@ -145,3 +149,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
 {
     return -ENOSYS;
 }
+#endif
diff --git a/qom/cpu.c b/qom/cpu.c
index e242dcb..0c76712 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -20,6 +20,7 @@
 
 #include "qom/cpu.h"
 #include "qemu-common.h"
+#include "sysemu/kvm.h"
 
 void cpu_reset_interrupt(CPUState *cpu, int mask)
 {
@@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
 
 static void cpu_common_realizefn(DeviceState *dev, Error **errp)
 {
+    if (dev->hotplugged) {
+        cpu_synchronize_post_init(CPU(dev));
+    }
 }
 
 static void cpu_class_init(ObjectClass *klass, void *data)
diff --git a/vl.c b/vl.c
index a8bba04..97f0349 100644
--- a/vl.c
+++ b/vl.c
@@ -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
     NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
 
 static bool tcg_allowed = true;
-bool kvm_allowed;
 bool xen_allowed;
 uint32_t xen_domid;
 enum xen_mode xen_mode = XEN_EMULATE;
-- 
1.8.1.4

[Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Igor Mammedov]

... and call it if defined from CPUClass.realize() if CPU was hotplugged

by default leave .resume() unset (i.e. NULL) and override it for softmmu
in qemu_init_vcpu() if it's still unset.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 cpus.c            | 15 ++++++++++++---
 include/qom/cpu.h |  2 ++
 qom/cpu.c         |  5 +++++
 3 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/cpus.c b/cpus.c
index 9b9a32f..6b793c5 100644
--- a/cpus.c
+++ b/cpus.c
@@ -973,6 +973,13 @@ void pause_all_vcpus(void)
     }
 }
 
+static void resume_vcpu(CPUState *cpu)
+{
+    cpu->stop = false;
+    cpu->stopped = false;
+    qemu_cpu_kick(cpu);
+}
+
 void resume_all_vcpus(void)
 {
     CPUArchState *penv = first_cpu;
@@ -980,9 +987,7 @@ void resume_all_vcpus(void)
     qemu_clock_enable(vm_clock, true);
     while (penv) {
         CPUState *pcpu = ENV_GET_CPU(penv);
-        pcpu->stop = false;
-        pcpu->stopped = false;
-        qemu_cpu_kick(pcpu);
+        resume_vcpu(pcpu);
         penv = penv->next_cpu;
     }
 }
@@ -1042,7 +1047,11 @@ void qemu_init_vcpu(void *_env)
 {
     CPUArchState *env = _env;
     CPUState *cpu = ENV_GET_CPU(env);
+    CPUClass *klass = CPU_GET_CLASS(cpu);
 
+    if (klass->resume == NULL) {
+        klass->resume = resume_vcpu;
+    }
     cpu->nr_cores = smp_cores;
     cpu->nr_threads = smp_threads;
     cpu->stopped = true;
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 3664a1b..6d6eb7a 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
  * instantiatable CPU type.
  * @reset: Callback to reset the #CPUState to its initial state.
  * @do_interrupt: Callback for interrupt handling.
+ * @resume: Callback for putting CPU in runable state
  * @vmsd: State description for migration.
  *
  * Represents a CPU family or model.
@@ -58,6 +59,7 @@ typedef struct CPUClass {
 
     void (*reset)(CPUState *cpu);
     void (*do_interrupt)(CPUState *cpu);
+    void (*resume)(CPUState *cpu);
 
     const struct VMStateDescription *vmsd;
 } CPUClass;
diff --git a/qom/cpu.c b/qom/cpu.c
index 0c76712..c062e00 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -58,8 +58,13 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
 
 static void cpu_common_realizefn(DeviceState *dev, Error **errp)
 {
+    CPUClass *klass = CPU_GET_CLASS(dev);
+
     if (dev->hotplugged) {
         cpu_synchronize_post_init(CPU(dev));
+        if (klass->resume != NULL) {
+            klass->resume(CPU(dev));
+        }
     }
 }
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 07/22] target-i386: kvmvapic: replace FROM_SYSBUS() with QOM type cast

[Igor Mammedov]

... and define type name and type cast macro for kvmvapic according
to accepted convention.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * s/VAPIC_DEVICE/VAPIC/; s/TYPE_VAPIC_DEVICE/TYPE_VAPIC/

Note: stray cleanup, since I excluded following patch that converted
kvmvapic to ICCDevice
---
 hw/i386/kvmvapic.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
index cc95e5c..c4be882 100644
--- a/hw/i386/kvmvapic.c
+++ b/hw/i386/kvmvapic.c
@@ -60,6 +60,9 @@ typedef struct VAPICROMState {
     bool rom_mapped_writable;
 } VAPICROMState;
 
+#define TYPE_VAPIC "kvmvapic"
+#define VAPIC(obj) OBJECT_CHECK(VAPICROMState, (obj), TYPE_VAPIC)
+
 #define TPR_INSTR_ABS_MODRM             0x1
 #define TPR_INSTR_MATCH_MODRM_REG       0x2
 
@@ -690,7 +693,7 @@ static const MemoryRegionOps vapic_ops = {
 
 static int vapic_init(SysBusDevice *dev)
 {
-    VAPICROMState *s = FROM_SYSBUS(VAPICROMState, dev);
+    VAPICROMState *s = VAPIC(dev);
 
     memory_region_init_io(&s->io, &vapic_ops, s, "kvmvapic", 2);
     sysbus_add_io(dev, VAPIC_IO_PORT, &s->io);
@@ -806,7 +809,7 @@ static void vapic_class_init(ObjectClass *klass, void *data)
 }
 
 static const TypeInfo vapic_type = {
-    .name          = "kvmvapic",
+    .name          = TYPE_VAPIC,
     .parent        = TYPE_SYS_BUS_DEVICE,
     .instance_size = sizeof(VAPICROMState),
     .class_init    = vapic_class_init,
-- 
1.8.1.4

[Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/ioapic_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
index d4aff29..561b987 100644
--- a/hw/ioapic_common.c
+++ b/hw/ioapic_common.c
@@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
 
 static int ioapic_init_common(SysBusDevice *dev)
 {
-    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
+    IOAPICCommonState *s = IOAPIC_COMMON(dev);
     IOAPICCommonClass *info;
     static int ioapic_no;
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 09/22] introduce CPU hot-plug notifier

[Igor Mammedov]

hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
---
 include/qom/cpu.h       |  2 ++
 include/sysemu/sysemu.h |  3 +++
 qom/cpu.c               | 12 ++++++++++++
 3 files changed, 17 insertions(+)

diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 6d6eb7a..210aca3 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -46,6 +46,7 @@ typedef struct CPUState CPUState;
  * @reset: Callback to reset the #CPUState to its initial state.
  * @do_interrupt: Callback for interrupt handling.
  * @resume: Callback for putting CPU in runable state
+ * @get_firmware_id: Callback for getting arch depended CPU id
  * @vmsd: State description for migration.
  *
  * Represents a CPU family or model.
@@ -60,6 +61,7 @@ typedef struct CPUClass {
     void (*reset)(CPUState *cpu);
     void (*do_interrupt)(CPUState *cpu);
     void (*resume)(CPUState *cpu);
+    void (*get_firmware_id)(CPUState *cpu);
 
     const struct VMStateDescription *vmsd;
 } CPUClass;
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 6578782..a8c3de1 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
 /* generic hotplug */
 void drive_hot_add(Monitor *mon, const QDict *qdict);
 
+/* CPU hotplug */
+void qemu_register_cpu_added_notifier(Notifier *notifier);
+
 /* pcie aer error injection */
 void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
 int do_pcie_aer_inject_error(Monitor *mon,
diff --git a/qom/cpu.c b/qom/cpu.c
index c062e00..10ceaed 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -21,6 +21,17 @@
 #include "qom/cpu.h"
 #include "qemu-common.h"
 #include "sysemu/kvm.h"
+#include "qemu/notify.h"
+#include "sysemu/sysemu.h"
+
+/* CPU hot-plug notifiers */
+static NotifierList cpu_added_notifiers =
+    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
+
+void qemu_register_cpu_added_notifier(Notifier *notifier)
+{
+    notifier_list_add(&cpu_added_notifiers, notifier);
+}
 
 void cpu_reset_interrupt(CPUState *cpu, int mask)
 {
@@ -65,6 +76,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
         if (klass->resume != NULL) {
             klass->resume(CPU(dev));
         }
+        notifier_list_notify(&cpu_added_notifiers, dev);
     }
 }
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 10/22] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

... so that on reboot BIOS could read current available CPU count

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
v2:
  * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
---
 hw/mc146818rtc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/hw/mc146818rtc.c b/hw/mc146818rtc.c
index a2119ad..c3a6a2d 100644
--- a/hw/mc146818rtc.c
+++ b/hw/mc146818rtc.c
@@ -82,6 +82,7 @@ typedef struct RTCState {
     Notifier clock_reset_notifier;
     LostTickPolicy lost_tick_policy;
     Notifier suspend_notifier;
+    Notifier cpu_added_notifier;
 } RTCState;
 
 static void rtc_set_time(RTCState *s);
@@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
     rtc_set_memory(&s->dev, 0xF, 0xFE);
 }
 
+static void rtc_notify_cpu_added(Notifier *notifier, void *data)
+{
+    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
+
+    /* increment the number of CPUs */
+    s->cmos_data[0x5f] += 1;
+}
+
 static void rtc_reset(void *opaque)
 {
     RTCState *s = opaque;
@@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
     s->suspend_notifier.notify = rtc_notify_suspend;
     qemu_register_suspend_notifier(&s->suspend_notifier);
 
+    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
+    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
+
     memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
     isa_register_ioport(dev, &s->io, base);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 11/22] cpu: introduce get_firmware_id() method and override it for target-i386

[Igor Mammedov]

get_firmware_id() adds possibily for generic code to get guest visible
CPI id without accessing CPUArchState. If target doesn't override it,
it will return cpu_index.

Override it on target-i386 to return APIC ID.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 * it will be used later by new cpu_exists() generic function and
   acpi_piix.
---
 include/qom/cpu.h |  4 ++--
 qom/cpu.c         |  6 ++++++
 target-i386/cpu.c | 10 ++++++++++
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 210aca3..0d33009 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -46,7 +46,7 @@ typedef struct CPUState CPUState;
  * @reset: Callback to reset the #CPUState to its initial state.
  * @do_interrupt: Callback for interrupt handling.
  * @resume: Callback for putting CPU in runable state
- * @get_firmware_id: Callback for getting arch depended CPU id
+ * @get_firmware_id: Callback for getting arch depended CPU ID
  * @vmsd: State description for migration.
  *
  * Represents a CPU family or model.
@@ -61,7 +61,7 @@ typedef struct CPUClass {
     void (*reset)(CPUState *cpu);
     void (*do_interrupt)(CPUState *cpu);
     void (*resume)(CPUState *cpu);
-    void (*get_firmware_id)(CPUState *cpu);
+    int64_t (*get_firmware_id)(CPUState *cpu);
 
     const struct VMStateDescription *vmsd;
 } CPUClass;
diff --git a/qom/cpu.c b/qom/cpu.c
index 10ceaed..a54d4d1 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -80,6 +80,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
     }
 }
 
+static int64_t cpu_common_get_firmware_id(CPUState *cpu)
+{
+    return cpu->cpu_index;
+}
+
 static void cpu_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -87,6 +92,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
 
     k->class_by_name = cpu_common_class_by_name;
     k->reset = cpu_common_reset;
+    k->get_firmware_id = cpu_common_get_firmware_id;
     dc->realize = cpu_common_realizefn;
     dc->no_user = 1;
 }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 269a681..858fd54 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2269,6 +2269,14 @@ static void x86_cpu_initfn(Object *obj)
     }
 }
 
+static int64_t x86_cpu_get_firmware_id(CPUState *cpu)
+{
+    X86CPU *x86cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86cpu->env;
+
+    return env->cpuid_apic_id;
+}
+
 static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 {
     X86CPUClass *xcc = X86_CPU_CLASS(oc);
@@ -2283,6 +2291,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 
     cc->do_interrupt = x86_cpu_do_interrupt;
     cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
+
+    cc->get_firmware_id = x86_cpu_get_firmware_id;
 }
 
 static const TypeInfo x86_cpu_type_info = {
-- 
1.8.1.4

[Qemu-devel] [PATCH 12/22] cpu: add helper cpu_exists(), to check if CPU with specified id exists

[Igor Mammedov]

... it should be used only on slow path since it does recursive search
    on /machine QOM tree for objects of TYPE_CPU

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 include/qom/cpu.h | 10 ++++++++++
 qom/cpu.c         | 21 +++++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 0d33009..5cac79b 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -227,6 +227,16 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
  */
 CPUState *qemu_get_cpu(int index);
 
+/**
+ * cpu_exists:
+ * @id - guest exposed CPU ID to lookup
+ *
+ * Search for CPU with specified ID.
+ *
+ * Returns: true - CPU is found, false - CPU isn't found
+ */
+bool cpu_exists(int64_t id);
+
 #ifndef CONFIG_USER_ONLY
 
 typedef void (*CPUInterruptHandler)(CPUState *, int);
diff --git a/qom/cpu.c b/qom/cpu.c
index a54d4d1..46b77d3 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -24,6 +24,27 @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 
+static int cpu_exist_cb(Object *obj, void *opaque)
+{
+    int64_t id = *(int64_t *)opaque;
+    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
+
+    if (cpu_obj) {
+        CPUState *cpu = CPU(cpu_obj);
+        CPUClass *klass = CPU_GET_CLASS(cpu);
+
+        if (klass->get_firmware_id && klass->get_firmware_id(cpu) == id) {
+            return 1;
+        }
+    }
+    return object_child_foreach(obj, cpu_exist_cb, opaque);
+}
+
+bool cpu_exists(int64_t id)
+{
+   return cpu_exist_cb(qdev_get_machine(), &id) ? true : false;
+}
+
 /* CPU hot-plug notifiers */
 static NotifierList cpu_added_notifiers =
     NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
-- 
1.8.1.4

[Qemu-devel] [PATCH 13/22] acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest

[Igor Mammedov]

* introduce processor status bitmask visible to guest at 0xaf00 addr,
  where Seabios expects it
* set bit corresponding to APIC ID in processor status bitmask on
  receiving CPU hot-plug notification
* trigger CPU hot-plug SCI, expected by Seabios on receiving CPU
  hot-plug notification

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * use CPUClass.get_firmware_id() to make code target independent
  * bump up vmstate_acpi version
---
 hw/acpi_piix4.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 111 insertions(+), 3 deletions(-)

diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
index 48a32b5..ccfc028 100644
--- a/hw/acpi_piix4.c
+++ b/hw/acpi_piix4.c
@@ -48,19 +48,28 @@
 #define PCI_EJ_BASE 0xae08
 #define PCI_RMV_BASE 0xae0c
 
+#define PROC_BASE 0xaf00
+#define PROC_LEN 32
+
 #define PIIX4_PCI_HOTPLUG_STATUS 2
+#define PIIX4_CPU_HOTPLUG_STATUS 4
 
 struct pci_status {
     uint32_t up; /* deprecated, maintained for migration compatibility */
     uint32_t down;
 };
 
+struct cpu_status {
+    uint8_t sts[PROC_LEN];
+};
+
 typedef struct PIIX4PMState {
     PCIDevice dev;
 
     MemoryRegion io;
     MemoryRegion io_gpe;
     MemoryRegion io_pci;
+    MemoryRegion io_cpu;
     ACPIREGS ar;
 
     APMState apm;
@@ -82,6 +91,9 @@ typedef struct PIIX4PMState {
     uint8_t disable_s3;
     uint8_t disable_s4;
     uint8_t s4_val;
+
+    struct cpu_status gpe_cpu;
+    Notifier cpu_add_notifier;
 } PIIX4PMState;
 
 static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
@@ -100,8 +112,8 @@ static void pm_update_sci(PIIX4PMState *s)
                    ACPI_BITMASK_POWER_BUTTON_ENABLE |
                    ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
                    ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
-        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0])
-          & PIIX4_PCI_HOTPLUG_STATUS) != 0);
+        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0]) &
+          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
 
     qemu_set_irq(s->irq, sci_level);
     /* schedule a timer interruption if needed */
@@ -257,6 +269,17 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+#define VMSTATE_CPU_STATUS_ARRAY(_field, _state)                             \
+ {                                                                           \
+     .name       = (stringify(_field)),                                      \
+     .version_id = 0,                                                        \
+     .num        = PROC_LEN,                                                 \
+     .info       = &vmstate_info_uint8,                                      \
+     .size       = sizeof(uint8_t),                                          \
+     .flags      = VMS_ARRAY,                                                \
+     .offset     = vmstate_offset_array(_state, _field, uint8_t, PROC_LEN),  \
+ }
+
 /* qemu-kvm 1.2 uses version 3 but advertised as 2
  * To support incoming qemu-kvm 1.2 migration, change version_id
  * and minimum_version_id to 2 below (which breaks migration from
@@ -265,7 +288,7 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
  */
 static const VMStateDescription vmstate_acpi = {
     .name = "piix4_pm",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .minimum_version_id_old = 1,
     .load_state_old = acpi_load_old,
@@ -281,6 +304,7 @@ static const VMStateDescription vmstate_acpi = {
         VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
         VMSTATE_STRUCT(pci0_status, PIIX4PMState, 2, vmstate_pci_status,
                        struct pci_status),
+        VMSTATE_CPU_STATUS_ARRAY(gpe_cpu.sts, PIIX4PMState),
         VMSTATE_END_OF_LIST()
     }
 };
@@ -585,6 +609,83 @@ static const MemoryRegionOps piix4_pci_ops = {
     },
 };
 
+static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned width)
+{
+    PIIX4PMState *s = opaque;
+    struct cpu_status *cpus = &s->gpe_cpu;
+    uint64_t val = cpus->sts[addr];
+    return val;
+}
+
+static void cpu_status_write(void *opaque, hwaddr addr, uint64_t data,
+                             unsigned int size)
+{
+    /* TODO: implement VCPU removal on guest signal that CPU can be removed */
+}
+
+static const MemoryRegionOps cpu_hotplug_ops = {
+    .read = cpu_status_read,
+    .write = cpu_status_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 1,
+        .max_access_size = 1,
+    },
+};
+
+typedef enum {
+    PLUG,
+    UNPLUG,
+} HotplugEventType;
+
+static void piix4_cpu_hotplug_req(PIIX4PMState *s, CPUState *cpu,
+                                  HotplugEventType action)
+{
+    struct cpu_status *g = &s->gpe_cpu;
+    ACPIGPE *gpe = &s->ar.gpe;
+    CPUClass *k = CPU_GET_CLASS(cpu);
+    int64_t cpu_id;
+
+    assert(s != NULL);
+
+    *gpe->sts = *gpe->sts | PIIX4_CPU_HOTPLUG_STATUS;
+    cpu_id = k->get_firmware_id(CPU(cpu));
+    if (action == PLUG) {
+        g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
+    } else {
+        g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
+    }
+    pm_update_sci(s);
+}
+
+static void piix4_cpu_add_req(Notifier *n, void *opaque)
+{
+    PIIX4PMState *s = container_of(n, PIIX4PMState, cpu_add_notifier);
+    piix4_cpu_hotplug_req(s, CPU(opaque), PLUG);
+}
+
+static int piix4_init_cpu_status(Object *obj, void *opaque)
+{
+    struct cpu_status *g = (struct cpu_status *)opaque;
+    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
+
+    if (cpu_obj) {
+        struct Error *error = NULL;
+        CPUClass *k = CPU_GET_CLASS(cpu_obj);
+        int64_t id = k->get_firmware_id(CPU(cpu_obj));
+
+        if (error) {
+            fprintf(stderr, "failed to initilize CPU status for ACPI: %s\n",
+                    error_get_pretty(error));
+            error_free(error);
+            abort();
+        }
+        g_assert((id / 8) < PROC_LEN);
+        g->sts[id / 8] |= (1 << (id % 8));
+    }
+    return object_child_foreach(obj, piix4_init_cpu_status, opaque);
+}
+
 static int piix4_device_hotplug(DeviceState *qdev, PCIDevice *dev,
                                 PCIHotplugState state);
 
@@ -600,6 +701,13 @@ static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
     memory_region_add_subregion(parent, PCI_HOTPLUG_ADDR,
                                 &s->io_pci);
     pci_bus_hotplug(bus, piix4_device_hotplug, &s->dev.qdev);
+
+    piix4_init_cpu_status(qdev_get_machine(), &s->gpe_cpu);
+    memory_region_init_io(&s->io_cpu, &cpu_hotplug_ops, s, "apci-cpu-hotplug",
+                          PROC_LEN);
+    memory_region_add_subregion(parent, PROC_BASE, &s->io_cpu);
+    s->cpu_add_notifier.notify = piix4_cpu_add_req;
+    qemu_register_cpu_added_notifier(&s->cpu_add_notifier);
 }
 
 static void enable_device(PIIX4PMState *s, int slot)
-- 
1.8.1.4

[Qemu-devel] [PATCH 14/22] target-i386: introduce apic-id property

[Igor Mammedov]

... and use it from board level to set APIC ID for CPUs
it creates.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
Note:
  * pc_new_cpu() function will be reused later in CPU hot-plug hook.

v2:
  * use generic cpu_exists() instead of custom one
  * make apic-id dynamic property, so it won't be possible to use it
    as global property, since it's instance specific
---
 hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
 target-i386/cpu.c | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index ebbf059..9d617b4 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
+static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+{
+    X86CPU *cpu;
+
+    cpu = cpu_x86_create(cpu_model, errp);
+    if (!cpu) {
+        return;
+    }
+
+    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
+    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
+
+    if (error_is_set(errp)) {
+        if (cpu != NULL) {
+            object_unref(OBJECT(cpu));
+        }
+    }
+}
+
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    Error *error = NULL;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
     }
 
     for (i = 0; i < smp_cpus; i++) {
-        if (!cpu_x86_init(cpu_model)) {
+        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        if (error) {
+            fprintf(stderr, "%s\n", error_get_pretty(error));
+            error_free(error);
             exit(1);
         }
     }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 858fd54..db56b52 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1272,6 +1272,41 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
     cpu->env.tsc_khz = value / 1000;
 }
 
+static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
+                                  const char *name, Error **errp)
+{
+    X86CPU *cpu = X86_CPU(obj);
+    int64_t value = cpu->env.cpuid_apic_id;
+
+    visit_type_int(v, &value, name, errp);
+}
+
+static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
+                                  const char *name, Error **errp)
+{
+    X86CPU *cpu = X86_CPU(obj);
+    const int64_t min = 0;
+    const int64_t max = UINT32_MAX;
+    int64_t value;
+
+    visit_type_int(v, &value, name, errp);
+    if (error_is_set(errp)) {
+        return;
+    }
+    if (value < min || value > max) {
+        error_setg(errp, "Property %s.%s doesn't take value %" PRId64
+                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
+                   object_get_typename(obj), name, value, min, max);
+        return;
+    }
+
+    if (cpu_exists(value)) {
+        error_setg(errp, "CPU with APIC ID %" PRIi64 " exists", value);
+        return;
+    }
+    cpu->env.cpuid_apic_id = value;
+}
+
 static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
 {
     x86_def_t *def;
@@ -2256,6 +2291,9 @@ static void x86_cpu_initfn(Object *obj)
     object_property_add(obj, "tsc-frequency", "int",
                         x86_cpuid_get_tsc_freq,
                         x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
+    object_property_add(obj, "apic-id", "int",
+                        x86_cpuid_get_apic_id,
+                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
 
     env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
 
-- 
1.8.1.4

[Qemu-devel] [PATCH 15/22] introduce ICC bus/device/bridge

[Igor Mammedov]

... to provide hotplug-able bus.

* icc-bridge will serve as a parent for icc-bus, provide
  mmio mapping services to child icc-device and create
  IOAPIC when requested.
* icc-device will replace SysBusDevice as a parent of APIC
  and IOAPIC devices.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/Makefile.objs |  2 +-
 hw/i386/pc_piix.c     |  7 +++++
 hw/i386/pc_q35.c      |  7 +++++
 hw/icc_bus.c          | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++
 hw/icc_bus.h          | 51 +++++++++++++++++++++++++++++++
 5 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 hw/icc_bus.c
 create mode 100644 hw/icc_bus.h

diff --git a/hw/i386/Makefile.objs b/hw/i386/Makefile.objs
index a78c0b2..316f999 100644
--- a/hw/i386/Makefile.objs
+++ b/hw/i386/Makefile.objs
@@ -1,5 +1,5 @@
 obj-y += mc146818rtc.o
-obj-y += apic_common.o apic.o
+obj-y += apic_common.o apic.o icc_bus.o
 obj-y += sga.o ioapic_common.o ioapic.o piix_pci.o
 obj-y += vmport.o
 obj-y += pci/pci-hotplug.o wdt_ib700.o
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 0abc9f1..8bf5440 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -37,6 +37,7 @@
 #include "hw/kvm/clock.h"
 #include "sysemu/sysemu.h"
 #include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/blockdev.h"
 #include "hw/smbus.h"
@@ -84,8 +85,13 @@ static void pc_init1(MemoryRegion *system_memory,
     MemoryRegion *ram_memory;
     MemoryRegion *pci_memory;
     MemoryRegion *rom_memory;
+    DeviceState *icc_bridge;
     void *fw_cfg = NULL;
 
+    icc_bridge = qdev_create(NULL, TYPE_ICC_BRIDGE);
+    object_property_add_child(qdev_get_machine(), "icc-bridge",
+                              OBJECT(icc_bridge), NULL);
+
     pc_cpus_init(cpu_model);
     pc_acpi_init("acpi-dsdt.aml");
 
@@ -160,6 +166,7 @@ static void pc_init1(MemoryRegion *system_memory,
     if (pci_enabled) {
         ioapic_init_gsi(gsi_state, "i440fx");
     }
+    qdev_init_nofail(icc_bridge);
 
     pc_register_ferr_irq(gsi[13]);
 
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 4f5f347..fb701b4 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -41,6 +41,7 @@
 #include "hw/ide/pci.h"
 #include "hw/ide/ahci.h"
 #include "hw/usb.h"
+#include "hw/icc_bus.h"
 
 /* ICH9 AHCI has 6 ports */
 #define MAX_SATA_PORTS     6
@@ -85,6 +86,11 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
     ICH9LPCState *ich9_lpc;
     PCIDevice *ahci;
     qemu_irq *cmos_s3;
+    DeviceState *icc_bridge;
+
+    icc_bridge = qdev_create(NULL, TYPE_ICC_BRIDGE);
+    object_property_add_child(qdev_get_machine(), "icc-bridge",
+                              OBJECT(icc_bridge), NULL);
 
     pc_cpus_init(cpu_model);
     pc_acpi_init("q35-acpi-dsdt.aml");
@@ -168,6 +174,7 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
     if (pci_enabled) {
         ioapic_init_gsi(gsi_state, NULL);
     }
+    qdev_init_nofail(icc_bridge);
 
     pc_register_ferr_irq(gsi[13]);
 
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
new file mode 100644
index 0000000..7e75a0e
--- /dev/null
+++ b/hw/icc_bus.c
@@ -0,0 +1,85 @@
+/* icc_bus.c
+ * emulate x86 ICC(INTERRUPT CONTROLLER COMMUNICATIONS) bus
+ *
+ * Copyright (c) 2013 Red Hat, Inc
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+#include "icc_bus.h"
+#include "sysbus.h"
+
+static void icc_bus_initfn(Object *obj)
+{
+    BusState *b = BUS(obj);
+    b->allow_hotplug = true;
+}
+
+static const TypeInfo icc_bus_info = {
+    .name = TYPE_ICC_BUS,
+    .parent = TYPE_BUS,
+    .instance_size = sizeof(ICCBus),
+    .instance_init = icc_bus_initfn,
+};
+
+static int icc_device_init(DeviceState *dev)
+{
+    ICCDevice *id = ICC_DEVICE(dev);
+    ICCDeviceClass *k = ICC_DEVICE_GET_CLASS(id);
+
+    return k->init(id);
+}
+
+static void icc_device_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *k = DEVICE_CLASS(klass);
+
+    k->init = icc_device_init;
+    k->bus_type = TYPE_ICC_BUS;
+}
+
+static const TypeInfo icc_device_info = {
+    .name = TYPE_ICC_DEVICE,
+    .parent = TYPE_DEVICE,
+    .abstract = true,
+    .instance_size = sizeof(ICCDevice),
+    .class_size = sizeof(ICCDeviceClass),
+    .class_init = icc_device_class_init,
+};
+
+typedef struct ICCBridgeState {
+    SysBusDevice busdev;
+} ICCBridgeState;
+#define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
+
+
+static void icc_bridge_initfn(Object *obj)
+{
+    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+}
+
+static const TypeInfo icc_bridge_info = {
+    .name  = "icc-bridge",
+    .parent = TYPE_SYS_BUS_DEVICE,
+    .instance_init  = icc_bridge_initfn,
+    .instance_size  = sizeof(ICCBridgeState),
+};
+
+static void icc_bus_register_types(void)
+{
+    type_register_static(&icc_bus_info);
+    type_register_static(&icc_device_info);
+    type_register_static(&icc_bridge_info);
+}
+
+type_init(icc_bus_register_types)
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
new file mode 100644
index 0000000..f959a43
--- /dev/null
+++ b/hw/icc_bus.h
@@ -0,0 +1,51 @@
+/* icc_bus.h
+ * emulate x86 ICC(INTERRUPT CONTROLLER COMMUNICATIONS) bus
+ *
+ * Copyright (c) 2013 Red Hat, Inc
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+#ifndef ICC_BUS_H
+#define ICC_BUS_H
+
+#include "hw/qdev-core.h"
+
+#define TYPE_ICC_BUS "icc-bus"
+
+#ifndef CONFIG_USER_ONLY
+typedef struct ICCBus {
+    BusState qbus;
+} ICCBus;
+#define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
+
+typedef struct ICCDevice {
+    DeviceState qdev;
+} ICCDevice;
+
+typedef struct ICCDeviceClass {
+    DeviceClass parent_class;
+    int (*init)(ICCDevice *dev);
+} ICCDeviceClass;
+#define TYPE_ICC_DEVICE "icc-device"
+#define ICC_DEVICE(obj) OBJECT_CHECK(ICCDevice, (obj), TYPE_ICC_DEVICE)
+#define ICC_DEVICE_CLASS(klass) \
+     OBJECT_CLASS_CHECK(ICCDeviceClass, (klass), TYPE_ICC_DEVICE)
+#define ICC_DEVICE_GET_CLASS(obj) \
+     OBJECT_GET_CLASS(ICCDeviceClass, (obj), TYPE_ICC_DEVICE)
+
+#define TYPE_ICC_BRIDGE "icc-bridge"
+
+#endif /* CONFIG_USER_ONLY */
+#endif
+
-- 
1.8.1.4

[Qemu-devel] [PATCH 16/22] target-i386: cpu: attach ICC bus to CPU on its creation

[Igor Mammedov]

... during startup,  so it would be possible to uplug it later
and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
attach hotplugged CPU to ICC bus.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 target-i386/cpu.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index db56b52..2b3c1f3 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,6 +41,7 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen.h"
 #include "hw/sysbus.h"
@@ -1604,6 +1605,7 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
     CPUX86State *env;
     gchar **model_pieces;
     char *name, *features;
+    Object *icc_bus = object_resolve_path_type("icc-bus", TYPE_ICC_BUS, NULL);
 
     model_pieces = g_strsplit(cpu_model, ",", 2);
     if (!model_pieces[0]) {
@@ -1614,6 +1616,10 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
     features = model_pieces[1];
 
     cpu = X86_CPU(object_new(TYPE_X86_CPU));
+    if (icc_bus) {
+        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
+        object_unref(OBJECT(cpu));
+    }
     env = &cpu->env;
     env->cpu_model_str = cpu_model;
 
@@ -2323,6 +2329,7 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 
     xcc->parent_realize = dc->realize;
     dc->realize = x86_cpu_realizefn;
+    dc->bus_type = TYPE_ICC_BUS;
 
     xcc->parent_reset = cc->reset;
     cc->reset = x86_cpu_reset;
-- 
1.8.1.4

[Qemu-devel] [PATCH 17/22] target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE

[Igor Mammedov]

... and put APIC_SPACE_SIZE in public header so that it could be
reused later elsewhere.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/apic.c          | 2 +-
 hw/apic_internal.h | 2 --
 hw/kvm/apic.c      | 2 +-
 hw/xen_apic.c      | 2 +-
 target-i386/cpu.h  | 1 +
 5 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/hw/apic.c b/hw/apic.c
index d2395f0..d99aeae 100644
--- a/hw/apic.c
+++ b/hw/apic.c
@@ -874,7 +874,7 @@ static const MemoryRegionOps apic_io_ops = {
 static void apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &apic_io_ops, s, "apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
     s->timer = qemu_new_timer_ns(vm_clock, apic_timer, s);
     local_apics[s->idx] = s;
diff --git a/hw/apic_internal.h b/hw/apic_internal.h
index 578241f..aac6290 100644
--- a/hw/apic_internal.h
+++ b/hw/apic_internal.h
@@ -66,8 +66,6 @@
 
 #define MAX_APICS 255
 
-#define MSI_SPACE_SIZE                  0x100000
-
 typedef struct APICCommonState APICCommonState;
 
 #define TYPE_APIC_COMMON "apic-common"
diff --git a/hw/kvm/apic.c b/hw/kvm/apic.c
index d994ea7..8e272e7 100644
--- a/hw/kvm/apic.c
+++ b/hw/kvm/apic.c
@@ -174,7 +174,7 @@ static const MemoryRegionOps kvm_apic_io_ops = {
 static void kvm_apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &kvm_apic_io_ops, s, "kvm-apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
     if (kvm_has_gsi_routing()) {
         msi_supported = true;
diff --git a/hw/xen_apic.c b/hw/xen_apic.c
index 8f387b6..86974b7 100644
--- a/hw/xen_apic.c
+++ b/hw/xen_apic.c
@@ -39,7 +39,7 @@ static const MemoryRegionOps xen_apic_io_ops = {
 static void xen_apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &xen_apic_io_ops, s, "xen-apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
 #if defined(CONFIG_XEN_CTRL_INTERFACE_VERSION) \
     && CONFIG_XEN_CTRL_INTERFACE_VERSION >= 420
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index b98efd2..df233e6 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -1269,5 +1269,6 @@ uint32_t x86_cpu_apic_id_from_index(unsigned int cpu_index);
 void enable_compat_apic_id_mode(void);
 
 #define APIC_DEFAULT_ADDRESS 0xfee00000
+#define APIC_SPACE_SIZE      0x100000
 
 #endif /* CPU_I386_H */
-- 
1.8.1.4

[Qemu-devel] [PATCH 18/22] target-i386: move APIC to ICC bus

[Igor Mammedov]

... to allow it to be hotplugged

 * map APIC's mmio at board level if it is present

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/apic_common.c   | 17 ++++++++++++-----
 hw/apic_internal.h |  6 +++---
 hw/i386/kvmvapic.c |  1 +
 hw/i386/pc.c       | 19 ++++++++++++++++---
 hw/icc_bus.c       | 15 ++++++++++++++-
 hw/icc_bus.h       |  3 ++-
 target-i386/cpu.c  | 16 +++-------------
 7 files changed, 51 insertions(+), 26 deletions(-)

diff --git a/hw/apic_common.c b/hw/apic_common.c
index 3798509..b2e84e6 100644
--- a/hw/apic_common.c
+++ b/hw/apic_common.c
@@ -21,6 +21,8 @@
 #include "hw/apic_internal.h"
 #include "trace.h"
 #include "sysemu/kvm.h"
+#include "qdev.h"
+#include "hw/sysbus.h"
 
 static int apic_irq_delivered;
 bool apic_report_tpr_access;
@@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque, int version_id)
     return 0;
 }
 
-static int apic_init_common(SysBusDevice *dev)
+static int apic_init_common(ICCDevice *dev)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
     static int apic_no;
+    static bool mmio_registered;
 
     if (apic_no >= MAX_APICS) {
         return -1;
@@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
 
     info = APIC_COMMON_GET_CLASS(s);
     info->init(s);
-
-    sysbus_init_mmio(dev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
 
     /* Note: We need at least 1M to map the VAPIC option ROM */
     if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
@@ -375,7 +382,7 @@ static Property apic_properties_common[] = {
 
 static void apic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     dc->vmsd = &vmstate_apic_common;
@@ -387,7 +394,7 @@ static void apic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo apic_common_type = {
     .name = TYPE_APIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(APICCommonState),
     .class_size = sizeof(APICCommonClass),
     .class_init = apic_common_class_init,
diff --git a/hw/apic_internal.h b/hw/apic_internal.h
index aac6290..172fc91 100644
--- a/hw/apic_internal.h
+++ b/hw/apic_internal.h
@@ -21,7 +21,7 @@
 #define QEMU_APIC_INTERNAL_H
 
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 #include "qemu/timer.h"
 
 /* APIC Local Vector Table */
@@ -78,7 +78,7 @@ typedef struct APICCommonState APICCommonState;
 
 typedef struct APICCommonClass
 {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
 
     void (*init)(APICCommonState *s);
     void (*set_base)(APICCommonState *s, uint64_t val);
@@ -92,7 +92,7 @@ typedef struct APICCommonClass
 } APICCommonClass;
 
 struct APICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
 
     MemoryRegion io_memory;
     X86CPU *cpu;
diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
index c4be882..81e0a75 100644
--- a/hw/i386/kvmvapic.c
+++ b/hw/i386/kvmvapic.c
@@ -12,6 +12,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/kvm.h"
 #include "hw/apic_internal.h"
+#include "hw/sysbus.h"
 
 #define VAPIC_IO_PORT           0x7e
 
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 9d617b4..5331c0f 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -52,6 +52,7 @@
 #include "sysemu/arch_init.h"
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
+#include "hw/icc_bus.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -869,13 +870,13 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
 {
     X86CPU *cpu;
 
     cpu = cpu_x86_create(cpu_model, errp);
     if (!cpu) {
-        return;
+        return cpu;
     }
 
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
@@ -884,14 +885,18 @@ static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
+            cpu = NULL;
         }
     }
+    return cpu;
 }
 
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    X86CPU *cpu;
     Error *error = NULL;
+    SysBusDevice *ib;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -902,14 +907,22 @@ void pc_cpus_init(const char *cpu_model)
 #endif
     }
 
+    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
+                                                 TYPE_ICC_BRIDGE, NULL));
+
     for (i = 0; i < smp_cpus; i++) {
-        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
             exit(1);
         }
     }
+
+    /* map APIC MMIO area if CPU has it */
+    if (cpu->env.apic_state) {
+        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
+    }
 }
 
 void pc_acpi_init(const char *default_dsdt)
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index 7e75a0e..a0c4e02 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -59,13 +59,26 @@ static const TypeInfo icc_device_info = {
 
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
+    MemoryRegion apic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
 
 static void icc_bridge_initfn(Object *obj)
 {
-    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+    SysBusDevice *sb = SYS_BUS_DEVICE(obj);
+    ICCBus *ibus;
+
+    ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
+
+    /* Do not change order of registering regions,
+     * APIC must be first registered region, board maps it by 0 index
+     */
+    memory_region_init(&s->apic_container, "icc-apic-container",
+                       APIC_SPACE_SIZE);
+    sysbus_init_mmio(sb, &s->apic_container);
+    ibus->apic_address_space = &s->apic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index f959a43..2c8f78f 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -19,6 +19,7 @@
 #ifndef ICC_BUS_H
 #define ICC_BUS_H
 
+#include "exec/memory.h"
 #include "hw/qdev-core.h"
 
 #define TYPE_ICC_BUS "icc-bus"
@@ -26,6 +27,7 @@
 #ifndef CONFIG_USER_ONLY
 typedef struct ICCBus {
     BusState qbus;
+    MemoryRegion *apic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
@@ -48,4 +50,3 @@ typedef struct ICCDeviceClass {
 
 #endif /* CONFIG_USER_ONLY */
 #endif
-
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 2b3c1f3..32f7aea 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,10 +41,10 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/qdev-properties.h"
 #include "hw/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen.h"
-#include "hw/sysbus.h"
 #include "hw/apic_internal.h"
 #endif
 
@@ -2104,6 +2104,7 @@ static void mce_init(X86CPU *cpu)
 static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
+    DeviceState *dev = DEVICE(cpu);
     APICCommonState *apic;
     const char *apic_type = "apic";
 
@@ -2113,7 +2114,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
         apic_type = "xen-apic";
     }
 
-    env->apic_state = qdev_try_create(NULL, apic_type);
+    env->apic_state = qdev_try_create(dev->parent_bus, apic_type);
     if (env->apic_state == NULL) {
         error_setg(errp, "APIC device '%s' could not be created", apic_type);
         return;
@@ -2130,7 +2131,6 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
-    static int apic_mapped;
 
     if (env->apic_state == NULL) {
         return;
@@ -2141,16 +2141,6 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
                    object_get_typename(OBJECT(env->apic_state)));
         return;
     }
-
-    /* XXX: mapping more APICs at the same memory location */
-    if (apic_mapped == 0) {
-        /* NOTE: the APIC is directly connected to the CPU - it is not
-           on the global memory bus. */
-        /* XXX: what if the base changes? */
-        sysbus_mmio_map_overlap(SYS_BUS_DEVICE(env->apic_state), 0,
-                                APIC_DEFAULT_ADDRESS, 0x1000);
-        apic_mapped = 1;
-    }
 }
 #else
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
-- 
1.8.1.4

[Qemu-devel] [PATCH 18/22] target-i386: move IOAPIC to ICC bus

[Igor Mammedov]

 * inherit IOAPICs from ICCDevice and attach them to ICC bus
 * map IOAPIC's mmio at board level
 * make IOAPIC a child device of icc-bridge

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c          | 23 +++++++++--------------
 hw/i386/pc_piix.c     |  2 +-
 hw/i386/pc_q35.c      |  2 +-
 hw/icc_bus.c          | 28 +++++++++++++++++++++++++++-
 hw/icc_bus.h          |  2 ++
 hw/ioapic_common.c    | 15 +++++++++++----
 hw/ioapic_internal.h  |  6 +++---
 hw/kvm/ioapic.c       |  2 +-
 hw/pc.h               |  2 +-
 include/exec/memory.h | 19 +++++++++++++++++++
 10 files changed, 75 insertions(+), 26 deletions(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7a608c5..fbc0bcf 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -51,6 +51,7 @@
 #include "exec/address-spaces.h"
 #include "sysemu/arch_init.h"
 #include "qemu/bitmap.h"
+#include "hw/icc_bus.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -1166,26 +1167,20 @@ void pc_pci_device_init(PCIBus *pci_bus)
     }
 }
 
-void ioapic_init_gsi(GSIState *gsi_state, const char *parent_name)
+void ioapic_init_gsi(GSIState *gsi_state, DeviceState *dev)
 {
-    DeviceState *dev;
-    SysBusDevice *d;
     unsigned int i;
+    DeviceState *ioapic;
+    const char *ioapic_name = "ioapic";
 
     if (kvm_irqchip_in_kernel()) {
-        dev = qdev_create(NULL, "kvm-ioapic");
-    } else {
-        dev = qdev_create(NULL, "ioapic");
-    }
-    if (parent_name) {
-        object_property_add_child(object_resolve_path(parent_name, NULL),
-                                  "ioapic", OBJECT(dev), NULL);
+        ioapic_name = "kvm-ioapic";
     }
-    qdev_init_nofail(dev);
-    d = SYS_BUS_DEVICE(dev);
-    sysbus_mmio_map(d, 0, IO_APIC_DEFAULT_ADDRESS);
+    object_property_set_str(OBJECT(dev), ioapic_name, "ioapic-type", NULL);
 
+    sysbus_mmio_map(SYS_BUS_DEVICE(dev), 1, IO_APIC_DEFAULT_ADDRESS);
+    ioapic = DEVICE(object_resolve_path_component(OBJECT(dev), "ioapic"));
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
-        gsi_state->ioapic_irq[i] = qdev_get_gpio_in(dev, i);
+        gsi_state->ioapic_irq[i] = qdev_get_gpio_in(ioapic, i);
     }
 }
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 8bf5440..bf1e88c 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -164,7 +164,7 @@ static void pc_init1(MemoryRegion *system_memory,
         gsi_state->i8259_irq[i] = i8259[i];
     }
     if (pci_enabled) {
-        ioapic_init_gsi(gsi_state, "i440fx");
+        ioapic_init_gsi(gsi_state, icc_bridge);
     }
     qdev_init_nofail(icc_bridge);
 
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index fb701b4..94a3952 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -172,7 +172,7 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
         gsi_state->i8259_irq[i] = i8259[i];
     }
     if (pci_enabled) {
-        ioapic_init_gsi(gsi_state, NULL);
+        ioapic_init_gsi(gsi_state, icc_bridge);
     }
     qdev_init_nofail(icc_bridge);
 
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index 7e75a0e..fc8d892 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -59,13 +59,39 @@ static const TypeInfo icc_device_info = {
 
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
+    MemoryRegion ioapic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
+static void icc_bridge_prop_set_ioapic_type(Object *obj, const char *value,
+                                            Error **errp)
+{
+    BusState *bus = BUS(object_resolve_path_component(obj, "icc-bus"));
+    DeviceState *ioapic;
+
+    if (value != NULL) {
+        ioapic = qdev_create(bus, value);
+        object_property_add_child(obj, "ioapic", OBJECT(ioapic), NULL);
+        qdev_init_nofail(ioapic);
+    }
+}
 
 static void icc_bridge_initfn(Object *obj)
 {
-    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+    SysBusDevice *sb = SYS_BUS_DEVICE(obj);
+    ICCBus *ibus;
+
+    object_property_add_str(obj, "ioapic-type",
+                             NULL,
+                             icc_bridge_prop_set_ioapic_type,
+                             NULL);
+
+    ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
+
+    memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
+    sysbus_init_mmio(sb, &s->ioapic_container);
+    ibus->ioapic_address_space = &s->ioapic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index f959a43..b8e2032 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -19,6 +19,7 @@
 #ifndef ICC_BUS_H
 #define ICC_BUS_H
 
+#include "exec/memory.h"
 #include "hw/qdev-core.h"
 
 #define TYPE_ICC_BUS "icc-bus"
@@ -26,6 +27,7 @@
 #ifndef CONFIG_USER_ONLY
 typedef struct ICCBus {
     BusState qbus;
+    MemoryRegion *ioapic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
index 561b987..a366bae 100644
--- a/hw/ioapic_common.c
+++ b/hw/ioapic_common.c
@@ -57,11 +57,13 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
     return 0;
 }
 
-static int ioapic_init_common(SysBusDevice *dev)
+static int ioapic_init_common(ICCDevice *dev)
 {
     IOAPICCommonState *s = IOAPIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     IOAPICCommonClass *info;
     static int ioapic_no;
+    static bool mmio_registered;
 
     if (ioapic_no >= MAX_IOAPICS) {
         return -1;
@@ -70,7 +72,12 @@ static int ioapic_init_common(SysBusDevice *dev)
     info = IOAPIC_COMMON_GET_CLASS(s);
     info->init(s, ioapic_no);
 
-    sysbus_init_mmio(&s->busdev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->ioapic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
+
     ioapic_no++;
 
     return 0;
@@ -95,7 +102,7 @@ static const VMStateDescription vmstate_ioapic_common = {
 
 static void ioapic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     sc->init = ioapic_init_common;
@@ -105,7 +112,7 @@ static void ioapic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo ioapic_common_type = {
     .name = TYPE_IOAPIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(IOAPICCommonState),
     .class_size = sizeof(IOAPICCommonClass),
     .class_init = ioapic_common_class_init,
diff --git a/hw/ioapic_internal.h b/hw/ioapic_internal.h
index 25576c8..c1db3f0 100644
--- a/hw/ioapic_internal.h
+++ b/hw/ioapic_internal.h
@@ -24,7 +24,7 @@
 
 #include "hw/hw.h"
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 
 #define MAX_IOAPICS                     1
 
@@ -82,14 +82,14 @@ typedef struct IOAPICCommonState IOAPICCommonState;
      OBJECT_GET_CLASS(IOAPICCommonClass, (obj), TYPE_IOAPIC_COMMON)
 
 typedef struct IOAPICCommonClass {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
     void (*init)(IOAPICCommonState *s, int instance_no);
     void (*pre_save)(IOAPICCommonState *s);
     void (*post_load)(IOAPICCommonState *s);
 } IOAPICCommonClass;
 
 struct IOAPICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
     MemoryRegion io_memory;
     uint8_t id;
     uint8_t ioregsel;
diff --git a/hw/kvm/ioapic.c b/hw/kvm/ioapic.c
index 23877d4..ca9b249 100644
--- a/hw/kvm/ioapic.c
+++ b/hw/kvm/ioapic.c
@@ -96,7 +96,7 @@ static void kvm_ioapic_put(IOAPICCommonState *s)
 
     kioapic->id = s->id;
     kioapic->ioregsel = s->ioregsel;
-    kioapic->base_address = s->busdev.mmio[0].addr;
+    kioapic->base_address = memory_region_get_address(&s->io_memory);
     kioapic->irr = s->irr;
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
         kioapic->redirtbl[i].bits = s->ioredtbl[i];
diff --git a/hw/pc.h b/hw/pc.h
index 8e1dd4c..dbd5794 100644
--- a/hw/pc.h
+++ b/hw/pc.h
@@ -105,7 +105,7 @@ void pc_pci_device_init(PCIBus *pci_bus);
 typedef void (*cpu_set_smm_t)(int smm, void *arg);
 void cpu_smm_register(cpu_set_smm_t callback, void *arg);
 
-void ioapic_init_gsi(GSIState *gsi_state, const char *parent_name);
+void ioapic_init_gsi(GSIState *gsi_state, DeviceState *ioapic);
 
 /* acpi.c */
 extern int acpi_enabled;
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 2322732..223008a5 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -711,6 +711,25 @@ void memory_region_set_enabled(MemoryRegion *mr, bool enabled);
 void memory_region_set_address(MemoryRegion *mr, hwaddr addr);
 
 /*
+ * memory_region_get_address: get current the address of a region
+ *
+ * Returns the absolute address of a region.
+ * May be used on regions that are currently part of a memory hierarchy.
+ *
+ * @mr: the region being queried
+ */
+static inline hwaddr memory_region_get_address(MemoryRegion *mr)
+{
+    hwaddr addr = mr->addr;
+
+    while (mr->parent) {
+        mr = mr->parent;
+        addr += mr->addr;
+    }
+    return addr;
+}
+
+/*
  * memory_region_set_alias_offset: dynamically update a memory alias's offset
  *
  * Dynamically updates the offset into the target region that an alias points
-- 
1.8.1.4

[Qemu-devel] [PATCH 19/22] target-i386: move APIC to ICC bus

[Igor Mammedov]

... to allow it to be hotplugged

 * map APIC's mmio at board level if it is present

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/apic_common.c   | 17 ++++++++++++-----
 hw/apic_internal.h |  6 +++---
 hw/i386/kvmvapic.c |  1 +
 hw/i386/pc.c       | 18 +++++++++++++++---
 hw/icc_bus.c       |  6 ++++++
 hw/icc_bus.h       |  2 +-
 target-i386/cpu.c  | 16 +++-------------
 7 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/hw/apic_common.c b/hw/apic_common.c
index 3798509..b2e84e6 100644
--- a/hw/apic_common.c
+++ b/hw/apic_common.c
@@ -21,6 +21,8 @@
 #include "hw/apic_internal.h"
 #include "trace.h"
 #include "sysemu/kvm.h"
+#include "qdev.h"
+#include "hw/sysbus.h"
 
 static int apic_irq_delivered;
 bool apic_report_tpr_access;
@@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque, int version_id)
     return 0;
 }
 
-static int apic_init_common(SysBusDevice *dev)
+static int apic_init_common(ICCDevice *dev)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
     static int apic_no;
+    static bool mmio_registered;
 
     if (apic_no >= MAX_APICS) {
         return -1;
@@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
 
     info = APIC_COMMON_GET_CLASS(s);
     info->init(s);
-
-    sysbus_init_mmio(dev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
 
     /* Note: We need at least 1M to map the VAPIC option ROM */
     if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
@@ -375,7 +382,7 @@ static Property apic_properties_common[] = {
 
 static void apic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     dc->vmsd = &vmstate_apic_common;
@@ -387,7 +394,7 @@ static void apic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo apic_common_type = {
     .name = TYPE_APIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(APICCommonState),
     .class_size = sizeof(APICCommonClass),
     .class_init = apic_common_class_init,
diff --git a/hw/apic_internal.h b/hw/apic_internal.h
index aac6290..172fc91 100644
--- a/hw/apic_internal.h
+++ b/hw/apic_internal.h
@@ -21,7 +21,7 @@
 #define QEMU_APIC_INTERNAL_H
 
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 #include "qemu/timer.h"
 
 /* APIC Local Vector Table */
@@ -78,7 +78,7 @@ typedef struct APICCommonState APICCommonState;
 
 typedef struct APICCommonClass
 {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
 
     void (*init)(APICCommonState *s);
     void (*set_base)(APICCommonState *s, uint64_t val);
@@ -92,7 +92,7 @@ typedef struct APICCommonClass
 } APICCommonClass;
 
 struct APICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
 
     MemoryRegion io_memory;
     X86CPU *cpu;
diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
index c4be882..81e0a75 100644
--- a/hw/i386/kvmvapic.c
+++ b/hw/i386/kvmvapic.c
@@ -12,6 +12,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/kvm.h"
 #include "hw/apic_internal.h"
+#include "hw/sysbus.h"
 
 #define VAPIC_IO_PORT           0x7e
 
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index fbc0bcf..5c0dd4f 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -869,13 +869,13 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
 {
     X86CPU *cpu;
 
     cpu = cpu_x86_create(cpu_model, errp);
     if (!cpu) {
-        return;
+        return cpu;
     }
 
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
@@ -884,14 +884,18 @@ static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
+            cpu = NULL;
         }
     }
+    return cpu;
 }
 
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    X86CPU *cpu;
     Error *error = NULL;
+    SysBusDevice *ib;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -902,14 +906,22 @@ void pc_cpus_init(const char *cpu_model)
 #endif
     }
 
+    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
+                                                 TYPE_ICC_BRIDGE, NULL));
+
     for (i = 0; i < smp_cpus; i++) {
-        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
             exit(1);
         }
     }
+
+    /* map APIC MMIO area if CPU has it */
+    if (cpu->env.apic_state) {
+        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
+    }
 }
 
 void pc_acpi_init(const char *default_dsdt)
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index fc8d892..bfe2a30 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -60,6 +60,7 @@ static const TypeInfo icc_device_info = {
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
     MemoryRegion ioapic_container;
+    MemoryRegion apic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
@@ -92,6 +93,11 @@ static void icc_bridge_initfn(Object *obj)
     memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
     sysbus_init_mmio(sb, &s->ioapic_container);
     ibus->ioapic_address_space = &s->ioapic_container;
+
+    memory_region_init(&s->apic_container, "icc-apic-container",
+                       APIC_SPACE_SIZE);
+    sysbus_init_mmio(sb, &s->apic_container);
+    ibus->apic_address_space = &s->apic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index b8e2032..a3d9e0a 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -28,6 +28,7 @@
 typedef struct ICCBus {
     BusState qbus;
     MemoryRegion *ioapic_address_space;
+    MemoryRegion *apic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
@@ -50,4 +51,3 @@ typedef struct ICCDeviceClass {
 
 #endif /* CONFIG_USER_ONLY */
 #endif
-
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 2b3c1f3..32f7aea 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,10 +41,10 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/qdev-properties.h"
 #include "hw/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen.h"
-#include "hw/sysbus.h"
 #include "hw/apic_internal.h"
 #endif
 
@@ -2104,6 +2104,7 @@ static void mce_init(X86CPU *cpu)
 static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
+    DeviceState *dev = DEVICE(cpu);
     APICCommonState *apic;
     const char *apic_type = "apic";
 
@@ -2113,7 +2114,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
         apic_type = "xen-apic";
     }
 
-    env->apic_state = qdev_try_create(NULL, apic_type);
+    env->apic_state = qdev_try_create(dev->parent_bus, apic_type);
     if (env->apic_state == NULL) {
         error_setg(errp, "APIC device '%s' could not be created", apic_type);
         return;
@@ -2130,7 +2131,6 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
-    static int apic_mapped;
 
     if (env->apic_state == NULL) {
         return;
@@ -2141,16 +2141,6 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
                    object_get_typename(OBJECT(env->apic_state)));
         return;
     }
-
-    /* XXX: mapping more APICs at the same memory location */
-    if (apic_mapped == 0) {
-        /* NOTE: the APIC is directly connected to the CPU - it is not
-           on the global memory bus. */
-        /* XXX: what if the base changes? */
-        sysbus_mmio_map_overlap(SYS_BUS_DEVICE(env->apic_state), 0,
-                                APIC_DEFAULT_ADDRESS, 0x1000);
-        apic_mapped = 1;
-    }
 }
 #else
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
-- 
1.8.1.4

[Qemu-devel] [PATCH 19/22] target-i386: move IOAPIC to ICC bus

[Igor Mammedov]

 * inherit IOAPICs from ICCDevice and attach them to ICC bus
 * map IOAPIC's mmio at board level
 * make IOAPIC a child device of icc-bridge

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c          | 22 ++++++++--------------
 hw/i386/pc_piix.c     |  2 +-
 hw/i386/pc_q35.c      |  2 +-
 hw/icc_bus.c          | 23 +++++++++++++++++++++++
 hw/icc_bus.h          |  1 +
 hw/ioapic_common.c    | 15 +++++++++++----
 hw/ioapic_internal.h  |  6 +++---
 hw/kvm/ioapic.c       |  2 +-
 hw/pc.h               |  2 +-
 include/exec/memory.h | 19 +++++++++++++++++++
 10 files changed, 69 insertions(+), 25 deletions(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 5331c0f..6f043a2 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1191,26 +1191,20 @@ void pc_pci_device_init(PCIBus *pci_bus)
     }
 }
 
-void ioapic_init_gsi(GSIState *gsi_state, const char *parent_name)
+void ioapic_init_gsi(GSIState *gsi_state, DeviceState *dev)
 {
-    DeviceState *dev;
-    SysBusDevice *d;
     unsigned int i;
+    DeviceState *ioapic;
+    const char *ioapic_name = "ioapic";
 
     if (kvm_irqchip_in_kernel()) {
-        dev = qdev_create(NULL, "kvm-ioapic");
-    } else {
-        dev = qdev_create(NULL, "ioapic");
-    }
-    if (parent_name) {
-        object_property_add_child(object_resolve_path(parent_name, NULL),
-                                  "ioapic", OBJECT(dev), NULL);
+        ioapic_name = "kvm-ioapic";
     }
-    qdev_init_nofail(dev);
-    d = SYS_BUS_DEVICE(dev);
-    sysbus_mmio_map(d, 0, IO_APIC_DEFAULT_ADDRESS);
+    object_property_set_str(OBJECT(dev), ioapic_name, "ioapic-type", NULL);
 
+    sysbus_mmio_map(SYS_BUS_DEVICE(dev), 1, IO_APIC_DEFAULT_ADDRESS);
+    ioapic = DEVICE(object_resolve_path_component(OBJECT(dev), "ioapic"));
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
-        gsi_state->ioapic_irq[i] = qdev_get_gpio_in(dev, i);
+        gsi_state->ioapic_irq[i] = qdev_get_gpio_in(ioapic, i);
     }
 }
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 8bf5440..bf1e88c 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -164,7 +164,7 @@ static void pc_init1(MemoryRegion *system_memory,
         gsi_state->i8259_irq[i] = i8259[i];
     }
     if (pci_enabled) {
-        ioapic_init_gsi(gsi_state, "i440fx");
+        ioapic_init_gsi(gsi_state, icc_bridge);
     }
     qdev_init_nofail(icc_bridge);
 
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index fb701b4..94a3952 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -172,7 +172,7 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
         gsi_state->i8259_irq[i] = i8259[i];
     }
     if (pci_enabled) {
-        ioapic_init_gsi(gsi_state, NULL);
+        ioapic_init_gsi(gsi_state, icc_bridge);
     }
     qdev_init_nofail(icc_bridge);
 
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index a0c4e02..fdf9504 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -60,9 +60,22 @@ static const TypeInfo icc_device_info = {
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
     MemoryRegion apic_container;
+    MemoryRegion ioapic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
+static void icc_bridge_prop_set_ioapic_type(Object *obj, const char *value,
+                                            Error **errp)
+{
+    BusState *bus = BUS(object_resolve_path_component(obj, "icc-bus"));
+    DeviceState *ioapic;
+
+    if (value != NULL) {
+        ioapic = qdev_create(bus, value);
+        object_property_add_child(obj, "ioapic", OBJECT(ioapic), NULL);
+        qdev_init_nofail(ioapic);
+    }
+}
 
 static void icc_bridge_initfn(Object *obj)
 {
@@ -70,6 +83,11 @@ static void icc_bridge_initfn(Object *obj)
     SysBusDevice *sb = SYS_BUS_DEVICE(obj);
     ICCBus *ibus;
 
+    object_property_add_str(obj, "ioapic-type",
+                             NULL,
+                             icc_bridge_prop_set_ioapic_type,
+                             NULL);
+
     ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
 
     /* Do not change order of registering regions,
@@ -79,6 +97,11 @@ static void icc_bridge_initfn(Object *obj)
                        APIC_SPACE_SIZE);
     sysbus_init_mmio(sb, &s->apic_container);
     ibus->apic_address_space = &s->apic_container;
+
+    /* must be second registered region, board maps it by 1 index */
+    memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
+    sysbus_init_mmio(sb, &s->ioapic_container);
+    ibus->ioapic_address_space = &s->ioapic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index 2c8f78f..69a0278 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -28,6 +28,7 @@
 typedef struct ICCBus {
     BusState qbus;
     MemoryRegion *apic_address_space;
+    MemoryRegion *ioapic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
index 561b987..a366bae 100644
--- a/hw/ioapic_common.c
+++ b/hw/ioapic_common.c
@@ -57,11 +57,13 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
     return 0;
 }
 
-static int ioapic_init_common(SysBusDevice *dev)
+static int ioapic_init_common(ICCDevice *dev)
 {
     IOAPICCommonState *s = IOAPIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     IOAPICCommonClass *info;
     static int ioapic_no;
+    static bool mmio_registered;
 
     if (ioapic_no >= MAX_IOAPICS) {
         return -1;
@@ -70,7 +72,12 @@ static int ioapic_init_common(SysBusDevice *dev)
     info = IOAPIC_COMMON_GET_CLASS(s);
     info->init(s, ioapic_no);
 
-    sysbus_init_mmio(&s->busdev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->ioapic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
+
     ioapic_no++;
 
     return 0;
@@ -95,7 +102,7 @@ static const VMStateDescription vmstate_ioapic_common = {
 
 static void ioapic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     sc->init = ioapic_init_common;
@@ -105,7 +112,7 @@ static void ioapic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo ioapic_common_type = {
     .name = TYPE_IOAPIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(IOAPICCommonState),
     .class_size = sizeof(IOAPICCommonClass),
     .class_init = ioapic_common_class_init,
diff --git a/hw/ioapic_internal.h b/hw/ioapic_internal.h
index 25576c8..c1db3f0 100644
--- a/hw/ioapic_internal.h
+++ b/hw/ioapic_internal.h
@@ -24,7 +24,7 @@
 
 #include "hw/hw.h"
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 
 #define MAX_IOAPICS                     1
 
@@ -82,14 +82,14 @@ typedef struct IOAPICCommonState IOAPICCommonState;
      OBJECT_GET_CLASS(IOAPICCommonClass, (obj), TYPE_IOAPIC_COMMON)
 
 typedef struct IOAPICCommonClass {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
     void (*init)(IOAPICCommonState *s, int instance_no);
     void (*pre_save)(IOAPICCommonState *s);
     void (*post_load)(IOAPICCommonState *s);
 } IOAPICCommonClass;
 
 struct IOAPICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
     MemoryRegion io_memory;
     uint8_t id;
     uint8_t ioregsel;
diff --git a/hw/kvm/ioapic.c b/hw/kvm/ioapic.c
index 23877d4..ca9b249 100644
--- a/hw/kvm/ioapic.c
+++ b/hw/kvm/ioapic.c
@@ -96,7 +96,7 @@ static void kvm_ioapic_put(IOAPICCommonState *s)
 
     kioapic->id = s->id;
     kioapic->ioregsel = s->ioregsel;
-    kioapic->base_address = s->busdev.mmio[0].addr;
+    kioapic->base_address = memory_region_get_address(&s->io_memory);
     kioapic->irr = s->irr;
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
         kioapic->redirtbl[i].bits = s->ioredtbl[i];
diff --git a/hw/pc.h b/hw/pc.h
index 55964ce..5796a28 100644
--- a/hw/pc.h
+++ b/hw/pc.h
@@ -105,7 +105,7 @@ void pc_pci_device_init(PCIBus *pci_bus);
 typedef void (*cpu_set_smm_t)(int smm, void *arg);
 void cpu_smm_register(cpu_set_smm_t callback, void *arg);
 
-void ioapic_init_gsi(GSIState *gsi_state, const char *parent_name);
+void ioapic_init_gsi(GSIState *gsi_state, DeviceState *ioapic);
 
 /* acpi.c */
 extern int acpi_enabled;
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 2322732..223008a5 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -711,6 +711,25 @@ void memory_region_set_enabled(MemoryRegion *mr, bool enabled);
 void memory_region_set_address(MemoryRegion *mr, hwaddr addr);
 
 /*
+ * memory_region_get_address: get current the address of a region
+ *
+ * Returns the absolute address of a region.
+ * May be used on regions that are currently part of a memory hierarchy.
+ *
+ * @mr: the region being queried
+ */
+static inline hwaddr memory_region_get_address(MemoryRegion *mr)
+{
+    hwaddr addr = mr->addr;
+
+    while (mr->parent) {
+        mr = mr->parent;
+        addr += mr->addr;
+    }
+    return addr;
+}
+
+/*
  * memory_region_set_alias_offset: dynamically update a memory alias's offset
  *
  * Dynamically updates the offset into the target region that an alias points
-- 
1.8.1.4

[Qemu-devel] [PATCH 20/22] qdev: set device's parent before calling realize() down inheritance chain.

[Igor Mammedov]

Currently device_set_realized() sets parent only after device was realized,
But qdev_device_add() sets it before device is realized.
Make behavior consistent and alter device_set_realized() to behave like
qdev_device_add().

It will allow to set link<> properties in realize() method in classes
inherited from DEVICE.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 * Usage examples
    - conversion to CPU back-link property in APIC
    - pre-allocate link<CPU[n]> at board level, and populate them
      at realize time when CPUs are created.
---
 hw/qdev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/hw/qdev.c b/hw/qdev.c
index e2bb37d..b9bd6f5 100644
--- a/hw/qdev.c
+++ b/hw/qdev.c
@@ -671,10 +671,6 @@ static void device_set_realized(Object *obj, bool value, Error **err)
     Error *local_err = NULL;
 
     if (value && !dev->realized) {
-        if (dc->realize) {
-            dc->realize(dev, &local_err);
-        }
-
         if (!obj->parent && local_err == NULL) {
             static int unattached_count;
             gchar *name = g_strdup_printf("device[%d]", unattached_count++);
@@ -685,6 +681,10 @@ static void device_set_realized(Object *obj, bool value, Error **err)
             g_free(name);
         }
 
+        if (dc->realize) {
+            dc->realize(dev, &local_err);
+        }
+
         if (qdev_get_vmsd(dev) && local_err == NULL) {
             vmstate_register_with_alias_id(dev, -1, qdev_get_vmsd(dev), dev,
                                            dev->instance_id_alias,
-- 
1.8.1.4

[Qemu-devel] [PATCH 21/22] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

... and leave links for not present CPUs empty

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c |  9 +++++++--
 hw/icc_bus.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 hw/icc_bus.h |  2 ++
 3 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 6f043a2..86e5365 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
+                          SysBusDevice *icc_bridge, Error **errp)
 {
     X86CPU *cpu;
 
@@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
     object_property_set_bool(OBJECT(cpu), true, "realized", errp);
 
+    if (icc_bridge != NULL) {
+        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
+    }
+
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
@@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
                                                  TYPE_ICC_BRIDGE, NULL));
 
     for (i = 0; i < smp_cpus; i++) {
-        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), ib, &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index fdf9504..fd4b237 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -18,6 +18,7 @@
  */
 #include "icc_bus.h"
 #include "sysbus.h"
+#include "sysemu/sysemu.h"
 
 static void icc_bus_initfn(Object *obj)
 {
@@ -61,6 +62,8 @@ typedef struct ICCBridgeState {
     SysBusDevice busdev;
     MemoryRegion apic_container;
     MemoryRegion ioapic_container;
+    Notifier cpu_add_notifier;
+    Object **links;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
@@ -77,11 +80,33 @@ static void icc_bridge_prop_set_ioapic_type(Object *obj, const char *value,
     }
 }
 
+void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
+{
+    gchar *name;
+    Error *error = NULL;
+    CPUState *cpu = CPU(cpu_obj);
+    int64_t id = CPU_GET_CLASS(cpu)->get_firmware_id(cpu);
+
+    name = g_strdup_printf("cpu[%" PRIu32 "]", x86_cpu_apic_id_from_index(id));
+    object_property_set_link(bridge, cpu_obj, name, &error);
+    g_free(name);
+
+    g_assert(error == NULL);
+}
+
+static void icc_bridge_cpu_add_req(Notifier *n, void *opaque)
+{
+    ICCBridgeState *s = container_of(n, ICCBridgeState, cpu_add_notifier);
+
+    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
+}
+
 static void icc_bridge_initfn(Object *obj)
 {
     ICCBridgeState *s = ICC_BRIGDE(obj);
     SysBusDevice *sb = SYS_BUS_DEVICE(obj);
     ICCBus *ibus;
+    int i;
 
     object_property_add_str(obj, "ioapic-type",
                              NULL,
@@ -102,12 +127,33 @@ static void icc_bridge_initfn(Object *obj)
     memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
     sysbus_init_mmio(sb, &s->ioapic_container);
     ibus->ioapic_address_space = &s->ioapic_container;
+
+    s->links = g_malloc0(sizeof(Object *) * max_cpus);
+    for (i = 0; i < max_cpus; i++) {
+        gchar *cpu_name;
+
+        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
+                                   x86_cpu_apic_id_from_index(i));
+        object_property_add_link(obj, cpu_name, TYPE_CPU, &s->links[i], NULL);
+        g_free(cpu_name);
+    }
+
+    s->cpu_add_notifier.notify = icc_bridge_cpu_add_req;
+    qemu_register_cpu_added_notifier(&s->cpu_add_notifier);
+}
+
+static void icc_bridge_fini(Object *obj)
+{
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+
+    g_free(s->links);
 }
 
 static const TypeInfo icc_bridge_info = {
     .name  = "icc-bridge",
     .parent = TYPE_SYS_BUS_DEVICE,
     .instance_init  = icc_bridge_initfn,
+    .instance_finalize  = icc_bridge_fini,
     .instance_size  = sizeof(ICCBridgeState),
 };
 
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index 69a0278..bc31cd9 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
 
 #define TYPE_ICC_BRIDGE "icc-bridge"
 
+void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
+
 #endif /* CONFIG_USER_ONLY */
 #endif
-- 
1.8.1.4

[Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

... via current_machine->cpu_hot_add() hook called by cpu-set QMP command,
for x86 target.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v4:
  * merge "qmp: add cpu-add qmp command" & "target-i386: implement CPU hot-add" patches
  * move notifier call to CPUCLass.realize()
  * add hook cpu_hot_add to QEMUMachine
  * make QEMUMachineInitArgs global and keep default cpu_model there

v3:
  * it appears that 'online/offline' in cpu-set are confusing people
    with what command actually does and users might have to distinguish
    if 'offline' is not implemented by parsing error message. To simplify
    things replace cpu-set with cpu-add command to show more clear what
    command does and just add cpu-del when CPU remove is implemented.

v2:
  * s/cpu_set/cpu-set/
  * qmp doc style fix
  * use bool type instead of opencodding online/offline string
     suggested-by: Eric Blake <eblake@redhat.com>
---
 hw/boards.h            |  3 +++
 hw/i386/pc.c           | 20 ++++++++++++++++++++
 qapi-schema.json       | 11 +++++++++++
 qmp-commands.hx        | 23 +++++++++++++++++++++++
 qmp.c                  | 10 ++++++++++
 stubs/do_cpu_hot_add.c |  7 +++++++
 vl.c                   |  6 +++++-
 7 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 stubs/do_cpu_hot_add.c

diff --git a/hw/boards.h b/hw/boards.h
index 425bdc7..de8f92a 100644
--- a/hw/boards.h
+++ b/hw/boards.h
@@ -18,6 +18,8 @@ typedef struct QEMUMachineInitArgs {
     const char *cpu_model;
 } QEMUMachineInitArgs;
 
+extern QEMUMachineInitArgs *machine_args;
+
 typedef void QEMUMachineInitFunc(QEMUMachineInitArgs *args);
 
 typedef void QEMUMachineResetFunc(void);
@@ -43,6 +45,7 @@ typedef struct QEMUMachine {
     GlobalProperty *compat_props;
     struct QEMUMachine *next;
     const char *hw_version;
+    void (*hot_add_cpu)(const int64_t id, Error **errp);
 } QEMUMachine;
 
 int qemu_register_machine(QEMUMachine *m);
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 86e5365..1f94134 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -53,6 +53,7 @@
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
 #include "hw/icc_bus.h"
+#include "hw/boards.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -896,6 +897,23 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
     return cpu;
 }
 
+static void do_cpu_hot_add(const int64_t id, Error **errp)
+{
+    if (cpu_exists(id)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", it already exists", id);
+        return;
+    }
+
+    if (id >= pc_apic_id_limit(max_cpus)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", max allowed: 0x%x", id, pc_apic_id_limit(max_cpus) - 1);
+        return;
+    }
+
+    pc_new_cpu(machine_args->cpu_model, id, NULL, errp);
+}
+
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
@@ -910,7 +928,9 @@ void pc_cpus_init(const char *cpu_model)
 #else
         cpu_model = "qemu32";
 #endif
+        machine_args->cpu_model = cpu_model;
     }
+    current_machine->hot_add_cpu = do_cpu_hot_add;
 
     ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
                                                  TYPE_ICC_BRIDGE, NULL));
diff --git a/qapi-schema.json b/qapi-schema.json
index db542f6..a760ed5 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1387,6 +1387,17 @@
 { 'command': 'cpu', 'data': {'index': 'int'} }
 
 ##
+# @cpu-add
+#
+# Adds CPU with specified id
+#
+# @id: cpu id of CPU to be created
+#
+# Returns: Nothing on success
+##
+{ 'command': 'cpu-add', 'data': {'id': 'int'} }
+
+##
 # @memsave:
 #
 # Save a portion of guest memory to a file.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 1e0e11e..79b2ba7 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -407,6 +407,29 @@ Example:
 EQMP
 
     {
+        .name       = "cpu-add",
+        .args_type  = "id:i",
+        .mhandler.cmd_new = qmp_marshal_input_cpu_add,
+    },
+
+SQMP
+cpu-add
+-------
+
+Adds virtual cpu
+
+Arguments:
+
+- "id": cpu id (json-int)
+
+Example:
+
+-> { "execute": "cpu-add", "arguments": { "id": 2 } }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "memsave",
         .args_type  = "val:l,size:i,filename:s,cpu:i?",
         .mhandler.cmd_new = qmp_marshal_input_memsave,
diff --git a/qmp.c b/qmp.c
index 55b056b..8a9fd9e 100644
--- a/qmp.c
+++ b/qmp.c
@@ -24,6 +24,7 @@
 #include "hw/qdev.h"
 #include "sysemu/blockdev.h"
 #include "qom/qom-qobject.h"
+#include "hw/boards.h"
 
 NameInfo *qmp_query_name(Error **errp)
 {
@@ -108,6 +109,15 @@ void qmp_cpu(int64_t index, Error **errp)
     /* Just do nothing */
 }
 
+void qmp_cpu_add(int64_t id, Error **errp)
+{
+    if (current_machine->hot_add_cpu) {
+        current_machine->hot_add_cpu(id, errp);
+    } else {
+        error_setg(errp, "Not supported");
+    }
+}
+
 #ifndef CONFIG_VNC
 /* If VNC support is enabled, the "true" query-vnc command is
    defined in the VNC subsystem */
diff --git a/stubs/do_cpu_hot_add.c b/stubs/do_cpu_hot_add.c
new file mode 100644
index 0000000..1f6d7a6
--- /dev/null
+++ b/stubs/do_cpu_hot_add.c
@@ -0,0 +1,7 @@
+#include "qapi/error.h"
+#include "sysemu/sysemu.h"
+
+void do_cpu_hot_add(const int64_t id, Error **errp)
+{
+    error_setg(errp, "Not implemented");
+}
diff --git a/vl.c b/vl.c
index 97f0349..b2ea58c 100644
--- a/vl.c
+++ b/vl.c
@@ -179,6 +179,8 @@ int main(int argc, char **argv)
 #define MAX_VIRTIO_CONSOLES 1
 #define MAX_SCLP_CONSOLES 1
 
+QEMUMachineInitArgs *machine_args;
+
 static const char *data_dir[16];
 static int data_dir_idx;
 const char *bios_name = NULL;
@@ -4310,13 +4312,15 @@ int main(int argc, char **argv, char **envp)
                                  .kernel_cmdline = kernel_cmdline,
                                  .initrd_filename = initrd_filename,
                                  .cpu_model = cpu_model };
+    machine_args = &args;
+    current_machine = machine;
+
     machine->init(&args);
 
     cpu_synchronize_all_post_init();
 
     set_numa_modes();
 
-    current_machine = machine;
 
     /* init USB devices */
     if (usb_enabled(false)) {
-- 
1.8.1.4

Re: [Qemu-devel] [PATCH 18/22] target-i386: move APIC to ICC bus

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:37:10PM +0200, Igor Mammedov wrote:
[...]
>  void pc_cpus_init(const char *cpu_model)
>  {
>      int i;
> +    X86CPU *cpu;
>      Error *error = NULL;
> +    SysBusDevice *ib;
>  
>      /* init CPUs */
>      if (cpu_model == NULL) {
> @@ -902,14 +907,22 @@ void pc_cpus_init(const char *cpu_model)
>  #endif
>      }
>  
> +    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
> +                                                 TYPE_ICC_BRIDGE, NULL));
> +
>      for (i = 0; i < smp_cpus; i++) {
> -        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
>          if (error) {
>              fprintf(stderr, "%s\n", error_get_pretty(error));
>              error_free(error);
>              exit(1);
>          }
>      }
> +
> +    /* map APIC MMIO area if CPU has it */
> +    if (cpu->env.apic_staiy) {
> +        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
> +    }

GCC doesn't like this:

make[1]: Entering directory `/home/ehabkost/rh/proj/virt/qemu/x86_64-softmmu'
cc -I. -I/home/ehabkost/rh/proj/virt/qemu -I/home/ehabkost/rh/proj/virt/qemu/include -I/home/ehabkost/rh/proj/virt/qemu/linux-headers -I/home/ehabkost/rh/proj/virt/qemu/tcg -I/home/ehabkost/rh/proj/virt/qemu/tcg/i386  -Werror -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing  -fstack-protector-all -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -I/usr/include/pixman-1   -DHAS_AUDIO -DHAS_AUDIO_CHOICE  -I/home/ehabkost/rh/proj/virt/qemu/hw/i386 -Ihw/i386 -I../linux-headers -I.. -I/home/ehabkost/rh/proj/virt/qemu/target-i386 -DNEED_CPU_H -I/home/ehabkost/rh/proj/virt/qemu/include -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -MMD -MP -MT hw/i386/pc.o -MF hw/i386/pc.d -O2 -D_FORTIFY_SOURCE=2 -g  -c -o hw/i386/pc.o /home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c
/home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c: In function ‘pc_cpus_init’:
/home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c:948:17: error: ‘cpu’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
cc1: all warnings being treated as errors
make[1]: *** [hw/i386/pc.o] Error 1


>  }
>  

-- 
Eduardo

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
[...]
> diff --git a/qapi-schema.json b/qapi-schema.json
> index db542f6..a760ed5 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -1387,6 +1387,17 @@
>  { 'command': 'cpu', 'data': {'index': 'int'} }
>  
>  ##
> +# @cpu-add
> +#
> +# Adds CPU with specified id
> +#
> +# @id: cpu id of CPU to be created

Can we have the semantics/constraints of "id" documented here? Is it an
arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
it have to be the index of the CPU in the CPU list? How the IDs of
existing CPUs set using "-smp" are allocated?

I am looking at the code right now to understand how this implementation
works, but the documentation could contain or point to documentation on
how the "id" parameter is used and interpreted.

> +#
> +# Returns: Nothing on success
> +##
> +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> +
> +##
-- 
Eduardo

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 02:10:54PM -0300, Eduardo Habkost wrote:
> On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index db542f6..a760ed5 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -1387,6 +1387,17 @@
> >  { 'command': 'cpu', 'data': {'index': 'int'} }
> >  
> >  ##
> > +# @cpu-add
> > +#
> > +# Adds CPU with specified id
> > +#
> > +# @id: cpu id of CPU to be created
> 
> Can we have the semantics/constraints of "id" documented here? Is it an
> arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> it have to be the index of the CPU in the CPU list? How the IDs of
> existing CPUs set using "-smp" are allocated?
> 
> I am looking at the code right now to understand how this implementation
> works, but the documentation could contain or point to documentation on
> how the "id" parameter is used and interpreted.

So, the answer to my own question seems to be on patch 21/22. Let me
check if I understand this correctly:

 @id: cpu id of the CPU to be created. It must be one of the the
      available IDs listed at the "/machine/icc-bridge/cpu[0..N]" links.

Is the above correct?

Now, my question is: suppose a caller starts QEMU as:

 $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
        -numa node,cpus=0-8   -numa node,cpus=9-17 \
        -numa node,cpus=18-26 -numa node,cpus=27-35

and the caller wants to hot-add all VCPUs in the last CPU socket (that
means: all the VCPUs in the last NUMA node, that means: CPU indexes
27-35). What should the caller do to find out which of the
/machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
VCPUs?


> 
> > +#
> > +# Returns: Nothing on success
> > +##
> > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > +
> > +##
> -- 
> Eduardo
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 18/22] target-i386: move APIC to ICC bus

[Igor Mammedov]

On Fri, 5 Apr 2013 13:15:03 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 04:37:10PM +0200, Igor Mammedov wrote:
> [...]
> >  void pc_cpus_init(const char *cpu_model)
> >  {
> >      int i;
> > +    X86CPU *cpu;
> >      Error *error = NULL;
> > +    SysBusDevice *ib;
> >  
> >      /* init CPUs */
> >      if (cpu_model == NULL) {
> > @@ -902,14 +907,22 @@ void pc_cpus_init(const char *cpu_model)
> >  #endif
> >      }
> >  
> > +    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
> > +                                                 TYPE_ICC_BRIDGE, NULL));
> > +
> >      for (i = 0; i < smp_cpus; i++) {
> > -        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> >          if (error) {
> >              fprintf(stderr, "%s\n", error_get_pretty(error));
> >              error_free(error);
> >              exit(1);
> >          }
> >      }
> > +
> > +    /* map APIC MMIO area if CPU has it */
> > +    if (cpu->env.apic_staiy) {
> > +        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
> > +    }
> 
> GCC doesn't like this:
> 
> make[1]: Entering directory `/home/ehabkost/rh/proj/virt/qemu/x86_64-softmmu'
> cc -I. -I/home/ehabkost/rh/proj/virt/qemu -I/home/ehabkost/rh/proj/virt/qemu/include -I/home/ehabkost/rh/proj/virt/qemu/linux-headers -I/home/ehabkost/rh/proj/virt/qemu/tcg -I/home/ehabkost/rh/proj/virt/qemu/tcg/i386  -Werror -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing  -fstack-protector-all -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -I/usr/include/pixman-1   -DHAS_AUDIO -DHAS_AUDIO_CHOICE  -I/home/ehabkost/rh/proj/virt/qemu/hw/i386 -Ihw/i386 -I../linux-headers -I.. -I/home/ehabkost/rh/proj/virt/qemu/target-i386 -DNEED_CPU_H -I/home/ehabkost/rh/proj/virt/qemu/include -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -MMD -MP -MT hw/i386/pc.o -MF hw/i386/pc.d -O2 -D_FORTIFY_SOURCE=2 -g  -c -o hw/i386/pc.o /home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c
> /home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c: In function ‘pc_cpus_init’:
> /home/ehabkost/rh/proj/virt/qemu/hw/i386/pc.c:948:17: error: ‘cpu’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
> cc1: all warnings being treated as errors
> make[1]: *** [hw/i386/pc.o] Error 1

I've tested it --enable-debug with which it compiles ok even wit
--enable-werror.
Thanks for finding it out though, I've  updated
https://github.com/imammedo/qemu/tree/cpu_add.v2 and
https://github.com/imammedo/qemu/tree/cpu_set.WIP with a fix and will post
fixed patch as reply to original one.

> 
> 
> >  }
> >  
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

[Qemu-devel] [PATCH 18/22] target-i386: move APIC to ICC bus

[Igor Mammedov]

... to allow it to be hotplugged

 * map APIC's mmio at board level if it is present

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * fix compile error "cpu may be used uninitialized"
     with --enable-werror but without --enable-debug.
---
 hw/apic_common.c   | 17 ++++++++++++-----
 hw/apic_internal.h |  6 +++---
 hw/i386/kvmvapic.c |  1 +
 hw/i386/pc.c       | 19 ++++++++++++++++---
 hw/icc_bus.c       | 15 ++++++++++++++-
 hw/icc_bus.h       |  3 ++-
 target-i386/cpu.c  | 16 +++-------------
 7 files changed, 51 insertions(+), 26 deletions(-)

diff --git a/hw/apic_common.c b/hw/apic_common.c
index 3798509..b2e84e6 100644
--- a/hw/apic_common.c
+++ b/hw/apic_common.c
@@ -21,6 +21,8 @@
 #include "hw/apic_internal.h"
 #include "trace.h"
 #include "sysemu/kvm.h"
+#include "qdev.h"
+#include "hw/sysbus.h"
 
 static int apic_irq_delivered;
 bool apic_report_tpr_access;
@@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque, int version_id)
     return 0;
 }
 
-static int apic_init_common(SysBusDevice *dev)
+static int apic_init_common(ICCDevice *dev)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
     static int apic_no;
+    static bool mmio_registered;
 
     if (apic_no >= MAX_APICS) {
         return -1;
@@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
 
     info = APIC_COMMON_GET_CLASS(s);
     info->init(s);
-
-    sysbus_init_mmio(dev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
 
     /* Note: We need at least 1M to map the VAPIC option ROM */
     if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
@@ -375,7 +382,7 @@ static Property apic_properties_common[] = {
 
 static void apic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     dc->vmsd = &vmstate_apic_common;
@@ -387,7 +394,7 @@ static void apic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo apic_common_type = {
     .name = TYPE_APIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(APICCommonState),
     .class_size = sizeof(APICCommonClass),
     .class_init = apic_common_class_init,
diff --git a/hw/apic_internal.h b/hw/apic_internal.h
index aac6290..172fc91 100644
--- a/hw/apic_internal.h
+++ b/hw/apic_internal.h
@@ -21,7 +21,7 @@
 #define QEMU_APIC_INTERNAL_H
 
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/icc_bus.h"
 #include "qemu/timer.h"
 
 /* APIC Local Vector Table */
@@ -78,7 +78,7 @@ typedef struct APICCommonState APICCommonState;
 
 typedef struct APICCommonClass
 {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
 
     void (*init)(APICCommonState *s);
     void (*set_base)(APICCommonState *s, uint64_t val);
@@ -92,7 +92,7 @@ typedef struct APICCommonClass
 } APICCommonClass;
 
 struct APICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
 
     MemoryRegion io_memory;
     X86CPU *cpu;
diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
index c4be882..81e0a75 100644
--- a/hw/i386/kvmvapic.c
+++ b/hw/i386/kvmvapic.c
@@ -12,6 +12,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/kvm.h"
 #include "hw/apic_internal.h"
+#include "hw/sysbus.h"
 
 #define VAPIC_IO_PORT           0x7e
 
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 9d617b4..7925a41 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -52,6 +52,7 @@
 #include "sysemu/arch_init.h"
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
+#include "hw/icc_bus.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -869,13 +870,13 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
 {
     X86CPU *cpu;
 
     cpu = cpu_x86_create(cpu_model, errp);
     if (!cpu) {
-        return;
+        return cpu;
     }
 
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
@@ -884,14 +885,18 @@ static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
+            cpu = NULL;
         }
     }
+    return cpu;
 }
 
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    X86CPU *cpu = NULL;
     Error *error = NULL;
+    SysBusDevice *ib;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -902,14 +907,22 @@ void pc_cpus_init(const char *cpu_model)
 #endif
     }
 
+    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
+                                                 TYPE_ICC_BRIDGE, NULL));
+
     for (i = 0; i < smp_cpus; i++) {
-        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
             exit(1);
         }
     }
+
+    /* map APIC MMIO area if CPU has it */
+    if (cpu && cpu->env.apic_state) {
+        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
+    }
 }
 
 void pc_acpi_init(const char *default_dsdt)
diff --git a/hw/icc_bus.c b/hw/icc_bus.c
index 7e75a0e..a0c4e02 100644
--- a/hw/icc_bus.c
+++ b/hw/icc_bus.c
@@ -59,13 +59,26 @@ static const TypeInfo icc_device_info = {
 
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
+    MemoryRegion apic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
 
 static void icc_bridge_initfn(Object *obj)
 {
-    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+    SysBusDevice *sb = SYS_BUS_DEVICE(obj);
+    ICCBus *ibus;
+
+    ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
+
+    /* Do not change order of registering regions,
+     * APIC must be first registered region, board maps it by 0 index
+     */
+    memory_region_init(&s->apic_container, "icc-apic-container",
+                       APIC_SPACE_SIZE);
+    sysbus_init_mmio(sb, &s->apic_container);
+    ibus->apic_address_space = &s->apic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/icc_bus.h b/hw/icc_bus.h
index f959a43..2c8f78f 100644
--- a/hw/icc_bus.h
+++ b/hw/icc_bus.h
@@ -19,6 +19,7 @@
 #ifndef ICC_BUS_H
 #define ICC_BUS_H
 
+#include "exec/memory.h"
 #include "hw/qdev-core.h"
 
 #define TYPE_ICC_BUS "icc-bus"
@@ -26,6 +27,7 @@
 #ifndef CONFIG_USER_ONLY
 typedef struct ICCBus {
     BusState qbus;
+    MemoryRegion *apic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
@@ -48,4 +50,3 @@ typedef struct ICCDeviceClass {
 
 #endif /* CONFIG_USER_ONLY */
 #endif
-
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 2b3c1f3..32f7aea 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,10 +41,10 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/qdev-properties.h"
 #include "hw/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen.h"
-#include "hw/sysbus.h"
 #include "hw/apic_internal.h"
 #endif
 
@@ -2104,6 +2104,7 @@ static void mce_init(X86CPU *cpu)
 static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
+    DeviceState *dev = DEVICE(cpu);
     APICCommonState *apic;
     const char *apic_type = "apic";
 
@@ -2113,7 +2114,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
         apic_type = "xen-apic";
     }
 
-    env->apic_state = qdev_try_create(NULL, apic_type);
+    env->apic_state = qdev_try_create(dev->parent_bus, apic_type);
     if (env->apic_state == NULL) {
         error_setg(errp, "APIC device '%s' could not be created", apic_type);
         return;
@@ -2130,7 +2131,6 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
-    static int apic_mapped;
 
     if (env->apic_state == NULL) {
         return;
@@ -2141,16 +2141,6 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
                    object_get_typename(OBJECT(env->apic_state)));
         return;
     }
-
-    /* XXX: mapping more APICs at the same memory location */
-    if (apic_mapped == 0) {
-        /* NOTE: the APIC is directly connected to the CPU - it is not
-           on the global memory bus. */
-        /* XXX: what if the base changes? */
-        sysbus_mmio_map_overlap(SYS_BUS_DEVICE(env->apic_state), 0,
-                                APIC_DEFAULT_ADDRESS, 0x1000);
-        apic_mapped = 1;
-    }
 }
 #else
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
-- 
1.8.1.4

Re: [Qemu-devel] [PATCH 11/22] cpu: introduce get_firmware_id() method and override it for target-i386

[li guang]

在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> get_firmware_id() adds possibily for generic code to get guest visible
> CPI id without accessing CPUArchState. If target doesn't override it,
> it will return cpu_index.
> 
> Override it on target-i386 to return APIC ID.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  * it will be used later by new cpu_exists() generic function and
>    acpi_piix.
> ---
>  include/qom/cpu.h |  4 ++--
>  qom/cpu.c         |  6 ++++++
>  target-i386/cpu.c | 10 ++++++++++
>  3 files changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 210aca3..0d33009 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -46,7 +46,7 @@ typedef struct CPUState CPUState;
>   * @reset: Callback to reset the #CPUState to its initial state.
>   * @do_interrupt: Callback for interrupt handling.
>   * @resume: Callback for putting CPU in runable state
> - * @get_firmware_id: Callback for getting arch depended CPU id
> + * @get_firmware_id: Callback for getting arch depended CPU ID

get_firmware_id ?
seems obscure to me.
can't it be get_arch_cpuid?

>   * @vmsd: State description for migration.
>   *
>   * Represents a CPU family or model.
> @@ -61,7 +61,7 @@ typedef struct CPUClass {
>      void (*reset)(CPUState *cpu);
>      void (*do_interrupt)(CPUState *cpu);
>      void (*resume)(CPUState *cpu);
> -    void (*get_firmware_id)(CPUState *cpu);
> +    int64_t (*get_firmware_id)(CPUState *cpu);
>  
>      const struct VMStateDescription *vmsd;
>  } CPUClass;
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 10ceaed..a54d4d1 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -80,6 +80,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>      }
>  }
>  
> +static int64_t cpu_common_get_firmware_id(CPUState *cpu)
> +{
> +    return cpu->cpu_index;
> +}
> +
>  static void cpu_class_init(ObjectClass *klass, void *data)
>  {
>      DeviceClass *dc = DEVICE_CLASS(klass);
> @@ -87,6 +92,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
>  
>      k->class_by_name = cpu_common_class_by_name;
>      k->reset = cpu_common_reset;
> +    k->get_firmware_id = cpu_common_get_firmware_id;
>      dc->realize = cpu_common_realizefn;
>      dc->no_user = 1;
>  }
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 269a681..858fd54 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -2269,6 +2269,14 @@ static void x86_cpu_initfn(Object *obj)
>      }
>  }
>  
> +static int64_t x86_cpu_get_firmware_id(CPUState *cpu)
> +{
> +    X86CPU *x86cpu = X86_CPU(cpu);
> +    CPUX86State *env = &x86cpu->env;
> +
> +    return env->cpuid_apic_id;
> +}
> +
>  static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>  {
>      X86CPUClass *xcc = X86_CPU_CLASS(oc);
> @@ -2283,6 +2291,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>  
>      cc->do_interrupt = x86_cpu_do_interrupt;
>      cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
> +
> +    cc->get_firmware_id = x86_cpu_get_firmware_id;
>  }
>  
>  static const TypeInfo x86_cpu_type_info = {

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[li guang]

This patch should be combined with [PATCH 07/22]

在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  hw/ioapic_common.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
> index d4aff29..561b987 100644
> --- a/hw/ioapic_common.c
> +++ b/hw/ioapic_common.c
> @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
>  
>  static int ioapic_init_common(SysBusDevice *dev)
>  {
> -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
> +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
>      IOAPICCommonClass *info;
>      static int ioapic_no;
>  

Re: [Qemu-devel] [PATCH 13/22] acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest

[li guang]

在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> * introduce processor status bitmask visible to guest at 0xaf00 addr,
>   where Seabios expects it
> * set bit corresponding to APIC ID in processor status bitmask on
>   receiving CPU hot-plug notification
> * trigger CPU hot-plug SCI, expected by Seabios on receiving CPU
>   hot-plug notification

expected by seabios?
IMHO, seems you are saying ACPI's asl code which
just wrapped in seabios.

> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * use CPUClass.get_firmware_id() to make code target independent
>   * bump up vmstate_acpi version
> ---
>  hw/acpi_piix4.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 111 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
> index 48a32b5..ccfc028 100644
> --- a/hw/acpi_piix4.c
> +++ b/hw/acpi_piix4.c
> @@ -48,19 +48,28 @@
>  #define PCI_EJ_BASE 0xae08
>  #define PCI_RMV_BASE 0xae0c
>  
> +#define PROC_BASE 0xaf00
> +#define PROC_LEN 32
> +
>  #define PIIX4_PCI_HOTPLUG_STATUS 2
> +#define PIIX4_CPU_HOTPLUG_STATUS 4
>  
>  struct pci_status {
>      uint32_t up; /* deprecated, maintained for migration compatibility */
>      uint32_t down;
>  };
>  
> +struct cpu_status {
> +    uint8_t sts[PROC_LEN];
> +};
> +
>  typedef struct PIIX4PMState {
>      PCIDevice dev;
>  
>      MemoryRegion io;
>      MemoryRegion io_gpe;
>      MemoryRegion io_pci;
> +    MemoryRegion io_cpu;
>      ACPIREGS ar;
>  
>      APMState apm;
> @@ -82,6 +91,9 @@ typedef struct PIIX4PMState {
>      uint8_t disable_s3;
>      uint8_t disable_s4;
>      uint8_t s4_val;
> +
> +    struct cpu_status gpe_cpu;
> +    Notifier cpu_add_notifier;

seems you named cpu_added_notifier at [PATCH 10/22],
should the notifer's name be unified?

>  } PIIX4PMState;
>  
>  static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
> @@ -100,8 +112,8 @@ static void pm_update_sci(PIIX4PMState *s)
>                     ACPI_BITMASK_POWER_BUTTON_ENABLE |
>                     ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
>                     ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
> -        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0])
> -          & PIIX4_PCI_HOTPLUG_STATUS) != 0);
> +        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0]) &
> +          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
>  
>      qemu_set_irq(s->irq, sci_level);
>      /* schedule a timer interruption if needed */
> @@ -257,6 +269,17 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
>      return ret;
>  }
>  
> +#define VMSTATE_CPU_STATUS_ARRAY(_field, _state)                             \
> + {                                                                           \
> +     .name       = (stringify(_field)),                                      \
> +     .version_id = 0,                                                        \
> +     .num        = PROC_LEN,                                                 \
> +     .info       = &vmstate_info_uint8,                                      \
> +     .size       = sizeof(uint8_t),                                          \
> +     .flags      = VMS_ARRAY,                                                \
> +     .offset     = vmstate_offset_array(_state, _field, uint8_t, PROC_LEN),  \
> + }
> +
>  /* qemu-kvm 1.2 uses version 3 but advertised as 2
>   * To support incoming qemu-kvm 1.2 migration, change version_id
>   * and minimum_version_id to 2 below (which breaks migration from
> @@ -265,7 +288,7 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
>   */
>  static const VMStateDescription vmstate_acpi = {
>      .name = "piix4_pm",
> -    .version_id = 3,
> +    .version_id = 4,
>      .minimum_version_id = 3,
>      .minimum_version_id_old = 1,
>      .load_state_old = acpi_load_old,
> @@ -281,6 +304,7 @@ static const VMStateDescription vmstate_acpi = {
>          VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
>          VMSTATE_STRUCT(pci0_status, PIIX4PMState, 2, vmstate_pci_status,
>                         struct pci_status),
> +        VMSTATE_CPU_STATUS_ARRAY(gpe_cpu.sts, PIIX4PMState),
>          VMSTATE_END_OF_LIST()
>      }
>  };
> @@ -585,6 +609,83 @@ static const MemoryRegionOps piix4_pci_ops = {
>      },
>  };
>  
> +static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned width)
> +{
> +    PIIX4PMState *s = opaque;
> +    struct cpu_status *cpus = &s->gpe_cpu;
> +    uint64_t val = cpus->sts[addr];
> +    return val;
> +}
> +
> +static void cpu_status_write(void *opaque, hwaddr addr, uint64_t data,
> +                             unsigned int size)
> +{
> +    /* TODO: implement VCPU removal on guest signal that CPU can be removed */
> +}
> +
> +static const MemoryRegionOps cpu_hotplug_ops = {
> +    .read = cpu_status_read,
> +    .write = cpu_status_write,
> +    .endianness = DEVICE_LITTLE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 1,
> +        .max_access_size = 1,
> +    },
> +};
> +
> +typedef enum {
> +    PLUG,
> +    UNPLUG,
> +} HotplugEventType;
> +
> +static void piix4_cpu_hotplug_req(PIIX4PMState *s, CPUState *cpu,
> +                                  HotplugEventType action)
> +{
> +    struct cpu_status *g = &s->gpe_cpu;
> +    ACPIGPE *gpe = &s->ar.gpe;
> +    CPUClass *k = CPU_GET_CLASS(cpu);
> +    int64_t cpu_id;
> +
> +    assert(s != NULL);
> +
> +    *gpe->sts = *gpe->sts | PIIX4_CPU_HOTPLUG_STATUS;
> +    cpu_id = k->get_firmware_id(CPU(cpu));
> +    if (action == PLUG) {
> +        g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
> +    } else {
> +        g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
> +    }
> +    pm_update_sci(s);
> +}
> +
> +static void piix4_cpu_add_req(Notifier *n, void *opaque)
> +{
> +    PIIX4PMState *s = container_of(n, PIIX4PMState, cpu_add_notifier);
> +    piix4_cpu_hotplug_req(s, CPU(opaque), PLUG);
> +}
> +
> +static int piix4_init_cpu_status(Object *obj, void *opaque)
> +{
> +    struct cpu_status *g = (struct cpu_status *)opaque;
> +    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
> +
> +    if (cpu_obj) {
> +        struct Error *error = NULL;
> +        CPUClass *k = CPU_GET_CLASS(cpu_obj);
> +        int64_t id = k->get_firmware_id(CPU(cpu_obj));
> +
> +        if (error) {
> +            fprintf(stderr, "failed to initilize CPU status for ACPI: %s\n",
> +                    error_get_pretty(error));
> +            error_free(error);
> +            abort();
> +        }
> +        g_assert((id / 8) < PROC_LEN);
> +        g->sts[id / 8] |= (1 << (id % 8));
> +    }
> +    return object_child_foreach(obj, piix4_init_cpu_status, opaque);
> +}
> +
>  static int piix4_device_hotplug(DeviceState *qdev, PCIDevice *dev,
>                                  PCIHotplugState state);
>  
> @@ -600,6 +701,13 @@ static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
>      memory_region_add_subregion(parent, PCI_HOTPLUG_ADDR,
>                                  &s->io_pci);
>      pci_bus_hotplug(bus, piix4_device_hotplug, &s->dev.qdev);
> +
> +    piix4_init_cpu_status(qdev_get_machine(), &s->gpe_cpu);
> +    memory_region_init_io(&s->io_cpu, &cpu_hotplug_ops, s, "apci-cpu-hotplug",
> +                          PROC_LEN);
> +    memory_region_add_subregion(parent, PROC_BASE, &s->io_cpu);
> +    s->cpu_add_notifier.notify = piix4_cpu_add_req;
> +    qemu_register_cpu_added_notifier(&s->cpu_add_notifier);
>  }
>  
>  static void enable_device(PIIX4PMState *s, int slot)

Re: [Qemu-devel] [PATCH 02/22] target-i386: split APIC creation from initialization in x86_cpu_realizefn()

[li guang]

Reviewed-by: liguang <lig.fnst@cn.fujitsu.com>

在 2013-04-05五的 16:36 +0200，Igor Mammedov写道：
> When APIC is hotplugged during CPU hotplug, device_set_realized()
> calls device_reset() on it. And if QEMU runs in KVM mode, following
> call chain will fail:
>     apic_reset_common()
>         -> kvm_apic_vapic_base_update()
>             -> kvm_vcpu_ioctl(cpu->kvm_fd,...)
> due to cpu->kvm_fd not being initialized yet.
> 
> cpu->kvm_fd is initialized during qemu_init_vcpu() call but x86_cpu_apic_init()
> can't be moved after it because kvm_init_vcpu() -> kvm_arch_reset_vcpu()
> relies on APIC to determine if CPU is BSP for setting initial env->mp_state.
> 
> So split APIC device creation from its initialization and realize APIC
> after CPU is created, when it's safe to call APIC's reset method.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * s/x86_cpu_apic_init()/x86_cpu_apic_realize()/
> ---
>  target-i386/cpu.c | 24 +++++++++++++++++++++---
>  1 file changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 8b348d0..41f0f47 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -2050,9 +2050,8 @@ static void mce_init(X86CPU *cpu)
>  }
>  
>  #ifndef CONFIG_USER_ONLY
> -static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
> +static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
>  {
> -    static int apic_mapped;
>      CPUX86State *env = &cpu->env;
>      APICCommonState *apic;
>      const char *apic_type = "apic";
> @@ -2075,6 +2074,16 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
>      /* TODO: convert to link<> */
>      apic = APIC_COMMON(env->apic_state);
>      apic->cpu = cpu;
> +}
> +
> +static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
> +{
> +    CPUX86State *env = &cpu->env;
> +    static int apic_mapped;
> +
> +    if (env->apic_state == NULL) {
> +        return;
> +    }
>  
>      if (qdev_init(env->apic_state)) {
>          error_setg(errp, "APIC device '%s' could not be initialized",
> @@ -2092,6 +2101,10 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
>          apic_mapped = 1;
>      }
>  }
> +#else
> +static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
> +{
> +}
>  #endif
>  
>  static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
> @@ -2142,7 +2155,7 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
>      qemu_register_reset(x86_cpu_machine_reset_cb, cpu);
>  
>      if (cpu->env.cpuid_features & CPUID_APIC || smp_cpus > 1) {
> -        x86_cpu_apic_init(cpu, &local_err);
> +        x86_cpu_apic_create(cpu, &local_err);
>          if (local_err != NULL) {
>              goto out;
>          }
> @@ -2151,6 +2164,11 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
>  
>      mce_init(cpu);
>      qemu_init_vcpu(&cpu->env);
> +
> +    x86_cpu_apic_realize(cpu, &local_err);
> +    if (local_err != NULL) {
> +        goto out;
> +    }
>      cpu_reset(CPU(cpu));
>  
>      xcc->parent_realize(dev, &local_err);

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[li guang]

Reviewed-by: liguang <lig.fnst@cn.fujitsu.com>

在 2013-04-05五的 16:36 +0200，Igor Mammedov写道：
> ... and call it if defined from CPUClass.realize() if CPU was hotplugged
> 
> by default leave .resume() unset (i.e. NULL) and override it for softmmu
> in qemu_init_vcpu() if it's still unset.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  cpus.c            | 15 ++++++++++++---
>  include/qom/cpu.h |  2 ++
>  qom/cpu.c         |  5 +++++
>  3 files changed, 19 insertions(+), 3 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 9b9a32f..6b793c5 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
>      }
>  }
>  
> +static void resume_vcpu(CPUState *cpu)
> +{
> +    cpu->stop = false;
> +    cpu->stopped = false;
> +    qemu_cpu_kick(cpu);
> +}
> +
>  void resume_all_vcpus(void)
>  {
>      CPUArchState *penv = first_cpu;
> @@ -980,9 +987,7 @@ void resume_all_vcpus(void)
>      qemu_clock_enable(vm_clock, true);
>      while (penv) {
>          CPUState *pcpu = ENV_GET_CPU(penv);
> -        pcpu->stop = false;
> -        pcpu->stopped = false;
> -        qemu_cpu_kick(pcpu);
> +        resume_vcpu(pcpu);
>          penv = penv->next_cpu;
>      }
>  }
> @@ -1042,7 +1047,11 @@ void qemu_init_vcpu(void *_env)
>  {
>      CPUArchState *env = _env;
>      CPUState *cpu = ENV_GET_CPU(env);
> +    CPUClass *klass = CPU_GET_CLASS(cpu);
>  
> +    if (klass->resume == NULL) {
> +        klass->resume = resume_vcpu;
> +    }
>      cpu->nr_cores = smp_cores;
>      cpu->nr_threads = smp_threads;
>      cpu->stopped = true;
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 3664a1b..6d6eb7a 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
>   * instantiatable CPU type.
>   * @reset: Callback to reset the #CPUState to its initial state.
>   * @do_interrupt: Callback for interrupt handling.
> + * @resume: Callback for putting CPU in runable state
>   * @vmsd: State description for migration.
>   *
>   * Represents a CPU family or model.
> @@ -58,6 +59,7 @@ typedef struct CPUClass {
>  
>      void (*reset)(CPUState *cpu);
>      void (*do_interrupt)(CPUState *cpu);
> +    void (*resume)(CPUState *cpu);
>  
>      const struct VMStateDescription *vmsd;
>  } CPUClass;
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 0c76712..c062e00 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -58,8 +58,13 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
>  
>  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>  {
> +    CPUClass *klass = CPU_GET_CLASS(dev);
> +
>      if (dev->hotplugged) {
>          cpu_synchronize_post_init(CPU(dev));
> +        if (klass->resume != NULL) {
> +            klass->resume(CPU(dev));
> +        }
>      }
>  }
>  

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Igor Mammedov]

On Mon, 08 Apr 2013 10:13:28 +0800
li guang <lig.fnst@cn.fujitsu.com> wrote:

> This patch should be combined with [PATCH 07/22]
> 
> 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  hw/ioapic_common.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
> > index d4aff29..561b987 100644
> > --- a/hw/ioapic_common.c
> > +++ b/hw/ioapic_common.c
> > @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int
> > version_id) 
> >  static int ioapic_init_common(SysBusDevice *dev)
> >  {
> > -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
> > +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
> >      IOAPICCommonClass *info;
> >      static int ioapic_no;
> >  
> 
> 
> 

sure, will do this on the next respin.

Re: [Qemu-devel] [PATCH 11/22] cpu: introduce get_firmware_id() method and override it for target-i386

[Igor Mammedov]

On Mon, 08 Apr 2013 10:02:21 +0800
li guang <lig.fnst@cn.fujitsu.com> wrote:

> 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> > get_firmware_id() adds possibily for generic code to get guest visible
> > CPI id without accessing CPUArchState. If target doesn't override it,
> > it will return cpu_index.
> > 
> > Override it on target-i386 to return APIC ID.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  * it will be used later by new cpu_exists() generic function and
> >    acpi_piix.
> > ---
> >  include/qom/cpu.h |  4 ++--
> >  qom/cpu.c         |  6 ++++++
> >  target-i386/cpu.c | 10 ++++++++++
> >  3 files changed, 18 insertions(+), 2 deletions(-)
> > 
> > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > index 210aca3..0d33009 100644
> > --- a/include/qom/cpu.h
> > +++ b/include/qom/cpu.h
> > @@ -46,7 +46,7 @@ typedef struct CPUState CPUState;
> >   * @reset: Callback to reset the #CPUState to its initial state.
> >   * @do_interrupt: Callback for interrupt handling.
> >   * @resume: Callback for putting CPU in runable state
> > - * @get_firmware_id: Callback for getting arch depended CPU id
> > + * @get_firmware_id: Callback for getting arch depended CPU ID
> 
> get_firmware_id ?
> seems obscure to me.
> can't it be get_arch_cpuid?

*_cpuid might be confused with cpuid term on x86, not sure athe it would be
better. Perhaps then get_{arch|}_id() will be sufficient, since it's a method
of CPUClass and CPU is implied.

Re: [Qemu-devel] [PATCH 13/22] acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest

[Igor Mammedov]

On Mon, 08 Apr 2013 10:24:50 +0800
li guang <lig.fnst@cn.fujitsu.com> wrote:

> 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> > * introduce processor status bitmask visible to guest at 0xaf00 addr,
> >   where Seabios expects it
> > * set bit corresponding to APIC ID in processor status bitmask on
> >   receiving CPU hot-plug notification
> > * trigger CPU hot-plug SCI, expected by Seabios on receiving CPU
> >   hot-plug notification
> 
> expected by seabios?
> IMHO, seems you are saying ACPI's asl code which
> just wrapped in seabios.
Yep, that's what ACPI asl code there expects. With current effort to move
ACPI tables into it might be better to s/Seabios/ACPI/ here.

> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> > v2:
> >   * use CPUClass.get_firmware_id() to make code target independent
> >   * bump up vmstate_acpi version
> > ---
> >  hw/acpi_piix4.c | 114
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed,
> > 111 insertions(+), 3 deletions(-)
> > 
> > diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
> > index 48a32b5..ccfc028 100644
> > --- a/hw/acpi_piix4.c
> > +++ b/hw/acpi_piix4.c
> > @@ -48,19 +48,28 @@
> >  #define PCI_EJ_BASE 0xae08
> >  #define PCI_RMV_BASE 0xae0c
> >  
> > +#define PROC_BASE 0xaf00
> > +#define PROC_LEN 32
> > +
> >  #define PIIX4_PCI_HOTPLUG_STATUS 2
> > +#define PIIX4_CPU_HOTPLUG_STATUS 4
> >  
> >  struct pci_status {
> >      uint32_t up; /* deprecated, maintained for migration compatibility */
> >      uint32_t down;
> >  };
> >  
> > +struct cpu_status {
> > +    uint8_t sts[PROC_LEN];
> > +};
> > +
> >  typedef struct PIIX4PMState {
> >      PCIDevice dev;
> >  
> >      MemoryRegion io;
> >      MemoryRegion io_gpe;
> >      MemoryRegion io_pci;
> > +    MemoryRegion io_cpu;
> >      ACPIREGS ar;
> >  
> >      APMState apm;
> > @@ -82,6 +91,9 @@ typedef struct PIIX4PMState {
> >      uint8_t disable_s3;
> >      uint8_t disable_s4;
> >      uint8_t s4_val;
> > +
> > +    struct cpu_status gpe_cpu;
> > +    Notifier cpu_add_notifier;
> 
> seems you named cpu_added_notifier at [PATCH 10/22],
> should the notifer's name be unified?
I'll fix it on next respin, thanks.

> 
> >  } PIIX4PMState;
> >  
> >  static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
> > @@ -100,8 +112,8 @@ static void pm_update_sci(PIIX4PMState *s)
> >                     ACPI_BITMASK_POWER_BUTTON_ENABLE |
> >                     ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
> >                     ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
> > -        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0])
> > -          & PIIX4_PCI_HOTPLUG_STATUS) != 0);
> > +        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0]) &
> > +          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
> >  
> >      qemu_set_irq(s->irq, sci_level);
> >      /* schedule a timer interruption if needed */
> > @@ -257,6 +269,17 @@ static int acpi_load_old(QEMUFile *f, void *opaque,
> > int version_id) return ret;
> >  }
> >  
> > +#define VMSTATE_CPU_STATUS_ARRAY(_field,
> > _state)                             \
> > +
> > {                                                                           \
> > +     .name       =
> > (stringify(_field)),                                      \
> > +     .version_id =
> > 0,                                                        \
> > +     .num        =
> > PROC_LEN,                                                 \
> > +     .info       =
> > &vmstate_info_uint8,                                      \
> > +     .size       =
> > sizeof(uint8_t),                                          \
> > +     .flags      =
> > VMS_ARRAY,                                                \
> > +     .offset     = vmstate_offset_array(_state, _field, uint8_t,
> > PROC_LEN),  \
> > + }
> > +
> >  /* qemu-kvm 1.2 uses version 3 but advertised as 2
> >   * To support incoming qemu-kvm 1.2 migration, change version_id
> >   * and minimum_version_id to 2 below (which breaks migration from
> > @@ -265,7 +288,7 @@ static int acpi_load_old(QEMUFile *f, void *opaque,
> > int version_id) */
> >  static const VMStateDescription vmstate_acpi = {
> >      .name = "piix4_pm",
> > -    .version_id = 3,
> > +    .version_id = 4,
> >      .minimum_version_id = 3,
> >      .minimum_version_id_old = 1,
> >      .load_state_old = acpi_load_old,
> > @@ -281,6 +304,7 @@ static const VMStateDescription vmstate_acpi = {
> >          VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
> >          VMSTATE_STRUCT(pci0_status, PIIX4PMState, 2, vmstate_pci_status,
> >                         struct pci_status),
> > +        VMSTATE_CPU_STATUS_ARRAY(gpe_cpu.sts, PIIX4PMState),
> >          VMSTATE_END_OF_LIST()
> >      }
> >  };
> > @@ -585,6 +609,83 @@ static const MemoryRegionOps piix4_pci_ops = {
> >      },
> >  };
> >  
> > +static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned
> > width) +{
> > +    PIIX4PMState *s = opaque;
> > +    struct cpu_status *cpus = &s->gpe_cpu;
> > +    uint64_t val = cpus->sts[addr];
> > +    return val;
> > +}
> > +
> > +static void cpu_status_write(void *opaque, hwaddr addr, uint64_t data,
> > +                             unsigned int size)
> > +{
> > +    /* TODO: implement VCPU removal on guest signal that CPU can be
> > removed */ +}
> > +
> > +static const MemoryRegionOps cpu_hotplug_ops = {
> > +    .read = cpu_status_read,
> > +    .write = cpu_status_write,
> > +    .endianness = DEVICE_LITTLE_ENDIAN,
> > +    .valid = {
> > +        .min_access_size = 1,
> > +        .max_access_size = 1,
> > +    },
> > +};
> > +
> > +typedef enum {
> > +    PLUG,
> > +    UNPLUG,
> > +} HotplugEventType;
> > +
> > +static void piix4_cpu_hotplug_req(PIIX4PMState *s, CPUState *cpu,
> > +                                  HotplugEventType action)
> > +{
> > +    struct cpu_status *g = &s->gpe_cpu;
> > +    ACPIGPE *gpe = &s->ar.gpe;
> > +    CPUClass *k = CPU_GET_CLASS(cpu);
> > +    int64_t cpu_id;
> > +
> > +    assert(s != NULL);
> > +
> > +    *gpe->sts = *gpe->sts | PIIX4_CPU_HOTPLUG_STATUS;
> > +    cpu_id = k->get_firmware_id(CPU(cpu));
> > +    if (action == PLUG) {
> > +        g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
> > +    } else {
> > +        g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
> > +    }
> > +    pm_update_sci(s);
> > +}
> > +
> > +static void piix4_cpu_add_req(Notifier *n, void *opaque)
> > +{
> > +    PIIX4PMState *s = container_of(n, PIIX4PMState, cpu_add_notifier);
> > +    piix4_cpu_hotplug_req(s, CPU(opaque), PLUG);
> > +}
> > +
> > +static int piix4_init_cpu_status(Object *obj, void *opaque)
> > +{
> > +    struct cpu_status *g = (struct cpu_status *)opaque;
> > +    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
> > +
> > +    if (cpu_obj) {
> > +        struct Error *error = NULL;
> > +        CPUClass *k = CPU_GET_CLASS(cpu_obj);
> > +        int64_t id = k->get_firmware_id(CPU(cpu_obj));
> > +
> > +        if (error) {
> > +            fprintf(stderr, "failed to initilize CPU status for ACPI:
> > %s\n",
> > +                    error_get_pretty(error));
> > +            error_free(error);
> > +            abort();
> > +        }
> > +        g_assert((id / 8) < PROC_LEN);
> > +        g->sts[id / 8] |= (1 << (id % 8));
> > +    }
> > +    return object_child_foreach(obj, piix4_init_cpu_status, opaque);
> > +}
> > +
> >  static int piix4_device_hotplug(DeviceState *qdev, PCIDevice *dev,
> >                                  PCIHotplugState state);
> >  
> > @@ -600,6 +701,13 @@ static void
> > piix4_acpi_system_hot_add_init(MemoryRegion *parent,
> > memory_region_add_subregion(parent, PCI_HOTPLUG_ADDR, &s->io_pci);
> >      pci_bus_hotplug(bus, piix4_device_hotplug, &s->dev.qdev);
> > +
> > +    piix4_init_cpu_status(qdev_get_machine(), &s->gpe_cpu);
> > +    memory_region_init_io(&s->io_cpu, &cpu_hotplug_ops, s,
> > "apci-cpu-hotplug",
> > +                          PROC_LEN);
> > +    memory_region_add_subregion(parent, PROC_BASE, &s->io_cpu);
> > +    s->cpu_add_notifier.notify = piix4_cpu_add_req;
> > +    qemu_register_cpu_added_notifier(&s->cpu_add_notifier);
> >  }
> >  
> >  static void enable_device(PIIX4PMState *s, int slot)
> 
> 

Re: [Qemu-devel] [PATCH 02/22] target-i386: split APIC creation from initialization in x86_cpu_realizefn()

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:36:54PM +0200, Igor Mammedov wrote:
> When APIC is hotplugged during CPU hotplug, device_set_realized()
> calls device_reset() on it. And if QEMU runs in KVM mode, following
> call chain will fail:
>     apic_reset_common()
>         -> kvm_apic_vapic_base_update()
>             -> kvm_vcpu_ioctl(cpu->kvm_fd,...)
> due to cpu->kvm_fd not being initialized yet.
> 
> cpu->kvm_fd is initialized during qemu_init_vcpu() call but x86_cpu_apic_init()
> can't be moved after it because kvm_init_vcpu() -> kvm_arch_reset_vcpu()
> relies on APIC to determine if CPU is BSP for setting initial env->mp_state.
> 
> So split APIC device creation from its initialization and realize APIC
> after CPU is created, when it's safe to call APIC's reset method.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>


> ---
> v2:
>   * s/x86_cpu_apic_init()/x86_cpu_apic_realize()/
> ---
>  target-i386/cpu.c | 24 +++++++++++++++++++++---
>  1 file changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 8b348d0..41f0f47 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -2050,9 +2050,8 @@ static void mce_init(X86CPU *cpu)
>  }
>  
>  #ifndef CONFIG_USER_ONLY
> -static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
> +static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
>  {
> -    static int apic_mapped;
>      CPUX86State *env = &cpu->env;
>      APICCommonState *apic;
>      const char *apic_type = "apic";
> @@ -2075,6 +2074,16 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
>      /* TODO: convert to link<> */
>      apic = APIC_COMMON(env->apic_state);
>      apic->cpu = cpu;
> +}
> +
> +static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
> +{
> +    CPUX86State *env = &cpu->env;
> +    static int apic_mapped;
> +
> +    if (env->apic_state == NULL) {
> +        return;
> +    }
>  
>      if (qdev_init(env->apic_state)) {
>          error_setg(errp, "APIC device '%s' could not be initialized",
> @@ -2092,6 +2101,10 @@ static void x86_cpu_apic_init(X86CPU *cpu, Error **errp)
>          apic_mapped = 1;
>      }
>  }
> +#else
> +static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
> +{
> +}
>  #endif
>  
>  static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
> @@ -2142,7 +2155,7 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
>      qemu_register_reset(x86_cpu_machine_reset_cb, cpu);
>  
>      if (cpu->env.cpuid_features & CPUID_APIC || smp_cpus > 1) {
> -        x86_cpu_apic_init(cpu, &local_err);
> +        x86_cpu_apic_create(cpu, &local_err);
>          if (local_err != NULL) {
>              goto out;
>          }
> @@ -2151,6 +2164,11 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
>  
>      mce_init(cpu);
>      qemu_init_vcpu(&cpu->env);
> +
> +    x86_cpu_apic_realize(cpu, &local_err);
> +    if (local_err != NULL) {
> +        goto out;
> +    }
>      cpu_reset(CPU(cpu));
>  
>      xcc->parent_realize(dev, &local_err);
> -- 
> 1.8.1.4
> 
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 03/22] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:36:55PM +0200, Igor Mammedov wrote:
> Move CPU creation and features parsing into a separate cpu_x86_create()
> function, so that board would be able to set board specific CPU
> properties before CPU is realized.
> 
> Keep cpu_x86_init() for compatibility with the code that uses cpu_init()
> and doesn't need to modify CPU properties.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  target-i386/cpu.c | 28 +++++++++++++++++++---------
>  target-i386/cpu.h |  1 +
>  2 files changed, 20 insertions(+), 9 deletions(-)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 41f0f47..269a681 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -1563,17 +1563,16 @@ static void cpu_x86_register(X86CPU *cpu, const char *name, Error **errp)
>      object_property_set_str(OBJECT(cpu), def->model_id, "model-id", errp);
>  }
>  
> -X86CPU *cpu_x86_init(const char *cpu_model)
> +X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
>  {
>      X86CPU *cpu = NULL;
>      CPUX86State *env;
>      gchar **model_pieces;
>      char *name, *features;
> -    Error *error = NULL;
>  
>      model_pieces = g_strsplit(cpu_model, ",", 2);
>      if (!model_pieces[0]) {
> -        error_setg(&error, "Invalid/empty CPU model name");
> +        error_setg(errp, "Invalid/empty CPU model name");
>          goto out;
>      }
>      name = model_pieces[0];
> @@ -1583,23 +1582,34 @@ X86CPU *cpu_x86_init(const char *cpu_model)
>      env = &cpu->env;
>      env->cpu_model_str = cpu_model;
>  
> -    cpu_x86_register(cpu, name, &error);
> -    if (error) {
> +    cpu_x86_register(cpu, name, errp);
> +    if (error_is_set(errp)) {

So the function now does error checking properly if and only if errp is
not NULL. Do we really want to do that?

>          goto out;
>      }
>  
> -    cpu_x86_parse_featurestr(cpu, features, &error);
> -    if (error) {
> +    cpu_x86_parse_featurestr(cpu, features, errp);
> +    if (error_is_set(errp)) {
>          goto out;
>      }
>  
> -    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
> +out:
> +    g_strfreev(model_pieces);

Any specific reason you didn't choose to keep 'Error *error = NULL'
inside cpu_x86_create() as well, and use error_propagate() here? I
believe it would make the patch simpler and easier to review, and at the
same time make cpu_x86_init() check for errors properly even if errp is
NULL. This is the opposite of what you did on x86_cpu_realizefn() at
patch 01/22.

I am not against it if you want to keep this style of error-checking,
but I believe an error_propagate()-based version would be simpler and
safer.


> +    return cpu;
> +}
> +
> +X86CPU *cpu_x86_init(const char *cpu_model)
> +{
> +    Error *error = NULL;
> +    X86CPU *cpu;
> +
> +    cpu = cpu_x86_create(cpu_model, &error);
>      if (error) {
>          goto out;
>      }
>  
> +    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
> +
>  out:
> -    g_strfreev(model_pieces);
>      if (error) {
>          fprintf(stderr, "%s\n", error_get_pretty(error));
>          error_free(error);
> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> index 069a2e2..b98efd2 100644
> --- a/target-i386/cpu.h
> +++ b/target-i386/cpu.h
> @@ -896,6 +896,7 @@ typedef struct CPUX86State {
>  #include "cpu-qom.h"
>  
>  X86CPU *cpu_x86_init(const char *cpu_model);
> +X86CPU *cpu_x86_create(const char *cpu_model, Error **errp);
>  int cpu_x86_exec(CPUX86State *s);
>  void x86_cpu_list(FILE *f, fprintf_function cpu_fprintf);
>  void x86_cpudef_setup(void);
> -- 
> 1.8.1.4
> 
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/22] cpu: Pass CPUState to *cpu_synchronize_post*()

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:36:56PM +0200, Igor Mammedov wrote:
> ... so it could be called from without requiring CPUArchState
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> ---
>  cpus.c               |  4 ++--
>  include/sysemu/kvm.h | 12 ++++++------
>  kvm-all.c            |  8 ++------
>  kvm-stub.c           |  4 ++--
>  4 files changed, 12 insertions(+), 16 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index e919dd7..9b9a32f 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -419,7 +419,7 @@ void cpu_synchronize_all_post_reset(void)
>      CPUArchState *cpu;
>  
>      for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
> -        cpu_synchronize_post_reset(cpu);
> +        cpu_synchronize_post_reset(ENV_GET_CPU(cpu));
>      }
>  }
>  
> @@ -428,7 +428,7 @@ void cpu_synchronize_all_post_init(void)
>      CPUArchState *cpu;
>  
>      for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
> -        cpu_synchronize_post_init(cpu);
> +        cpu_synchronize_post_init(ENV_GET_CPU(cpu));
>      }
>  }
>  
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index f2d97b5..495e6f8 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -250,8 +250,8 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
>  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
>                                        uint32_t index, int reg);
>  void kvm_cpu_synchronize_state(CPUArchState *env);
> -void kvm_cpu_synchronize_post_reset(CPUArchState *env);
> -void kvm_cpu_synchronize_post_init(CPUArchState *env);
> +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> +void kvm_cpu_synchronize_post_init(CPUState *cpu);
>  
>  /* generic hooks - to be moved/refactored once there are more users */
>  
> @@ -262,17 +262,17 @@ static inline void cpu_synchronize_state(CPUArchState *env)
>      }
>  }
>  
> -static inline void cpu_synchronize_post_reset(CPUArchState *env)
> +static inline void cpu_synchronize_post_reset(CPUState *cpu)
>  {
>      if (kvm_enabled()) {
> -        kvm_cpu_synchronize_post_reset(env);
> +        kvm_cpu_synchronize_post_reset(cpu);
>      }
>  }
>  
> -static inline void cpu_synchronize_post_init(CPUArchState *env)
> +static inline void cpu_synchronize_post_init(CPUState *cpu)
>  {
>      if (kvm_enabled()) {
> -        kvm_cpu_synchronize_post_init(env);
> +        kvm_cpu_synchronize_post_init(cpu);
>      }
>  }
>  
> diff --git a/kvm-all.c b/kvm-all.c
> index 9b433d3..fc4e17c 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -1510,18 +1510,14 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
>      }
>  }
>  
> -void kvm_cpu_synchronize_post_reset(CPUArchState *env)
> +void kvm_cpu_synchronize_post_reset(CPUState *cpu)
>  {
> -    CPUState *cpu = ENV_GET_CPU(env);
> -
>      kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE);
>      cpu->kvm_vcpu_dirty = false;
>  }
>  
> -void kvm_cpu_synchronize_post_init(CPUArchState *env)
> +void kvm_cpu_synchronize_post_init(CPUState *cpu)
>  {
> -    CPUState *cpu = ENV_GET_CPU(env);
> -
>      kvm_arch_put_registers(cpu, KVM_PUT_FULL_STATE);
>      cpu->kvm_vcpu_dirty = false;
>  }
> diff --git a/kvm-stub.c b/kvm-stub.c
> index 760aadc..82875dd 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -42,11 +42,11 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
>  {
>  }
>  
> -void kvm_cpu_synchronize_post_reset(CPUArchState *env)
> +void kvm_cpu_synchronize_post_reset(CPUState *env)
>  {
>  }
>  
> -void kvm_cpu_synchronize_post_init(CPUArchState *env)
> +void kvm_cpu_synchronize_post_init(CPUState *cpu)
>  {
>  }
>  
> -- 
> 1.8.1.4
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 05/22] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:36:57PM +0200, Igor Mammedov wrote:
> ... to synchronize CPU state to KVM
> 
> * in addition link kvm-stub.o to *-user target and fix related compiling issues.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  Makefile.target      |  6 ++++++
>  include/sysemu/kvm.h | 22 ++++++++++++----------
>  kvm-all.c            |  1 +
>  kvm-stub.c           |  5 +++++
>  qom/cpu.c            |  4 ++++
>  vl.c                 |  1 -
>  6 files changed, 28 insertions(+), 11 deletions(-)
> 
> diff --git a/Makefile.target b/Makefile.target
> index 2bd6d14..0c2c24a 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -76,6 +76,12 @@ obj-y += disas.o
>  obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
>  
>  #########################################################
> +# user emulator target
> +ifdef CONFIG_USER_ONLY
> +obj-y += kvm-stub.o
> +endif #CONFIG_USER_ONLY

Why you didn't include it on libqemustub instead? You said on a previous
thread that it is not possible to include all of kvm-stub.c into
libqemustub. Why not?

> +
> +#########################################################
>  # Linux user emulator target
>  
>  ifdef CONFIG_LINUX_USER
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 495e6f8..8534e44 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
>  #if !defined(CONFIG_USER_ONLY)
>  void *kvm_vmalloc(ram_addr_t size);
>  void *kvm_arch_vmalloc(ram_addr_t size);
> -void kvm_setup_guest_memory(void *start, size_t size);
> +#endif
>  
> +void kvm_setup_guest_memory(void *start, size_t size);
>  void kvm_flush_coalesced_mmio_buffer(void);
> -#endif
>  
>  int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
>                            target_ulong len, int type);
> @@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
>  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
>                                        uint32_t index, int reg);
>  void kvm_cpu_synchronize_state(CPUArchState *env);
> -void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> -void kvm_cpu_synchronize_post_init(CPUState *cpu);
>  
>  /* generic hooks - to be moved/refactored once there are more users */
>  
> @@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
>      }
>  }
>  
> +#if !defined(CONFIG_USER_ONLY)
> +int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> +                                       hwaddr *phys_addr);
> +#endif
> +
> +#endif
> +
> +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> +void kvm_cpu_synchronize_post_init(CPUState *cpu);
> +
>  static inline void cpu_synchronize_post_reset(CPUState *cpu)
>  {
>      if (kvm_enabled()) {
> @@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
>  }
>  
>  
> -#if !defined(CONFIG_USER_ONLY)
> -int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> -                                       hwaddr *phys_addr);
> -#endif
> -
> -#endif
>  int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
>                             uint32_t size);
>  
> diff --git a/kvm-all.c b/kvm-all.c
> index fc4e17c..1d17128 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;
>  
>  static const KVMCapabilityInfo kvm_required_capabilites[] = {
>      KVM_CAP_INFO(USER_MEMORY),
> diff --git a/kvm-stub.c b/kvm-stub.c
> index 82875dd..f3e9438 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -12,7 +12,9 @@
>  
>  #include "qemu-common.h"
>  #include "hw/hw.h"
> +#ifndef CONFIG_USER_ONLY
>  #include "hw/pci/msi.h"
> +#endif
>  #include "cpu.h"
>  #include "exec/gdbstub.h"
>  #include "sysemu/kvm.h"
> @@ -23,6 +25,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;
>  
>  int kvm_init_vcpu(CPUState *cpu)
>  {
> @@ -122,6 +125,7 @@ int kvm_on_sigbus(int code, void *addr)
>      return 1;
>  }
>  
> +#ifndef CONFIG_USER_ONLY
>  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
>  {
>      return -ENOSYS;
> @@ -145,3 +149,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
>  {
>      return -ENOSYS;
>  }
> +#endif
> diff --git a/qom/cpu.c b/qom/cpu.c
> index e242dcb..0c76712 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -20,6 +20,7 @@
>  
>  #include "qom/cpu.h"
>  #include "qemu-common.h"
> +#include "sysemu/kvm.h"
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
>  
>  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>  {
> +    if (dev->hotplugged) {
> +        cpu_synchronize_post_init(CPU(dev)); +    } }
>  
>  static void cpu_class_init(ObjectClass *klass, void *data) diff --git
>  a/vl.c b/vl.c index a8bba04..97f0349 100644 --- a/vl.c +++ b/vl.c @@
>  -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
>  NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
>  
>  static bool tcg_allowed = true; -bool kvm_allowed; bool xen_allowed;
>  uint32_t xen_domid; enum xen_mode xen_mode = XEN_EMULATE; -- 
> 1.8.1.4
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Eduardo Habkost]

On Fri, Apr 05, 2013 at 04:36:58PM +0200, Igor Mammedov wrote:
> ... and call it if defined from CPUClass.realize() if CPU was hotplugged
> 
> by default leave .resume() unset (i.e. NULL) and override it for softmmu
> in qemu_init_vcpu() if it's still unset.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
[...]
> diff --git a/cpus.c b/cpus.c
> index 9b9a32f..6b793c5 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
[...]
> @@ -1042,7 +1047,11 @@ void qemu_init_vcpu(void *_env)
>  {
>      CPUArchState *env = _env;
>      CPUState *cpu = ENV_GET_CPU(env);
> +    CPUClass *klass = CPU_GET_CLASS(cpu);
>  
> +    if (klass->resume == NULL) {
> +        klass->resume = resume_vcpu;
> +    }

So you are initializing a field of CPUClass struct inside a CPU object
initialization function. And that's a function that is not even
converted to QOM yet, and buried inside a non-trivial function call tree
(hence easy to be called at the wrong time if one day we reorder the
initialization steps).

Can't we do this on class_init(), where it belongs? If we need different
implementations for softmmu/user, we can add a stub for *-user. I think
even an explicit #ifdef inside resume_vcpu() would be preferable to
this.


>      cpu->nr_cores = smp_cores;
>      cpu->nr_threads = smp_threads;
>      cpu->stopped = true;
[...]

-- 
Eduardo

Re: [Qemu-devel] [PATCH 05/22] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Igor Mammedov]

On Mon, 8 Apr 2013 16:45:01 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 04:36:57PM +0200, Igor Mammedov wrote:
> > ... to synchronize CPU state to KVM
> > 
> > * in addition link kvm-stub.o to *-user target and fix related compiling
> > issues.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  Makefile.target      |  6 ++++++
> >  include/sysemu/kvm.h | 22 ++++++++++++----------
> >  kvm-all.c            |  1 +
> >  kvm-stub.c           |  5 +++++
> >  qom/cpu.c            |  4 ++++
> >  vl.c                 |  1 -
> >  6 files changed, 28 insertions(+), 11 deletions(-)
> > 
> > diff --git a/Makefile.target b/Makefile.target
> > index 2bd6d14..0c2c24a 100644
> > --- a/Makefile.target
> > +++ b/Makefile.target
> > @@ -76,6 +76,12 @@ obj-y += disas.o
> >  obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
> >  
> >  #########################################################
> > +# user emulator target
> > +ifdef CONFIG_USER_ONLY
> > +obj-y += kvm-stub.o
> > +endif #CONFIG_USER_ONLY
> 
> Why you didn't include it on libqemustub instead? You said on a previous
> thread that it is not possible to include all of kvm-stub.c into
> libqemustub. Why not?
It wasn't possible, but after this patch it's, I guess.
Paolo objected to including CPU related stuff in libqemustub, hence
this hunk in Makefile.

> 
> > +
> > +#########################################################
> >  # Linux user emulator target
> >  
> >  ifdef CONFIG_LINUX_USER
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index 495e6f8..8534e44 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
> >  #if !defined(CONFIG_USER_ONLY)
> >  void *kvm_vmalloc(ram_addr_t size);
> >  void *kvm_arch_vmalloc(ram_addr_t size);
> > -void kvm_setup_guest_memory(void *start, size_t size);
> > +#endif
> >  
> > +void kvm_setup_guest_memory(void *start, size_t size);
> >  void kvm_flush_coalesced_mmio_buffer(void);
> > -#endif
> >  
> >  int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
> >                            target_ulong len, int type);
> > @@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int
> > extension); uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t
> > function, uint32_t index, int reg);
> >  void kvm_cpu_synchronize_state(CPUArchState *env);
> > -void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > -void kvm_cpu_synchronize_post_init(CPUState *cpu);
> >  
> >  /* generic hooks - to be moved/refactored once there are more users */
> >  
> > @@ -262,6 +260,16 @@ static inline void
> > cpu_synchronize_state(CPUArchState *env) }
> >  }
> >  
> > +#if !defined(CONFIG_USER_ONLY)
> > +int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > +                                       hwaddr *phys_addr);
> > +#endif
> > +
> > +#endif
> > +
> > +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > +void kvm_cpu_synchronize_post_init(CPUState *cpu);
> > +
> >  static inline void cpu_synchronize_post_reset(CPUState *cpu)
> >  {
> >      if (kvm_enabled()) {
> > @@ -277,12 +285,6 @@ static inline void
> > cpu_synchronize_post_init(CPUState *cpu) }
> >  
> >  
> > -#if !defined(CONFIG_USER_ONLY)
> > -int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > -                                       hwaddr *phys_addr);
> > -#endif
> > -
> > -#endif
> >  int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool
> > assign, uint32_t size);
> >  
> > diff --git a/kvm-all.c b/kvm-all.c
> > index fc4e17c..1d17128 100644
> > --- a/kvm-all.c
> > +++ b/kvm-all.c
> > @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> >  
> >  static const KVMCapabilityInfo kvm_required_capabilites[] = {
> >      KVM_CAP_INFO(USER_MEMORY),
> > diff --git a/kvm-stub.c b/kvm-stub.c
> > index 82875dd..f3e9438 100644
> > --- a/kvm-stub.c
> > +++ b/kvm-stub.c
> > @@ -12,7 +12,9 @@
> >  
> >  #include "qemu-common.h"
> >  #include "hw/hw.h"
> > +#ifndef CONFIG_USER_ONLY
> >  #include "hw/pci/msi.h"
> > +#endif
> >  #include "cpu.h"
> >  #include "exec/gdbstub.h"
> >  #include "sysemu/kvm.h"
> > @@ -23,6 +25,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> >  
> >  int kvm_init_vcpu(CPUState *cpu)
> >  {
> > @@ -122,6 +125,7 @@ int kvm_on_sigbus(int code, void *addr)
> >      return 1;
> >  }
> >  
> > +#ifndef CONFIG_USER_ONLY
> >  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
> >  {
> >      return -ENOSYS;
> > @@ -145,3 +149,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s,
> > EventNotifier *n, int virq) {
> >      return -ENOSYS;
> >  }
> > +#endif
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index e242dcb..0c76712 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -20,6 +20,7 @@
> >  
> >  #include "qom/cpu.h"
> >  #include "qemu-common.h"
> > +#include "sysemu/kvm.h"
> >  
> >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> >  {
> > @@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char
> > *cpu_model) 
> >  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> >  {
> > +    if (dev->hotplugged) {
> > +        cpu_synchronize_post_init(CPU(dev)); +    } }
> >  
> >  static void cpu_class_init(ObjectClass *klass, void *data) diff --git
> >  a/vl.c b/vl.c index a8bba04..97f0349 100644 --- a/vl.c +++ b/vl.c @@
> >  -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
> >  NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
> >  
> >  static bool tcg_allowed = true; -bool kvm_allowed; bool xen_allowed;
> >  uint32_t xen_domid; enum xen_mode xen_mode = XEN_EMULATE; -- 
> > 1.8.1.4
> > 
> 

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Igor Mammedov]

On Mon, 8 Apr 2013 17:13:11 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 04:36:58PM +0200, Igor Mammedov wrote:
> > ... and call it if defined from CPUClass.realize() if CPU was hotplugged
> > 
> > by default leave .resume() unset (i.e. NULL) and override it for softmmu
> > in qemu_init_vcpu() if it's still unset.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> [...]
> > diff --git a/cpus.c b/cpus.c
> > index 9b9a32f..6b793c5 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
> [...]
> > @@ -1042,7 +1047,11 @@ void qemu_init_vcpu(void *_env)
> >  {
> >      CPUArchState *env = _env;
> >      CPUState *cpu = ENV_GET_CPU(env);
> > +    CPUClass *klass = CPU_GET_CLASS(cpu);
> >  
> > +    if (klass->resume == NULL) {
> > +        klass->resume = resume_vcpu;
> > +    }
> 
> So you are initializing a field of CPUClass struct inside a CPU object
> initialization function. And that's a function that is not even
> converted to QOM yet, and buried inside a non-trivial function call tree
> (hence easy to be called at the wrong time if one day we reorder the
> initialization steps).
init step are not likely to change this drastically but main reason why
it's here see below.

> 
> Can't we do this on class_init(), where it belongs? If we need different
> implementations for softmmu/user, we can add a stub for *-user. I think
> even an explicit #ifdef inside resume_vcpu() would be preferable to
> this.
Generally I agree with you that class_init() would be more correct, but
ifdefs in qom/cpu.c defeat purpose to build qom/cpu.c only once for all
targets, that Andreas are working towards.

It asks surely asks for resume_vcpu() stub in libqemustubs, and I'd do it
there weren't objections to it.

Paolo,
 it looks like a good candidate for libqemustubs, would it be ok to add stub
 there?

> 
> 
> >      cpu->nr_cores = smp_cores;
> >      cpu->nr_threads = smp_threads;
> >      cpu->stopped = true;
> [...]
> 

Re: [Qemu-devel] [PATCH 03/22] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Paolo Bonzini]

Il 08/04/2013 20:30, Eduardo Habkost ha scritto:
>> > -    cpu_x86_register(cpu, name, &error);
>> > -    if (error) {
>> > +    cpu_x86_register(cpu, name, errp);
>> > +    if (error_is_set(errp)) {
> So the function now does error checking properly if and only if errp is
> not NULL. Do we really want to do that?

No, using error_propagate is the correct idiom indeed.

Paolo

>> >          goto out;
>> >      }
>> >  
>> > -    cpu_x86_parse_featurestr(cpu, features, &error);
>> > -    if (error) {
>> > +    cpu_x86_parse_featurestr(cpu, features, errp);
>> > +    if (error_is_set(errp)) {
>> >          goto out;
>> >      }
>> >  
>> > -    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
>> > +out:
>> > +    g_strfreev(model_pieces);
> Any specific reason you didn't choose to keep 'Error *error = NULL'
> inside cpu_x86_create() as well, and use error_propagate() here? I
> believe it would make the patch simpler and easier to review, and at the
> same time make cpu_x86_init() check for errors properly even if errp is
> NULL. This is the opposite of what you did on x86_cpu_realizefn() at
> patch 01/22.

Re: [Qemu-devel] [PATCH 03/22] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Igor Mammedov]

On Tue, 09 Apr 2013 12:30:21 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 08/04/2013 20:30, Eduardo Habkost ha scritto:
> >> > -    cpu_x86_register(cpu, name, &error);
> >> > -    if (error) {
> >> > +    cpu_x86_register(cpu, name, errp);
> >> > +    if (error_is_set(errp)) {
> > So the function now does error checking properly if and only if errp is
> > not NULL. Do we really want to do that?
> 
> No, using error_propagate is the correct idiom indeed.
Ok, I'll use  error_propagate() in next version.

> 
> Paolo
> 
> >> >          goto out;
> >> >      }
> >> >  
> >> > -    cpu_x86_parse_featurestr(cpu, features, &error);
> >> > -    if (error) {
> >> > +    cpu_x86_parse_featurestr(cpu, features, errp);
> >> > +    if (error_is_set(errp)) {
> >> >          goto out;
> >> >      }
> >> >  
> >> > -    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
> >> > +out:
> >> > +    g_strfreev(model_pieces);
> > Any specific reason you didn't choose to keep 'Error *error = NULL'
> > inside cpu_x86_create() as well, and use error_propagate() here? I
> > believe it would make the patch simpler and easier to review, and at the
> > same time make cpu_x86_init() check for errors properly even if errp is
> > NULL. This is the opposite of what you did on x86_cpu_realizefn() at
> > patch 01/22.
> 

Re: [Qemu-devel] [PATCH 05/22] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Paolo Bonzini]

Il 05/04/2013 16:36, Igor Mammedov ha scritto:
> ... to synchronize CPU state to KVM
> 
> * in addition link kvm-stub.o to *-user target and fix related compiling issues.

Please make it a separate patch.

The Makefile.target parts should look like this, I think:

diff --git a/Makefile.target b/Makefile.target
index 2bd6d14..9bde8e5 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -64,6 +64,12 @@ all: $(PROGS) stap
 # Dummy command so that make thinks it has done something
 	@true
 
+CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
+CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
+CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
+CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
+CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
+
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
@@ -74,6 +80,7 @@ obj-y += fpu/softfloat.o
 obj-y += target-$(TARGET_BASE_ARCH)/
 obj-y += disas.o
 obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
+obj-$(CONFIG_NO_KVM) += kvm-stub.o
 
 #########################################################
 # Linux user emulator target
@@ -102,18 +109,11 @@ endif #CONFIG_BSD_USER
 #########################################################
 # System emulator target
 ifdef CONFIG_SOFTMMU
-CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
-CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
-CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
-CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
-CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
-
 obj-y += arch_init.o cpus.o monitor.o gdbstub.o balloon.o ioport.o
 obj-y += qtest.o
 obj-y += hw/
 obj-$(CONFIG_FDT) += device_tree.o
 obj-$(CONFIG_KVM) += kvm-all.o
-obj-$(CONFIG_NO_KVM) += kvm-stub.o
 obj-y += memory.o savevm.o cputlb.o
 obj-$(CONFIG_HAVE_GET_MEMORY_MAPPING) += memory_mapping.o
 obj-$(CONFIG_HAVE_CORE_DUMP) += dump.o

Basically, kvm-stub.o can be in the common section that is used for
both user and softmmu targets.

Paolo

Re: [Qemu-devel] [PATCH 05/22] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Paolo Bonzini]

Il 09/04/2013 12:13, Igor Mammedov ha scritto:
> > Why you didn't include it on libqemustub instead? You said on a previous
> > thread that it is not possible to include all of kvm-stub.c into
> > libqemustub. Why not?
> 
> It wasn't possible, but after this patch it's, I guess.
> Paolo objected to including CPU related stuff in libqemustub, hence
> this hunk in Makefile.

I objected to including target-specific stuff, but general APIs
certainly can go there.  That is certainly the case for kvm-stub.c.

Still, please make this a separate patch.

Paolo

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Paolo Bonzini]

Il 08/04/2013 22:13, Eduardo Habkost ha scritto:
> So you are initializing a field of CPUClass struct inside a CPU object
> initialization function. And that's a function that is not even
> converted to QOM yet, and buried inside a non-trivial function call tree
> (hence easy to be called at the wrong time if one day we reorder the
> initialization steps).
> 
> Can't we do this on class_init(), where it belongs? If we need different
> implementations for softmmu/user, we can add a stub for *-user.

Yes, please add a stub for the new function and override it in cpus.c.

> I think even an explicit #ifdef inside resume_vcpu() would be
> preferable to this.

Using an #ifdef basically means putting it in exec.c.  I'm not sure
about that, it seems to fit more in cpus.c.

Paolo

Re: [Qemu-devel] [PATCH 09/22] introduce CPU hot-plug notifier

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
> ---
>  include/qom/cpu.h       |  2 ++
>  include/sysemu/sysemu.h |  3 +++
>  qom/cpu.c               | 12 ++++++++++++
>  3 files changed, 17 insertions(+)
> 
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 6d6eb7a..210aca3 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -46,6 +46,7 @@ typedef struct CPUState CPUState;
>   * @reset: Callback to reset the #CPUState to its initial state.
>   * @do_interrupt: Callback for interrupt handling.
>   * @resume: Callback for putting CPU in runable state
> + * @get_firmware_id: Callback for getting arch depended CPU id
>   * @vmsd: State description for migration.
>   *
>   * Represents a CPU family or model.
> @@ -60,6 +61,7 @@ typedef struct CPUClass {
>      void (*reset)(CPUState *cpu);
>      void (*do_interrupt)(CPUState *cpu);
>      void (*resume)(CPUState *cpu);
> +    void (*get_firmware_id)(CPUState *cpu);
>  
>      const struct VMStateDescription *vmsd;
>  } CPUClass;

This should be in patch 11, other than that

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 6578782..a8c3de1 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
>  /* generic hotplug */
>  void drive_hot_add(Monitor *mon, const QDict *qdict);
>  
> +/* CPU hotplug */
> +void qemu_register_cpu_added_notifier(Notifier *notifier);
> +
>  /* pcie aer error injection */
>  void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
>  int do_pcie_aer_inject_error(Monitor *mon,
> diff --git a/qom/cpu.c b/qom/cpu.c
> index c062e00..10ceaed 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -21,6 +21,17 @@
>  #include "qom/cpu.h"
>  #include "qemu-common.h"
>  #include "sysemu/kvm.h"
> +#include "qemu/notify.h"
> +#include "sysemu/sysemu.h"
> +
> +/* CPU hot-plug notifiers */
> +static NotifierList cpu_added_notifiers =
> +    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> +
> +void qemu_register_cpu_added_notifier(Notifier *notifier)
> +{
> +    notifier_list_add(&cpu_added_notifiers, notifier);
> +}
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -65,6 +76,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>          if (klass->resume != NULL) {
>              klass->resume(CPU(dev));
>          }
> +        notifier_list_notify(&cpu_added_notifiers, dev);
>      }
>  }
>  
> 

Re: [Qemu-devel] [PATCH 12/22] cpu: add helper cpu_exists(), to check if CPU with specified id exists

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> ... it should be used only on slow path since it does recursive search
>     on /machine QOM tree for objects of TYPE_CPU
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  include/qom/cpu.h | 10 ++++++++++
>  qom/cpu.c         | 21 +++++++++++++++++++++
>  2 files changed, 31 insertions(+)
> 
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 0d33009..5cac79b 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -227,6 +227,16 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
>   */
>  CPUState *qemu_get_cpu(int index);
>  
> +/**
> + * cpu_exists:
> + * @id - guest exposed CPU ID to lookup
> + *
> + * Search for CPU with specified ID.
> + *
> + * Returns: true - CPU is found, false - CPU isn't found
> + */
> +bool cpu_exists(int64_t id);
> +
>  #ifndef CONFIG_USER_ONLY
>  
>  typedef void (*CPUInterruptHandler)(CPUState *, int);
> diff --git a/qom/cpu.c b/qom/cpu.c
> index a54d4d1..46b77d3 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -24,6 +24,27 @@
>  #include "qemu/notify.h"
>  #include "sysemu/sysemu.h"
>  
> +static int cpu_exist_cb(Object *obj, void *opaque)
> +{
> +    int64_t id = *(int64_t *)opaque;
> +    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
> +
> +    if (cpu_obj) {
> +        CPUState *cpu = CPU(cpu_obj);
> +        CPUClass *klass = CPU_GET_CLASS(cpu);
> +
> +        if (klass->get_firmware_id && klass->get_firmware_id(cpu) == id) {
> +            return 1;
> +        }
> +    }
> +    return object_child_foreach(obj, cpu_exist_cb, opaque);
> +}
> +
> +bool cpu_exists(int64_t id)
> +{
> +   return cpu_exist_cb(qdev_get_machine(), &id) ? true : false;
> +}
> +
>  /* CPU hot-plug notifiers */
>  static NotifierList cpu_added_notifiers =
>      NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [PATCH 14/22] target-i386: introduce apic-id property

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> ... and use it from board level to set APIC ID for CPUs
> it creates.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> Note:
>   * pc_new_cpu() function will be reused later in CPU hot-plug hook.
> 
> v2:
>   * use generic cpu_exists() instead of custom one
>   * make apic-id dynamic property, so it won't be possible to use it
>     as global property, since it's instance specific
> ---
>  hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
>  target-i386/cpu.c | 38 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 62 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index ebbf059..9d617b4 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
>      }
>  }
>  
> +static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> +{
> +    X86CPU *cpu;
> +
> +    cpu = cpu_x86_create(cpu_model, errp);
> +    if (!cpu) {
> +        return;
> +    }
> +
> +    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> +    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> +
> +    if (error_is_set(errp)) {
> +        if (cpu != NULL) {
> +            object_unref(OBJECT(cpu));
> +        }
> +    }
> +}
> +
>  void pc_cpus_init(const char *cpu_model)
>  {
>      int i;
> +    Error *error = NULL;
>  
>      /* init CPUs */
>      if (cpu_model == NULL) {
> @@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
>      }
>  
>      for (i = 0; i < smp_cpus; i++) {
> -        if (!cpu_x86_init(cpu_model)) {
> +        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> +        if (error) {
> +            fprintf(stderr, "%s\n", error_get_pretty(error));
> +            error_free(error);
>              exit(1);
>          }
>      }
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 858fd54..db56b52 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -1272,6 +1272,41 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
>      cpu->env.tsc_khz = value / 1000;
>  }
>  
> +static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
> +                                  const char *name, Error **errp)
> +{
> +    X86CPU *cpu = X86_CPU(obj);
> +    int64_t value = cpu->env.cpuid_apic_id;
> +
> +    visit_type_int(v, &value, name, errp);
> +}
> +
> +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> +                                  const char *name, Error **errp)
> +{
> +    X86CPU *cpu = X86_CPU(obj);
> +    const int64_t min = 0;
> +    const int64_t max = UINT32_MAX;
> +    int64_t value;
> +
> +    visit_type_int(v, &value, name, errp);
> +    if (error_is_set(errp)) {
> +        return;
> +    }

Please use a local Error* function and error_propagate.

> +    if (value < min || value > max) {
> +        error_setg(errp, "Property %s.%s doesn't take value %" PRId64
> +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> +                   object_get_typename(obj), name, value, min, max);
> +        return;
> +    }
> +
> +    if (cpu_exists(value)) {
> +        error_setg(errp, "CPU with APIC ID %" PRIi64 " exists", value);
> +        return;
> +    }
> +    cpu->env.cpuid_apic_id = value;
> +}
> +
>  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
>  {
>      x86_def_t *def;
> @@ -2256,6 +2291,9 @@ static void x86_cpu_initfn(Object *obj)
>      object_property_add(obj, "tsc-frequency", "int",
>                          x86_cpuid_get_tsc_freq,
>                          x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
> +    object_property_add(obj, "apic-id", "int",
> +                        x86_cpuid_get_apic_id,
> +                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
>  
>      env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
>  
> 

With the above change,

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [PATCH 18/22] target-i386: move APIC to ICC bus

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> @@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque, int version_id)
>      return 0;
>  }
>  
> -static int apic_init_common(SysBusDevice *dev)
> +static int apic_init_common(ICCDevice *dev)
>  {
>      APICCommonState *s = APIC_COMMON(dev);
> +    DeviceState *d = DEVICE(dev);
>      APICCommonClass *info;
>      static DeviceState *vapic;
>      static int apic_no;
> +    static bool mmio_registered;
>  
>      if (apic_no >= MAX_APICS) {
>          return -1;
> @@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
>  
>      info = APIC_COMMON_GET_CLASS(s);
>      info->init(s);
> -
> -    sysbus_init_mmio(dev, &s->io_memory);
> +    if (!mmio_registered) {
> +        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
> +        memory_region_add_subregion(as, 0, &s->io_memory);
> +        mmio_registered = true;
> +    }

Ok for now.  When we get IOMMU regions, we could use them to dispatch on
the current CPU's APIC.

Paolo

Re: [Qemu-devel] [PATCH 18/22] target-i386: move IOAPIC to ICC bus

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> +    const char *ioapic_name = "ioapic";
>  
>      if (kvm_irqchip_in_kernel()) {
> -        dev = qdev_create(NULL, "kvm-ioapic");
> -    } else {
> -        dev = qdev_create(NULL, "ioapic");
> -    }
> -    if (parent_name) {
> -        object_property_add_child(object_resolve_path(parent_name, NULL),
> -                                  "ioapic", OBJECT(dev), NULL);
> +        ioapic_name = "kvm-ioapic";
>      }
> -    qdev_init_nofail(dev);
> -    d = SYS_BUS_DEVICE(dev);
> -    sysbus_mmio_map(d, 0, IO_APIC_DEFAULT_ADDRESS);
> +    object_property_set_str(OBJECT(dev), ioapic_name, "ioapic-type", NULL);

Why do you need this?

Having the IOAPIC as a QOM child of the icc-bridge is not too important,
I think.  Perhaps not even too correct...

Paolo

Re: [Qemu-devel] [PATCH 20/22] qdev: set device's parent before calling realize() down inheritance chain.

[Paolo Bonzini]

Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> Currently device_set_realized() sets parent only after device was realized,
> But qdev_device_add() sets it before device is realized.
> Make behavior consistent and alter device_set_realized() to behave like
> qdev_device_add().
> 
> It will allow to set link<> properties in realize() method in classes
> inherited from DEVICE.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  * Usage examples
>     - conversion to CPU back-link property in APIC
>     - pre-allocate link<CPU[n]> at board level, and populate them
>       at realize time when CPUs are created.
> ---
>  hw/qdev.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/hw/qdev.c b/hw/qdev.c
> index e2bb37d..b9bd6f5 100644
> --- a/hw/qdev.c
> +++ b/hw/qdev.c
> @@ -671,10 +671,6 @@ static void device_set_realized(Object *obj, bool value, Error **err)
>      Error *local_err = NULL;
>  
>      if (value && !dev->realized) {
> -        if (dc->realize) {
> -            dc->realize(dev, &local_err);
> -        }
> -
>          if (!obj->parent && local_err == NULL) {
>              static int unattached_count;
>              gchar *name = g_strdup_printf("device[%d]", unattached_count++);
> @@ -685,6 +681,10 @@ static void device_set_realized(Object *obj, bool value, Error **err)
>              g_free(name);
>          }
>  
> +        if (dc->realize) {
> +            dc->realize(dev, &local_err);
> +        }
> +
>          if (qdev_get_vmsd(dev) && local_err == NULL) {
>              vmstate_register_with_alias_id(dev, -1, qdev_get_vmsd(dev), dev,
>                                             dev->instance_id_alias,
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Paolo Bonzini]

Il 08/04/2013 13:32, Igor Mammedov ha scritto:
> 
>> > This patch should be combined with [PATCH 07/22]
>> > 
>> > 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
>>> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>> > > ---
>>> > >  hw/ioapic_common.c | 2 +-
>>> > >  1 file changed, 1 insertion(+), 1 deletion(-)
>>> > > 
>>> > > diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
>>> > > index d4aff29..561b987 100644
>>> > > --- a/hw/ioapic_common.c
>>> > > +++ b/hw/ioapic_common.c
>>> > > @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int
>>> > > version_id) 
>>> > >  static int ioapic_init_common(SysBusDevice *dev)
>>> > >  {
>>> > > -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
>>> > > +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
>>> > >      IOAPICCommonClass *info;
>>> > >      static int ioapic_no;
>>> > >  
>> > 
>> > 
>> > 
> sure, will do this on the next respin.

Actually no, why?  It's two different files.

Paolo

Re: [Qemu-devel] [PATCH 19/22] target-i386: move APIC to ICC bus

[Igor Mammedov]

On Fri,  5 Apr 2013 16:37:12 +0200
Igor Mammedov <imammedo@redhat.com> wrote:

> ... to allow it to be hotplugged
> 
>  * map APIC's mmio at board level if it is present
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  hw/apic_common.c   | 17 ++++++++++++-----
>  hw/apic_internal.h |  6 +++---
>  hw/i386/kvmvapic.c |  1 +
>  hw/i386/pc.c       | 18 +++++++++++++++---
>  hw/icc_bus.c       |  6 ++++++
>  hw/icc_bus.h       |  2 +-
>  target-i386/cpu.c  | 16 +++-------------
>  7 files changed, 41 insertions(+), 25 deletions(-)
> 
> diff --git a/hw/apic_common.c b/hw/apic_common.c
> index 3798509..b2e84e6 100644
> --- a/hw/apic_common.c
> +++ b/hw/apic_common.c
> @@ -21,6 +21,8 @@
>  #include "hw/apic_internal.h"
>  #include "trace.h"
>  #include "sysemu/kvm.h"
> +#include "qdev.h"
> +#include "hw/sysbus.h"
>  
>  static int apic_irq_delivered;
>  bool apic_report_tpr_access;
> @@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque,
> int version_id) return 0;
>  }
>  
> -static int apic_init_common(SysBusDevice *dev)
> +static int apic_init_common(ICCDevice *dev)
>  {
>      APICCommonState *s = APIC_COMMON(dev);
> +    DeviceState *d = DEVICE(dev);
>      APICCommonClass *info;
>      static DeviceState *vapic;
>      static int apic_no;
> +    static bool mmio_registered;
>  
>      if (apic_no >= MAX_APICS) {
>          return -1;
> @@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
>  
>      info = APIC_COMMON_GET_CLASS(s);
>      info->init(s);
> -
> -    sysbus_init_mmio(dev, &s->io_memory);
> +    if (!mmio_registered) {
> +        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
> +        memory_region_add_subregion(as, 0, &s->io_memory);
> +        mmio_registered = true;
> +    }
>  
>      /* Note: We need at least 1M to map the VAPIC option ROM */
>      if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
> @@ -375,7 +382,7 @@ static Property apic_properties_common[] = {
>  
>  static void apic_common_class_init(ObjectClass *klass, void *data)
>  {
> -    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
> +    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
>      DeviceClass *dc = DEVICE_CLASS(klass);
>  
>      dc->vmsd = &vmstate_apic_common;
> @@ -387,7 +394,7 @@ static void apic_common_class_init(ObjectClass *klass,
> void *data) 
>  static const TypeInfo apic_common_type = {
>      .name = TYPE_APIC_COMMON,
> -    .parent = TYPE_SYS_BUS_DEVICE,
> +    .parent = TYPE_ICC_DEVICE,
>      .instance_size = sizeof(APICCommonState),
>      .class_size = sizeof(APICCommonClass),
>      .class_init = apic_common_class_init,
> diff --git a/hw/apic_internal.h b/hw/apic_internal.h
> index aac6290..172fc91 100644
> --- a/hw/apic_internal.h
> +++ b/hw/apic_internal.h
> @@ -21,7 +21,7 @@
>  #define QEMU_APIC_INTERNAL_H
>  
>  #include "exec/memory.h"
> -#include "hw/sysbus.h"
> +#include "hw/icc_bus.h"
>  #include "qemu/timer.h"
>  
>  /* APIC Local Vector Table */
> @@ -78,7 +78,7 @@ typedef struct APICCommonState APICCommonState;
>  
>  typedef struct APICCommonClass
>  {
> -    SysBusDeviceClass parent_class;
> +    ICCDeviceClass parent_class;
>  
>      void (*init)(APICCommonState *s);
>      void (*set_base)(APICCommonState *s, uint64_t val);
> @@ -92,7 +92,7 @@ typedef struct APICCommonClass
>  } APICCommonClass;
>  
>  struct APICCommonState {
> -    SysBusDevice busdev;
> +    ICCDevice busdev;
>  
>      MemoryRegion io_memory;
>      X86CPU *cpu;
> diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
> index c4be882..81e0a75 100644
> --- a/hw/i386/kvmvapic.c
> +++ b/hw/i386/kvmvapic.c
> @@ -12,6 +12,7 @@
>  #include "sysemu/cpus.h"
>  #include "sysemu/kvm.h"
>  #include "hw/apic_internal.h"
> +#include "hw/sysbus.h"
>  
>  #define VAPIC_IO_PORT           0x7e
>  
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index fbc0bcf..5c0dd4f 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -869,13 +869,13 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int
> level) }
>  }
>  
> -static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error
> **errp) +static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
> Error **errp) {
>      X86CPU *cpu;
>  
>      cpu = cpu_x86_create(cpu_model, errp);
>      if (!cpu) {
> -        return;
> +        return cpu;
>      }
>  
>      object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> @@ -884,14 +884,18 @@ static void pc_new_cpu(const char *cpu_model, int64_t
> apic_id, Error **errp) if (error_is_set(errp)) {
>          if (cpu != NULL) {
>              object_unref(OBJECT(cpu));
> +            cpu = NULL;
>          }
>      }
> +    return cpu;
>  }
>  
>  void pc_cpus_init(const char *cpu_model)
>  {
>      int i;
> +    X86CPU *cpu;
>      Error *error = NULL;
> +    SysBusDevice *ib;
>  
>      /* init CPUs */
>      if (cpu_model == NULL) {
> @@ -902,14 +906,22 @@ void pc_cpus_init(const char *cpu_model)
>  #endif
>      }
>  
> +    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
> +                                                 TYPE_ICC_BRIDGE, NULL));
> +
>      for (i = 0; i < smp_cpus; i++) {
> -        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
>          if (error) {
>              fprintf(stderr, "%s\n", error_get_pretty(error));
>              error_free(error);
>              exit(1);
>          }
>      }
> +
> +    /* map APIC MMIO area if CPU has it */
> +    if (cpu->env.apic_state) {
> +        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
> +    }
>  }
>  
>  void pc_acpi_init(const char *default_dsdt)
> diff --git a/hw/icc_bus.c b/hw/icc_bus.c
> index fc8d892..bfe2a30 100644
> --- a/hw/icc_bus.c
> +++ b/hw/icc_bus.c
> @@ -60,6 +60,7 @@ static const TypeInfo icc_device_info = {
>  typedef struct ICCBridgeState {
>      SysBusDevice busdev;
>      MemoryRegion ioapic_container;
> +    MemoryRegion apic_container;
>  } ICCBridgeState;
>  #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj),
> TYPE_ICC_BRIDGE) 
> @@ -92,6 +93,11 @@ static void icc_bridge_initfn(Object *obj)
>      memory_region_init(&s->ioapic_container, "icc-ioapic-container",
> 0x1000); sysbus_init_mmio(sb, &s->ioapic_container);
>      ibus->ioapic_address_space = &s->ioapic_container;
> +
> +    memory_region_init(&s->apic_container, "icc-apic-container",
> +                       APIC_SPACE_SIZE);
> +    sysbus_init_mmio(sb, &s->apic_container);
> +    ibus->apic_address_space = &s->apic_container;
>  }
>  
>  static const TypeInfo icc_bridge_info = {
> diff --git a/hw/icc_bus.h b/hw/icc_bus.h
> index b8e2032..a3d9e0a 100644
> --- a/hw/icc_bus.h
> +++ b/hw/icc_bus.h
> @@ -28,6 +28,7 @@
>  typedef struct ICCBus {
>      BusState qbus;
>      MemoryRegion *ioapic_address_space;
> +    MemoryRegion *apic_address_space;
>  } ICCBus;
>  #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
>  
> @@ -50,4 +51,3 @@ typedef struct ICCDeviceClass {
>  
>  #endif /* CONFIG_USER_ONLY */
>  #endif
> -
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 2b3c1f3..32f7aea 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -41,10 +41,10 @@
>  #endif
>  
>  #include "sysemu/sysemu.h"
> +#include "hw/qdev-properties.h"
>  #include "hw/icc_bus.h"
>  #ifndef CONFIG_USER_ONLY
>  #include "hw/xen.h"
> -#include "hw/sysbus.h"
>  #include "hw/apic_internal.h"
>  #endif
>  
> @@ -2104,6 +2104,7 @@ static void mce_init(X86CPU *cpu)
>  static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
>  {
>      CPUX86State *env = &cpu->env;
> +    DeviceState *dev = DEVICE(cpu);
>      APICCommonState *apic;
>      const char *apic_type = "apic";
>  
> @@ -2113,7 +2114,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error
> **errp) apic_type = "xen-apic";
>      }
>  
> -    env->apic_state = qdev_try_create(NULL, apic_type);
> +    env->apic_state = qdev_try_create(dev->parent_bus, apic_type);
>      if (env->apic_state == NULL) {
>          error_setg(errp, "APIC device '%s' could not be created",
> apic_type); return;
> @@ -2130,7 +2131,6 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error
> **errp) static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
>  {
>      CPUX86State *env = &cpu->env;
> -    static int apic_mapped;
>  
>      if (env->apic_state == NULL) {
>          return;
> @@ -2141,16 +2141,6 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error
> **errp) object_get_typename(OBJECT(env->apic_state)));
>          return;
>      }
> -
> -    /* XXX: mapping more APICs at the same memory location */
> -    if (apic_mapped == 0) {
> -        /* NOTE: the APIC is directly connected to the CPU - it is not
> -           on the global memory bus. */
> -        /* XXX: what if the base changes? */
> -        sysbus_mmio_map_overlap(SYS_BUS_DEVICE(env->apic_state), 0,
> -                                APIC_DEFAULT_ADDRESS, 0x1000);
> -        apic_mapped = 1;
> -    }
>  }
>  #else
>  static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)

Ignore this patch, pls.
"[PATCH 18/22] target-i386: move APIC to ICC bus" is the correct one.

Re: [Qemu-devel] [PATCH 18/22] target-i386: move IOAPIC to ICC bus

[Igor Mammedov]

On Tue, 09 Apr 2013 13:33:56 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 05/04/2013 16:37, Igor Mammedov ha scritto:
> > +    const char *ioapic_name = "ioapic";
> >  
> >      if (kvm_irqchip_in_kernel()) {
> > -        dev = qdev_create(NULL, "kvm-ioapic");
> > -    } else {
> > -        dev = qdev_create(NULL, "ioapic");
> > -    }
> > -    if (parent_name) {
> > -        object_property_add_child(object_resolve_path(parent_name, NULL),
> > -                                  "ioapic", OBJECT(dev), NULL);
> > +        ioapic_name = "kvm-ioapic";
> >      }
> > -    qdev_init_nofail(dev);
> > -    d = SYS_BUS_DEVICE(dev);
> > -    sysbus_mmio_map(d, 0, IO_APIC_DEFAULT_ADDRESS);
> > +    object_property_set_str(OBJECT(dev), ioapic_name, "ioapic-type",
> > NULL);
> 
> Why do you need this?
> 
> Having the IOAPIC as a QOM child of the icc-bridge is not too important,
> I think.  Perhaps not even too correct...
icc-bridge provides address space for IOAPIC, so it looked like a right thing
to make sure that destruction order would be IOAPIC first and only then parent
icc-bridge.

This patch is not really necessary for hot-plug, we could easily drop it, if
you prefer.

BTW:
 looks like this pre-reordering patch slipped in
 Correct one is "[PATCH 19/22] target-i386: move IOAPIC to ICC bus"

> 
> Paolo
> 

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Andreas Färber]

Am 09.04.2013 12:26, schrieb Igor Mammedov:
> On Mon, 8 Apr 2013 17:13:11 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
>> On Fri, Apr 05, 2013 at 04:36:58PM +0200, Igor Mammedov wrote:
>>> ... and call it if defined from CPUClass.realize() if CPU was hotplugged
>>>
>>> by default leave .resume() unset (i.e. NULL) and override it for softmmu
>>> in qemu_init_vcpu() if it's still unset.
>>>
>>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>> [...]
>>> diff --git a/cpus.c b/cpus.c
>>> index 9b9a32f..6b793c5 100644
>>> --- a/cpus.c
>>> +++ b/cpus.c
>>> @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
>> [...]
>>> @@ -1042,7 +1047,11 @@ void qemu_init_vcpu(void *_env)
>>>  {
>>>      CPUArchState *env = _env;
>>>      CPUState *cpu = ENV_GET_CPU(env);
>>> +    CPUClass *klass = CPU_GET_CLASS(cpu);
>>>  
>>> +    if (klass->resume == NULL) {
>>> +        klass->resume = resume_vcpu;
>>> +    }
>>
>> So you are initializing a field of CPUClass struct inside a CPU object
>> initialization function. And that's a function that is not even
>> converted to QOM yet, and buried inside a non-trivial function call tree
>> (hence easy to be called at the wrong time if one day we reorder the
>> initialization steps).
> init step are not likely to change this drastically but main reason why
> it's here see below.
> 
>>
>> Can't we do this on class_init(), where it belongs? If we need different
>> implementations for softmmu/user, we can add a stub for *-user. I think
>> even an explicit #ifdef inside resume_vcpu() would be preferable to
>> this.
> Generally I agree with you that class_init() would be more correct, but
> ifdefs in qom/cpu.c defeat purpose to build qom/cpu.c only once for all
> targets, that Andreas are working towards.

Indeed. Anthony had suggested to introduce a cpu-softmmu.c or something
like that it we ever need to distinguish. Not sure how they would
interface though. ;)

Andreas

> 
> It asks surely asks for resume_vcpu() stub in libqemustubs, and I'd do it
> there weren't objections to it.
> 
> Paolo,
>  it looks like a good candidate for libqemustubs, would it be ok to add stub
>  there?
> 
>>
>>
>>>      cpu->nr_cores = smp_cores;
>>>      cpu->nr_threads = smp_threads;
>>>      cpu->stopped = true;
>> [...]
>>
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 03/22] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Andreas Färber]

Am 09.04.2013 12:33, schrieb Igor Mammedov:
> On Tue, 09 Apr 2013 12:30:21 +0200
> Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
>> Il 08/04/2013 20:30, Eduardo Habkost ha scritto:
>>>>> -    cpu_x86_register(cpu, name, &error);
>>>>> -    if (error) {
>>>>> +    cpu_x86_register(cpu, name, errp);
>>>>> +    if (error_is_set(errp)) {
>>> So the function now does error checking properly if and only if errp is
>>> not NULL. Do we really want to do that?
>>
>> No, using error_propagate is the correct idiom indeed.
> Ok, I'll use  error_propagate() in next version.

I remember accepting some CPU refactoring patch but asking you to fix up
the same issue as follow-up - I don't remember having received a single
fix-up patch, could you please check on whether that is hidden in some
series or still missing? Thanks!

Andreas

> 
>>
>> Paolo
>>
>>>>>          goto out;
>>>>>      }
>>>>>  
>>>>> -    cpu_x86_parse_featurestr(cpu, features, &error);
>>>>> -    if (error) {
>>>>> +    cpu_x86_parse_featurestr(cpu, features, errp);
>>>>> +    if (error_is_set(errp)) {
>>>>>          goto out;
>>>>>      }
>>>>>  
>>>>> -    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
>>>>> +out:
>>>>> +    g_strfreev(model_pieces);
>>> Any specific reason you didn't choose to keep 'Error *error = NULL'
>>> inside cpu_x86_create() as well, and use error_propagate() here? I
>>> believe it would make the patch simpler and easier to review, and at the
>>> same time make cpu_x86_init() check for errors properly even if errp is
>>> NULL. This is the opposite of what you did on x86_cpu_realizefn() at
>>> patch 01/22.
>>
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 01/22] target-i386: consolidate error propagation in x86_cpu_realizefn()

[Andreas Färber]

Am 05.04.2013 16:36, schrieb Igor Mammedov:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Thanks, applied to qom-cpu:
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 02/22] target-i386: split APIC creation from initialization in x86_cpu_realizefn()

[Andreas Färber]

Am 05.04.2013 16:36, schrieb Igor Mammedov:
> When APIC is hotplugged during CPU hotplug, device_set_realized()
> calls device_reset() on it. And if QEMU runs in KVM mode, following
> call chain will fail:
>     apic_reset_common()
>         -> kvm_apic_vapic_base_update()
>             -> kvm_vcpu_ioctl(cpu->kvm_fd,...)
> due to cpu->kvm_fd not being initialized yet.
> 
> cpu->kvm_fd is initialized during qemu_init_vcpu() call but x86_cpu_apic_init()
> can't be moved after it because kvm_init_vcpu() -> kvm_arch_reset_vcpu()
> relies on APIC to determine if CPU is BSP for setting initial env->mp_state.
> 
> So split APIC device creation from its initialization and realize APIC
> after CPU is created, when it's safe to call APIC's reset method.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * s/x86_cpu_apic_init()/x86_cpu_apic_realize()/

Thanks, applied to qom-cpu (dropping "call" from commit message to
squeeze it into 76 chars):
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

On Fri, 5 Apr 2013 14:10:54 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index db542f6..a760ed5 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -1387,6 +1387,17 @@
> >  { 'command': 'cpu', 'data': {'index': 'int'} }
> >  
> >  ##
> > +# @cpu-add
> > +#
> > +# Adds CPU with specified id
> > +#
> > +# @id: cpu id of CPU to be created
> 
> Can we have the semantics/constraints of "id" documented here? Is it an
> arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
it's generic function so documenting it as APIC ID is not appropriate.

I for sure should document it on cpu-hotplug wiki page though, for x86 use
case for starters. i.e. how to use QMP to get a list of available/free IDs.
and in which order to use them.

> it have to be the index of the CPU in the CPU list? How the IDs of
> existing CPUs set using "-smp" are allocated?
With current -smp implementation the same way as it was before,
and for migration to work hot-plugged CPU has to be the next unused APIC
ID in their sequence, so that target qemu could be started with "-smp n+1".

But -smp along with -numa should be reworked to allow specifying guest visible
CPU IDs for arbitrary CPU hotplug to work.

when we done with QOMifying CPUs it might be possible to use -device for them
and keeping -smp for compat/shorcut purposes.

> 
> I am looking at the code right now to understand how this implementation
> works, but the documentation could contain or point to documentation on
> how the "id" parameter is used and interpreted.
I'll add pointer to wiki and describe there target-i386 use-case.

> 
> > +#
> > +# Returns: Nothing on success
> > +##
> > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > +
> > +##
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Eduardo Habkost]

On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> On Fri, 5 Apr 2013 14:10:54 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > [...]
> > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > index db542f6..a760ed5 100644
> > > --- a/qapi-schema.json
> > > +++ b/qapi-schema.json
> > > @@ -1387,6 +1387,17 @@
> > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > >  
> > >  ##
> > > +# @cpu-add
> > > +#
> > > +# Adds CPU with specified id
> > > +#
> > > +# @id: cpu id of CPU to be created
> > 
> > Can we have the semantics/constraints of "id" documented here? Is it an
> > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> it's generic function so documenting it as APIC ID is not appropriate.
> 
> I for sure should document it on cpu-hotplug wiki page though, for x86 use
> case for starters. i.e. how to use QMP to get a list of available/free IDs.
> and in which order to use them.
> 
> > it have to be the index of the CPU in the CPU list? How the IDs of
> > existing CPUs set using "-smp" are allocated?
> With current -smp implementation the same way as it was before,
> and for migration to work hot-plugged CPU has to be the next unused APIC
> ID in their sequence, so that target qemu could be started with "-smp n+1".

The problem is that it's hard to find out what's the APIC ID for each
CPU mentioned in the command-line.

For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
be:

online on startup:
package 0, core 0: 0 1 2
package 0, core 1: 4 5 6
package 0, core 2: 8 9 10
package 1, core 0: 16 17 18
package 1, core 1: 20 21 22
package 1, core 2: 24 25 26

offline on startup:
package 2, core 0: 32 33 34
package 2, core 1: 36 37 38
package 2, core 2: 40 41 42
package 3, core 0: 48 49 50
package 3, core 1: 52 53 54
package 3, core 2: 56 57 58


What should the caller do to find out the correct ID for each of the 36
VCPUs? This should be clearly documented.


> 
> But -smp along with -numa should be reworked to allow specifying guest visible
> CPU IDs for arbitrary CPU hotplug to work.

I'm curious how you plan to make this work while keeping command-line
compatibility. See the question I sent on my other message, about how to
map the IDs used on -numa (that are "CPU indexes") to the IDs required
by cpu-add.


> 
> when we done with QOMifying CPUs it might be possible to use -device for them
> and keeping -smp for compat/shorcut purposes.
> 
> > 
> > I am looking at the code right now to understand how this implementation
> > works, but the documentation could contain or point to documentation on
> > how the "id" parameter is used and interpreted.
> I'll add pointer to wiki and describe there target-i386 use-case.

Thanks! Could you try to document it succintly inside qapi-schema.json
as well? Maybe just a pointer to other documents would be useful.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

On Tue, 9 Apr 2013 17:46:21 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > On Fri, 5 Apr 2013 14:10:54 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > [...]
> > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > index db542f6..a760ed5 100644
> > > > --- a/qapi-schema.json
> > > > +++ b/qapi-schema.json
> > > > @@ -1387,6 +1387,17 @@
> > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > >  
> > > >  ##
> > > > +# @cpu-add
> > > > +#
> > > > +# Adds CPU with specified id
> > > > +#
> > > > +# @id: cpu id of CPU to be created
> > > 
> > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > it's generic function so documenting it as APIC ID is not appropriate.
> > 
> > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > and in which order to use them.
> > 
> > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > existing CPUs set using "-smp" are allocated?
> > With current -smp implementation the same way as it was before,
> > and for migration to work hot-plugged CPU has to be the next unused APIC
> > ID in their sequence, so that target qemu could be started with "-smp n+1".
> 
> The problem is that it's hard to find out what's the APIC ID for each
> CPU mentioned in the command-line.
> 
> For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> be:
> 
> online on startup:
> package 0, core 0: 0 1 2
> package 0, core 1: 4 5 6
> package 0, core 2: 8 9 10
> package 1, core 0: 16 17 18
> package 1, core 1: 20 21 22
> package 1, core 2: 24 25 26
> 
> offline on startup:
> package 2, core 0: 32 33 34
> package 2, core 1: 36 37 38
> package 2, core 2: 40 41 42
> package 3, core 0: 48 49 50
> package 3, core 1: 52 53 54
> package 3, core 2: 56 57 58
> 
> 
> What should the caller do to find out the correct ID for each of the 36
> VCPUs? This should be clearly documented.
Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
as /machine/icc-bridge/cpu[0..n]
And since in above sequence IDs are monotonously increasing it's enough to use
the next unused APIC ID to make -smp n+1 work. 
> 
> 
> > 
> > But -smp along with -numa should be reworked to allow specifying guest visible
> > CPU IDs for arbitrary CPU hotplug to work.
> 
> I'm curious how you plan to make this work while keeping command-line
> compatibility. See the question I sent on my other message, about how to
> map the IDs used on -numa (that are "CPU indexes") to the IDs required
> by cpu-add.
It doesn't mean that we should stick to bad/insufficient interface forever. 
We could add new one that does it right and keep old one for a time being for
compatibility.

> 
> > 
> > when we done with QOMifying CPUs it might be possible to use -device for them
> > and keeping -smp for compat/shorcut purposes.
> > 
> > > 
> > > I am looking at the code right now to understand how this implementation
> > > works, but the documentation could contain or point to documentation on
> > > how the "id" parameter is used and interpreted.
> > I'll add pointer to wiki and describe there target-i386 use-case.
> 
> Thanks! Could you try to document it succintly inside qapi-schema.json
> as well? Maybe just a pointer to other documents would be useful.
It's very target specific, so separate document probably would make more sense.

> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[li guang]

在 2013-04-09二的 13:36 +0200，Paolo Bonzini写道：
> Il 08/04/2013 13:32, Igor Mammedov ha scritto:
> > 
> >> > This patch should be combined with [PATCH 07/22]
> >> > 
> >> > 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> >>> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> >>> > > ---
> >>> > >  hw/ioapic_common.c | 2 +-
> >>> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> >>> > > 
> >>> > > diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
> >>> > > index d4aff29..561b987 100644
> >>> > > --- a/hw/ioapic_common.c
> >>> > > +++ b/hw/ioapic_common.c
> >>> > > @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int
> >>> > > version_id) 
> >>> > >  static int ioapic_init_common(SysBusDevice *dev)
> >>> > >  {
> >>> > > -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
> >>> > > +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
> >>> > >      IOAPICCommonClass *info;
> >>> > >      static int ioapic_no;
> >>> > >  
> >> > 
> >> > 
> >> > 
> > sure, will do this on the next respin.
> 
> Actually no, why?  It's two different files.

because they do the same trivial thing,
do you want some mechanic changes separated?

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Paolo Bonzini]

Il 10/04/2013 02:21, li guang ha scritto:
>>>>> > >> > 
>>>>> > >> > 
>>> > > sure, will do this on the next respin.
>> > 
>> > Actually no, why?  It's two different files.
> because they do the same trivial thing,
> do you want some mechanic changes separated?

If it makes sense, yes.

Paolo

Re: [Qemu-devel] [PATCH 06/22] cpu: introduce CPUClass.resume() method

[Igor Mammedov]

On Tue, 09 Apr 2013 13:20:27 +0200
Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 08/04/2013 22:13, Eduardo Habkost ha scritto:
> > So you are initializing a field of CPUClass struct inside a CPU object
> > initialization function. And that's a function that is not even
> > converted to QOM yet, and buried inside a non-trivial function call tree
> > (hence easy to be called at the wrong time if one day we reorder the
> > initialization steps).
> > 
> > Can't we do this on class_init(), where it belongs? If we need different
> > implementations for softmmu/user, we can add a stub for *-user.
> 
> Yes, please add a stub for the new function and override it in cpus.c.
Ok, I'll revert it to a way as it was in RFC
http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg03774.html 
minus crept in cpu_synchronize_post_init() line.

There is no much use in introducing resume method when there are
only one implementation and stub.

> 
> > I think even an explicit #ifdef inside resume_vcpu() would be
> > preferable to this.
> 
> Using an #ifdef basically means putting it in exec.c.  I'm not sure
> about that, it seems to fit more in cpus.c.
> 
> Paolo
> 

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Igor Mammedov]

On Wed, 10 Apr 2013 08:21:17 +0800
li guang <lig.fnst@cn.fujitsu.com> wrote:

> 在 2013-04-09二的 13:36 +0200，Paolo Bonzini写道：
> > Il 08/04/2013 13:32, Igor Mammedov ha scritto:
> > > 
> > >> > This patch should be combined with [PATCH 07/22]
> > >> > 
> > >> > 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
> > >>> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > >>> > > ---
> > >>> > >  hw/ioapic_common.c | 2 +-
> > >>> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >>> > > 
> > >>> > > diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
> > >>> > > index d4aff29..561b987 100644
> > >>> > > --- a/hw/ioapic_common.c
> > >>> > > +++ b/hw/ioapic_common.c
> > >>> > > @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void
> > >>> > > *opaque, int version_id) 
> > >>> > >  static int ioapic_init_common(SysBusDevice *dev)
> > >>> > >  {
> > >>> > > -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
> > >>> > > +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
> > >>> > >      IOAPICCommonClass *info;
> > >>> > >      static int ioapic_no;
> > >>> > >  
> > >> > 
> > >> > 
> > >> > 
> > > sure, will do this on the next respin.
> > 
> > Actually no, why?  It's two different files.
> 
> because they do the same trivial thing,
> do you want some mechanic changes separated?

There is no point arguing,
Andreas applied the first patch to qom-cpu tree already,
and seems he is fine taking them as separate patches.

Re: [Qemu-devel] [PATCH 07/22] target-i386: kvmvapic: replace FROM_SYSBUS() with QOM type cast

[Andreas Färber]

Am 05.04.2013 16:36, schrieb Igor Mammedov:
> ... and define type name and type cast macro for kvmvapic according
> to accepted convention.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * s/VAPIC_DEVICE/VAPIC/; s/TYPE_VAPIC_DEVICE/TYPE_VAPIC/
> 
> Note: stray cleanup, since I excluded following patch that converted
> kvmvapic to ICCDevice

It's a valid cleanup of a CPU-related device, so I'm happily taking it.

Thanks, applied to qom-cpu (dropping "target-i386"):
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

> ---
>  hw/i386/kvmvapic.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
> index cc95e5c..c4be882 100644
> --- a/hw/i386/kvmvapic.c
> +++ b/hw/i386/kvmvapic.c
> @@ -60,6 +60,9 @@ typedef struct VAPICROMState {
>      bool rom_mapped_writable;
>  } VAPICROMState;
>  
> +#define TYPE_VAPIC "kvmvapic"
> +#define VAPIC(obj) OBJECT_CHECK(VAPICROMState, (obj), TYPE_VAPIC)
> +
>  #define TPR_INSTR_ABS_MODRM             0x1
>  #define TPR_INSTR_MATCH_MODRM_REG       0x2
>  
> @@ -690,7 +693,7 @@ static const MemoryRegionOps vapic_ops = {
>  
>  static int vapic_init(SysBusDevice *dev)
>  {
> -    VAPICROMState *s = FROM_SYSBUS(VAPICROMState, dev);
> +    VAPICROMState *s = VAPIC(dev);
>  
>      memory_region_init_io(&s->io, &vapic_ops, s, "kvmvapic", 2);
>      sysbus_add_io(dev, VAPIC_IO_PORT, &s->io);
> @@ -806,7 +809,7 @@ static void vapic_class_init(ObjectClass *klass, void *data)
>  }
>  
>  static const TypeInfo vapic_type = {
> -    .name          = "kvmvapic",
> +    .name          = TYPE_VAPIC,
>      .parent        = TYPE_SYS_BUS_DEVICE,
>      .instance_size = sizeof(VAPICROMState),
>      .class_init    = vapic_class_init,
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Andreas Färber]

Am 05.04.2013 16:37, schrieb Igor Mammedov:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Thanks, applied to qom-cpu (dropping "target-i386"):
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

> ---
>  hw/ioapic_common.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
> index d4aff29..561b987 100644
> --- a/hw/ioapic_common.c
> +++ b/hw/ioapic_common.c
> @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
>  
>  static int ioapic_init_common(SysBusDevice *dev)
>  {
> -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
> +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
>      IOAPICCommonClass *info;
>      static int ioapic_no;
>  

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 08/22] target-i386: ioapic: replace FROM_SYSBUS() with QOM type cast

[Andreas Färber]

Am 10.04.2013 18:12, schrieb Igor Mammedov:
> On Wed, 10 Apr 2013 08:21:17 +0800
> li guang <lig.fnst@cn.fujitsu.com> wrote:
> 
>> 在 2013-04-09二的 13:36 +0200，Paolo Bonzini写道：
>>> Il 08/04/2013 13:32, Igor Mammedov ha scritto:
>>>>
>>>>>> This patch should be combined with [PATCH 07/22]
>>>>>>
>>>>>> 在 2013-04-05五的 16:37 +0200，Igor Mammedov写道：
>>>>>>>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>>>>>>> ---
>>>>>>>>  hw/ioapic_common.c | 2 +-
>>>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/hw/ioapic_common.c b/hw/ioapic_common.c
>>>>>>>> index d4aff29..561b987 100644
>>>>>>>> --- a/hw/ioapic_common.c
>>>>>>>> +++ b/hw/ioapic_common.c
>>>>>>>> @@ -59,7 +59,7 @@ static int ioapic_dispatch_post_load(void
>>>>>>>> *opaque, int version_id) 
>>>>>>>>  static int ioapic_init_common(SysBusDevice *dev)
>>>>>>>>  {
>>>>>>>> -    IOAPICCommonState *s = FROM_SYSBUS(IOAPICCommonState, dev);
>>>>>>>> +    IOAPICCommonState *s = IOAPIC_COMMON(dev);
>>>>>>>>      IOAPICCommonClass *info;
>>>>>>>>      static int ioapic_no;
>>>>>>>>  
>>>>>>
>>>>>>
>>>>>>
>>>> sure, will do this on the next respin.
>>>
>>> Actually no, why?  It's two different files.
>>
>> because they do the same trivial thing,
>> do you want some mechanic changes separated?
> 
> There is no point arguing,
> Andreas applied the first patch to qom-cpu tree already,
> and seems he is fine taking them as separate patches.

Yes, I had the second one in my testing queue, too. It is exactly the
way I'd expect it, modulo that it doesn't touch target-i386 directory it
mentions. That is assuming that there are no further FROM_SYSBUS() or
DO_UPCAST()s hidden elsewhere in the respective device, which I didn't
check yet.

Finding an appropriate subject for an equivalent change touching two
files is one aspect for separate patches, individually being able to
bisect problems another. If it were not two but a hundred one-line
patches, that were a different issue of course. But unfortunately there
are quite a lot of FROM_SYSBUS() users, most of which require more work
than this one here, so doing all in one "sysbus:" patch seems unrealistic.

BTW since these are SysBus devices, if someone has time to convert the
initfns to QOM realize as follow-ups, I would be happy to review.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Eduardo Habkost]

On Tue, Apr 09, 2013 at 11:05:26PM +0200, Igor Mammedov wrote:
> On Tue, 9 Apr 2013 17:46:21 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > > On Fri, 5 Apr 2013 14:10:54 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > > [...]
> > > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > > index db542f6..a760ed5 100644
> > > > > --- a/qapi-schema.json
> > > > > +++ b/qapi-schema.json
> > > > > @@ -1387,6 +1387,17 @@
> > > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > > >  
> > > > >  ##
> > > > > +# @cpu-add
> > > > > +#
> > > > > +# Adds CPU with specified id
> > > > > +#
> > > > > +# @id: cpu id of CPU to be created
> > > > 
> > > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > > it's generic function so documenting it as APIC ID is not appropriate.
> > > 
> > > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > > and in which order to use them.
> > > 
> > > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > > existing CPUs set using "-smp" are allocated?
> > > With current -smp implementation the same way as it was before,
> > > and for migration to work hot-plugged CPU has to be the next unused APIC
> > > ID in their sequence, so that target qemu could be started with "-smp n+1".
> > 
> > The problem is that it's hard to find out what's the APIC ID for each
> > CPU mentioned in the command-line.
> > 
> > For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> > will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> > be:
> > 
> > online on startup:
> > package 0, core 0: 0 1 2
> > package 0, core 1: 4 5 6
> > package 0, core 2: 8 9 10
> > package 1, core 0: 16 17 18
> > package 1, core 1: 20 21 22
> > package 1, core 2: 24 25 26
> > 
> > offline on startup:
> > package 2, core 0: 32 33 34
> > package 2, core 1: 36 37 38
> > package 2, core 2: 40 41 42
> > package 3, core 0: 48 49 50
> > package 3, core 1: 52 53 54
> > package 3, core 2: 56 57 58
> > 
> > 
> > What should the caller do to find out the correct ID for each of the 36
> > VCPUs? This should be clearly documented.
> Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
> as /machine/icc-bridge/cpu[0..n]
> And since in above sequence IDs are monotonously increasing it's enough to use
> the next unused APIC ID to make -smp n+1 work. 

This can be one method to map CPU indexes to APIC IDs, yes. I find it
hard to explain and hard to use, and it imposes constraints on the way
the target-specific IDs are calculated (requiring them to be
monotonically increasing). But it may be a reasonable solution by now.

I have one additional question about the icc-bridge paths: are the link
paths on icc-bridge explicitly documented/required to be APIC IDs, or
the caller should assume they are arbitrary IDs with no specific
meaning?


> > 
> > 
> > > 
> > > But -smp along with -numa should be reworked to allow specifying guest visible
> > > CPU IDs for arbitrary CPU hotplug to work.
> > 
> > I'm curious how you plan to make this work while keeping command-line
> > compatibility. See the question I sent on my other message, about how to
> > map the IDs used on -numa (that are "CPU indexes") to the IDs required
> > by cpu-add.
> It doesn't mean that we should stick to bad/insufficient interface forever. 
> We could add new one that does it right and keep old one for a time being for
> compatibility.

The problem is that I only see three possible kinds of CPU identifiers
that could work in the command-line:

 * Arbitrary user-defined IDs
 * CPU indexes (the current interface)
 * Topology-based identifiers/paths (e.g.
   "/machine/numa_node[0]/cpu_socket[1]/core[2]/thread[1]")

I don't think we can asily use APIC IDs on the command-line because the
caller simply doesn't know what will be the APIC ID for each VCPU.

But this is a problem we can discuss and solve later. By now, we are
stuck with the legacy CPU-index-based interfaces.

> 
> > 
> > > 
> > > when we done with QOMifying CPUs it might be possible to use -device for them
> > > and keeping -smp for compat/shorcut purposes.
> > > 
> > > > 
> > > > I am looking at the code right now to understand how this implementation
> > > > works, but the documentation could contain or point to documentation on
> > > > how the "id" parameter is used and interpreted.
> > > I'll add pointer to wiki and describe there target-i386 use-case.
> > 
> > Thanks! Could you try to document it succintly inside qapi-schema.json
> > as well? Maybe just a pointer to other documents would be useful.
> It's very target specific, so separate document probably would make more sense.

The IDs are target-specific, but do we really need to make the interface
specification/usage to be target-specific? We could have a
target-independent interface to find out what are the available/valid
IDs to use on cpu-add.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

On Fri, 5 Apr 2013 14:24:35 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 05, 2013 at 02:10:54PM -0300, Eduardo Habkost wrote:
> > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > [...]
> > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > index db542f6..a760ed5 100644
> > > --- a/qapi-schema.json
> > > +++ b/qapi-schema.json
> > > @@ -1387,6 +1387,17 @@
> > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > >  
> > >  ##
> > > +# @cpu-add
> > > +#
> > > +# Adds CPU with specified id
> > > +#
> > > +# @id: cpu id of CPU to be created
> > 
> > Can we have the semantics/constraints of "id" documented here? Is it an
> > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > it have to be the index of the CPU in the CPU list? How the IDs of
> > existing CPUs set using "-smp" are allocated?
> > 
> > I am looking at the code right now to understand how this implementation
> > works, but the documentation could contain or point to documentation on
> > how the "id" parameter is used and interpreted.
> 
> So, the answer to my own question seems to be on patch 21/22. Let me
> check if I understand this correctly:
> 
>  @id: cpu id of the CPU to be created. It must be one of the the
>       available IDs listed at the "/machine/icc-bridge/cpu[0..N]" links.
> 
> Is the above correct?
yes.

> 
> Now, my question is: suppose a caller starts QEMU as:
> 
>  $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
>         -numa node,cpus=0-8   -numa node,cpus=9-17 \
>         -numa node,cpus=18-26 -numa node,cpus=27-35
> 
> and the caller wants to hot-add all VCPUs in the last CPU socket (that
> means: all the VCPUs in the last NUMA node, that means: CPU indexes
> 27-35). What should the caller do to find out which of the
> /machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
> VCPUs?
If one wants plug in a specific CPU, It won't work with -numa yet.

to make it work we need to specify sockets on -numa cmd. line and then
construct synthetic QOM containers tree that could looklike:

/machine/numa_nodes/[0..N]/sockets/[0..M]/cpus/[0..X]

then command line could look like:
$ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
       -numa node,sockets=0-1 -numa node,sockets=2-3
> 
> 
> > 
> > > +#
> > > +# Returns: Nothing on success
> > > +##
> > > +{ 'command': 'cpu-add', 'data': {'id': 'int'} }
> > > +
> > > +##
> > -- 
> > Eduardo
> > 
> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

On Thu, 11 Apr 2013 12:12:37 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Tue, Apr 09, 2013 at 11:05:26PM +0200, Igor Mammedov wrote:
> > On Tue, 9 Apr 2013 17:46:21 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Tue, Apr 09, 2013 at 10:19:11PM +0200, Igor Mammedov wrote:
> > > > On Fri, 5 Apr 2013 14:10:54 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Fri, Apr 05, 2013 at 04:37:16PM +0200, Igor Mammedov wrote:
> > > > > [...]
> > > > > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > > > > index db542f6..a760ed5 100644
> > > > > > --- a/qapi-schema.json
> > > > > > +++ b/qapi-schema.json
> > > > > > @@ -1387,6 +1387,17 @@
> > > > > >  { 'command': 'cpu', 'data': {'index': 'int'} }
> > > > > >  
> > > > > >  ##
> > > > > > +# @cpu-add
> > > > > > +#
> > > > > > +# Adds CPU with specified id
> > > > > > +#
> > > > > > +# @id: cpu id of CPU to be created
> > > > > 
> > > > > Can we have the semantics/constraints of "id" documented here? Is it an
> > > > > arbitrary ID chosen by the caller? Does it have to be the APIC ID? Does
> > > > it's generic function so documenting it as APIC ID is not appropriate.
> > > > 
> > > > I for sure should document it on cpu-hotplug wiki page though, for x86 use
> > > > case for starters. i.e. how to use QMP to get a list of available/free IDs.
> > > > and in which order to use them.
> > > > 
> > > > > it have to be the index of the CPU in the CPU list? How the IDs of
> > > > > existing CPUs set using "-smp" are allocated?
> > > > With current -smp implementation the same way as it was before,
> > > > and for migration to work hot-plugged CPU has to be the next unused APIC
> > > > ID in their sequence, so that target qemu could be started with "-smp n+1".
> > > 
> > > The problem is that it's hard to find out what's the APIC ID for each
> > > CPU mentioned in the command-line.
> > > 
> > > For example, if you use "-smp 18,cores=3,threads=3,maxcpus=36" thread ID
> > > will use 2 bits, core ID will use 2 bits, the APIC IDs on startup will
> > > be:
> > > 
> > > online on startup:
> > > package 0, core 0: 0 1 2
> > > package 0, core 1: 4 5 6
> > > package 0, core 2: 8 9 10
> > > package 1, core 0: 16 17 18
> > > package 1, core 1: 20 21 22
> > > package 1, core 2: 24 25 26
> > > 
> > > offline on startup:
> > > package 2, core 0: 32 33 34
> > > package 2, core 1: 36 37 38
> > > package 2, core 2: 40 41 42
> > > package 3, core 0: 48 49 50
> > > package 3, core 1: 52 53 54
> > > package 3, core 2: 56 57 58
> > > 
> > > 
> > > What should the caller do to find out the correct ID for each of the 36
> > > VCPUs? This should be clearly documented.
> > Patch 21/22 exposes all APIC IDs (including offline) as links via QOM
> > as /machine/icc-bridge/cpu[0..n]
> > And since in above sequence IDs are monotonously increasing it's enough to use
> > the next unused APIC ID to make -smp n+1 work. 
> 
> This can be one method to map CPU indexes to APIC IDs, yes. I find it
> hard to explain and hard to use, and it imposes constraints on the way
> the target-specific IDs are calculated (requiring them to be
> monotonically increasing). But it may be a reasonable solution by now.
Arbitrary CPU hotplug + migration, require ability to specify which CPU
to create on target. It's probably post CPU-unplug topic or might be
fixed with it.

> 
> I have one additional question about the icc-bridge paths: are the link
> paths on icc-bridge explicitly documented/required to be APIC IDs, or
> the caller should assume they are arbitrary IDs with no specific
> meaning?
since it's on icc-bridge, they are APIC IDs, but from user's POV I wouldn't
care and treat it as opaque.

If we do it in a generic way, then I'll say they are arbitrary IDs.

> 
> > > 
> > > 
> > > > 
> > > > But -smp along with -numa should be reworked to allow specifying guest visible
> > > > CPU IDs for arbitrary CPU hotplug to work.
> > > 
> > > I'm curious how you plan to make this work while keeping command-line
> > > compatibility. See the question I sent on my other message, about how to
> > > map the IDs used on -numa (that are "CPU indexes") to the IDs required
> > > by cpu-add.
> > It doesn't mean that we should stick to bad/insufficient interface forever. 
> > We could add new one that does it right and keep old one for a time being for
> > compatibility.
> 
> The problem is that I only see three possible kinds of CPU identifiers
> that could work in the command-line:
> 
>  * Arbitrary user-defined IDs
>  * CPU indexes (the current interface)
>  * Topology-based identifiers/paths (e.g.
>    "/machine/numa_node[0]/cpu_socket[1]/core[2]/thread[1]")
+1 to topology, others will allow to specify bad configuration

> 
> I don't think we can asily use APIC IDs on the command-line because the
> caller simply doesn't know what will be the APIC ID for each VCPU.
Agree, especially regarding -numa option, it should consume sockets instead.

user might still potentially use opaque ID in cmd line if he will add CPUs
with -device apic_id=xxx, but first he should probe qemu with the same topology
and read complete topology with IDs qemu provides.

> 
> But this is a problem we can discuss and solve later. By now, we are
> stuck with the legacy CPU-index-based interfaces.
> 
> > 
> > > 
> > > > 
> > > > when we done with QOMifying CPUs it might be possible to use -device for them
> > > > and keeping -smp for compat/shorcut purposes.
> > > > 
> > > > > 
> > > > > I am looking at the code right now to understand how this implementation
> > > > > works, but the documentation could contain or point to documentation on
> > > > > how the "id" parameter is used and interpreted.
> > > > I'll add pointer to wiki and describe there target-i386 use-case.
> > > 
> > > Thanks! Could you try to document it succintly inside qapi-schema.json
> > > as well? Maybe just a pointer to other documents would be useful.
> > It's very target specific, so separate document probably would make more sense.
> 
> The IDs are target-specific, but do we really need to make the interface
> specification/usage to be target-specific? We could have a
> target-independent interface to find out what are the available/valid
> IDs to use on cpu-add.
Answered to this in another email about -numa and showed how ID discovery
could be made in unified cross target way.

> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 22/22] add cpu-add qmp command and implement CPU hot-add for target-i386

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 05:17:29PM +0200, Igor Mammedov wrote:
[...]
> > 
> > Now, my question is: suppose a caller starts QEMU as:
> > 
> >  $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
> >         -numa node,cpus=0-8   -numa node,cpus=9-17 \
> >         -numa node,cpus=18-26 -numa node,cpus=27-35
> > 
> > and the caller wants to hot-add all VCPUs in the last CPU socket (that
> > means: all the VCPUs in the last NUMA node, that means: CPU indexes
> > 27-35). What should the caller do to find out which of the
> > /machine/icc-bridge/cpu[0..N] links/IDs really correspond to those
> > VCPUs?
> If one wants plug in a specific CPU, It won't work with -numa yet.

OK, documenting it as unsupported by now would be enough to me. :-)

It looks like -numa is the only interface that requires CPUs to be
individually identified in the command-line, right?

> 
> to make it work we need to specify sockets on -numa cmd. line and then
> construct synthetic QOM containers tree that could looklike:
> 
> /machine/numa_nodes/[0..N]/sockets/[0..M]/cpus/[0..X]
> 
> then command line could look like:
> $ qemu -smp 18,cores=3,threads=3,maxcpus=36 \
>        -numa node,sockets=0-1 -numa node,sockets=2-3

Makes sense to me. Using socket numbers is one way of making the CPU
identifiers be topology-based (as mentioned on a previous message), and
also make it stricter to allow only topologies that actually make sense
for a "/machine/numa_nodes/.../sockets/.../threads/..." tree.

> [...]

-- 
Eduardo