[Qemu-devel] [PATCH RFC 00/14]pc: q35: x2APIC support in kvm_apic mode

[Qemu-devel] [PATCH RFC 00/14]pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

Series is based on top cpu hotplug refactoring(*1),
and extends current CPU/kvm_apic/generic pc machine
code to support x2APIC and upto 288 VCPUs when QEMU
is used with KVM's lapic.

Due to FW_CFG_MAX_CPUS (which is actually apic_id_limit)
being limited to uint16_t, the max possible APIC ID is
limitted to 2^16 with this series but that should
be sufficient for bumping VCPUs number for quite a while.


git gree for testing:
https://github.com/imammedo/qemu.git x2apic_rfc

To play with the feature, one would also need
seabios counterpart:
https://github.com/imammedo/seabios.git x2apic_rfc

Radim's
[RFC 0/9] KVM: x86: break the xAPIC barrier
http://www.spinics.net/lists/kvm/msg132036.html

and Peter's IRQ remapping series.

*1)
[Qemu-devel] [RFC 00/42] ACPI CPU hotplug refactoring to support more than 255 CPUs and PXM/OST methods
https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg00033.html

Igor Mammedov (14):
  pc: acpi: x2APIC support for MADT table
  pc: acpi: x2APIC support for SRAT table
  acpi: cphp: add support for x2APIC in _MAT method
  pc: leave max apic_id_limit only in legacy cpu hotplug code
  pc: apic: move MAX_APICS check to 'apic' class
  pc: apic_common: extend APIC ID property to 32bit
  pc: apic_common: restore APIC ID to initial ID on reset
  pc: apic_common: reset APIC ID to initial ID when switching into
    x2APIC mode
  pc: kvm_apic: pass APIC ID depending on xAPIC/x2APIC mode
  pc: clarify FW_CFG_MAX_CPUS usage comment
  increase MAX_CPUMASK_BITS from 255 to 288
  pc: add 'etc/boot-cpus' fw_cfg file for machine with more than 255
    CPUs
  pc: q35: bump max_cpus to 288
  DO NOT APPLY: hack to make 2nd CPU APIC ID to be x2APIC one

 hw/acpi/cpu.c                   |   5 ++
 hw/acpi/cpu_hotplug.c           |   7 ++-
 hw/arm/virt.c                   |   2 +-
 hw/i386/acpi-build.c            | 114 +++++++++++++++++++++++++++-------------
 hw/i386/kvm/apic.c              |  12 ++++-
 hw/i386/pc.c                    |  34 ++++++------
 hw/i386/pc_piix.c               |   1 +
 hw/i386/pc_q35.c                |   1 +
 hw/intc/apic.c                  |  10 ++++
 hw/intc/apic_common.c           |  54 +++++++++++++++----
 hw/ppc/spapr.c                  |   2 +-
 include/hw/acpi/acpi-defs.h     |  32 +++++++++++
 include/hw/i386/apic_internal.h |   7 ++-
 include/sysemu/sysemu.h         |   2 +-
 target-i386/cpu.c               |   2 +-
 target-i386/cpu.h               |   1 +
 16 files changed, 212 insertions(+), 74 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 01/14] pc: acpi: x2APIC support for MADT table

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/acpi-build.c        | 78 +++++++++++++++++++++++++++++++--------------
 include/hw/acpi/acpi-defs.h | 19 +++++++++++
 2 files changed, 73 insertions(+), 24 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 93f7b5b..668b6bc 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -338,24 +338,38 @@ build_fadt(GArray *table_data, GArray *linker, AcpiPmInfo *pm,
 void pc_madt_cpu_entry(AcpiDeviceIf *adev, int uid,
                        CPUArchIdList *apic_ids, GArray *entry)
 {
-    int apic_id;
-    AcpiMadtProcessorApic *apic = acpi_data_push(entry, sizeof *apic);
-
-    apic_id = apic_ids->cpus[uid].arch_id;
-    apic->type = ACPI_APIC_PROCESSOR;
-    apic->length = sizeof(*apic);
-    apic->processor_id = uid;
-    apic->local_apic_id = apic_id;
-    if (apic_ids->cpus[uid].cpu != NULL) {
-        apic->flags = cpu_to_le32(1);
+    uint32_t apic_id = apic_ids->cpus[uid].arch_id;
+
+    /* ACPI spec says that LAPIC entry for non present
+     * CPU may be omitted from MADT or it must be marked
+     * as disabled. However omitting non present CPU from
+     * MADT breaks hotplug on linux. So possible CPUs
+     * should be put in MADT but kept disabled.
+     */
+    if (apic_id < 255) {
+        AcpiMadtProcessorApic *apic = acpi_data_push(entry, sizeof *apic);
+
+        apic->type = ACPI_APIC_PROCESSOR;
+        apic->length = sizeof(*apic);
+        apic->processor_id = uid;
+        apic->local_apic_id = apic_id;
+        if (apic_ids->cpus[uid].cpu != NULL) {
+            apic->flags = cpu_to_le32(1);
+        } else {
+            apic->flags = cpu_to_le32(0);
+        }
     } else {
-        /* ACPI spec says that LAPIC entry for non present
-         * CPU may be omitted from MADT or it must be marked
-         * as disabled. However omitting non present CPU from
-         * MADT breaks hotplug on linux. So possible CPUs
-         * should be put in MADT but kept disabled.
-         */
-        apic->flags = cpu_to_le32(0);
+        AcpiMadtProcessorX2Apic *apic = acpi_data_push(entry, sizeof *apic);
+
+        apic->type = ACPI_APIC_LOCAL_X2APIC;
+        apic->length = sizeof(*apic);
+        apic->uid = uid;
+        apic->x2apic_id = apic_id;
+        if (apic_ids->cpus[uid].cpu != NULL) {
+            apic->flags = cpu_to_le32(1);
+        } else {
+            apic->flags = cpu_to_le32(0);
+        }
     }
 }
 
@@ -367,11 +381,11 @@ build_madt(GArray *table_data, GArray *linker, PCMachineState *pcms)
     int madt_start = table_data->len;
     AcpiDeviceIfClass *adevc = ACPI_DEVICE_IF_GET_CLASS(pcms->acpi_dev);
     AcpiDeviceIf *adev = ACPI_DEVICE_IF(pcms->acpi_dev);
+    bool x2apic_mode = false;
 
     AcpiMultipleApicTable *madt;
     AcpiMadtIoApic *io_apic;
     AcpiMadtIntsrcovr *intsrcovr;
-    AcpiMadtLocalNmi *local_nmi;
     int i;
 
     madt = acpi_data_push(table_data, sizeof *madt);
@@ -380,6 +394,9 @@ build_madt(GArray *table_data, GArray *linker, PCMachineState *pcms)
 
     for (i = 0; i < apic_ids->len; i++) {
         adevc->madt_cpu(adev, i, apic_ids, table_data);
+        if (apic_ids->cpus[i].arch_id > 254) {
+            x2apic_mode = true;
+        }
     }
     g_free(apic_ids);
 
@@ -413,12 +430,25 @@ build_madt(GArray *table_data, GArray *linker, PCMachineState *pcms)
         intsrcovr->flags  = cpu_to_le16(0xd); /* active high, level triggered */
     }
 
-    local_nmi = acpi_data_push(table_data, sizeof *local_nmi);
-    local_nmi->type         = ACPI_APIC_LOCAL_NMI;
-    local_nmi->length       = sizeof(*local_nmi);
-    local_nmi->processor_id = 0xff; /* all processors */
-    local_nmi->flags        = cpu_to_le16(0);
-    local_nmi->lint         = 1; /* ACPI_LINT1 */
+    if (x2apic_mode) {
+        AcpiMadtLocalX2ApicNmi *local_nmi;
+
+        local_nmi = acpi_data_push(table_data, sizeof *local_nmi);
+        local_nmi->type   = ACPI_APIC_LOCAL_X2APIC_NMI;
+        local_nmi->length = sizeof(*local_nmi);
+        local_nmi->uid    = 0xFFFFFFFF; /* all processors */
+        local_nmi->flags  = cpu_to_le16(0);
+        local_nmi->lint   = 1; /* ACPI_LINT1 */
+    } else {
+        AcpiMadtLocalNmi *local_nmi;
+
+        local_nmi = acpi_data_push(table_data, sizeof *local_nmi);
+        local_nmi->type         = ACPI_APIC_LOCAL_NMI;
+        local_nmi->length       = sizeof(*local_nmi);
+        local_nmi->processor_id = 0xff; /* all processors */
+        local_nmi->flags        = cpu_to_le16(0);
+        local_nmi->lint         = 1; /* ACPI_LINT1 */
+    }
 
     build_header(linker, table_data,
                  (void *)(table_data->data + madt_start), "APIC",
diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index c7a03d4..a821248 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -342,6 +342,25 @@ struct AcpiMadtLocalNmi {
 } QEMU_PACKED;
 typedef struct AcpiMadtLocalNmi AcpiMadtLocalNmi;
 
+struct AcpiMadtProcessorX2Apic
+{
+    ACPI_SUB_HEADER_DEF
+    uint16_t reserved;
+    uint32_t x2apic_id;              /* Processor's local x2APIC ID */
+    uint32_t flags;
+    uint32_t uid;                    /* Processor object _UID */
+} QEMU_PACKED;
+typedef struct AcpiMadtProcessorX2Apic AcpiMadtProcessorX2Apic;
+
+struct AcpiMadtLocalX2ApicNmi {
+    ACPI_SUB_HEADER_DEF
+    uint16_t flags;                  /* MPS INTI flags */
+    uint32_t uid;                    /* Processor object _UID */
+    uint8_t  lint;                   /* Local APIC LINT# */
+    uint8_t  reserved[3];            /* Local APIC LINT# */
+} QEMU_PACKED;
+typedef struct AcpiMadtLocalX2ApicNmi AcpiMadtLocalX2ApicNmi;
+
 struct AcpiMadtGenericInterrupt {
     ACPI_SUB_HEADER_DEF
     uint16_t reserved;
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 02/14] pc: acpi: x2APIC support for SRAT table

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/acpi-build.c        | 36 ++++++++++++++++++++++++------------
 include/hw/acpi/acpi-defs.h | 13 +++++++++++++
 2 files changed, 37 insertions(+), 12 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 668b6bc..e5ba704 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -2366,7 +2366,6 @@ static void
 build_srat(GArray *table_data, GArray *linker, MachineState *machine)
 {
     AcpiSystemResourceAffinityTable *srat;
-    AcpiSratProcessorAffinity *core;
     AcpiSratMemoryAffinity *numamem;
 
     int i;
@@ -2386,17 +2385,30 @@ build_srat(GArray *table_data, GArray *linker, MachineState *machine)
     srat->reserved1 = cpu_to_le32(1);
 
     for (i = 0; i < apic_ids->len; i++) {
-        int apic_id = apic_ids->cpus[i].arch_id;
-
-        core = acpi_data_push(table_data, sizeof *core);
-        core->type = ACPI_SRAT_PROCESSOR;
-        core->length = sizeof(*core);
-        core->local_apic_id = apic_id;
-        curnode = pcms->node_cpu[i];
-        core->proximity_lo = curnode;
-        memset(core->proximity_hi, 0, 3);
-        core->local_sapic_eid = 0;
-        core->flags = cpu_to_le32(1);
+        uint32_t apic_id = apic_ids->cpus[i].arch_id;
+
+        if (apic_id < 255) {
+            AcpiSratProcessorAffinity *core;
+
+            core = acpi_data_push(table_data, sizeof *core);
+            core->type = ACPI_SRAT_PROCESSOR;
+            core->length = sizeof(*core);
+            core->local_apic_id = apic_id;
+            curnode = pcms->node_cpu[i];
+            core->proximity_lo = curnode;
+            memset(core->proximity_hi, 0, 3);
+            core->local_sapic_eid = 0;
+            core->flags = cpu_to_le32(1);
+        } else {
+            AcpiSratProcessorX2ApicAffinity *core;
+
+            core = acpi_data_push(table_data, sizeof *core);
+            core->type = ACPI_SRAT_PROCESSOR_X2APIC;
+            core->length = sizeof(*core);
+            core->x2apic_id = apic_id;
+            core->proximity_domain = cpu_to_le32(pcms->node_cpu[i]);
+            core->flags = cpu_to_le32(1);
+        }
     }
 
 
diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index a821248..9af349c 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -476,6 +476,7 @@ typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
 #define ACPI_SRAT_PROCESSOR          0
 #define ACPI_SRAT_MEMORY             1
+#define ACPI_SRAT_PROCESSOR_X2APIC   2
 
 struct AcpiSratProcessorAffinity
 {
@@ -489,6 +490,18 @@ struct AcpiSratProcessorAffinity
 } QEMU_PACKED;
 typedef struct AcpiSratProcessorAffinity AcpiSratProcessorAffinity;
 
+struct AcpiSratProcessorX2ApicAffinity
+{
+    ACPI_SUB_HEADER_DEF
+    uint16_t    reserved;
+    uint32_t    proximity_domain;
+    uint32_t    x2apic_id;
+    uint32_t    flags;
+    uint32_t    clk_domain;
+    uint32_t    reserved2;
+} QEMU_PACKED;
+typedef struct AcpiSratProcessorX2ApicAffinity AcpiSratProcessorX2ApicAffinity;
+
 struct AcpiSratMemoryAffinity
 {
     ACPI_SUB_HEADER_DEF
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 03/14] acpi: cphp: add support for x2APIC in _MAT method

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/acpi/cpu.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index e10903b..01405e7 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -563,6 +563,11 @@ void build_cpus_aml(Aml *table, MachineState *machine, bool acpi1_compat,
                         apic->flags = cpu_to_le32(1);
                         break;
                     }
+                    case ACPI_APIC_LOCAL_X2APIC: {
+                        AcpiMadtProcessorX2Apic *apic = (void *)buf->data;
+                        apic->flags = cpu_to_le32(1);
+                        break;
+                    }
                     default:
                         assert(0);
                     }
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 04/14] pc: leave max apic_id_limit only in legacy cpu hotplug code

[Igor Mammedov]

that's enough to make old code that depends on it
to prevent QEMU starting with more than 255 CPUs.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/acpi/cpu_hotplug.c | 7 ++++++-
 hw/i386/pc.c          | 7 -------
 2 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/hw/acpi/cpu_hotplug.c b/hw/acpi/cpu_hotplug.c
index 6c0b555..bd8d915 100644
--- a/hw/acpi/cpu_hotplug.c
+++ b/hw/acpi/cpu_hotplug.c
@@ -15,6 +15,7 @@
 #include "qapi/error.h"
 #include "qom/cpu.h"
 #include "hw/i386/pc.h"
+#include "qemu/error-report.h"
 
 #define CPU_EJECT_METHOD "CPEJ"
 #define CPU_MAT_METHOD "CPMA"
@@ -215,7 +216,11 @@ void build_legacy_cpu_hotplug_aml(Aml *ctx, MachineState *machine,
     /* The current AML generator can cover the APIC ID range [0..255],
      * inclusive, for VCPU hotplug. */
     QEMU_BUILD_BUG_ON(ACPI_CPU_HOTPLUG_ID_LIMIT > 256);
-    g_assert(pcms->apic_id_limit <= ACPI_CPU_HOTPLUG_ID_LIMIT);
+    if (pcms->apic_id_limit > ACPI_CPU_HOTPLUG_ID_LIMIT) {
+        error_report("max_cpus is too large. APIC ID of last CPU is %u",
+                     pcms->apic_id_limit - 1);
+        exit(1);
+    }
 
     /* create PCI0.PRES device and its _CRS to reserve CPU hotplug MMIO */
     dev = aml_device("PCI0." stringify(CPU_HOTPLUG_RESOURCE_DEVICE));
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7072fb5..1baf01b 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -780,7 +780,6 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
     numa_fw_cfg[0] = cpu_to_le64(nb_numa_nodes);
     for (i = 0; i < max_cpus; i++) {
         unsigned int apic_id = x86_cpu_apic_id_from_index(i);
-        assert(apic_id < pcms->apic_id_limit);
         for (j = 0; j < nb_numa_nodes; j++) {
             if (test_bit(i, numa_info[j].node_cpu)) {
                 numa_fw_cfg[apic_id + 1] = cpu_to_le64(j);
@@ -1133,12 +1132,6 @@ void pc_cpus_init(PCMachineState *pcms)
      * This is used for FW_CFG_MAX_CPUS. See comments on bochs_bios_init().
      */
     pcms->apic_id_limit = x86_cpu_apic_id_from_index(max_cpus - 1) + 1;
-    if (pcms->apic_id_limit > ACPI_CPU_HOTPLUG_ID_LIMIT) {
-        error_report("max_cpus is too large. APIC ID of last CPU is %u",
-                     pcms->apic_id_limit - 1);
-        exit(1);
-    }
-
     pcms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
                                     sizeof(CPUArchId) * max_cpus);
     pcms->node_cpu = g_malloc0(max_cpus * sizeof *pcms->node_cpu);
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 05/14] pc: apic: move MAX_APICS check to 'apic' class

[Igor Mammedov]

MAX_APICS is only used by child 'apic' class and not
by its parent TYPE_APIC_COMMON or any other derived
class.
Move check into end user 'apic' class so it won't
get in the way of other APIC implementations
if they support more then MAX_APICS.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/intc/apic.c                  | 10 ++++++++++
 hw/intc/apic_common.c           |  8 --------
 include/hw/i386/apic_internal.h |  4 +---
 3 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/hw/intc/apic.c b/hw/intc/apic.c
index 28c2ea5..a9d2117 100644
--- a/hw/intc/apic.c
+++ b/hw/intc/apic.c
@@ -26,7 +26,9 @@
 #include "trace.h"
 #include "hw/i386/pc.h"
 #include "hw/i386/apic-msidef.h"
+#include "qapi/error.h"
 
+#define MAX_APICS 255
 #define MAX_APIC_WORDS 8
 
 #define SYNC_FROM_VAPIC                 0x1
@@ -867,6 +869,14 @@ static const MemoryRegionOps apic_io_ops = {
 static void apic_realize(DeviceState *dev, Error **errp)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    static int apic_no;
+
+    if (apic_no >= MAX_APICS) {
+        error_setg(errp, "%s initialization failed.",
+                   object_get_typename(OBJECT(dev)));
+        return;
+    }
+    s->idx = apic_no++;
 
     memory_region_init_io(&s->io_memory, OBJECT(s), &apic_io_ops, s, "apic-msi",
                           APIC_SPACE_SIZE);
diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index 4abe145..fb556cb 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -297,14 +297,6 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     APICCommonState *s = APIC_COMMON(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
-    static int apic_no;
-
-    if (apic_no >= MAX_APICS) {
-        error_setg(errp, "%s initialization failed.",
-                   object_get_typename(OBJECT(dev)));
-        return;
-    }
-    s->idx = apic_no++;
 
     info = APIC_COMMON_GET_CLASS(s);
     info->realize(dev, errp);
diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
index 74fe935..5d3be9a 100644
--- a/include/hw/i386/apic_internal.h
+++ b/include/hw/i386/apic_internal.h
@@ -120,8 +120,6 @@
 #define VAPIC_ENABLE_BIT                0
 #define VAPIC_ENABLE_MASK               (1 << VAPIC_ENABLE_BIT)
 
-#define MAX_APICS 255
-
 typedef struct APICCommonState APICCommonState;
 
 #define TYPE_APIC_COMMON "apic-common"
@@ -175,7 +173,7 @@ struct APICCommonState {
     uint32_t initial_count;
     int64_t initial_count_load_time;
     int64_t next_time;
-    int idx;
+    int idx; /* not actually common, used only by 'apic' derived class */
     QEMUTimer *timer;
     int64_t timer_expiry;
     int sipi_vector;
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 06/14] pc: apic_common: extend APIC ID property to 32bit

[Igor Mammedov]

ACPI ID is 32 bit wide on CPUs with x2APIC support.
Extend 'id' property to support it.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/intc/apic_common.c           | 40 +++++++++++++++++++++++++++++++++++++++-
 include/hw/i386/apic_internal.h |  3 ++-
 target-i386/cpu.c               |  2 +-
 target-i386/cpu.h               |  1 +
 4 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index fb556cb..b33a823 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -19,6 +19,7 @@
  */
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "qapi/visitor.h"
 #include "hw/i386/apic.h"
 #include "hw/i386/apic_internal.h"
 #include "trace.h"
@@ -404,13 +405,49 @@ static const VMStateDescription vmstate_apic_common = {
 };
 
 static Property apic_properties_common[] = {
-    DEFINE_PROP_UINT8("id", APICCommonState, id, -1),
     DEFINE_PROP_UINT8("version", APICCommonState, version, 0x14),
     DEFINE_PROP_BIT("vapic", APICCommonState, vapic_control, VAPIC_ENABLE_BIT,
                     true),
     DEFINE_PROP_END_OF_LIST(),
 };
 
+static void apic_common_get_id(Object *obj, Visitor *v, const char *name,
+                               void *opaque, Error **errp)
+{
+    APICCommonState *s = APIC_COMMON(obj);
+    int64_t value;
+
+    value = s->apicbase & MSR_IA32_APICBASE_EXTD ? s->initial_apic_id : s->id;
+    visit_type_int(v, name, &value, errp);
+}
+
+static void apic_common_set_id(Object *obj, Visitor *v, const char *name,
+                               void *opaque, Error **errp)
+{
+    APICCommonState *s = APIC_COMMON(obj);
+    Error *local_err = NULL;
+    int64_t value;
+
+    visit_type_int(v, name, &value, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    s->initial_apic_id = value;
+    s->id = (uint8_t)value;
+}
+
+static void apic_common_initfn(Object *obj)
+{
+    APICCommonState *s = APIC_COMMON(obj);
+
+    s->id = s->initial_apic_id = -1;
+    object_property_add(obj, "id", "int",
+                        apic_common_get_id,
+                        apic_common_set_id, NULL, NULL, NULL);
+}
+
 static void apic_common_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -430,6 +467,7 @@ static const TypeInfo apic_common_type = {
     .name = TYPE_APIC_COMMON,
     .parent = TYPE_DEVICE,
     .instance_size = sizeof(APICCommonState),
+    .instance_init = apic_common_initfn,
     .class_size = sizeof(APICCommonClass),
     .class_init = apic_common_class_init,
     .abstract = true,
diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
index 5d3be9a..d7859e1 100644
--- a/include/hw/i386/apic_internal.h
+++ b/include/hw/i386/apic_internal.h
@@ -154,7 +154,8 @@ struct APICCommonState {
     MemoryRegion io_memory;
     X86CPU *cpu;
     uint32_t apicbase;
-    uint8_t id;
+    uint8_t id; /* legacy APIC ID */
+    uint32_t initial_apic_id;
     uint8_t version;
     uint8_t arb_id;
     uint8_t tpr;
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 060131a..1b81cfa 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2860,7 +2860,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 
     object_property_add_child(OBJECT(cpu), "apic",
                               OBJECT(cpu->apic_state), NULL);
-    qdev_prop_set_uint8(cpu->apic_state, "id", cpu->apic_id);
+    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);
     /* TODO: convert to link<> */
     apic = APIC_COMMON(cpu->apic_state);
     apic->cpu = cpu;
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index 732eb6d..93d4864 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -322,6 +322,7 @@
 #define MSR_IA32_APICBASE               0x1b
 #define MSR_IA32_APICBASE_BSP           (1<<8)
 #define MSR_IA32_APICBASE_ENABLE        (1<<11)
+#define MSR_IA32_APICBASE_EXTD          (1<<10)
 #define MSR_IA32_APICBASE_BASE          (0xfffffU<<12)
 #define MSR_IA32_FEATURE_CONTROL        0x0000003a
 #define MSR_TSC_ADJUST                  0x0000003b
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 07/14] pc: apic_common: restore APIC ID to initial ID on reset

[Igor Mammedov]

APIC ID should be restored to initial APIC ID
state after Reset and Power-On.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/intc/apic_common.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index b33a823..e17b92c 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -240,6 +240,7 @@ static void apic_reset_common(DeviceState *dev)
 
     bsp = s->apicbase & MSR_IA32_APICBASE_BSP;
     s->apicbase = APIC_DEFAULT_ADDRESS | bsp | MSR_IA32_APICBASE_ENABLE;
+    s->id = s->initial_apic_id;
 
     s->vapic_paddr = 0;
     info->vapic_base_update(s);
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 08/14] pc: apic_common: reset APIC ID to initial ID when switching into x2APIC mode

[Igor Mammedov]

SDM: x2APIC State Transitions:
         State Changes From xAPIC Mode to x2APIC Mode
"
Any APIC ID value written to the memory-mapped
local APIC ID register is not preserved
"

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/intc/apic_common.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index e17b92c..eefb57d 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -37,6 +37,11 @@ void cpu_set_apic_base(DeviceState *dev, uint64_t val)
     if (dev) {
         APICCommonState *s = APIC_COMMON(dev);
         APICCommonClass *info = APIC_COMMON_GET_CLASS(s);
+        /* switching to x2APIC, reset possibly modified xAPIC ID */
+        if (!(s->apicbase & MSR_IA32_APICBASE_EXTD) &&
+            (val & MSR_IA32_APICBASE_EXTD)) {
+            s->id = s->initial_apic_id;
+        }
         info->set_base(s, val);
     }
 }
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 09/14] pc: kvm_apic: pass APIC ID depending on xAPIC/x2APIC mode

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/kvm/apic.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/hw/i386/kvm/apic.c b/hw/i386/kvm/apic.c
index 3c7c8fa..c6891b8 100644
--- a/hw/i386/kvm/apic.c
+++ b/hw/i386/kvm/apic.c
@@ -32,7 +32,11 @@ void kvm_put_apic_state(DeviceState *dev, struct kvm_lapic_state *kapic)
     int i;
 
     memset(kapic, 0, sizeof(*kapic));
-    kvm_apic_set_reg(kapic, 0x2, s->id << 24);
+    if (s->apicbase & MSR_IA32_APICBASE_EXTD) {
+        kvm_apic_set_reg(kapic, 0x2, cpu_to_be32(s->initial_apic_id));
+    } else {
+        kvm_apic_set_reg(kapic, 0x2, s->id << 24);
+    }
     kvm_apic_set_reg(kapic, 0x8, s->tpr);
     kvm_apic_set_reg(kapic, 0xd, s->log_dest << 24);
     kvm_apic_set_reg(kapic, 0xe, s->dest_mode << 28 | 0x0fffffff);
@@ -57,7 +61,11 @@ void kvm_get_apic_state(DeviceState *dev, struct kvm_lapic_state *kapic)
     APICCommonState *s = APIC_COMMON(dev);
     int i, v;
 
-    s->id = kvm_apic_get_reg(kapic, 0x2) >> 24;
+    if (s->apicbase & MSR_IA32_APICBASE_EXTD) {
+        assert(be32_to_cpu(kvm_apic_get_reg(kapic, 0x2)) == s->initial_apic_id);
+    } else {
+        s->id = kvm_apic_get_reg(kapic, 0x2) >> 24;
+    }
     s->tpr = kvm_apic_get_reg(kapic, 0x8);
     s->arb_id = kvm_apic_get_reg(kapic, 0x9);
     s->log_dest = kvm_apic_get_reg(kapic, 0xd) >> 24;
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 10/14] pc: clarify FW_CFG_MAX_CPUS usage comment

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 1baf01b..5fa5668 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -746,17 +746,15 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
 
     /* FW_CFG_MAX_CPUS is a bit confusing/problematic on x86:
      *
-     * SeaBIOS needs FW_CFG_MAX_CPUS for CPU hotplug, but the CPU hotplug
-     * QEMU<->SeaBIOS interface is not based on the "CPU index", but on the APIC
-     * ID of hotplugged CPUs[1]. This means that FW_CFG_MAX_CPUS is not the
-     * "maximum number of CPUs", but the "limit to the APIC ID values SeaBIOS
-     * may see".
+     * For machine types prior to 1.8, SeaBIOS needs FW_CFG_MAX_CPUS for
+     * building MPTable, ACPI MADT, ACPI CPU hotplug and ACPI SRAT table,
+     * that tables are based on xAPIC ID and QEMU<->SeaBIOS interface
+     * for CPU hotplug also uses APIC ID and not "CPU index".
+     * This means that FW_CFG_MAX_CPUS is not the "maximum number of CPUs",
+     * but the "limit to the APIC ID values SeaBIOS may see".
      *
-     * So, this means we must not use max_cpus, here, but the maximum possible
-     * APIC ID value, plus one.
-     *
-     * [1] The only kind of "CPU identifier" used between SeaBIOS and QEMU is
-     *     the APIC ID, not the "CPU index"
+     * So for compatibility reasons with old BIOSes we are stuck with
+     * "etc/max-cpus" actually being apic_id_limit
      */
     fw_cfg_add_i16(fw_cfg, FW_CFG_MAX_CPUS, (uint16_t)pcms->apic_id_limit);
     fw_cfg_add_i64(fw_cfg, FW_CFG_RAM_SIZE, (uint64_t)ram_size);
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 11/14] increase MAX_CPUMASK_BITS from 255 to 288

[Igor Mammedov]

so that it would be possible to increase maxcpus limit
for x86 target. Keep spapr/virt_arm at limit they used
to have 255.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/arm/virt.c           | 2 +-
 hw/ppc/spapr.c          | 2 +-
 include/sysemu/sysemu.h | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 56d35c7..438e09c 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1358,7 +1358,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
      * it later in machvirt_init, where we have more information about the
      * configuration of the particular instance.
      */
-    mc->max_cpus = MAX_CPUMASK_BITS;
+    mc->max_cpus = 255;
     mc->has_dynamic_sysbus = true;
     mc->block_default_type = IF_VIRTIO;
     mc->no_cdrom = 1;
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index b69995e..d61ece2 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2280,7 +2280,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     mc->init = ppc_spapr_init;
     mc->reset = ppc_spapr_reset;
     mc->block_default_type = IF_SCSI;
-    mc->max_cpus = MAX_CPUMASK_BITS;
+    mc->max_cpus = 255;
     mc->no_parallel = 1;
     mc->default_boot_order = "";
     mc->default_ram_size = 512 * M_BYTE;
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 38fb3ca..5edcc72 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -185,7 +185,7 @@ extern int mem_prealloc;
  *
  * Note that cpu->get_arch_id() may be larger than MAX_CPUMASK_BITS.
  */
-#define MAX_CPUMASK_BITS 255
+#define MAX_CPUMASK_BITS 288
 
 #define MAX_OPTION_ROMS 16
 typedef struct QEMUOptionRom {
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 12/14] pc: add 'etc/boot-cpus' fw_cfg file for machine with more than 255 CPUs

[Igor Mammedov]

Currently firmware uses 1 byte at 0x5F offset in RTC CMOS
to get number of CPUs present at boot. However 1 byte is
not enough to handle more than 255 CPUs.  So add a fw_cfg
file that would allow QEMU to tell it.
For compat reasons add file only for machine types that
support more than 255 CPUs.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 5fa5668..7161485 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -738,6 +738,7 @@ static void pc_build_smbios(FWCfgState *fw_cfg)
 
 static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
 {
+    MachineClass *mc = MACHINE_GET_CLASS(pcms);
     FWCfgState *fw_cfg;
     uint64_t *numa_fw_cfg;
     int i, j;
@@ -770,6 +771,11 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
                     sizeof(struct e820_entry) * e820_entries);
 
     fw_cfg_add_bytes(fw_cfg, FW_CFG_HPET, &hpet_cfg, sizeof(hpet_cfg));
+    if (mc->max_cpus > 255) {
+        static uint16_t boot_cpus;
+        boot_cpus = cpu_to_le16(smp_cpus);
+        fw_cfg_add_file(fw_cfg, "etc/boot-cpus", &smp_cpus, sizeof(boot_cpus));
+    }
     /* allocate memory for the NUMA channel: one (64bit) word for the number
      * of nodes, one word for each VCPU->node and one word for each node to
      * hold the amount of memory.
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 13/14] pc: q35: bump max_cpus to 288

[Igor Mammedov]

along with it for machine versions 2.6 and older keep
it at 255 and also freeze that limit at 255 for pc_piix4
machine as it won't be getting more than 255 CPUs.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c      | 2 +-
 hw/i386/pc_piix.c | 1 +
 hw/i386/pc_q35.c  | 1 +
 3 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7161485..567f3e2 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -2064,7 +2064,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data)
     mc->possible_cpu_arch_ids = pc_possible_cpu_arch_ids;
     mc->default_boot_order = "cad";
     mc->hot_add_cpu = pc_hot_add_cpu;
-    mc->max_cpus = 255;
+    mc->max_cpus = 288;
     mc->reset = pc_machine_reset;
     hc->pre_plug = pc_machine_device_pre_plug_cb;
     hc->plug = pc_machine_device_plug_cb;
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 860a83d..6299c60 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -409,6 +409,7 @@ static void pc_xen_hvm_init(MachineState *machine)
 
 static void pc_i440fx_machine_options(MachineClass *m)
 {
+    m->max_cpus = 255;
     m->family = "pc_piix";
     m->desc = "Standard PC (i440FX + PIIX, 1996)";
     m->hot_add_cpu = pc_hot_add_cpu;
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 0dc4ed5..5dd07a4 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -297,6 +297,7 @@ static void pc_q35_2_6_machine_options(MachineClass *m)
 {
     pc_q35_2_7_machine_options(m);
     m->alias = NULL;
+    m->max_cpus = 255;
     SET_MACHINE_COMPAT(m, PC_COMPAT_2_6);
 }
 
-- 
1.8.3.1

[Qemu-devel] [PATCH RFC 14/14] DO NOT APPLY: hack to make 2nd CPU APIC ID to be x2APIC one

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 567f3e2..7d301a2 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -695,6 +695,7 @@ static uint32_t x86_cpu_apic_id_from_index(unsigned int cpu_index)
         }
         return cpu_index;
     } else {
+if(correct_id == 1) correct_id = 280;
         return correct_id;
     }
 }
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH RFC 11/14] increase MAX_CPUMASK_BITS from 255 to 288

[Greg Kurz]

On Mon,  9 May 2016 11:26:57 +0200
Igor Mammedov <imammedo@redhat.com> wrote:

> so that it would be possible to increase maxcpus limit
> for x86 target. Keep spapr/virt_arm at limit they used
> to have 255.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---

As said in my other post, bumping up to 1024 seems reasonable for pseries.

Would it be a problem for the x86 target ? If it is okay for you, I'd rather
try to push this separately from your series.

Anyway,

Reviewed-by: Greg Kurz <gkurz@linux.vnet.ibm.com>

>  hw/arm/virt.c           | 2 +-
>  hw/ppc/spapr.c          | 2 +-
>  include/sysemu/sysemu.h | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 56d35c7..438e09c 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1358,7 +1358,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
>       * it later in machvirt_init, where we have more information about the
>       * configuration of the particular instance.
>       */
> -    mc->max_cpus = MAX_CPUMASK_BITS;
> +    mc->max_cpus = 255;
>      mc->has_dynamic_sysbus = true;
>      mc->block_default_type = IF_VIRTIO;
>      mc->no_cdrom = 1;
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index b69995e..d61ece2 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -2280,7 +2280,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
>      mc->init = ppc_spapr_init;
>      mc->reset = ppc_spapr_reset;
>      mc->block_default_type = IF_SCSI;
> -    mc->max_cpus = MAX_CPUMASK_BITS;
> +    mc->max_cpus = 255;
>      mc->no_parallel = 1;
>      mc->default_boot_order = "";
>      mc->default_ram_size = 512 * M_BYTE;
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 38fb3ca..5edcc72 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -185,7 +185,7 @@ extern int mem_prealloc;
>   *
>   * Note that cpu->get_arch_id() may be larger than MAX_CPUMASK_BITS.
>   */
> -#define MAX_CPUMASK_BITS 255
> +#define MAX_CPUMASK_BITS 288
> 
>  #define MAX_OPTION_ROMS 16
>  typedef struct QEMUOptionRom {