[Qemu-devel] [PATCH v2 00/24] numa: add '-numa cpu' option

[Qemu-devel] [PATCH v2 00/24] numa: add '-numa cpu' option

[Igor Mammedov]

Changes since v1:
   * arm (Drew)
       - s/virt_idx2mp_affinity/virt_cpu_mp_affinity/
       - add arm_cpu_mp_affinity() to reduce code duplication
       - use cpu list returned by possible_cpu_arch_ids() instead of
         directly accessing MachineState::possible_cpus field
   * various checkpatch/cleanups/spelling fixes (Eric/Drew)
   * simplify spapr_cpu_index_to_props() (David)
   * add/use numa_[has_]node_id() wrappers (Drew)
   * use new NumaCpuOptions instead of CpuInstanceProperties
     in NumaOptions (Eduardo)
   * better documment NumaCpuOptions.node-id in qapi-schema.json (Eduardo)
Changes since RFC:
    * convert all targets that support numa (Eduardo)
    * add numa CLI tests
    * support wildcard matching with "-numa cpu,..." (Paolo)

Series introduces a new CLI option to allow mapping cpus to numa
nodes using public properties [socket|core|thread]-ids instead of
internal cpu_index and moving internal handling of cpu<->node
mapping from cpu_index based global bitmaps to MachineState.

New '-numa cpu' option is supported only on PC and SPAPR
machines that implement hotpluggable-cpus query.
ARM machine user-facing interface stays cpu_index based due to
lack of hotpluggable-cpus support, but internally cpu<->node
mapping will be using the common for PC/SPAPR/ARM approach
(i.e. store mapping info in MachineState:possible_cpus)

It only provides CLI interface to do mapping, there is no QMP
one as I haven't found a suitable place/way to update/set mapping
after machine_done for QEMU started with -S (stopped mode) so that
mgmt could query hopluggable-cpus first, then map them to numa nodes
in runtime before actually allowing guest to run.

Another alternative I've been considering is to add CLI option
similar to -S but that would pause initialization before machine_init()
callback is run so that user can get CPU layout with hopluggable-cpus,
then map CPUs to numa nodes and unpause to let machine_init() initialize
machine using previously predefined numa mapping.
Such option might also be useful for other usecases.


git repo for testing:
   https://github.com/imammedo/qemu.git cphp_numa_cfg_v2
reference to v1:
   https://www.mail-archive.com/qemu-devel@nongnu.org/msg438955.html

CC: Eduardo Habkost <ehabkost@redhat.com>
CC: Peter Maydell <peter.maydell@linaro.org>
CC: Andrew Jones <drjones@redhat.com>
CC: David Gibson <david@gibson.dropbear.id.au>
CC: Eric Blake <eblake@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Shannon Zhao <zhaoshenglong@huawei.com>
CC: qemu-arm@nongnu.org
CC: qemu-ppc@nongnu.org

Igor Mammedov (24):
  tests: add CPUs to numa node mapping test
  hw/arm/virt: extract mp-affinity calculation in separate function
  hw/arm/virt: use machine->possible_cpus for storing possible topology
    info
  hw/arm/virt: explicitly allocate cpu_index for cpus
  numa: move source of default CPUs to NUMA node mapping into boards
  spapr: add node-id property to sPAPR core
  pc: add node-id property to CPU
  virt-arm: add node-id property to CPU
  numa: add check that board supports cpu_index to node mapping
  numa: mirror cpu to node mapping in MachineState::possible_cpus
  numa: do default mapping based on possible_cpus instead of node_cpu
    bitmaps
  numa: add numa_[has_]node_id() wrappers
  pc: get numa node mapping from possible_cpus instead of
    numa_get_node_for_cpu()
  spapr: get numa node mapping from possible_cpus instead of
    numa_get_node_for_cpu()
  virt-arm: get numa node mapping from possible_cpus instead of
    numa_get_node_for_cpu()
  QMP: include CpuInstanceProperties into query_cpus output output
  tests: numa: add case for QMP command query-cpus
  numa: remove no longer used numa_get_node_for_cpu()
  numa: remove no longer need numa_post_machine_init()
  machine: call machine init from wrapper
  numa: use possible_cpus for not mapped CPUs check
  numa: remove node_cpu bitmaps as they are no longer used
  numa: add '-numa cpu,...' option for property based node mapping
  tests: check -numa node,cpu=props_list usecase

 include/hw/boards.h             |  11 +-
 include/hw/ppc/spapr_cpu_core.h |   1 +
 include/qom/cpu.h               |   2 +
 include/sysemu/numa.h           |  16 ++-
 target/arm/cpu.h                |   2 +
 cpus.c                          |  10 ++
 hw/acpi/cpu.c                   |   7 +-
 hw/arm/virt-acpi-build.c        |  19 +--
 hw/arm/virt.c                   | 123 +++++++++++++---
 hw/core/machine.c               | 132 ++++++++++++++++++
 hw/i386/acpi-build.c            |  11 +-
 hw/i386/pc.c                    |  54 +++++--
 hw/ppc/spapr.c                  |  44 +++++-
 hw/ppc/spapr_cpu_core.c         |  21 ++-
 numa.c                          | 157 +++++++++------------
 qapi-schema.json                |  27 +++-
 qemu-options.hx                 |  23 ++-
 target/arm/cpu.c                |  13 +-
 target/i386/cpu.c               |   1 +
 tests/Makefile.include          |   5 +
 tests/numa-test.c               | 302 ++++++++++++++++++++++++++++++++++++++++
 vl.c                            |   6 +-
 22 files changed, 796 insertions(+), 191 deletions(-)
 create mode 100644 tests/numa-test.c

-- 
2.7.4

[Qemu-devel] [PATCH v2 01/24] tests: add CPUs to numa node mapping test

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 tests/Makefile.include |   5 +++
 tests/numa-test.c      | 106 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 111 insertions(+)
 create mode 100644 tests/numa-test.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 31931c0..16ff8f3 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -260,6 +260,7 @@ check-qtest-i386-y += tests/test-filter-mirror$(EXESUF)
 check-qtest-i386-y += tests/test-filter-redirector$(EXESUF)
 check-qtest-i386-y += tests/postcopy-test$(EXESUF)
 check-qtest-i386-y += tests/test-x86-cpuid-compat$(EXESUF)
+check-qtest-i386-y += tests/numa-test$(EXESUF)
 check-qtest-x86_64-y += $(check-qtest-i386-y)
 gcov-files-i386-y += i386-softmmu/hw/timer/mc146818rtc.c
 gcov-files-x86_64-y = $(subst i386-softmmu/,x86_64-softmmu/,$(gcov-files-i386-y))
@@ -300,6 +301,7 @@ check-qtest-ppc64-y += tests/test-netfilter$(EXESUF)
 check-qtest-ppc64-y += tests/test-filter-mirror$(EXESUF)
 check-qtest-ppc64-y += tests/test-filter-redirector$(EXESUF)
 check-qtest-ppc64-y += tests/display-vga-test$(EXESUF)
+check-qtest-ppc64-y += tests/numa-test$(EXESUF)
 check-qtest-ppc64-$(CONFIG_EVENTFD) += tests/ivshmem-test$(EXESUF)
 
 check-qtest-sh4-y = tests/endianness-test$(EXESUF)
@@ -324,6 +326,8 @@ gcov-files-arm-y += arm-softmmu/hw/block/virtio-blk.c
 check-qtest-arm-y += tests/test-arm-mptimer$(EXESUF)
 gcov-files-arm-y += hw/timer/arm_mptimer.c
 
+check-qtest-aarch64-y = tests/numa-test$(EXESUF)
+
 check-qtest-microblazeel-y = $(check-qtest-microblaze-y)
 
 check-qtest-xtensaeb-y = $(check-qtest-xtensa-y)
@@ -753,6 +757,7 @@ tests/vhost-user-bridge$(EXESUF): tests/vhost-user-bridge.o contrib/libvhost-use
 tests/test-uuid$(EXESUF): tests/test-uuid.o $(test-util-obj-y)
 tests/test-arm-mptimer$(EXESUF): tests/test-arm-mptimer.o
 tests/test-qapi-util$(EXESUF): tests/test-qapi-util.o $(test-util-obj-y)
+tests/numa-test$(EXESUF): tests/numa-test.o
 
 tests/migration/stress$(EXESUF): tests/migration/stress.o
 	$(call quiet-command, $(LINKPROG) -static -O3 $(PTHREAD_LIB) -o $@ $< ,"LINK","$(TARGET_DIR)$@")
diff --git a/tests/numa-test.c b/tests/numa-test.c
new file mode 100644
index 0000000..f5da0c8
--- /dev/null
+++ b/tests/numa-test.c
@@ -0,0 +1,106 @@
+/*
+ * NUMA configuration test cases
+ *
+ * Copyright (c) 2017 Red Hat Inc.
+ * Authors:
+ *  Igor Mammedov <imammedo@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+
+static char *make_cli(const char *generic_cli, const char *test_cli)
+{
+    return g_strdup_printf("%s %s", generic_cli ? generic_cli : "", test_cli);
+}
+
+static char *hmp_info_numa(void)
+{
+    QDict *resp;
+    char *s;
+
+    resp = qmp("{ 'execute': 'human-monitor-command', 'arguments': "
+                      "{ 'command-line': 'info numa '} }");
+    g_assert(resp);
+    g_assert(qdict_haskey(resp, "return"));
+    s = g_strdup(qdict_get_str(resp, "return"));
+    g_assert(s);
+    QDECREF(resp);
+    return s;
+}
+
+static void test_mon_explicit(const void *data)
+{
+    char *s;
+    char *cli;
+
+    cli = make_cli(data, "-smp 8 "
+                   "-numa node,nodeid=0,cpus=0-3 "
+                   "-numa node,nodeid=1,cpus=4-7 ");
+    qtest_start(cli);
+
+    s = hmp_info_numa();
+    g_assert(strstr(s, "node 0 cpus: 0 1 2 3"));
+    g_assert(strstr(s, "node 1 cpus: 4 5 6 7"));
+    g_free(s);
+
+    qtest_end();
+    g_free(cli);
+}
+
+static void test_mon_default(const void *data)
+{
+    char *s;
+    char *cli;
+
+    cli = make_cli(data, "-smp 8 -numa node -numa node");
+    qtest_start(cli);
+
+    s = hmp_info_numa();
+    g_assert(strstr(s, "node 0 cpus: 0 2 4 6"));
+    g_assert(strstr(s, "node 1 cpus: 1 3 5 7"));
+    g_free(s);
+
+    qtest_end();
+    g_free(cli);
+}
+
+static void test_mon_partial(const void *data)
+{
+    char *s;
+    char *cli;
+
+    cli = make_cli(data, "-smp 8 "
+                   "-numa node,nodeid=0,cpus=0-1 "
+                   "-numa node,nodeid=1,cpus=4-5 ");
+    qtest_start(cli);
+
+    s = hmp_info_numa();
+    g_assert(strstr(s, "node 0 cpus: 0 1 2 3 6 7"));
+    g_assert(strstr(s, "node 1 cpus: 4 5"));
+    g_free(s);
+
+    qtest_end();
+    g_free(cli);
+}
+
+int main(int argc, char **argv)
+{
+    const char *args = NULL;
+    const char *arch = qtest_get_arch();
+
+    if (strcmp(arch, "aarch64") == 0) {
+        args = "-machine virt";
+    }
+
+    g_test_init(&argc, &argv, NULL);
+
+    qtest_add_data_func("/numa/mon/default", args, test_mon_default);
+    qtest_add_data_func("/numa/mon/cpus/explicit", args, test_mon_explicit);
+    qtest_add_data_func("/numa/mon/cpus/partial", args, test_mon_partial);
+
+    return g_test_run();
+}
-- 
2.7.4

[Qemu-devel] [PATCH v2 02/24] hw/arm/virt: extract mp-affinity calculation in separate function

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
 (Drew)
   - s/virt_idx2mp_affinity/virt_cpu_mp_affinity/
   - add arm_cpu_mp_affinity() to reduce code duplication
---
 target/arm/cpu.h |  2 ++
 hw/arm/virt.c    | 43 ++++++++++++++++++++++++++-----------------
 target/arm/cpu.c | 12 +++++++++---
 3 files changed, 37 insertions(+), 20 deletions(-)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index 1055bfe..048faed 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -710,6 +710,8 @@ static inline ARMCPU *arm_env_get_cpu(CPUARMState *env)
     return container_of(env, ARMCPU, env);
 }
 
+uint64_t arm_cpu_mp_affinity(int idx, uint8_t clustersz);
+
 #define ENV_GET_CPU(e) CPU(arm_env_get_cpu(e))
 
 #define ENV_OFFSET offsetof(ARMCPU, env)
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 5f62a03..61ae437 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1194,6 +1194,29 @@ void virt_machine_done(Notifier *notifier, void *data)
     virt_build_smbios(vms);
 }
 
+static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
+{
+    uint8_t clustersz = ARM_DEFAULT_CPUS_PER_CLUSTER;
+    VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+
+    if (!vmc->disallow_affinity_adjustment) {
+        /* Adjust MPIDR like 64-bit KVM hosts, which incorporate the
+         * GIC's target-list limitations. 32-bit KVM hosts currently
+         * always create clusters of 4 CPUs, but that is expected to
+         * change when they gain support for gicv3. When KVM is enabled
+         * it will override the changes we make here, therefore our
+         * purposes are to make TCG consistent (with 64-bit KVM hosts)
+         * and to improve SGI efficiency.
+         */
+        if (vms->gic_version == 3) {
+            clustersz = GICV3_TARGETLIST_BITS;
+        } else {
+            clustersz = GIC_TARGETLIST_BITS;
+        }
+    }
+    return arm_cpu_mp_affinity(idx, clustersz);
+}
+
 static void machvirt_init(MachineState *machine)
 {
     VirtMachineState *vms = VIRT_MACHINE(machine);
@@ -1210,7 +1233,6 @@ static void machvirt_init(MachineState *machine)
     CPUClass *cc;
     Error *err = NULL;
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
-    uint8_t clustersz;
 
     if (!cpu_model) {
         cpu_model = "cortex-a15";
@@ -1263,10 +1285,8 @@ static void machvirt_init(MachineState *machine)
      */
     if (vms->gic_version == 3) {
         virt_max_cpus = vms->memmap[VIRT_GIC_REDIST].size / 0x20000;
-        clustersz = GICV3_TARGETLIST_BITS;
     } else {
         virt_max_cpus = GIC_NCPU;
-        clustersz = GIC_TARGETLIST_BITS;
     }
 
     if (max_cpus > virt_max_cpus) {
@@ -1326,20 +1346,9 @@ static void machvirt_init(MachineState *machine)
 
     for (n = 0; n < smp_cpus; n++) {
         Object *cpuobj = object_new(typename);
-        if (!vmc->disallow_affinity_adjustment) {
-            /* Adjust MPIDR like 64-bit KVM hosts, which incorporate the
-             * GIC's target-list limitations. 32-bit KVM hosts currently
-             * always create clusters of 4 CPUs, but that is expected to
-             * change when they gain support for gicv3. When KVM is enabled
-             * it will override the changes we make here, therefore our
-             * purposes are to make TCG consistent (with 64-bit KVM hosts)
-             * and to improve SGI efficiency.
-             */
-            uint8_t aff1 = n / clustersz;
-            uint8_t aff0 = n % clustersz;
-            object_property_set_int(cpuobj, (aff1 << ARM_AFF1_SHIFT) | aff0,
-                                    "mp-affinity", NULL);
-        }
+
+        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
+                                "mp-affinity", NULL);
 
         if (!vms->secure) {
             object_property_set_bool(cpuobj, false, "has_el3", NULL);
diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index b357aee..ee1406d 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -458,6 +458,13 @@ static void arm_disas_set_info(CPUState *cpu, disassemble_info *info)
     }
 }
 
+uint64_t arm_cpu_mp_affinity(int idx, uint8_t clustersz)
+{
+    uint32_t Aff1 = idx / clustersz;
+    uint32_t Aff0 = idx % clustersz;
+    return (Aff1 << ARM_AFF1_SHIFT) | Aff0;
+}
+
 static void arm_cpu_initfn(Object *obj)
 {
     CPUState *cs = CPU(obj);
@@ -709,9 +716,8 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
      * so these bits always RAZ.
      */
     if (cpu->mp_affinity == ARM64_AFFINITY_INVALID) {
-        uint32_t Aff1 = cs->cpu_index / ARM_DEFAULT_CPUS_PER_CLUSTER;
-        uint32_t Aff0 = cs->cpu_index % ARM_DEFAULT_CPUS_PER_CLUSTER;
-        cpu->mp_affinity = (Aff1 << ARM_AFF1_SHIFT) | Aff0;
+        cpu->mp_affinity = arm_cpu_mp_affinity(cs->cpu_index,
+                                               ARM_DEFAULT_CPUS_PER_CLUSTER);
     }
 
     if (cpu->reset_hivecs) {
-- 
2.7.4

[Qemu-devel] [PATCH v2 03/24] hw/arm/virt: use machine->possible_cpus for storing possible topology info

[Igor Mammedov]

for now precalculate and store mp_afinity in possible_cpus
as ARM cpus don't have socket/core/thread-id properties yet.
In follow patches possible_cpus will be used for storing
and setting NUMA node mapping and replace legacy bitmap
based numa_info[node_id].node_cpu/numa_get_node_for_cpu()

For the lack of better idea, this patch cannibalizes
possible_cpus.cpus[x].props.thread_id so that
*_cpu_index_to_props() callback could return addressable
by props CPU which will be used by machine_set_cpu_numa_node()
in follow up patches to assign a CPU to node. But
cannibalizing is fine for now as that thread_id isn't exposed
to users (no hotpluggable_cpus callback support for ARM yet)
and it will be used only internally until 'device_add cpu'
is supported where we can decide on which properties to use.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  (Drew)
    - discarding result of possible_cpu_arch_ids() makes
      call not obvious and is confusing. Instead assign
      possible_cpu_arch_ids() result to local var and use
      it instead of direct access to machine->possible_cpus
      field, as it's done in pc.c
---
 hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 61ae437..e2c5626 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1221,6 +1221,8 @@ static void machvirt_init(MachineState *machine)
 {
     VirtMachineState *vms = VIRT_MACHINE(machine);
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine);
+    MachineClass *mc = MACHINE_GET_CLASS(machine);
+    const CPUArchIdList *possible_cpus;
     qemu_irq pic[NUM_IRQS];
     MemoryRegion *sysmem = get_system_memory();
     MemoryRegion *secure_sysmem = NULL;
@@ -1344,10 +1346,16 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    for (n = 0; n < smp_cpus; n++) {
-        Object *cpuobj = object_new(typename);
+    possible_cpus = mc->possible_cpu_arch_ids(machine);
+    for (n = 0; n < possible_cpus->len; n++) {
+        Object *cpuobj;
 
-        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
+        if (n >= smp_cpus) {
+            break;
+        }
+
+        cpuobj = object_new(typename);
+        object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
                                 "mp-affinity", NULL);
 
         if (!vms->secure) {
@@ -1527,6 +1535,31 @@ static void virt_set_gic_version(Object *obj, const char *value, Error **errp)
     }
 }
 
+static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
+{
+    int n;
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+
+    if (ms->possible_cpus) {
+        assert(ms->possible_cpus->len == max_cpus);
+        return ms->possible_cpus;
+    }
+
+    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
+                                  sizeof(CPUArchId) * max_cpus);
+    ms->possible_cpus->len = max_cpus;
+    for (n = 0; n < ms->possible_cpus->len; n++) {
+        ms->possible_cpus->cpus[n].arch_id =
+            virt_cpu_mp_affinity(vms, n);
+        ms->possible_cpus->cpus[n].props.has_thread_id = true;
+        ms->possible_cpus->cpus[n].props.thread_id = n;
+
+        /* TODO: add 'has_node/node' here to describe
+           to which node core belongs */
+    }
+    return ms->possible_cpus;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1543,6 +1576,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->pci_allow_0_address = true;
     /* We know we will never create a pre-ARMv7 CPU which needs 1K pages */
     mc->minimum_page_bits = 12;
+    mc->possible_cpu_arch_ids = virt_possible_cpu_arch_ids;
 }
 
 static const TypeInfo virt_machine_info = {
-- 
2.7.4

[Qemu-devel] [PATCH v2 04/24] hw/arm/virt: explicitly allocate cpu_index for cpus

[Igor Mammedov]

Currently cpu_index is implicitly auto assigned during
cpu.realize() time cpu_exec_realizefn()->cpu_list_add().

It happens to match index in possible_cpus so take
control over it and make board initialize cpu_index
to possible_cpus index explicitly. It will at least
document that board is in control of it and when
'-device cpu' support comes it will keep cpu_index
stable regardless of order cpus are created so it won't
break migration.
Within this series it will be used for internal
conversion from storing cpu_index based NUMA node
bitmaps to property based mapping with possible_cpus,
And will allow map cpu_index to a CPU entry in
possible_cpus array.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 hw/arm/virt.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index e2c5626..acc748e 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1349,6 +1349,7 @@ static void machvirt_init(MachineState *machine)
     possible_cpus = mc->possible_cpu_arch_ids(machine);
     for (n = 0; n < possible_cpus->len; n++) {
         Object *cpuobj;
+        CPUState *cs;
 
         if (n >= smp_cpus) {
             break;
@@ -1358,6 +1359,9 @@ static void machvirt_init(MachineState *machine)
         object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
                                 "mp-affinity", NULL);
 
+        cs = CPU(cpuobj);
+        cs->cpu_index = n;
+
         if (!vms->secure) {
             object_property_set_bool(cpuobj, false, "has_el3", NULL);
         }
-- 
2.7.4

[Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

Originally CPU threads were by default assigned in
round-robin fashion. However it was causing issues in
guest since CPU threads from the same socket/core could
be placed on different NUMA nodes.
Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
fixed it by grouping threads within a socket on the same node
introducing cpu_index_to_socket_id() callback and commit
20bb648d (spapr: Fix default NUMA node allocation for threads)
reused callback to fix similar issues for SPAPR machine
even though socket doesn't make much sense there.

As result QEMU ended up having 3 default distribution rules
used by 3 targets /virt-arm, spapr, pc/.

In effort of moving NUMA mapping for CPUs into possible_cpus,
generalize default mapping in numa.c by making boards decide
on default mapping and let them explicitly tell generic
numa code to which node a CPU thread belongs to by replacing
cpu_index_to_socket_id() with @cpu_index_to_instance_props()
which provides default node_id assigned by board to specified
cpu_index.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
Patch only moves source of default mapping to possible_cpus[]
and leaves the rest of NUMA handling to numa_info[node_id].node_cpu
bitmaps. It's up to follow up patches to replace bitmaps
with possible_cpus[] internally.

v3:
  - drop duplicate ';' (Drew)
v2:
  - use cpu_index instead of core_id directly in spapr_cpu_index_to_props()
       (David Gibson)
  - ammend comment message s/board/numa init code/
       (David Gibson)
---
 include/hw/boards.h   |  8 ++++++--
 include/sysemu/numa.h |  2 +-
 hw/arm/virt.c         | 20 ++++++++++++++++++--
 hw/i386/pc.c          | 23 +++++++++++++++++------
 hw/ppc/spapr.c        | 27 ++++++++++++++++++++-------
 numa.c                | 15 +++++++++------
 vl.c                  |  2 +-
 7 files changed, 72 insertions(+), 25 deletions(-)

diff --git a/include/hw/boards.h b/include/hw/boards.h
index 31d9c72..5d6af21 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -74,7 +74,10 @@ typedef struct {
  *    of HotplugHandler object, which handles hotplug operation
  *    for a given @dev. It may return NULL if @dev doesn't require
  *    any actions to be performed by hotplug handler.
- * @cpu_index_to_socket_id:
+ * @cpu_index_to_instance_props:
+ *    used to provide @cpu_index to socket/core/thread number mapping, allowing
+ *    legacy code to perform maping from cpu_index to topology properties
+ *    Returns: tuple of socket/core/thread ids given cpu_index belongs to.
  *    used to provide @cpu_index to socket number mapping, allowing
  *    a machine to group CPU threads belonging to the same socket/package
  *    Returns: socket number given cpu_index belongs to.
@@ -139,7 +142,8 @@ struct MachineClass {
 
     HotplugHandler *(*get_hotplug_handler)(MachineState *machine,
                                            DeviceState *dev);
-    unsigned (*cpu_index_to_socket_id)(unsigned cpu_index);
+    CpuInstanceProperties (*cpu_index_to_instance_props)(MachineState *machine,
+                                                         unsigned cpu_index);
     const CPUArchIdList *(*possible_cpu_arch_ids)(MachineState *machine);
 };
 
diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index 8f09dcf..46ea6c7 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -24,7 +24,7 @@ typedef struct node_info {
 } NodeInfo;
 
 extern NodeInfo numa_info[MAX_NODES];
-void parse_numa_opts(MachineClass *mc);
+void parse_numa_opts(MachineState *ms);
 void numa_post_machine_init(void);
 void query_numa_node_mem(uint64_t node_mem[]);
 extern QemuOptsList qemu_numa_opts;
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index acc748e..3e19b5f 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1539,6 +1539,16 @@ static void virt_set_gic_version(Object *obj, const char *value, Error **errp)
     }
 }
 
+static CpuInstanceProperties
+virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
+{
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
+    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
+
+    assert(cpu_index < possible_cpus->len);
+    return possible_cpus->cpus[cpu_index].props;;
+}
+
 static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
 {
     int n;
@@ -1558,8 +1568,13 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
         ms->possible_cpus->cpus[n].props.has_thread_id = true;
         ms->possible_cpus->cpus[n].props.thread_id = n;
 
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+        /* default distribution of CPUs over NUMA nodes */
+        if (nb_numa_nodes) {
+            /* preset values but do not enable them i.e. 'has_node_id = false',
+             * numa init code will enable them later if manual mapping wasn't
+             * present on CLI */
+            ms->possible_cpus->cpus[n].props.node_id = n % nb_numa_nodes;
+        }
     }
     return ms->possible_cpus;
 }
@@ -1581,6 +1596,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     /* We know we will never create a pre-ARMv7 CPU which needs 1K pages */
     mc->minimum_page_bits = 12;
     mc->possible_cpu_arch_ids = virt_possible_cpu_arch_ids;
+    mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
 }
 
 static const TypeInfo virt_machine_info = {
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index f3b372a..9cec0c1 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -2243,12 +2243,14 @@ static void pc_machine_reset(void)
     }
 }
 
-static unsigned pc_cpu_index_to_socket_id(unsigned cpu_index)
+static CpuInstanceProperties
+pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 {
-    X86CPUTopoInfo topo;
-    x86_topo_ids_from_idx(smp_cores, smp_threads, cpu_index,
-                          &topo);
-    return topo.pkg_id;
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
+    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
+
+    assert(cpu_index < possible_cpus->len);
+    return possible_cpus->cpus[cpu_index].props;;
 }
 
 static const CPUArchIdList *pc_possible_cpu_arch_ids(MachineState *ms)
@@ -2280,6 +2282,15 @@ static const CPUArchIdList *pc_possible_cpu_arch_ids(MachineState *ms)
         ms->possible_cpus->cpus[i].props.core_id = topo.core_id;
         ms->possible_cpus->cpus[i].props.has_thread_id = true;
         ms->possible_cpus->cpus[i].props.thread_id = topo.smt_id;
+
+        /* default distribution of CPUs over NUMA nodes */
+        if (nb_numa_nodes) {
+            /* preset values but do not enable them i.e. 'has_node_id = false',
+             * numa init code will enable them later if manual mapping wasn't
+             * present on CLI */
+            ms->possible_cpus->cpus[i].props.node_id =
+                topo.pkg_id % nb_numa_nodes;
+        }
     }
     return ms->possible_cpus;
 }
@@ -2322,7 +2333,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data)
     pcmc->acpi_data_size = 0x20000 + 0x8000;
     pcmc->save_tsc_khz = true;
     mc->get_hotplug_handler = pc_get_hotpug_handler;
-    mc->cpu_index_to_socket_id = pc_cpu_index_to_socket_id;
+    mc->cpu_index_to_instance_props = pc_cpu_index_to_props;
     mc->possible_cpu_arch_ids = pc_possible_cpu_arch_ids;
     mc->has_hotpluggable_cpus = true;
     mc->default_boot_order = "cad";
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 80d12d0..33405a0 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2981,11 +2981,17 @@ static HotplugHandler *spapr_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
-static unsigned spapr_cpu_index_to_socket_id(unsigned cpu_index)
+static CpuInstanceProperties
+spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
 {
-    /* Allocate to NUMA nodes on a "socket" basis (not that concept of
-     * socket means much for the paravirtualized PAPR platform) */
-    return cpu_index / smp_threads / smp_cores;
+    CPUArchId *core_slot;
+    MachineClass *mc = MACHINE_GET_CLASS(machine);
+
+    /* make sure possible_cpu are intialized */
+    mc->possible_cpu_arch_ids(machine);
+    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
+    assert(core_slot);
+    return core_slot->props;
 }
 
 static const CPUArchIdList *spapr_possible_cpu_arch_ids(MachineState *machine)
@@ -3012,8 +3018,15 @@ static const CPUArchIdList *spapr_possible_cpu_arch_ids(MachineState *machine)
         machine->possible_cpus->cpus[i].arch_id = core_id;
         machine->possible_cpus->cpus[i].props.has_core_id = true;
         machine->possible_cpus->cpus[i].props.core_id = core_id;
-        /* TODO: add 'has_node/node' here to describe
-           to which node core belongs */
+
+        /* default distribution of CPUs over NUMA nodes */
+        if (nb_numa_nodes) {
+            /* preset values but do not enable them i.e. 'has_node_id = false',
+             * numa init code will enable them later if manual mapping wasn't
+             * present on CLI */
+            machine->possible_cpus->cpus[i].props.node_id =
+                core_id / smp_threads / smp_cores % nb_numa_nodes;
+        }
     }
     return machine->possible_cpus;
 }
@@ -3138,7 +3151,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     hc->pre_plug = spapr_machine_device_pre_plug;
     hc->plug = spapr_machine_device_plug;
     hc->unplug = spapr_machine_device_unplug;
-    mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
+    mc->cpu_index_to_instance_props = spapr_cpu_index_to_props;
     mc->possible_cpu_arch_ids = spapr_possible_cpu_arch_ids;
     hc->unplug_request = spapr_machine_device_unplug_request;
 
diff --git a/numa.c b/numa.c
index 6fc2393..ab1661d 100644
--- a/numa.c
+++ b/numa.c
@@ -294,9 +294,10 @@ static void validate_numa_cpus(void)
     g_free(seen_cpus);
 }
 
-void parse_numa_opts(MachineClass *mc)
+void parse_numa_opts(MachineState *ms)
 {
     int i;
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
 
     for (i = 0; i < MAX_NODES; i++) {
         numa_info[i].node_cpu = bitmap_new(max_cpus);
@@ -378,14 +379,16 @@ void parse_numa_opts(MachineClass *mc)
          * rule grouping VCPUs by socket so that VCPUs from the same socket
          * would be on the same node.
          */
+        if (!mc->cpu_index_to_instance_props) {
+            error_report("default CPUs to NUMA node mapping isn't supported");
+            exit(1);
+        }
         if (i == nb_numa_nodes) {
             for (i = 0; i < max_cpus; i++) {
-                unsigned node_id = i % nb_numa_nodes;
-                if (mc->cpu_index_to_socket_id) {
-                    node_id = mc->cpu_index_to_socket_id(i) % nb_numa_nodes;
-                }
+                CpuInstanceProperties props;
+                props = mc->cpu_index_to_instance_props(ms, i);
 
-                set_bit(i, numa_info[node_id].node_cpu);
+                set_bit(i, numa_info[props.node_id].node_cpu);
             }
         }
 
diff --git a/vl.c b/vl.c
index f46e070..c63f4d5 100644
--- a/vl.c
+++ b/vl.c
@@ -4506,7 +4506,7 @@ int main(int argc, char **argv, char **envp)
     default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
     default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
 
-    parse_numa_opts(machine_class);
+    parse_numa_opts(current_machine);
 
     if (qemu_opts_foreach(qemu_find_opts("mon"),
                           mon_init_func, NULL, NULL)) {
-- 
2.7.4

[Qemu-devel] [PATCH v2 06/24] spapr: add node-id property to sPAPR core

[Igor Mammedov]

it will allow switching from cpu_index to core based numa
mapping in follow up patches.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 include/hw/ppc/spapr_cpu_core.h |  1 +
 include/qom/cpu.h               |  2 ++
 hw/ppc/spapr.c                  | 17 +++++++++++++++++
 hw/ppc/spapr_cpu_core.c         | 11 ++++++++---
 4 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
index 3c35665..93051e9 100644
--- a/include/hw/ppc/spapr_cpu_core.h
+++ b/include/hw/ppc/spapr_cpu_core.h
@@ -27,6 +27,7 @@ typedef struct sPAPRCPUCore {
 
     /*< public >*/
     void *threads;
+    int node_id;
 } sPAPRCPUCore;
 
 typedef struct sPAPRCPUCoreClass {
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 5d10359..55214ce 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -258,6 +258,8 @@ typedef void (*run_on_cpu_func)(CPUState *cpu, run_on_cpu_data data);
 
 struct qemu_work_item;
 
+#define CPU_UNSET_NUMA_NODE_ID -1
+
 /**
  * CPUState:
  * @cpu_index: CPU index (informative).
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 33405a0..7f58ee4 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2824,9 +2824,11 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
     MachineClass *mc = MACHINE_GET_CLASS(hotplug_dev);
     Error *local_err = NULL;
     CPUCore *cc = CPU_CORE(dev);
+    sPAPRCPUCore *sc = SPAPR_CPU_CORE(dev);
     char *base_core_type = spapr_get_cpu_core_type(machine->cpu_model);
     const char *type = object_get_typename(OBJECT(dev));
     CPUArchId *core_slot;
+    int node_id;
     int index;
 
     if (dev->hotplugged && !mc->has_hotpluggable_cpus) {
@@ -2861,6 +2863,21 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
         goto out;
     }
 
+    node_id = numa_get_node_for_cpu(cc->core_id);
+    if (node_id == nb_numa_nodes) {
+        /* by default CPUState::numa_node was 0 if it's not set via CLI
+         * keep it this way for now but in future we probably should
+         * refuse to start up with incomplete numa mapping */
+        node_id = 0;
+    }
+    if (sc->node_id == CPU_UNSET_NUMA_NODE_ID) {
+        sc->node_id = node_id;
+    } else if (sc->node_id != node_id) {
+        error_setg(&local_err, "node-id %d must match numa node specified"
+            "with -numa option for cpu-index %d", sc->node_id, cc->core_id);
+        goto out;
+    }
+
 out:
     g_free(base_core_type);
     error_propagate(errp, local_err);
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 4389ef4..9de7a56 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -176,7 +176,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
     const char *typename = object_class_get_name(scc->cpu_class);
     size_t size = object_type_get_instance_size(typename);
     Error *local_err = NULL;
-    int core_node_id = numa_get_node_for_cpu(cc->core_id);;
     void *obj;
     int i, j;
 
@@ -194,10 +193,10 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
 
         /* Set NUMA node for the added CPUs  */
         node_id = numa_get_node_for_cpu(cs->cpu_index);
-        if (node_id != core_node_id) {
+        if (node_id != sc->node_id) {
             error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
                 " on CPU[core-id: %d, node-id: %d], node-id must be the same",
-                 node_id, cs->cpu_index, cc->core_id, core_node_id);
+                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
             goto err;
         }
         if (node_id < nb_numa_nodes) {
@@ -263,6 +262,11 @@ static const char *spapr_core_models[] = {
     "POWER9_v1.0",
 };
 
+static Property spapr_cpu_core_properties[] = {
+    DEFINE_PROP_INT32("node-id", sPAPRCPUCore, node_id, CPU_UNSET_NUMA_NODE_ID),
+    DEFINE_PROP_END_OF_LIST()
+};
+
 void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(oc);
@@ -270,6 +274,7 @@ void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
 
     dc->realize = spapr_cpu_core_realize;
     dc->unrealize = spapr_cpu_core_unrealizefn;
+    dc->props = spapr_cpu_core_properties;
     scc->cpu_class = cpu_class_by_name(TYPE_POWERPC_CPU, data);
     g_assert(scc->cpu_class);
 }
-- 
2.7.4

[Qemu-devel] [PATCH v2 07/24] pc: add node-id property to CPU

[Igor Mammedov]

it will allow switching from cpu_index to property based
numa mapping in follow up patches.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/pc.c      | 17 +++++++++++++++++
 target/i386/cpu.c |  1 +
 2 files changed, 18 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 9cec0c1..4a4fb1c 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1893,6 +1893,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
                             DeviceState *dev, Error **errp)
 {
     int idx;
+    int node_id;
     CPUState *cs;
     CPUArchId *cpu_slot;
     X86CPUTopoInfo topo;
@@ -1982,6 +1983,22 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
 
     cs = CPU(cpu);
     cs->cpu_index = idx;
+
+    node_id = numa_get_node_for_cpu(cs->cpu_index);
+    if (node_id == nb_numa_nodes) {
+        /* by default CPUState::numa_node was 0 if it's not set via CLI
+         * keep it this way for now but in future we probably should
+         * refuse to start up with incomplete numa mapping */
+        node_id = 0;
+    }
+    if (cs->numa_node == CPU_UNSET_NUMA_NODE_ID) {
+        cs->numa_node = node_id;
+    } else if (cs->numa_node != node_id) {
+            error_setg(errp, "node-id %d must match numa node specified"
+                "with -numa option for cpu-index %d",
+                cs->numa_node, cs->cpu_index);
+            return;
+    }
 }
 
 static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 13c0985..007a5bd 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -3982,6 +3982,7 @@ static Property x86_cpu_properties[] = {
     DEFINE_PROP_INT32("core-id", X86CPU, core_id, -1),
     DEFINE_PROP_INT32("socket-id", X86CPU, socket_id, -1),
 #endif
+    DEFINE_PROP_INT32("node-id", CPUState, numa_node, CPU_UNSET_NUMA_NODE_ID),
     DEFINE_PROP_BOOL("pmu", X86CPU, enable_pmu, false),
     { .name  = "hv-spinlocks", .info  = &qdev_prop_spinlocks },
     DEFINE_PROP_BOOL("hv-relaxed", X86CPU, hyperv_relaxed_timing, false),
-- 
2.7.4

[Qemu-devel] [PATCH v2 08/24] virt-arm: add node-id property to CPU

[Igor Mammedov]

it will allow switching from cpu_index to property based
numa mapping in follow up patches.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/arm/virt.c    | 15 +++++++++++++++
 target/arm/cpu.c |  1 +
 2 files changed, 16 insertions(+)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 3e19b5f..05a2822 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1350,6 +1350,7 @@ static void machvirt_init(MachineState *machine)
     for (n = 0; n < possible_cpus->len; n++) {
         Object *cpuobj;
         CPUState *cs;
+        int node_id;
 
         if (n >= smp_cpus) {
             break;
@@ -1362,6 +1363,20 @@ static void machvirt_init(MachineState *machine)
         cs = CPU(cpuobj);
         cs->cpu_index = n;
 
+        node_id = numa_get_node_for_cpu(cs->cpu_index);
+        if (node_id == nb_numa_nodes) {
+            /* by default CPUState::numa_node was 0 if it's not set via CLI
+             * keep it this way for now but in future we probably should
+             * refuse to start up with incomplete numa mapping */
+             node_id = 0;
+        }
+        if (cs->numa_node == CPU_UNSET_NUMA_NODE_ID) {
+            cs->numa_node = node_id;
+        } else {
+            /* CPU isn't device_add compatible yet, this shouldn't happen */
+            error_setg(&error_abort, "user set node-id not implemented");
+        }
+
         if (!vms->secure) {
             object_property_set_bool(cpuobj, false, "has_el3", NULL);
         }
diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index ee1406d..c185eb1 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -1573,6 +1573,7 @@ static Property arm_cpu_properties[] = {
     DEFINE_PROP_UINT32("midr", ARMCPU, midr, 0),
     DEFINE_PROP_UINT64("mp-affinity", ARMCPU,
                         mp_affinity, ARM64_AFFINITY_INVALID),
+    DEFINE_PROP_INT32("node-id", CPUState, numa_node, CPU_UNSET_NUMA_NODE_ID),
     DEFINE_PROP_END_OF_LIST()
 };
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Igor Mammedov]

Default node mapping initialization already checks that board
supports cpu_index to node mapping and refuses to start if
it's not supported. Do the same for explicitly provided
mapping "-numa node,cpus=..."

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 numa.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/numa.c b/numa.c
index ab1661d..b517870 100644
--- a/numa.c
+++ b/numa.c
@@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
     return -1;
 }
 
-static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
+static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
+                            QemuOpts *opts, Error **errp)
 {
     uint16_t nodenr;
     uint16List *cpus = NULL;
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
 
     if (node->has_nodeid) {
         nodenr = node->nodeid;
@@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
         return;
     }
 
+    if (!mc->cpu_index_to_instance_props) {
+        error_report("CPUs to NUMA node mapping isn't supported");
+        exit(1);
+    }
     for (cpus = node->cpus; cpus; cpus = cpus->next) {
         if (cpus->value >= max_cpus) {
             error_setg(errp,
@@ -215,6 +221,7 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
 static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
 {
     NumaOptions *object = NULL;
+    MachineState *ms = opaque;
     Error *err = NULL;
 
     {
@@ -229,7 +236,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
 
     switch (object->type) {
     case NUMA_OPTIONS_TYPE_NODE:
-        numa_node_parse(&object->u.node, opts, &err);
+        numa_node_parse(ms, &object->u.node, opts, &err);
         if (err) {
             goto end;
         }
@@ -303,7 +310,7 @@ void parse_numa_opts(MachineState *ms)
         numa_info[i].node_cpu = bitmap_new(max_cpus);
     }
 
-    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, NULL, NULL)) {
+    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, NULL)) {
         exit(1);
     }
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

Introduce machine_set_cpu_numa_node() helper that stores
node mapping for CPU in MachineState::possible_cpus.
CPU and node it belongs to is specified by 'props' argument.

Patch doesn't remove old way of storing mapping in
numa_info[X].node_cpu as removing it at the same time
makes patch rather big. Instead it just mirrors mapping
in possible_cpus and follow up per target patches will
switch to possible_cpus and numa_info[X].node_cpu will
be removed once there isn't any users left.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 include/hw/boards.h |  2 ++
 hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 numa.c              |  8 +++++++
 3 files changed, 78 insertions(+)

diff --git a/include/hw/boards.h b/include/hw/boards.h
index 5d6af21..1f518a1 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
 bool machine_mem_merge(MachineState *machine);
 void machine_register_compat_props(MachineState *machine);
 HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
+void machine_set_cpu_numa_node(MachineState *machine,
+                               CpuInstanceProperties *props, Error **errp);
 
 /**
  * CPUArchId:
diff --git a/hw/core/machine.c b/hw/core/machine.c
index ada9eea..a63f17b 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
     return head;
 }
 
+void machine_set_cpu_numa_node(MachineState *machine,
+                               CpuInstanceProperties *props, Error **errp)
+{
+    MachineClass *mc = MACHINE_GET_CLASS(machine);
+    bool match = false;
+    int i;
+
+    if (!mc->possible_cpu_arch_ids) {
+        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
+        return;
+    }
+
+    /* force board to initialize possible_cpus if it hasn't been done yet */
+    mc->possible_cpu_arch_ids(machine);
+
+    for (i = 0; i < machine->possible_cpus->len; i++) {
+        CPUArchId *slot = &machine->possible_cpus->cpus[i];
+
+        /* reject unsupported by board properties */
+        if (props->has_thread_id && !slot->props.has_thread_id) {
+            error_setg(errp, "thread-id is not supported");
+            return;
+        }
+
+        if (props->has_core_id && !slot->props.has_core_id) {
+            error_setg(errp, "core-id is not supported");
+            return;
+        }
+
+        if (props->has_socket_id && !slot->props.has_socket_id) {
+            error_setg(errp, "socket-id is not supported");
+            return;
+        }
+
+        /* skip slots with explicit mismatch */
+        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
+                continue;
+        }
+
+        if (props->has_core_id && props->core_id != slot->props.core_id) {
+                continue;
+        }
+
+        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
+                continue;
+        }
+
+        /* reject assignment if slot is already assigned, for compatibility
+         * of legacy cpu_index mapping with SPAPR core based mapping do not
+         * error out if cpu thread and matched core have the same node-id */
+        if (slot->props.has_node_id &&
+            slot->props.node_id != props->node_id) {
+            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
+                       slot->props.node_id);
+            return;
+        }
+
+        /* assign slot to node as it's matched '-numa cpu' key */
+        match = true;
+        slot->props.node_id = props->node_id;
+        slot->props.has_node_id = props->has_node_id;
+    }
+
+    if (!match) {
+        error_setg(errp, "no match found");
+    }
+}
+
 static void machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
diff --git a/numa.c b/numa.c
index b517870..5ff1212 100644
--- a/numa.c
+++ b/numa.c
@@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
         exit(1);
     }
     for (cpus = node->cpus; cpus; cpus = cpus->next) {
+        CpuInstanceProperties props;
         if (cpus->value >= max_cpus) {
             error_setg(errp,
                        "CPU index (%" PRIu16 ")"
@@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
             return;
         }
         bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
+        props = mc->cpu_index_to_instance_props(ms, cpus->value);
+        props.node_id = nodenr;
+        props.has_node_id = true;
+        machine_set_cpu_numa_node(ms, &props, &error_fatal);
     }
 
     if (node->has_mem && node->has_memdev) {
@@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
         if (i == nb_numa_nodes) {
             for (i = 0; i < max_cpus; i++) {
                 CpuInstanceProperties props;
+                /* fetch default mapping from board and enable it */
                 props = mc->cpu_index_to_instance_props(ms, i);
+                props.has_node_id = true;
 
                 set_bit(i, numa_info[props.node_id].node_cpu);
+                machine_set_cpu_numa_node(ms, &props, &error_fatal);
             }
         }
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 11/24] numa: do default mapping based on possible_cpus instead of node_cpu bitmaps

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 numa.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/numa.c b/numa.c
index 5ff1212..c7e3e0a 100644
--- a/numa.c
+++ b/numa.c
@@ -309,6 +309,7 @@ static void validate_numa_cpus(void)
 void parse_numa_opts(MachineState *ms)
 {
     int i;
+    const CPUArchIdList *possible_cpus;
     MachineClass *mc = MACHINE_GET_CLASS(ms);
 
     for (i = 0; i < MAX_NODES; i++) {
@@ -379,11 +380,6 @@ void parse_numa_opts(MachineState *ms)
 
         numa_set_mem_ranges();
 
-        for (i = 0; i < nb_numa_nodes; i++) {
-            if (!bitmap_empty(numa_info[i].node_cpu, max_cpus)) {
-                break;
-            }
-        }
         /* Historically VCPUs were assigned in round-robin order to NUMA
          * nodes. However it causes issues with guest not handling it nice
          * in case where cores/threads from a multicore CPU appear on
@@ -391,11 +387,20 @@ void parse_numa_opts(MachineState *ms)
          * rule grouping VCPUs by socket so that VCPUs from the same socket
          * would be on the same node.
          */
-        if (!mc->cpu_index_to_instance_props) {
+        if (!mc->cpu_index_to_instance_props || !mc->possible_cpu_arch_ids) {
             error_report("default CPUs to NUMA node mapping isn't supported");
             exit(1);
         }
-        if (i == nb_numa_nodes) {
+
+        possible_cpus = mc->possible_cpu_arch_ids(ms);
+        for (i = 0; i < possible_cpus->len; i++) {
+            if (possible_cpus->cpus[i].props.has_node_id) {
+                break;
+            }
+        }
+
+        /* no CPUs are assigned to NUMA nodes */
+        if (i == possible_cpus->len) {
             for (i = 0; i < max_cpus; i++) {
                 CpuInstanceProperties props;
                 /* fetch default mapping from board and enable it */
-- 
2.7.4

[Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Igor Mammedov]

wrappers should make access to [has]node_id fields more readable

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 follow up patches will use this wrappers
 v2:
    - add wrappers (Drew)
---
 include/sysemu/numa.h | 10 ++++++++++
 numa.c                |  2 +-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index 46ea6c7..98d01e6 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -35,4 +35,14 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp);
 /* on success returns node index in numa_info,
  * on failure returns nb_numa_nodes */
 int numa_get_node_for_cpu(int idx);
+
+static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
+{
+    return possible_cpus->cpus[idx].props.has_node_id;
+}
+
+static inline int numa_node_id(const CPUArchIdList *possible_cpus, int idx)
+{
+    return possible_cpus->cpus[idx].props.node_id;
+}
 #endif
diff --git a/numa.c b/numa.c
index c7e3e0a..872ee0d 100644
--- a/numa.c
+++ b/numa.c
@@ -394,7 +394,7 @@ void parse_numa_opts(MachineState *ms)
 
         possible_cpus = mc->possible_cpu_arch_ids(ms);
         for (i = 0; i < possible_cpus->len; i++) {
-            if (possible_cpus->cpus[i].props.has_node_id) {
+            if (numa_has_node_id(possible_cpus, i)) {
                 break;
             }
         }
-- 
2.7.4

[Qemu-devel] [PATCH v2 13/24] pc: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
   use numa_[has_]node_id() wrappers (Drew)
---
 hw/acpi/cpu.c        |  7 +++----
 hw/i386/acpi-build.c | 11 ++++-------
 hw/i386/pc.c         | 18 ++++++++++--------
 3 files changed, 17 insertions(+), 19 deletions(-)

diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index 8c719d3..2002198 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -503,7 +503,6 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
 
         /* build Processor object for each processor */
         for (i = 0; i < arch_ids->len; i++) {
-            int j;
             Aml *dev;
             Aml *uid = aml_int(i);
             GArray *madt_buf = g_array_new(0, 1, 1);
@@ -557,9 +556,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
              * as a result _PXM is required for all CPUs which might
              * be hot-plugged. For simplicity, add it for all CPUs.
              */
-            j = numa_get_node_for_cpu(i);
-            if (j < nb_numa_nodes) {
-                aml_append(dev, aml_name_decl("_PXM", aml_int(j)));
+            if (numa_has_node_id(arch_ids, i)) {
+                aml_append(dev,
+                    aml_name_decl("_PXM", aml_int(numa_node_id(arch_ids, i))));
             }
 
             aml_append(cpus_dev, dev);
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 2073108..a66a968 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -2306,7 +2306,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
     srat->reserved1 = cpu_to_le32(1);
 
     for (i = 0; i < apic_ids->len; i++) {
-        int j = numa_get_node_for_cpu(i);
+        int node_id = numa_has_node_id(apic_ids, i) ?
+            numa_node_id(apic_ids, i) : 0;
         uint32_t apic_id = apic_ids->cpus[i].arch_id;
 
         if (apic_id < 255) {
@@ -2316,9 +2317,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
             core->type = ACPI_SRAT_PROCESSOR_APIC;
             core->length = sizeof(*core);
             core->local_apic_id = apic_id;
-            if (j < nb_numa_nodes) {
-                core->proximity_lo = j;
-            }
+            core->proximity_lo = node_id;
             memset(core->proximity_hi, 0, 3);
             core->local_sapic_eid = 0;
             core->flags = cpu_to_le32(1);
@@ -2329,9 +2328,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
             core->type = ACPI_SRAT_PROCESSOR_x2APIC;
             core->length = sizeof(*core);
             core->x2apic_id = cpu_to_le32(apic_id);
-            if (j < nb_numa_nodes) {
-                core->proximity_domain = cpu_to_le32(j);
-            }
+            core->proximity_domain = cpu_to_le32(node_id);
             core->flags = cpu_to_le32(1);
         }
     }
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 4a4fb1c..aeecf4b 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -747,7 +747,9 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
 {
     FWCfgState *fw_cfg;
     uint64_t *numa_fw_cfg;
-    int i, j;
+    int i;
+    const CPUArchIdList *cpus;
+    MachineClass *mc = MACHINE_GET_CLASS(pcms);
 
     fw_cfg = fw_cfg_init_io_dma(FW_CFG_IO_BASE, FW_CFG_IO_BASE + 4, as);
     fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, pcms->boot_cpus);
@@ -782,12 +784,12 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
      */
     numa_fw_cfg = g_new0(uint64_t, 1 + pcms->apic_id_limit + nb_numa_nodes);
     numa_fw_cfg[0] = cpu_to_le64(nb_numa_nodes);
-    for (i = 0; i < max_cpus; i++) {
-        unsigned int apic_id = x86_cpu_apic_id_from_index(i);
+    cpus = mc->possible_cpu_arch_ids(MACHINE(pcms));
+    for (i = 0; i < cpus->len; i++) {
+        unsigned int apic_id = cpus->cpus[i].arch_id;
         assert(apic_id < pcms->apic_id_limit);
-        j = numa_get_node_for_cpu(i);
-        if (j < nb_numa_nodes) {
-            numa_fw_cfg[apic_id + 1] = cpu_to_le64(j);
+        if (numa_has_node_id(cpus, i)) {
+            numa_fw_cfg[apic_id + 1] = cpu_to_le64(numa_node_id(cpus, i));
         }
     }
     for (i = 0; i < nb_numa_nodes; i++) {
@@ -1984,8 +1986,8 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
     cs = CPU(cpu);
     cs->cpu_index = idx;
 
-    node_id = numa_get_node_for_cpu(cs->cpu_index);
-    if (node_id == nb_numa_nodes) {
+    node_id = cpu_slot->props.node_id;
+    if (!cpu_slot->props.has_node_id) {
         /* by default CPUState::numa_node was 0 if it's not set via CLI
          * keep it this way for now but in future we probably should
          * refuse to start up with incomplete numa mapping */
-- 
2.7.4

[Qemu-devel] [PATCH v2 14/24] spapr: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[Igor Mammedov]

it's safe to remove thread node_id != core node_id error
branch as machine_set_cpu_numa_node() also does mismatch
check and is called even before any CPU is created.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/ppc/spapr.c          |  4 ++--
 hw/ppc/spapr_cpu_core.c | 14 ++------------
 2 files changed, 4 insertions(+), 14 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 7f58ee4..bcb91e7 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2863,8 +2863,8 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
         goto out;
     }
 
-    node_id = numa_get_node_for_cpu(cc->core_id);
-    if (node_id == nb_numa_nodes) {
+    node_id = core_slot->props.node_id;
+    if (!core_slot->props.has_node_id) {
         /* by default CPUState::numa_node was 0 if it's not set via CLI
          * keep it this way for now but in future we probably should
          * refuse to start up with incomplete numa mapping */
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 9de7a56..a17ea07 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -181,7 +181,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
 
     sc->threads = g_malloc0(size * cc->nr_threads);
     for (i = 0; i < cc->nr_threads; i++) {
-        int node_id;
         char id[32];
         CPUState *cs;
 
@@ -191,17 +190,8 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
         cs = CPU(obj);
         cs->cpu_index = cc->core_id + i;
 
-        /* Set NUMA node for the added CPUs  */
-        node_id = numa_get_node_for_cpu(cs->cpu_index);
-        if (node_id != sc->node_id) {
-            error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
-                " on CPU[core-id: %d, node-id: %d], node-id must be the same",
-                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
-            goto err;
-        }
-        if (node_id < nb_numa_nodes) {
-            cs->numa_node = node_id;
-        }
+        /* Set NUMA node for the threads belonged to core  */
+        cs->numa_node = sc->node_id;
 
         snprintf(id, sizeof(id), "thread[%d]", i);
         object_property_add_child(OBJECT(sc), id, obj, &local_err);
-- 
2.7.4

[Qemu-devel] [PATCH v2 15/24] virt-arm: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
   use numa_[has_]node_id() wrappers (Drew)
---
 hw/arm/virt-acpi-build.c | 19 +++++++------------
 hw/arm/virt.c            | 13 +++++++------
 2 files changed, 14 insertions(+), 18 deletions(-)

diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 0835e59..2404bb1 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -486,30 +486,25 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     AcpiSystemResourceAffinityTable *srat;
     AcpiSratProcessorGiccAffinity *core;
     AcpiSratMemoryAffinity *numamem;
-    int i, j, srat_start;
+    int i, srat_start;
     uint64_t mem_base;
-    uint32_t *cpu_node = g_malloc0(vms->smp_cpus * sizeof(uint32_t));
-
-    for (i = 0; i < vms->smp_cpus; i++) {
-        j = numa_get_node_for_cpu(i);
-        if (j < nb_numa_nodes) {
-                cpu_node[i] = j;
-        }
-    }
+    MachineClass *mc = MACHINE_GET_CLASS(vms);
+    const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids(MACHINE(vms));
 
     srat_start = table_data->len;
     srat = acpi_data_push(table_data, sizeof(*srat));
     srat->reserved1 = cpu_to_le32(1);
 
-    for (i = 0; i < vms->smp_cpus; ++i) {
+    for (i = 0; i < cpu_list->len; ++i) {
+        int node_id = numa_has_node_id(cpu_list, i) ?
+            numa_node_id(cpu_list, i) : 0;
         core = acpi_data_push(table_data, sizeof(*core));
         core->type = ACPI_SRAT_PROCESSOR_GICC;
         core->length = sizeof(*core);
-        core->proximity = cpu_to_le32(cpu_node[i]);
+        core->proximity = cpu_to_le32(node_id);
         core->acpi_processor_uid = cpu_to_le32(i);
         core->flags = cpu_to_le32(1);
     }
-    g_free(cpu_node);
 
     mem_base = vms->memmap[VIRT_MEM].base;
     for (i = 0; i < nb_numa_nodes; ++i) {
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 05a2822..233e8a7 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -338,7 +338,7 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
 {
     int cpu;
     int addr_cells = 1;
-    unsigned int i;
+    const MachineState *ms = MACHINE(vms);
 
     /*
      * From Documentation/devicetree/bindings/arm/cpus.txt
@@ -369,6 +369,7 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
     for (cpu = vms->smp_cpus - 1; cpu >= 0; cpu--) {
         char *nodename = g_strdup_printf("/cpus/cpu@%d", cpu);
         ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(cpu));
+        CPUState *cs = CPU(armcpu);
 
         qemu_fdt_add_subnode(vms->fdt, nodename);
         qemu_fdt_setprop_string(vms->fdt, nodename, "device_type", "cpu");
@@ -389,9 +390,9 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
                                   armcpu->mp_affinity);
         }
 
-        i = numa_get_node_for_cpu(cpu);
-        if (i < nb_numa_nodes) {
-            qemu_fdt_setprop_cell(vms->fdt, nodename, "numa-node-id", i);
+        if (numa_has_node_id(ms->possible_cpus, cs->cpu_index)) {
+            qemu_fdt_setprop_cell(vms->fdt, nodename, "numa-node-id",
+                numa_node_id(ms->possible_cpus, cs->cpu_index));
         }
 
         g_free(nodename);
@@ -1363,8 +1364,8 @@ static void machvirt_init(MachineState *machine)
         cs = CPU(cpuobj);
         cs->cpu_index = n;
 
-        node_id = numa_get_node_for_cpu(cs->cpu_index);
-        if (node_id == nb_numa_nodes) {
+        node_id = numa_node_id(possible_cpus, cs->cpu_index);
+        if (!numa_has_node_id(possible_cpus, cs->cpu_index)) {
             /* by default CPUState::numa_node was 0 if it's not set via CLI
              * keep it this way for now but in future we probably should
              * refuse to start up with incomplete numa mapping */
-- 
2.7.4

[Qemu-devel] [PATCH v2 16/24] QMP: include CpuInstanceProperties into query_cpus output output

[Igor Mammedov]

if board supports CpuInstanceProperties, report them for
each CPU thread listed. Main motivation for this is to
provide these properties introspection via QMP interface
for using in test cases to verify numa node to cpu mapping,
which includes not only boards that support cpu hotplug
and have this info in query-hotpluggable-cpus (pc/spapr)
but also for boards that don't not support hotpluggable-cpus
but support numa mapping (virt-arm).

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
v2:
  * fix checkpatch error and remove extra space after = (Eric)
---
 cpus.c           | 10 ++++++++++
 qapi-schema.json |  6 +++++-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index 740b8dc..4f91d25 100644
--- a/cpus.c
+++ b/cpus.c
@@ -50,6 +50,7 @@
 #include "qapi-event.h"
 #include "hw/nmi.h"
 #include "sysemu/replay.h"
+#include "hw/boards.h"
 
 #ifdef CONFIG_LINUX
 
@@ -1859,6 +1860,8 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
 
 CpuInfoList *qmp_query_cpus(Error **errp)
 {
+    MachineState *ms = MACHINE(qdev_get_machine());
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
     CpuInfoList *head = NULL, *cur_item = NULL;
     CPUState *cpu;
 
@@ -1909,6 +1912,13 @@ CpuInfoList *qmp_query_cpus(Error **errp)
 #else
         info->value->arch = CPU_INFO_ARCH_OTHER;
 #endif
+        info->value->has_props = !!mc->cpu_index_to_instance_props;
+        if (info->value->has_props) {
+            CpuInstanceProperties *props;
+            props = g_malloc0(sizeof(*props));
+            *props = mc->cpu_index_to_instance_props(ms, cpu->cpu_index);
+            info->value->props = props;
+        }
 
         /* XXX: waiting for the qapi to support GSList */
         if (!cur_item) {
diff --git a/qapi-schema.json b/qapi-schema.json
index 01b087f..76d137d 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1325,6 +1325,9 @@
 #
 # @thread_id: ID of the underlying host thread
 #
+# @props: properties describing to which node/socket/core/thread
+#         virtual CPU belongs to, provided if supported by board (since 2.10)
+#
 # @arch: architecture of the cpu, which determines which additional fields
 #        will be listed (since 2.6)
 #
@@ -1335,7 +1338,8 @@
 ##
 { 'union': 'CpuInfo',
   'base': {'CPU': 'int', 'current': 'bool', 'halted': 'bool',
-           'qom_path': 'str', 'thread_id': 'int', 'arch': 'CpuInfoArch' },
+           'qom_path': 'str', 'thread_id': 'int',
+           '*props': 'CpuInstanceProperties', 'arch': 'CpuInfoArch' },
   'discriminator': 'arch',
   'data': { 'x86': 'CpuInfoX86',
             'sparc': 'CpuInfoSPARC',
-- 
2.7.4

[Qemu-devel] [PATCH v2 17/24] tests: numa: add case for QMP command query-cpus

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
 * fix checkpatch error, move { to newline
---
 tests/numa-test.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/tests/numa-test.c b/tests/numa-test.c
index f5da0c8..2722687 100644
--- a/tests/numa-test.c
+++ b/tests/numa-test.c
@@ -87,6 +87,50 @@ static void test_mon_partial(const void *data)
     g_free(cli);
 }
 
+static QList *get_cpus(QDict **resp)
+{
+    *resp = qmp("{ 'execute': 'query-cpus' }");
+    g_assert(*resp);
+    g_assert(qdict_haskey(*resp, "return"));
+    return  qdict_get_qlist(*resp, "return");
+}
+
+static void test_query_cpus(const void *data)
+{
+    char *cli;
+    QDict *resp;
+    QList *cpus;
+    const QObject *e;
+
+    cli = make_cli(data, "-smp 8 -numa node,cpus=0-3 -numa node,cpus=4-7");
+    qtest_start(cli);
+    cpus = get_cpus(&resp);
+    g_assert(cpus);
+
+    while ((e = qlist_pop(cpus))) {
+        QDict *cpu, *props;
+        int64_t cpu_idx, node;
+
+        cpu = qobject_to_qdict(e);
+        g_assert(qdict_haskey(cpu, "CPU"));
+        g_assert(qdict_haskey(cpu, "props"));
+
+        cpu_idx = qdict_get_int(cpu, "CPU");
+        props = qdict_get_qdict(cpu, "props");
+        g_assert(qdict_haskey(props, "node-id"));
+        node = qdict_get_int(props, "node-id");
+        if (cpu_idx >= 0 && cpu_idx < 4) {
+            g_assert_cmpint(node, ==, 0);
+        } else {
+            g_assert_cmpint(node, ==, 1);
+        }
+    }
+
+    QDECREF(resp);
+    qtest_end();
+    g_free(cli);
+}
+
 int main(int argc, char **argv)
 {
     const char *args = NULL;
@@ -101,6 +145,7 @@ int main(int argc, char **argv)
     qtest_add_data_func("/numa/mon/default", args, test_mon_default);
     qtest_add_data_func("/numa/mon/cpus/explicit", args, test_mon_explicit);
     qtest_add_data_func("/numa/mon/cpus/partial", args, test_mon_partial);
+    qtest_add_data_func("/numa/qmp/cpus/query-cpus", args, test_query_cpus);
 
     return g_test_run();
 }
-- 
2.7.4

[Qemu-devel] [PATCH v2 18/24] numa: remove no longer used numa_get_node_for_cpu()

[Igor Mammedov]

it's been replaced by fetching mapping info from possible_cpus

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 include/sysemu/numa.h |  4 ----
 numa.c                | 14 --------------
 2 files changed, 18 deletions(-)

diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index 98d01e6..9077bb2 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -32,10 +32,6 @@ void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
 
-/* on success returns node index in numa_info,
- * on failure returns nb_numa_nodes */
-int numa_get_node_for_cpu(int idx);
-
 static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
 {
     return possible_cpus->cpus[idx].props.has_node_id;
diff --git a/numa.c b/numa.c
index 872ee0d..06c42e9 100644
--- a/numa.c
+++ b/numa.c
@@ -583,20 +583,6 @@ MemdevList *qmp_query_memdev(Error **errp)
     return list;
 }
 
-int numa_get_node_for_cpu(int idx)
-{
-    int i;
-
-    assert(idx < max_cpus);
-
-    for (i = 0; i < nb_numa_nodes; i++) {
-        if (test_bit(idx, numa_info[i].node_cpu)) {
-            break;
-        }
-    }
-    return i;
-}
-
 void ram_block_notifier_add(RAMBlockNotifier *n)
 {
     QLIST_INSERT_HEAD(&ram_list.ramblock_notifiers, n, next);
-- 
2.7.4

[Qemu-devel] [PATCH v2 19/24] numa: remove no longer need numa_post_machine_init()

[Igor Mammedov]

CPUState::numa_node is still in use but now it's set by
board when it creates CPU objects. So there isn't any
need to set it again after all CPU's are created,
since it's been already set.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 include/sysemu/numa.h |  1 -
 numa.c                | 15 ---------------
 vl.c                  |  2 --
 3 files changed, 18 deletions(-)

diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index 9077bb2..0b0a044 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -25,7 +25,6 @@ typedef struct node_info {
 
 extern NodeInfo numa_info[MAX_NODES];
 void parse_numa_opts(MachineState *ms);
-void numa_post_machine_init(void);
 void query_numa_node_mem(uint64_t node_mem[]);
 extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
diff --git a/numa.c b/numa.c
index 06c42e9..f8c849d 100644
--- a/numa.c
+++ b/numa.c
@@ -418,21 +418,6 @@ void parse_numa_opts(MachineState *ms)
     }
 }
 
-void numa_post_machine_init(void)
-{
-    CPUState *cpu;
-    int i;
-
-    CPU_FOREACH(cpu) {
-        for (i = 0; i < nb_numa_nodes; i++) {
-            assert(cpu->cpu_index < max_cpus);
-            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
-                cpu->numa_node = i;
-            }
-        }
-    }
-}
-
 static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
                                            const char *name,
                                            uint64_t ram_size)
diff --git a/vl.c b/vl.c
index c63f4d5..fe4741d 100644
--- a/vl.c
+++ b/vl.c
@@ -4595,8 +4595,6 @@ int main(int argc, char **argv, char **envp)
 
     cpu_synchronize_all_post_init();
 
-    numa_post_machine_init();
-
     rom_reset_order_override();
 
     /*
-- 
2.7.4

[Qemu-devel] [PATCH v2 20/24] machine: call machine init from wrapper

[Igor Mammedov]

add machine_run_board_init() wrapper that calls machine
init for now but in follow up patches it will be used
to run generic machine code that should run before
machine init.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 include/hw/boards.h | 1 +
 hw/core/machine.c   | 6 ++++++
 vl.c                | 2 +-
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/include/hw/boards.h b/include/hw/boards.h
index 1f518a1..0e74484 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -32,6 +32,7 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
 MachineClass *find_default_machine(void);
 extern MachineState *current_machine;
 
+void machine_run_board_init(MachineState *machine);
 bool machine_usb(MachineState *machine);
 bool machine_kernel_irqchip_allowed(MachineState *machine);
 bool machine_kernel_irqchip_required(MachineState *machine);
diff --git a/hw/core/machine.c b/hw/core/machine.c
index a63f17b..217b4f4 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -648,6 +648,12 @@ bool machine_mem_merge(MachineState *machine)
     return machine->mem_merge;
 }
 
+void machine_run_board_init(MachineState *machine)
+{
+    MachineClass *machine_class = MACHINE_GET_CLASS(machine);
+    machine_class->init(machine);
+}
+
 static void machine_class_finalize(ObjectClass *klass, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(klass);
diff --git a/vl.c b/vl.c
index fe4741d..ac46d6e 100644
--- a/vl.c
+++ b/vl.c
@@ -4562,7 +4562,7 @@ int main(int argc, char **argv, char **envp)
     current_machine->boot_order = boot_order;
     current_machine->cpu_model = cpu_model;
 
-    machine_class->init(current_machine);
+    machine_run_board_init(current_machine);
 
     realtime_init();
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 21/24] numa: use possible_cpus for not mapped CPUs check

[Igor Mammedov]

and remove corresponding part in numa.c that uses
node_cpu bitmaps.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
It's one more less user of node_cpu bitmpas, following
commit will remove the last user along with
node_cpu itself.
---
 hw/core/machine.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 numa.c            | 10 ----------
 2 files changed, 58 insertions(+), 10 deletions(-)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index 217b4f4..4229054 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -19,6 +19,7 @@
 #include "sysemu/sysemu.h"
 #include "qemu/error-report.h"
 #include "qemu/cutils.h"
+#include "sysemu/numa.h"
 
 static char *machine_get_accel(Object *obj, Error **errp)
 {
@@ -648,9 +649,66 @@ bool machine_mem_merge(MachineState *machine)
     return machine->mem_merge;
 }
 
+static char *cpu_slot_to_string(const CPUArchId *cpu)
+{
+    GString *s = g_string_new(NULL);
+    if (cpu->props.has_socket_id) {
+        g_string_append_printf(s, "socket-id: %"PRId64, cpu->props.socket_id);
+    }
+    if (cpu->props.has_core_id) {
+        if (s->len) {
+            g_string_append_printf(s, ", ");
+        }
+        g_string_append_printf(s, "core-id: %"PRId64, cpu->props.core_id);
+    }
+    if (cpu->props.has_thread_id) {
+        if (s->len) {
+            g_string_append_printf(s, ", ");
+        }
+        g_string_append_printf(s, "thread-id: %"PRId64, cpu->props.thread_id);
+    }
+    return g_string_free(s, false);
+}
+
+static void machine_numa_validate(MachineState *machine)
+{
+    int i;
+    GString *s = g_string_new(NULL);
+    MachineClass *mc = MACHINE_GET_CLASS(machine);
+    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(machine);
+
+    assert(nb_numa_nodes);
+    for (i = 0; i < possible_cpus->len; i++) {
+        const CPUArchId *cpu_slot = &possible_cpus->cpus[i];
+
+        /* at this point numa mappings are initilized by CLI options
+         * or with default mappings so it's sufficient to list
+         * all not yet mapped CPUs here */
+        /* TODO: make it hard error in future */
+        if (!cpu_slot->props.has_node_id) {
+            char *cpu_str = cpu_slot_to_string(cpu_slot);
+            g_string_append_printf(s, "%sCPU %d [%s]", s->len ? ", " : "", i,
+                                   cpu_str);
+            g_free(cpu_str);
+        }
+    }
+    if (s->len) {
+        error_report("warning: CPU(s) not present in any NUMA nodes: %s",
+                     s->str);
+        error_report("warning: All CPU(s) up to maxcpus should be described "
+                     "in NUMA config, ability to start up with partial NUMA "
+                     "mappings is obsoleted and will be removed in future");
+    }
+    g_string_free(s, true);
+}
+
 void machine_run_board_init(MachineState *machine)
 {
     MachineClass *machine_class = MACHINE_GET_CLASS(machine);
+
+    if (nb_numa_nodes) {
+        machine_numa_validate(machine);
+    }
     machine_class->init(machine);
 }
 
diff --git a/numa.c b/numa.c
index f8c849d..d7aa662 100644
--- a/numa.c
+++ b/numa.c
@@ -293,16 +293,6 @@ static void validate_numa_cpus(void)
         bitmap_or(seen_cpus, seen_cpus,
                   numa_info[i].node_cpu, max_cpus);
     }
-
-    if (!bitmap_full(seen_cpus, max_cpus)) {
-        char *msg;
-        bitmap_complement(seen_cpus, seen_cpus, max_cpus);
-        msg = enumerate_cpus(seen_cpus, max_cpus);
-        error_report("warning: CPU(s) not present in any NUMA nodes: %s", msg);
-        error_report("warning: All CPU(s) up to maxcpus should be described "
-                     "in NUMA config");
-        g_free(msg);
-    }
     g_free(seen_cpus);
 }
 
-- 
2.7.4

[Qemu-devel] [PATCH v2 22/24] numa: remove node_cpu bitmaps as they are no longer used

[Igor Mammedov]

Postfactum "CPU(s) present in multiple NUMA nodes" check
was the last user of node_cpu bitmaps, but it's not need
as machine_set_cpu_numa_node() does the similar check at
the time mapping is set for cpus (i.e. when -numa cpus=
is parsed) and ensures that cpu can be mapped only to
one node.

Remove duplicate check based on node_cpu bitmaps and
since the last user is gone remove node_cpu as well,
which completes internal transition from legacy bitmap
based mapping storage to possible_cpus storage.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 include/sysemu/numa.h |  1 -
 numa.c                | 42 ------------------------------------------
 2 files changed, 43 deletions(-)

diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index 0b0a044..7f2d397 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -17,7 +17,6 @@ struct numa_addr_range {
 
 typedef struct node_info {
     uint64_t node_mem;
-    unsigned long *node_cpu;
     struct HostMemoryBackend *node_memdev;
     bool present;
     QLIST_HEAD(, numa_addr_range) addr; /* List to store address ranges */
diff --git a/numa.c b/numa.c
index d7aa662..40e9f44 100644
--- a/numa.c
+++ b/numa.c
@@ -177,7 +177,6 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
                        cpus->value, max_cpus);
             return;
         }
-        bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
         props = mc->cpu_index_to_instance_props(ms, cpus->value);
         props.node_id = nodenr;
         props.has_node_id = true;
@@ -261,51 +260,12 @@ end:
     return 0;
 }
 
-static char *enumerate_cpus(unsigned long *cpus, int max_cpus)
-{
-    int cpu;
-    bool first = true;
-    GString *s = g_string_new(NULL);
-
-    for (cpu = find_first_bit(cpus, max_cpus);
-        cpu < max_cpus;
-        cpu = find_next_bit(cpus, max_cpus, cpu + 1)) {
-        g_string_append_printf(s, "%s%d", first ? "" : " ", cpu);
-        first = false;
-    }
-    return g_string_free(s, FALSE);
-}
-
-static void validate_numa_cpus(void)
-{
-    int i;
-    unsigned long *seen_cpus = bitmap_new(max_cpus);
-
-    for (i = 0; i < nb_numa_nodes; i++) {
-        if (bitmap_intersects(seen_cpus, numa_info[i].node_cpu, max_cpus)) {
-            bitmap_and(seen_cpus, seen_cpus,
-                       numa_info[i].node_cpu, max_cpus);
-            error_report("CPU(s) present in multiple NUMA nodes: %s",
-                         enumerate_cpus(seen_cpus, max_cpus));
-            g_free(seen_cpus);
-            exit(EXIT_FAILURE);
-        }
-        bitmap_or(seen_cpus, seen_cpus,
-                  numa_info[i].node_cpu, max_cpus);
-    }
-    g_free(seen_cpus);
-}
-
 void parse_numa_opts(MachineState *ms)
 {
     int i;
     const CPUArchIdList *possible_cpus;
     MachineClass *mc = MACHINE_GET_CLASS(ms);
 
-    for (i = 0; i < MAX_NODES; i++) {
-        numa_info[i].node_cpu = bitmap_new(max_cpus);
-    }
-
     if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, NULL)) {
         exit(1);
     }
@@ -397,12 +357,10 @@ void parse_numa_opts(MachineState *ms)
                 props = mc->cpu_index_to_instance_props(ms, i);
                 props.has_node_id = true;
 
-                set_bit(i, numa_info[props.node_id].node_cpu);
                 machine_set_cpu_numa_node(ms, &props, &error_fatal);
             }
         }
 
-        validate_numa_cpus();
     } else {
         numa_set_mem_node_id(0, ram_size, 0);
     }
-- 
2.7.4

[Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Igor Mammedov]

legacy cpu to node mapping is using cpu index values to map
VCPU to node with help of '-numa node,nodeid=node,cpus=x[-y]'
option. However cpu index is internal concept and QEMU users
have to guess /reimplement qemu's logic/ to map it to
a concrete cpu socket/core/thread to make sane CPUs
placement across numa nodes.

This patch allows to map cpu objects to numa nodes using
the same properties as used for cpus with -device/device_add
(socket-id/core-id/thread-id/node-id).

At present valid properties/values to address CPUs could be
fetched using hotpluggable-cpus monitor/qmp command, it will
require user to start qemu twice when creating domain to fetch
possible CPUs for a machine type/-smp layout first and
then the second time with numa explicit mapping for actual
usage. The first step results could be saved and reused to
set/change mapping later as far as machine type/-smp stays
the same.

Proposed impl. supports exact and wildcard matching to
simplify CLI and allow to set mapping for a specific cpu
or group of cpu objects specified by matched properties.

For example:

   # exact mapping x86
   -numa cpu,node-id=x,socket-id=y,core-id=z,thread-id=n

   # exact mapping SPAPR
   -numa cpu,node-id=x,core-id=y

   # wildcard mapping, all cpu objects that match socket-id=y
   # are mapped to node-id=x
   -numa cpu,node-id=x,socket-id=y

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  - use new NumaCpuOptions instead of CpuInstanceProperties in
    NumaOptions, so that in future we could decouple both
    if needed. (Eduardo Habkost <ehabkost@redhat.com>)
  - clarify effect of NumaCpuOptions.node-id in qapi-schema.json
---
 numa.c           | 25 +++++++++++++++++++++++++
 qapi-schema.json | 21 +++++++++++++++++++--
 qemu-options.hx  | 23 ++++++++++++++++++++++-
 3 files changed, 66 insertions(+), 3 deletions(-)

diff --git a/numa.c b/numa.c
index 40e9f44..61521f5 100644
--- a/numa.c
+++ b/numa.c
@@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
     NumaOptions *object = NULL;
     MachineState *ms = opaque;
     Error *err = NULL;
+    CpuInstanceProperties cpu;
 
     {
         Visitor *v = opts_visitor_new(opts);
@@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
         }
         nb_numa_nodes++;
         break;
+    case NUMA_OPTIONS_TYPE_CPU:
+        if (!object->u.cpu.has_node_id) {
+            error_setg(&err, "Missing mandatory node-id property");
+            goto end;
+        }
+        if (!numa_info[object->u.cpu.node_id].present) {
+            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
+                "defined with -numa node,nodeid=ID before it's used with "
+                "-numa cpu,node-id=ID", object->u.cpu.node_id);
+            goto end;
+        }
+
+        memset(&cpu, 0, sizeof(cpu));
+        cpu.has_node_id = object->u.cpu.has_node_id;
+        cpu.node_id = object->u.cpu.node_id;
+        cpu.has_socket_id = object->u.cpu.has_socket_id;
+        cpu.socket_id = object->u.cpu.socket_id;
+        cpu.has_core_id = object->u.cpu.has_core_id;
+        cpu.core_id = object->u.cpu.core_id;
+        cpu.has_thread_id = object->u.cpu.has_thread_id;
+        cpu.thread_id = object->u.cpu.thread_id;
+
+        machine_set_cpu_numa_node(ms, &cpu, &err);
+        break;
     default:
         abort();
     }
diff --git a/qapi-schema.json b/qapi-schema.json
index 76d137d..5baf3a4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -5680,10 +5680,12 @@
 ##
 # @NumaOptionsType:
 #
+# @cpu: property based CPU(s) to node mapping (Since: 2.10)
+#
 # Since: 2.1
 ##
 { 'enum': 'NumaOptionsType',
-  'data': [ 'node' ] }
+  'data': [ 'node', 'cpu' ] }
 
 ##
 # @NumaOptions:
@@ -5696,7 +5698,8 @@
   'base': { 'type': 'NumaOptionsType' },
   'discriminator': 'type',
   'data': {
-    'node': 'NumaNodeOptions' }}
+    'node': 'NumaNodeOptions',
+    'cpu': 'NumaCpuOptions' }}
 
 ##
 # @NumaNodeOptions:
@@ -5725,6 +5728,20 @@
    '*memdev': 'str' }}
 
 ##
+# @NumaCpuOptions:
+#
+# Option "-numa cpu" overrides default cpu to node mapping.
+# It accepts the same set of cpu properties as returned by
+# query-hotpluggable-cpus[].props, where node-id could be used to
+# override default node mapping.
+#
+# Since: 2.10
+##
+{ 'struct': 'NumaCpuOptions',
+   'base': 'CpuInstanceProperties',
+   'data' : {} }
+
+##
 # @HostMemPolicy:
 #
 # Host memory policy types
diff --git a/qemu-options.hx b/qemu-options.hx
index 787b9c3..e88f534 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -139,13 +139,16 @@ ETEXI
 
 DEF("numa", HAS_ARG, QEMU_OPTION_numa,
     "-numa node[,mem=size][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n"
-    "-numa node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n", QEMU_ARCH_ALL)
+    "-numa node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n"
+    "-numa cpu,node-id=node[,socket-id=x][,core-id=y][,thread-id=z]\n", QEMU_ARCH_ALL)
 STEXI
 @item -numa node[,mem=@var{size}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}]
 @itemx -numa node[,memdev=@var{id}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}]
+@itemx -numa cpu,node-id=@var{node}[,socket-id=@var{x}][,core-id=@var{y}][,thread-id=@var{z}]
 @findex -numa
 Define a NUMA node and assign RAM and VCPUs to it.
 
+Legacy VCPU assignment uses @samp{cpus} option where
 @var{firstcpu} and @var{lastcpu} are CPU indexes. Each
 @samp{cpus} option represent a contiguous range of CPU indexes
 (or a single VCPU if @var{lastcpu} is omitted). A non-contiguous
@@ -159,6 +162,24 @@ a NUMA node:
 -numa node,cpus=0-2,cpus=5
 @end example
 
+@samp{cpu} option is a new alternative to @samp{cpus} option
+which uses @samp{socket-id|core-id|thread-id} properties to assign
+CPU objects to a @var{node} using topology layout properties of CPU.
+The set of properties is machine specific, and depends on used
+machine type/@samp{smp} options. It could be queried with
+@samp{hotpluggable-cpus} monitor command.
+@samp{node-id} property specifies @var{node} to which CPU object
+will be assigned, it's required for @var{node} to be declared
+with @samp{node} option before it's used with @samp{cpu} option.
+
+For example:
+@example
+-M pc \
+-smp 1,sockets=2,maxcpus=2 \
+-numa node,nodeid=0 -numa node,nodeid=1 \
+-numa cpu,node-id=0,socket-id=0 -numa cpu,node-id=1,socket-id=1
+@end example
+
 @samp{mem} assigns a given RAM amount to a node. @samp{memdev}
 assigns RAM from a given memory backend device to a node. If
 @samp{mem} and @samp{memdev} are omitted in all nodes, RAM is
-- 
2.7.4

[Qemu-devel] [PATCH v2 24/24] tests: check -numa node, cpu=props_list usecase

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 tests/numa-test.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 151 insertions(+)

diff --git a/tests/numa-test.c b/tests/numa-test.c
index 2722687..c3475d6 100644
--- a/tests/numa-test.c
+++ b/tests/numa-test.c
@@ -131,6 +131,144 @@ static void test_query_cpus(const void *data)
     g_free(cli);
 }
 
+static void pc_numa_cpu(const void *data)
+{
+    char *cli;
+    QDict *resp;
+    QList *cpus;
+    const QObject *e;
+
+    cli = make_cli(data, "-cpu pentium -smp 8,sockets=2,cores=2,threads=2 "
+        "-numa node,nodeid=0 -numa node,nodeid=1 "
+        "-numa cpu,node-id=1,socket-id=0 "
+        "-numa cpu,node-id=0,socket-id=1,core-id=0 "
+        "-numa cpu,node-id=0,socket-id=1,core-id=1,thread-id=0 "
+        "-numa cpu,node-id=1,socket-id=1,core-id=1,thread-id=1");
+    qtest_start(cli);
+    cpus = get_cpus(&resp);
+    g_assert(cpus);
+
+    while ((e = qlist_pop(cpus))) {
+        QDict *cpu, *props;
+        int64_t socket, core, thread, node;
+
+        cpu = qobject_to_qdict(e);
+        g_assert(qdict_haskey(cpu, "props"));
+        props = qdict_get_qdict(cpu, "props");
+
+        g_assert(qdict_haskey(props, "node-id"));
+        node = qdict_get_int(props, "node-id");
+        g_assert(qdict_haskey(props, "socket-id"));
+        socket = qdict_get_int(props, "socket-id");
+        g_assert(qdict_haskey(props, "core-id"));
+        core = qdict_get_int(props, "core-id");
+        g_assert(qdict_haskey(props, "thread-id"));
+        thread = qdict_get_int(props, "thread-id");
+
+        if (socket == 0) {
+            g_assert_cmpint(node, ==, 1);
+        } else if (socket == 1 && core == 0) {
+            g_assert_cmpint(node, ==, 0);
+        } else if (socket == 1 && core == 1 && thread == 0) {
+            g_assert_cmpint(node, ==, 0);
+        } else if (socket == 1 && core == 1 && thread == 1) {
+            g_assert_cmpint(node, ==, 1);
+        } else {
+            g_assert(false);
+        }
+    }
+
+    QDECREF(resp);
+    qtest_end();
+    g_free(cli);
+}
+
+static void spapr_numa_cpu(const void *data)
+{
+    char *cli;
+    QDict *resp;
+    QList *cpus;
+    const QObject *e;
+
+    cli = make_cli(data, "-smp 4,cores=4 "
+        "-numa node,nodeid=0 -numa node,nodeid=1 "
+        "-numa cpu,node-id=0,core-id=0 "
+        "-numa cpu,node-id=0,core-id=1 "
+        "-numa cpu,node-id=0,core-id=2 "
+        "-numa cpu,node-id=1,core-id=3");
+    qtest_start(cli);
+    cpus = get_cpus(&resp);
+    g_assert(cpus);
+
+    while ((e = qlist_pop(cpus))) {
+        QDict *cpu, *props;
+        int64_t core, node;
+
+        cpu = qobject_to_qdict(e);
+        g_assert(qdict_haskey(cpu, "props"));
+        props = qdict_get_qdict(cpu, "props");
+
+        g_assert(qdict_haskey(props, "node-id"));
+        node = qdict_get_int(props, "node-id");
+        g_assert(qdict_haskey(props, "core-id"));
+        core = qdict_get_int(props, "core-id");
+
+        if (core >= 0 && core < 3) {
+            g_assert_cmpint(node, ==, 0);
+        } else if (core == 3) {
+            g_assert_cmpint(node, ==, 1);
+        } else {
+            g_assert(false);
+        }
+    }
+
+    QDECREF(resp);
+    qtest_end();
+    g_free(cli);
+}
+
+static void aarch64_numa_cpu(const void *data)
+{
+    char *cli;
+    QDict *resp;
+    QList *cpus;
+    const QObject *e;
+
+    cli = make_cli(data, "-smp 2 "
+        "-numa node,nodeid=0 -numa node,nodeid=1 "
+        "-numa cpu,node-id=1,thread-id=0 "
+        "-numa cpu,node-id=0,thread-id=1");
+    qtest_start(cli);
+    cpus = get_cpus(&resp);
+    g_assert(cpus);
+
+    while ((e = qlist_pop(cpus))) {
+        QDict *cpu, *props;
+        int64_t thread, node;
+
+        cpu = qobject_to_qdict(e);
+        g_assert(qdict_haskey(cpu, "props"));
+        props = qdict_get_qdict(cpu, "props");
+
+        g_assert(qdict_haskey(props, "node-id"));
+        node = qdict_get_int(props, "node-id");
+        g_assert(qdict_haskey(props, "thread-id"));
+        thread = qdict_get_int(props, "thread-id");
+
+        if (thread == 0) {
+            g_assert_cmpint(node, ==, 1);
+        } else if (thread == 1) {
+            g_assert_cmpint(node, ==, 0);
+        } else {
+            g_assert(false);
+        }
+    }
+
+    QDECREF(resp);
+    qtest_end();
+    g_free(cli);
+}
+
 int main(int argc, char **argv)
 {
     const char *args = NULL;
@@ -147,5 +285,18 @@ int main(int argc, char **argv)
     qtest_add_data_func("/numa/mon/cpus/partial", args, test_mon_partial);
     qtest_add_data_func("/numa/qmp/cpus/query-cpus", args, test_query_cpus);
 
+    if (!strcmp(arch, "i386") || !strcmp(arch, "x86_64")) {
+        qtest_add_data_func("/numa/pc/cpu/explicit", args, pc_numa_cpu);
+    }
+
+    if (!strcmp(arch, "ppc64")) {
+        qtest_add_data_func("/numa/spapr/cpu/explicit", args, spapr_numa_cpu);
+    }
+
+    if (!strcmp(arch, "aarch64")) {
+        qtest_add_data_func("/numa/aarch64/cpu/explicit", args,
+                            aarch64_numa_cpu);
+    }
+
     return g_test_run();
 }
-- 
2.7.4

Re: [Qemu-devel] [PATCH v2 01/24] tests: add CPUs to numa node mapping test

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:55PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
> Originally CPU threads were by default assigned in
> round-robin fashion. However it was causing issues in
> guest since CPU threads from the same socket/core could
> be placed on different NUMA nodes.
> Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
> fixed it by grouping threads within a socket on the same node
> introducing cpu_index_to_socket_id() callback and commit
> 20bb648d (spapr: Fix default NUMA node allocation for threads)
> reused callback to fix similar issues for SPAPR machine
> even though socket doesn't make much sense there.
> 
> As result QEMU ended up having 3 default distribution rules
> used by 3 targets /virt-arm, spapr, pc/.
> 
> In effort of moving NUMA mapping for CPUs into possible_cpus,
> generalize default mapping in numa.c by making boards decide
> on default mapping and let them explicitly tell generic
> numa code to which node a CPU thread belongs to by replacing
> cpu_index_to_socket_id() with @cpu_index_to_instance_props()
> which provides default node_id assigned by board to specified
> cpu_index.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Just two extra comments below:

[...]
> +static CpuInstanceProperties
> +virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> +{
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> +
> +    assert(cpu_index < possible_cpus->len);
> +    return possible_cpus->cpus[cpu_index].props;;
> +}
> +
[...]
> +static CpuInstanceProperties
> +pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
>  {
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> +
> +    assert(cpu_index < possible_cpus->len);
> +    return possible_cpus->cpus[cpu_index].props;;
>  }

The fact that these two implementations look exactly the same
made me wonder:

1) Why this isn't the default implementation;
2) Why exactly spapr needs a different implementation.

Then I noticed that there's nothing in the common machine code
that specifies that possible_cpus->cpus[] is indexed by
cpu_index. This means it is indeed safer to require each machine
to provide its own cpu_index_to_props implementation than having
a default implementation that can unexpectedly break (e.g. if
granularity at possible_cpus is not at VCPU/thread level).

I would still like to have an abstraction that wouldn't require
writing machine-specific code (e.g. cpu_index ranges to
possible_cpus like David suggested), but that's for a follow-up
series.

[...]
> +static CpuInstanceProperties
> +spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
>  {
> +    CPUArchId *core_slot;
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +
> +    /* make sure possible_cpu are intialized */
> +    mc->possible_cpu_arch_ids(machine);
> +    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
> +    assert(core_slot);
> +    return core_slot->props;
>  }

If you need to submit v3, maybe a comment here explaining why
spapr needs a different cpu_index_to_props implementation would
be helpful. I took a while to figure it out.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 06/24] spapr: add node-id property to sPAPR core

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:00PM +0200, Igor Mammedov wrote:
> it will allow switching from cpu_index to core based numa
> mapping in follow up patches.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

Have you considered adding this to TYPE_CPU instead of
duplicating the same code on multiple architectures/boards?

> ---
>  include/hw/ppc/spapr_cpu_core.h |  1 +
>  include/qom/cpu.h               |  2 ++
>  hw/ppc/spapr.c                  | 17 +++++++++++++++++
>  hw/ppc/spapr_cpu_core.c         | 11 ++++++++---
>  4 files changed, 28 insertions(+), 3 deletions(-)
> 
> diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
> index 3c35665..93051e9 100644
> --- a/include/hw/ppc/spapr_cpu_core.h
> +++ b/include/hw/ppc/spapr_cpu_core.h
> @@ -27,6 +27,7 @@ typedef struct sPAPRCPUCore {
>  
>      /*< public >*/
>      void *threads;
> +    int node_id;
>  } sPAPRCPUCore;
>  
>  typedef struct sPAPRCPUCoreClass {
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 5d10359..55214ce 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -258,6 +258,8 @@ typedef void (*run_on_cpu_func)(CPUState *cpu, run_on_cpu_data data);
>  
>  struct qemu_work_item;
>  
> +#define CPU_UNSET_NUMA_NODE_ID -1
> +
>  /**
>   * CPUState:
>   * @cpu_index: CPU index (informative).
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 33405a0..7f58ee4 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -2824,9 +2824,11 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
>      MachineClass *mc = MACHINE_GET_CLASS(hotplug_dev);
>      Error *local_err = NULL;
>      CPUCore *cc = CPU_CORE(dev);
> +    sPAPRCPUCore *sc = SPAPR_CPU_CORE(dev);
>      char *base_core_type = spapr_get_cpu_core_type(machine->cpu_model);
>      const char *type = object_get_typename(OBJECT(dev));
>      CPUArchId *core_slot;
> +    int node_id;
>      int index;
>  
>      if (dev->hotplugged && !mc->has_hotpluggable_cpus) {
> @@ -2861,6 +2863,21 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
>          goto out;
>      }
>  
> +    node_id = numa_get_node_for_cpu(cc->core_id);
> +    if (node_id == nb_numa_nodes) {
> +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> +         * keep it this way for now but in future we probably should
> +         * refuse to start up with incomplete numa mapping */
> +        node_id = 0;
> +    }
> +    if (sc->node_id == CPU_UNSET_NUMA_NODE_ID) {
> +        sc->node_id = node_id;
> +    } else if (sc->node_id != node_id) {
> +        error_setg(&local_err, "node-id %d must match numa node specified"
> +            "with -numa option for cpu-index %d", sc->node_id, cc->core_id);
> +        goto out;
> +    }
> +
>  out:
>      g_free(base_core_type);
>      error_propagate(errp, local_err);
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index 4389ef4..9de7a56 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -176,7 +176,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
>      const char *typename = object_class_get_name(scc->cpu_class);
>      size_t size = object_type_get_instance_size(typename);
>      Error *local_err = NULL;
> -    int core_node_id = numa_get_node_for_cpu(cc->core_id);;
>      void *obj;
>      int i, j;
>  
> @@ -194,10 +193,10 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
>  
>          /* Set NUMA node for the added CPUs  */
>          node_id = numa_get_node_for_cpu(cs->cpu_index);
> -        if (node_id != core_node_id) {
> +        if (node_id != sc->node_id) {
>              error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
>                  " on CPU[core-id: %d, node-id: %d], node-id must be the same",
> -                 node_id, cs->cpu_index, cc->core_id, core_node_id);
> +                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
>              goto err;
>          }
>          if (node_id < nb_numa_nodes) {
> @@ -263,6 +262,11 @@ static const char *spapr_core_models[] = {
>      "POWER9_v1.0",
>  };
>  
> +static Property spapr_cpu_core_properties[] = {
> +    DEFINE_PROP_INT32("node-id", sPAPRCPUCore, node_id, CPU_UNSET_NUMA_NODE_ID),
> +    DEFINE_PROP_END_OF_LIST()
> +};
> +
>  void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
>  {
>      DeviceClass *dc = DEVICE_CLASS(oc);
> @@ -270,6 +274,7 @@ void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
>  
>      dc->realize = spapr_cpu_core_realize;
>      dc->unrealize = spapr_cpu_core_unrealizefn;
> +    dc->props = spapr_cpu_core_properties;
>      scc->cpu_class = cpu_class_by_name(TYPE_POWERPC_CPU, data);
>      g_assert(scc->cpu_class);
>  }
> -- 
> 2.7.4
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
[...]
> @@ -378,14 +379,16 @@ void parse_numa_opts(MachineClass *mc)
>           * rule grouping VCPUs by socket so that VCPUs from the same socket
>           * would be on the same node.
>           */
> +        if (!mc->cpu_index_to_instance_props) {
> +            error_report("default CPUs to NUMA node mapping isn't supported");
> +            exit(1);
> +        }

This will make people trying to use -numa on unsupported machines
see a misleading error message: instead of telling them that the
machine doesn't support NUMA at all, the message seems to imply
that NUMA may be supported and we just don't have default NUMA node
mapping support.

Probably a more generic "NUMA is not supported by this
machine-type" message before even trying to parse -numa would be
clearer. (I don't know if another patch in this series already
does that.)

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:
> Default node mapping initialization already checks that board
> supports cpu_index to node mapping and refuses to start if
> it's not supported. Do the same for explicitly provided
> mapping "-numa node,cpus=..."
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  numa.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/numa.c b/numa.c
> index ab1661d..b517870 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
>      return -1;
>  }
>  
> -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> +                            QemuOpts *opts, Error **errp)
>  {
>      uint16_t nodenr;
>      uint16List *cpus = NULL;
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
>      if (node->has_nodeid) {
>          nodenr = node->nodeid;
> @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
>          return;
>      }
>  
> +    if (!mc->cpu_index_to_instance_props) {
> +        error_report("CPUs to NUMA node mapping isn't supported");
> +        exit(1);
> +    }

The error message sounds very confusing me. Maybe this could
become just: "NUMA is not supported by the %s machine-type"?

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
[...]
> diff --git a/numa.c b/numa.c
> index 6fc2393..ab1661d 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -294,9 +294,10 @@ static void validate_numa_cpus(void)
>      g_free(seen_cpus);
>  }
>  
> -void parse_numa_opts(MachineClass *mc)
> +void parse_numa_opts(MachineState *ms)
>  {
>      int i;
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
>      for (i = 0; i < MAX_NODES; i++) {
>          numa_info[i].node_cpu = bitmap_new(max_cpus);
> @@ -378,14 +379,16 @@ void parse_numa_opts(MachineClass *mc)


Expanding diff context:

>          /* Historically VCPUs were assigned in round-robin order to NUMA
>           * nodes. However it causes issues with guest not handling it nice
>           * in case where cores/threads from a multicore CPU appear on
>           * different nodes. So allow boards to override default distribution
>           * rule grouping VCPUs by socket so that VCPUs from the same socket
>           * would be on the same node.
>           */

The above comment looks obsolete, as we are removing the code inside
parse_numa_opts() that deals with grouping VCPUs by socket.

(Can be fixed by a follow-up patch, if necessary.)

> +        if (!mc->cpu_index_to_instance_props) {
> +            error_report("default CPUs to NUMA node mapping isn't supported");
> +            exit(1);
> +        }
>          if (i == nb_numa_nodes) {
>              for (i = 0; i < max_cpus; i++) {
> -                unsigned node_id = i % nb_numa_nodes;
> -                if (mc->cpu_index_to_socket_id) {
> -                    node_id = mc->cpu_index_to_socket_id(i) % nb_numa_nodes;
> -                }
> +                CpuInstanceProperties props;
> +                props = mc->cpu_index_to_instance_props(ms, i);
>  
> -                set_bit(i, numa_info[node_id].node_cpu);
> +                set_bit(i, numa_info[props.node_id].node_cpu);
>              }
>          }
>  
[...]

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> Introduce machine_set_cpu_numa_node() helper that stores
> node mapping for CPU in MachineState::possible_cpus.
> CPU and node it belongs to is specified by 'props' argument.
> 
> Patch doesn't remove old way of storing mapping in
> numa_info[X].node_cpu as removing it at the same time
> makes patch rather big. Instead it just mirrors mapping
> in possible_cpus and follow up per target patches will
> switch to possible_cpus and numa_info[X].node_cpu will
> be removed once there isn't any users left.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  include/hw/boards.h |  2 ++
>  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  numa.c              |  8 +++++++
>  3 files changed, 78 insertions(+)
> 
> diff --git a/include/hw/boards.h b/include/hw/boards.h
> index 5d6af21..1f518a1 100644
> --- a/include/hw/boards.h
> +++ b/include/hw/boards.h
> @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
>  bool machine_mem_merge(MachineState *machine);
>  void machine_register_compat_props(MachineState *machine);
>  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> +void machine_set_cpu_numa_node(MachineState *machine,
> +                               CpuInstanceProperties *props, Error **errp);
>  
>  /**
>   * CPUArchId:
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index ada9eea..a63f17b 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
>      return head;
>  }
>  
> +void machine_set_cpu_numa_node(MachineState *machine,
> +                               CpuInstanceProperties *props, Error **errp)

As the semantics of this function aren't trivial, it would be
nice to have a comment explaining what exactly this function do.

e.g.:
* make it clear that it could affect multiple CPU slots;
* make it clear what does it mean to have props->has_node_id=false as
  argument (is it really valid?);
* make it clear that it will refuse to change an existing mapping.


> +{
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +    bool match = false;
> +    int i;
> +
> +    if (!mc->possible_cpu_arch_ids) {
> +        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
> +        return;
> +    }
> +
> +    /* force board to initialize possible_cpus if it hasn't been done yet */
> +    mc->possible_cpu_arch_ids(machine);
> +
> +    for (i = 0; i < machine->possible_cpus->len; i++) {
> +        CPUArchId *slot = &machine->possible_cpus->cpus[i];
> +
> +        /* reject unsupported by board properties */
> +        if (props->has_thread_id && !slot->props.has_thread_id) {
> +            error_setg(errp, "thread-id is not supported");
> +            return;
> +        }
> +
> +        if (props->has_core_id && !slot->props.has_core_id) {
> +            error_setg(errp, "core-id is not supported");
> +            return;
> +        }
> +
> +        if (props->has_socket_id && !slot->props.has_socket_id) {
> +            error_setg(errp, "socket-id is not supported");
> +            return;
> +        }
> +
> +        /* skip slots with explicit mismatch */
> +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> +                continue;
> +        }
> +
> +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> +                continue;
> +        }
> +
> +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> +                continue;
> +        }
> +
> +        /* reject assignment if slot is already assigned, for compatibility
> +         * of legacy cpu_index mapping with SPAPR core based mapping do not
> +         * error out if cpu thread and matched core have the same node-id */
> +        if (slot->props.has_node_id &&
> +            slot->props.node_id != props->node_id) {
> +            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
> +                       slot->props.node_id);
> +            return;
> +        }
> +
> +        /* assign slot to node as it's matched '-numa cpu' key */
> +        match = true;
> +        slot->props.node_id = props->node_id;
> +        slot->props.has_node_id = props->has_node_id;

Is it really valid to have has_node_id=false? Maybe an
assert(props->has_node_id) at the beginning of the function would
be useful.

> +    }
> +
> +    if (!match) {
> +        error_setg(errp, "no match found");
> +    }
> +}
> +
>  static void machine_class_init(ObjectClass *oc, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(oc);
> diff --git a/numa.c b/numa.c
> index b517870..5ff1212 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
>          exit(1);
>      }
>      for (cpus = node->cpus; cpus; cpus = cpus->next) {
> +        CpuInstanceProperties props;
>          if (cpus->value >= max_cpus) {
>              error_setg(errp,
>                         "CPU index (%" PRIu16 ")"
> @@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
>              return;
>          }
>          bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
> +        props = mc->cpu_index_to_instance_props(ms, cpus->value);
> +        props.node_id = nodenr;
> +        props.has_node_id = true;
> +        machine_set_cpu_numa_node(ms, &props, &error_fatal);
>      }
>  
>      if (node->has_mem && node->has_memdev) {
> @@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
>          if (i == nb_numa_nodes) {
>              for (i = 0; i < max_cpus; i++) {
>                  CpuInstanceProperties props;
> +                /* fetch default mapping from board and enable it */
>                  props = mc->cpu_index_to_instance_props(ms, i);
> +                props.has_node_id = true;
>  
>                  set_bit(i, numa_info[props.node_id].node_cpu);
> +                machine_set_cpu_numa_node(ms, &props, &error_fatal);
>              }
>          }
>  
> -- 
> 2.7.4
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

On Wed, 3 May 2017 11:42:40 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
> > Originally CPU threads were by default assigned in
> > round-robin fashion. However it was causing issues in
> > guest since CPU threads from the same socket/core could
> > be placed on different NUMA nodes.
> > Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
> > fixed it by grouping threads within a socket on the same node
> > introducing cpu_index_to_socket_id() callback and commit
> > 20bb648d (spapr: Fix default NUMA node allocation for threads)
> > reused callback to fix similar issues for SPAPR machine
> > even though socket doesn't make much sense there.
> > 
> > As result QEMU ended up having 3 default distribution rules
> > used by 3 targets /virt-arm, spapr, pc/.
> > 
> > In effort of moving NUMA mapping for CPUs into possible_cpus,
> > generalize default mapping in numa.c by making boards decide
> > on default mapping and let them explicitly tell generic
> > numa code to which node a CPU thread belongs to by replacing
> > cpu_index_to_socket_id() with @cpu_index_to_instance_props()
> > which provides default node_id assigned by board to specified
> > cpu_index.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> Just two extra comments below:
> 
> [...]
> > +static CpuInstanceProperties
> > +virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > +
> > +    assert(cpu_index < possible_cpus->len);
> > +    return possible_cpus->cpus[cpu_index].props;;
> > +}
> > +  
> [...]
> > +static CpuInstanceProperties
> > +pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> >  {
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > +
> > +    assert(cpu_index < possible_cpus->len);
> > +    return possible_cpus->cpus[cpu_index].props;;
> >  }  
> 
> The fact that these two implementations look exactly the same
> made me wonder:
> 
> 1) Why this isn't the default implementation;
> 2) Why exactly spapr needs a different implementation.
> 
> Then I noticed that there's nothing in the common machine code
> that specifies that possible_cpus->cpus[] is indexed by
> cpu_index. This means it is indeed safer to require each machine
> to provide its own cpu_index_to_props implementation than having
> a default implementation that can unexpectedly break (e.g. if
> granularity at possible_cpus is not at VCPU/thread level).
> 
> I would still like to have an abstraction that wouldn't require
> writing machine-specific code (e.g. cpu_index ranges to
> possible_cpus like David suggested), but that's for a follow-up
> series.
I've left generalizing this part until we have defined/stable
topology for aarch64.
Also I hope, that instead of generalizing we might
depricate/drop -numa node,cpus=0,1,... in future and just drop
this hooks altogether as they are there only for compatibility
with old cpu_index based mapping


> [...]
> > +static CpuInstanceProperties
> > +spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
> >  {
> > +    CPUArchId *core_slot;
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +
> > +    /* make sure possible_cpu are intialized */
> > +    mc->possible_cpu_arch_ids(machine);
> > +    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
> > +    assert(core_slot);
> > +    return core_slot->props;
> >  }  
> 
> If you need to submit v3, maybe a comment here explaining why
> spapr needs a different cpu_index_to_props implementation would
> be helpful. I took a while to figure it out.
I'll add comment if series is respinned.

Re: [Qemu-devel] [PATCH v2 06/24] spapr: add node-id property to sPAPR core

[Igor Mammedov]

On Wed, 3 May 2017 11:46:44 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:00PM +0200, Igor Mammedov wrote:
> > it will allow switching from cpu_index to core based numa
> > mapping in follow up patches.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>  
> 
> Have you considered adding this to TYPE_CPU instead of
> duplicating the same code on multiple architectures/boards?
it was in TYPE_CPU at RFC time, but it adds public node-id
property to every CPU.
So I've rewrote it they way it would affect only necessary CPUs,

it might be possible to generalize node_id sanity checks at
pre_plug time into a common wrapper, I can do it on top so
it would be visible what it's generalized (or an extra patch on respin)

> 
> > ---
> >  include/hw/ppc/spapr_cpu_core.h |  1 +
> >  include/qom/cpu.h               |  2 ++
> >  hw/ppc/spapr.c                  | 17 +++++++++++++++++
> >  hw/ppc/spapr_cpu_core.c         | 11 ++++++++---
> >  4 files changed, 28 insertions(+), 3 deletions(-)
> > 
> > diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
> > index 3c35665..93051e9 100644
> > --- a/include/hw/ppc/spapr_cpu_core.h
> > +++ b/include/hw/ppc/spapr_cpu_core.h
> > @@ -27,6 +27,7 @@ typedef struct sPAPRCPUCore {
> >  
> >      /*< public >*/
> >      void *threads;
> > +    int node_id;
> >  } sPAPRCPUCore;
> >  
> >  typedef struct sPAPRCPUCoreClass {
> > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > index 5d10359..55214ce 100644
> > --- a/include/qom/cpu.h
> > +++ b/include/qom/cpu.h
> > @@ -258,6 +258,8 @@ typedef void (*run_on_cpu_func)(CPUState *cpu, run_on_cpu_data data);
> >  
> >  struct qemu_work_item;
> >  
> > +#define CPU_UNSET_NUMA_NODE_ID -1
> > +
> >  /**
> >   * CPUState:
> >   * @cpu_index: CPU index (informative).
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index 33405a0..7f58ee4 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -2824,9 +2824,11 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> >      MachineClass *mc = MACHINE_GET_CLASS(hotplug_dev);
> >      Error *local_err = NULL;
> >      CPUCore *cc = CPU_CORE(dev);
> > +    sPAPRCPUCore *sc = SPAPR_CPU_CORE(dev);
> >      char *base_core_type = spapr_get_cpu_core_type(machine->cpu_model);
> >      const char *type = object_get_typename(OBJECT(dev));
> >      CPUArchId *core_slot;
> > +    int node_id;
> >      int index;
> >  
> >      if (dev->hotplugged && !mc->has_hotpluggable_cpus) {
> > @@ -2861,6 +2863,21 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> >          goto out;
> >      }
> >  
> > +    node_id = numa_get_node_for_cpu(cc->core_id);
> > +    if (node_id == nb_numa_nodes) {
> > +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> > +         * keep it this way for now but in future we probably should
> > +         * refuse to start up with incomplete numa mapping */
> > +        node_id = 0;
> > +    }
> > +    if (sc->node_id == CPU_UNSET_NUMA_NODE_ID) {
> > +        sc->node_id = node_id;
> > +    } else if (sc->node_id != node_id) {
> > +        error_setg(&local_err, "node-id %d must match numa node specified"
> > +            "with -numa option for cpu-index %d", sc->node_id, cc->core_id);
> > +        goto out;
> > +    }
> > +
> >  out:
> >      g_free(base_core_type);
> >      error_propagate(errp, local_err);
> > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > index 4389ef4..9de7a56 100644
> > --- a/hw/ppc/spapr_cpu_core.c
> > +++ b/hw/ppc/spapr_cpu_core.c
> > @@ -176,7 +176,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> >      const char *typename = object_class_get_name(scc->cpu_class);
> >      size_t size = object_type_get_instance_size(typename);
> >      Error *local_err = NULL;
> > -    int core_node_id = numa_get_node_for_cpu(cc->core_id);;
> >      void *obj;
> >      int i, j;
> >  
> > @@ -194,10 +193,10 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> >  
> >          /* Set NUMA node for the added CPUs  */
> >          node_id = numa_get_node_for_cpu(cs->cpu_index);
> > -        if (node_id != core_node_id) {
> > +        if (node_id != sc->node_id) {
> >              error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
> >                  " on CPU[core-id: %d, node-id: %d], node-id must be the same",
> > -                 node_id, cs->cpu_index, cc->core_id, core_node_id);
> > +                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
> >              goto err;
> >          }
> >          if (node_id < nb_numa_nodes) {
> > @@ -263,6 +262,11 @@ static const char *spapr_core_models[] = {
> >      "POWER9_v1.0",
> >  };
> >  
> > +static Property spapr_cpu_core_properties[] = {
> > +    DEFINE_PROP_INT32("node-id", sPAPRCPUCore, node_id, CPU_UNSET_NUMA_NODE_ID),
> > +    DEFINE_PROP_END_OF_LIST()
> > +};
> > +
> >  void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> >  {
> >      DeviceClass *dc = DEVICE_CLASS(oc);
> > @@ -270,6 +274,7 @@ void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> >  
> >      dc->realize = spapr_cpu_core_realize;
> >      dc->unrealize = spapr_cpu_core_unrealizefn;
> > +    dc->props = spapr_cpu_core_properties;
> >      scc->cpu_class = cpu_class_by_name(TYPE_POWERPC_CPU, data);
> >      g_assert(scc->cpu_class);
> >  }
> > -- 
> > 2.7.4
> >   
> 

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

On Wed, 3 May 2017 11:59:35 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
> [...]
> > @@ -378,14 +379,16 @@ void parse_numa_opts(MachineClass *mc)
> >           * rule grouping VCPUs by socket so that VCPUs from the same socket
> >           * would be on the same node.
> >           */
> > +        if (!mc->cpu_index_to_instance_props) {
> > +            error_report("default CPUs to NUMA node mapping isn't supported");
> > +            exit(1);
> > +        }  
> 
> This will make people trying to use -numa on unsupported machines
> see a misleading error message: instead of telling them that the
> machine doesn't support NUMA at all, the message seems to imply
> that NUMA may be supported and we just don't have default NUMA node
> mapping support.
> 
> Probably a more generic "NUMA is not supported by this
> machine-type" message before even trying to parse -numa would be
> clearer. (I don't know if another patch in this series already
> does that.)
no, other places should error out with other error messages if it's
not supported.

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Igor Mammedov]

On Wed, 3 May 2017 12:04:29 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:
> > Default node mapping initialization already checks that board
> > supports cpu_index to node mapping and refuses to start if
> > it's not supported. Do the same for explicitly provided
> > mapping "-numa node,cpus=..."
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  numa.c | 13 ++++++++++---
> >  1 file changed, 10 insertions(+), 3 deletions(-)
> > 
> > diff --git a/numa.c b/numa.c
> > index ab1661d..b517870 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
> >      return -1;
> >  }
> >  
> > -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> > +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> > +                            QemuOpts *opts, Error **errp)
> >  {
> >      uint16_t nodenr;
> >      uint16List *cpus = NULL;
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> >  
> >      if (node->has_nodeid) {
> >          nodenr = node->nodeid;
> > @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> >          return;
> >      }
> >  
> > +    if (!mc->cpu_index_to_instance_props) {
> > +        error_report("CPUs to NUMA node mapping isn't supported");
> > +        exit(1);
> > +    }  
> 
> The error message sounds very confusing me. Maybe this could
> become just: "NUMA is not supported by the %s machine-type"?
If I replace this specific error message with a generic one and do
the same in 5/24, it will be harder for us/user to figure out
where exactly error happened.

Would be following clearer:
"option -numa node,cpus= is not supported by the %s machine-type"

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:17PM +0200, Igor Mammedov wrote:
[...]
> diff --git a/numa.c b/numa.c
> index 40e9f44..61521f5 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>      NumaOptions *object = NULL;
>      MachineState *ms = opaque;
>      Error *err = NULL;
> +    CpuInstanceProperties cpu;
>  
>      {
>          Visitor *v = opts_visitor_new(opts);
> @@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>          }
>          nb_numa_nodes++;
>          break;
> +    case NUMA_OPTIONS_TYPE_CPU:
> +        if (!object->u.cpu.has_node_id) {
> +            error_setg(&err, "Missing mandatory node-id property");
> +            goto end;
> +        }
> +        if (!numa_info[object->u.cpu.node_id].present) {
> +            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
> +                "defined with -numa node,nodeid=ID before it's used with "
> +                "-numa cpu,node-id=ID", object->u.cpu.node_id);
> +            goto end;
> +        }
> +
> +        memset(&cpu, 0, sizeof(cpu));
> +        cpu.has_node_id = object->u.cpu.has_node_id;
> +        cpu.node_id = object->u.cpu.node_id;
> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> +        cpu.socket_id = object->u.cpu.socket_id;
> +        cpu.has_core_id = object->u.cpu.has_core_id;
> +        cpu.core_id = object->u.cpu.core_id;
> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> +        cpu.thread_id = object->u.cpu.thread_id;

We don't have a way to avoid copying each field individually?
Some visitor trick, maybe?

Eric, Markus, Michael, do you have any suggestions?

> +
> +        machine_set_cpu_numa_node(ms, &cpu, &err);
> +        break;
>      default:
>          abort();
>      }
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 76d137d..5baf3a4 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -5680,10 +5680,12 @@
>  ##
>  # @NumaOptionsType:
>  #
> +# @cpu: property based CPU(s) to node mapping (Since: 2.10)
> +#
>  # Since: 2.1
>  ##
>  { 'enum': 'NumaOptionsType',
> -  'data': [ 'node' ] }
> +  'data': [ 'node', 'cpu' ] }
>  
>  ##
>  # @NumaOptions:
> @@ -5696,7 +5698,8 @@
>    'base': { 'type': 'NumaOptionsType' },
>    'discriminator': 'type',
>    'data': {
> -    'node': 'NumaNodeOptions' }}
> +    'node': 'NumaNodeOptions',
> +    'cpu': 'NumaCpuOptions' }}
>  
>  ##
>  # @NumaNodeOptions:
> @@ -5725,6 +5728,20 @@
>     '*memdev': 'str' }}
>  
>  ##
> +# @NumaCpuOptions:
> +#
> +# Option "-numa cpu" overrides default cpu to node mapping.
> +# It accepts the same set of cpu properties as returned by
> +# query-hotpluggable-cpus[].props, where node-id could be used to
> +# override default node mapping.
> +#
> +# Since: 2.10
> +##
> +{ 'struct': 'NumaCpuOptions',
> +   'base': 'CpuInstanceProperties',
> +   'data' : {} }
> +
> +##
>  # @HostMemPolicy:
>  #
>  # Host memory policy types
[...]

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On 05/03/2017 11:35 AM, Eduardo Habkost wrote:

>> +
>> +        memset(&cpu, 0, sizeof(cpu));
>> +        cpu.has_node_id = object->u.cpu.has_node_id;
>> +        cpu.node_id = object->u.cpu.node_id;
>> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
>> +        cpu.socket_id = object->u.cpu.socket_id;
>> +        cpu.has_core_id = object->u.cpu.has_core_id;
>> +        cpu.core_id = object->u.cpu.core_id;
>> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
>> +        cpu.thread_id = object->u.cpu.thread_id;
> 
> We don't have a way to avoid copying each field individually?
> Some visitor trick, maybe?
> 
> Eric, Markus, Michael, do you have any suggestions?

Markus just added QAPI_CLONE_MEMBERS(), which sounds like what you want:

https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg04867.html

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Eduardo Habkost]

On Wed, May 03, 2017 at 06:19:50PM +0200, Igor Mammedov wrote:
> On Wed, 3 May 2017 12:04:29 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:
> > > Default node mapping initialization already checks that board
> > > supports cpu_index to node mapping and refuses to start if
> > > it's not supported. Do the same for explicitly provided
> > > mapping "-numa node,cpus=..."
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > >  numa.c | 13 ++++++++++---
> > >  1 file changed, 10 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/numa.c b/numa.c
> > > index ab1661d..b517870 100644
> > > --- a/numa.c
> > > +++ b/numa.c
> > > @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
> > >      return -1;
> > >  }
> > >  
> > > -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> > > +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> > > +                            QemuOpts *opts, Error **errp)
> > >  {
> > >      uint16_t nodenr;
> > >      uint16List *cpus = NULL;
> > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > >  
> > >      if (node->has_nodeid) {
> > >          nodenr = node->nodeid;
> > > @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> > >          return;
> > >      }
> > >  
> > > +    if (!mc->cpu_index_to_instance_props) {
> > > +        error_report("CPUs to NUMA node mapping isn't supported");
> > > +        exit(1);
> > > +    }  
> > 
> > The error message sounds very confusing me. Maybe this could
> > become just: "NUMA is not supported by the %s machine-type"?
> If I replace this specific error message with a generic one and do
> the same in 5/24, it will be harder for us/user to figure out
> where exactly error happened.

I don't think we need to change the message in 5/24 if we change
this one.

> 
> Would be following clearer:
> "option -numa node,cpus= is not supported by the %s machine-type"

It would work if this was really the case. But the error message
is unconditional here, appearing even if cpus= is not set:

  $ qemu-system-x86_64 -m 1G -smp 1 -numa node,mem=1G -machine none
  qemu-system-x86_64: -numa node,mem=1G: CPUs to NUMA node mapping isn't supported

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Eduardo Habkost]

On Wed, May 03, 2017 at 11:39:10AM -0500, Eric Blake wrote:
> On 05/03/2017 11:35 AM, Eduardo Habkost wrote:
> 
> >> +
> >> +        memset(&cpu, 0, sizeof(cpu));
> >> +        cpu.has_node_id = object->u.cpu.has_node_id;
> >> +        cpu.node_id = object->u.cpu.node_id;
> >> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> >> +        cpu.socket_id = object->u.cpu.socket_id;
> >> +        cpu.has_core_id = object->u.cpu.has_core_id;
> >> +        cpu.core_id = object->u.cpu.core_id;
> >> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> >> +        cpu.thread_id = object->u.cpu.thread_id;
> > 
> > We don't have a way to avoid copying each field individually?
> > Some visitor trick, maybe?
> > 
> > Eric, Markus, Michael, do you have any suggestions?
> 
> Markus just added QAPI_CLONE_MEMBERS(), which sounds like what you want:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg04867.html

Not sure if it would work in this case. Note that cpu and
object->u.cpu have different types. 'cpu' is CpuInstanceProps,
but object->u.cpu is NodeCpuOptions.

NodeCpuOptions has { 'base': 'CpuInstanceProps' }, and
CpuInstanceProps field declarations are duplicated in struct
NodeCpuOptions. Do you know why struct inheritance is implemented
by duplicating the fields of the base struct instead of embedding
the base struct?

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:38:41PM -0300, Eduardo Habkost wrote:
> On Wed, May 03, 2017 at 11:39:10AM -0500, Eric Blake wrote:
> > On 05/03/2017 11:35 AM, Eduardo Habkost wrote:
> > 
> > >> +
> > >> +        memset(&cpu, 0, sizeof(cpu));
> > >> +        cpu.has_node_id = object->u.cpu.has_node_id;
> > >> +        cpu.node_id = object->u.cpu.node_id;
> > >> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> > >> +        cpu.socket_id = object->u.cpu.socket_id;
> > >> +        cpu.has_core_id = object->u.cpu.has_core_id;
> > >> +        cpu.core_id = object->u.cpu.core_id;
> > >> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> > >> +        cpu.thread_id = object->u.cpu.thread_id;
> > > 
> > > We don't have a way to avoid copying each field individually?
> > > Some visitor trick, maybe?
> > > 
> > > Eric, Markus, Michael, do you have any suggestions?
> > 
> > Markus just added QAPI_CLONE_MEMBERS(), which sounds like what you want:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg04867.html
> 
> Not sure if it would work in this case. Note that cpu and
> object->u.cpu have different types. 'cpu' is CpuInstanceProps,
> but object->u.cpu is NodeCpuOptions.
> 
> NodeCpuOptions has { 'base': 'CpuInstanceProps' }, and
> CpuInstanceProps field declarations are duplicated in struct
> NodeCpuOptions. Do you know why struct inheritance is implemented
> by duplicating the fields of the base struct instead of embedding
> the base struct?

Nevermind, I just found out that QAPI generates a
qapi_NumaCpuOptions_base() upcast helper. So this can be solved
with no data copying at all:

 machine_set_cpu_numa_nodes(ms, qapi_NumaCpuOptions_base(&object->u.cpu), &err);

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4448 bytes --]

On Wed, May 03, 2017 at 11:42:40AM -0300, Eduardo Habkost wrote:
> On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
> > Originally CPU threads were by default assigned in
> > round-robin fashion. However it was causing issues in
> > guest since CPU threads from the same socket/core could
> > be placed on different NUMA nodes.
> > Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
> > fixed it by grouping threads within a socket on the same node
> > introducing cpu_index_to_socket_id() callback and commit
> > 20bb648d (spapr: Fix default NUMA node allocation for threads)
> > reused callback to fix similar issues for SPAPR machine
> > even though socket doesn't make much sense there.
> > 
> > As result QEMU ended up having 3 default distribution rules
> > used by 3 targets /virt-arm, spapr, pc/.
> > 
> > In effort of moving NUMA mapping for CPUs into possible_cpus,
> > generalize default mapping in numa.c by making boards decide
> > on default mapping and let them explicitly tell generic
> > numa code to which node a CPU thread belongs to by replacing
> > cpu_index_to_socket_id() with @cpu_index_to_instance_props()
> > which provides default node_id assigned by board to specified
> > cpu_index.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> Just two extra comments below:
> 
> [...]
> > +static CpuInstanceProperties
> > +virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > +
> > +    assert(cpu_index < possible_cpus->len);
> > +    return possible_cpus->cpus[cpu_index].props;;
> > +}
> > +
> [...]
> > +static CpuInstanceProperties
> > +pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> >  {
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > +
> > +    assert(cpu_index < possible_cpus->len);
> > +    return possible_cpus->cpus[cpu_index].props;;
> >  }
> 
> The fact that these two implementations look exactly the same
> made me wonder:
> 
> 1) Why this isn't the default implementation;
> 2) Why exactly spapr needs a different implementation.
> 
> Then I noticed that there's nothing in the common machine code
> that specifies that possible_cpus->cpus[] is indexed by
> cpu_index. This means it is indeed safer to require each machine
> to provide its own cpu_index_to_props implementation than having
> a default implementation that can unexpectedly break (e.g. if
> granularity at possible_cpus is not at VCPU/thread level).
> 
> I would still like to have an abstraction that wouldn't require
> writing machine-specific code (e.g. cpu_index ranges to
> possible_cpus like David suggested), but that's for a follow-up
> series.

Yeah, that similarity bothered me to, but like you I realised the
problem is that spapr simply doesn't have the same granularity of
information as x86 and ARM - there's only one entry per core for PAPR
instead of one per thread.

So, we do need a machine specific mapping of cpu_index to location
properties, which is what the callback is for.

It does occur to me that another way of accomplishing that would be
for possible_cpu_arch_ids() to create a cpu_index->props mapping as a
simple array ofpointers, in addition to the list of possiblee props
structures.

Not sure if that would end up looking better or not.

> [...]
> > +static CpuInstanceProperties
> > +spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
> >  {
> > +    CPUArchId *core_slot;
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +
> > +    /* make sure possible_cpu are intialized */
> > +    mc->possible_cpu_arch_ids(machine);
> > +    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
> > +    assert(core_slot);
> > +    return core_slot->props;
> >  }
> 
> If you need to submit v3, maybe a comment here explaining why
> spapr needs a different cpu_index_to_props implementation would
> be helpful. I took a while to figure it out.
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

On Wed, 3 May 2017 12:20:22 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> > Introduce machine_set_cpu_numa_node() helper that stores
> > node mapping for CPU in MachineState::possible_cpus.
> > CPU and node it belongs to is specified by 'props' argument.
> > 
> > Patch doesn't remove old way of storing mapping in
> > numa_info[X].node_cpu as removing it at the same time
> > makes patch rather big. Instead it just mirrors mapping
> > in possible_cpus and follow up per target patches will
> > switch to possible_cpus and numa_info[X].node_cpu will
> > be removed once there isn't any users left.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > ---
> >  include/hw/boards.h |  2 ++
> >  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  numa.c              |  8 +++++++
> >  3 files changed, 78 insertions(+)
> > 
> > diff --git a/include/hw/boards.h b/include/hw/boards.h
> > index 5d6af21..1f518a1 100644
> > --- a/include/hw/boards.h
> > +++ b/include/hw/boards.h
> > @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
> >  bool machine_mem_merge(MachineState *machine);
> >  void machine_register_compat_props(MachineState *machine);
> >  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp);
> >  
> >  /**
> >   * CPUArchId:
> > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > index ada9eea..a63f17b 100644
> > --- a/hw/core/machine.c
> > +++ b/hw/core/machine.c
> > @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
> >      return head;
> >  }
> >  
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp)  
> 
> As the semantics of this function aren't trivial, it would be
> nice to have a comment explaining what exactly this function do.
> 
> e.g.:
> * make it clear that it could affect multiple CPU slots;
> * make it clear what does it mean to have props->has_node_id=false as
>   argument (is it really valid?);
> * make it clear that it will refuse to change an existing mapping.
sure, I'll add comment.

> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    bool match = false;
> > +    int i;
> > +
> > +    if (!mc->possible_cpu_arch_ids) {
> > +        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
> > +        return;
> > +    }
> > +
> > +    /* force board to initialize possible_cpus if it hasn't been done yet */
> > +    mc->possible_cpu_arch_ids(machine);
> > +
> > +    for (i = 0; i < machine->possible_cpus->len; i++) {
> > +        CPUArchId *slot = &machine->possible_cpus->cpus[i];
> > +
> > +        /* reject unsupported by board properties */
> > +        if (props->has_thread_id && !slot->props.has_thread_id) {
> > +            error_setg(errp, "thread-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_core_id && !slot->props.has_core_id) {
> > +            error_setg(errp, "core-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_socket_id && !slot->props.has_socket_id) {
> > +            error_setg(errp, "socket-id is not supported");
> > +            return;
> > +        }
> > +
> > +        /* skip slots with explicit mismatch */
> > +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> > +                continue;
> > +        }
> > +
> > +        /* reject assignment if slot is already assigned, for compatibility
> > +         * of legacy cpu_index mapping with SPAPR core based mapping do not
> > +         * error out if cpu thread and matched core have the same node-id */
> > +        if (slot->props.has_node_id &&
> > +            slot->props.node_id != props->node_id) {
> > +            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
> > +                       slot->props.node_id);
> > +            return;
> > +        }
> > +
> > +        /* assign slot to node as it's matched '-numa cpu' key */
> > +        match = true;
> > +        slot->props.node_id = props->node_id;
> > +        slot->props.has_node_id = props->has_node_id;  
> 
> Is it really valid to have has_node_id=false? Maybe an
> assert(props->has_node_id) at the beginning of the function would
> be useful.
yep, it shouldn't be false for props->has_node_id
I'll add assert

> 
> > +    }
> > +
> > +    if (!match) {
> > +        error_setg(errp, "no match found");
> > +    }
> > +}
> > +
> >  static void machine_class_init(ObjectClass *oc, void *data)
> >  {
> >      MachineClass *mc = MACHINE_CLASS(oc);
> > diff --git a/numa.c b/numa.c
> > index b517870..5ff1212 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >          exit(1);
> >      }
> >      for (cpus = node->cpus; cpus; cpus = cpus->next) {
> > +        CpuInstanceProperties props;
> >          if (cpus->value >= max_cpus) {
> >              error_setg(errp,
> >                         "CPU index (%" PRIu16 ")"
> > @@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >              return;
> >          }
> >          bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
> > +        props = mc->cpu_index_to_instance_props(ms, cpus->value);
> > +        props.node_id = nodenr;
> > +        props.has_node_id = true;
> > +        machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >      }
> >  
> >      if (node->has_mem && node->has_memdev) {
> > @@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
> >          if (i == nb_numa_nodes) {
> >              for (i = 0; i < max_cpus; i++) {
> >                  CpuInstanceProperties props;
> > +                /* fetch default mapping from board and enable it */
> >                  props = mc->cpu_index_to_instance_props(ms, i);
> > +                props.has_node_id = true;
> >  
> >                  set_bit(i, numa_info[props.node_id].node_cpu);
> > +                machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >              }
> >          }
> >  
> > -- 
> > 2.7.4
> >   
> 

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Igor Mammedov]

On Wed, 3 May 2017 14:31:09 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 06:19:50PM +0200, Igor Mammedov wrote:
> > On Wed, 3 May 2017 12:04:29 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> >   
> > > On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:  
> > > > Default node mapping initialization already checks that board
> > > > supports cpu_index to node mapping and refuses to start if
> > > > it's not supported. Do the same for explicitly provided
> > > > mapping "-numa node,cpus=..."
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > ---
> > > >  numa.c | 13 ++++++++++---
> > > >  1 file changed, 10 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/numa.c b/numa.c
> > > > index ab1661d..b517870 100644
> > > > --- a/numa.c
> > > > +++ b/numa.c
> > > > @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
> > > >      return -1;
> > > >  }
> > > >  
> > > > -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> > > > +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> > > > +                            QemuOpts *opts, Error **errp)
> > > >  {
> > > >      uint16_t nodenr;
> > > >      uint16List *cpus = NULL;
> > > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > > >  
> > > >      if (node->has_nodeid) {
> > > >          nodenr = node->nodeid;
> > > > @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> > > >          return;
> > > >      }
> > > >  
> > > > +    if (!mc->cpu_index_to_instance_props) {
> > > > +        error_report("CPUs to NUMA node mapping isn't supported");
> > > > +        exit(1);
> > > > +    }    
> > > 
> > > The error message sounds very confusing me. Maybe this could
> > > become just: "NUMA is not supported by the %s machine-type"?  
> > If I replace this specific error message with a generic one and do
> > the same in 5/24, it will be harder for us/user to figure out
> > where exactly error happened.  
> 
> I don't think we need to change the message in 5/24 if we change
> this one.
ok

> > 
> > Would be following clearer:
> > "option -numa node,cpus= is not supported by the %s machine-type"  
> 
> It would work if this was really the case. But the error message
> is unconditional here, appearing even if cpus= is not set:
> 
>   $ qemu-system-x86_64 -m 1G -smp 1 -numa node,mem=1G -machine none
>   qemu-system-x86_64: -numa node,mem=1G: CPUs to NUMA node mapping isn't supported
I'll change error message as you suggested before.

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

On Wed, 3 May 2017 12:13:21 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/numa.c b/numa.c
> > index 6fc2393..ab1661d 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -294,9 +294,10 @@ static void validate_numa_cpus(void)
> >      g_free(seen_cpus);
> >  }
> >  
> > -void parse_numa_opts(MachineClass *mc)
> > +void parse_numa_opts(MachineState *ms)
> >  {
> >      int i;
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> >  
> >      for (i = 0; i < MAX_NODES; i++) {
> >          numa_info[i].node_cpu = bitmap_new(max_cpus);
> > @@ -378,14 +379,16 @@ void parse_numa_opts(MachineClass *mc)  
> 
> 
> Expanding diff context:
> 
> >          /* Historically VCPUs were assigned in round-robin order to NUMA
> >           * nodes. However it causes issues with guest not handling it nice
> >           * in case where cores/threads from a multicore CPU appear on
> >           * different nodes. So allow boards to override default distribution
> >           * rule grouping VCPUs by socket so that VCPUs from the same socket
> >           * would be on the same node.
> >           */  
> 
> The above comment looks obsolete, as we are removing the code inside
> parse_numa_opts() that deals with grouping VCPUs by socket.
> 
> (Can be fixed by a follow-up patch, if necessary.)
it looks like, I'll respin series.
So I'll drop it on respin.

Looking at setting default mapping more,
it should be possible to remove it from parse_numa_opts()
altogether after this series.
Enable predefined by machine default mapping won't need
cpu_index_to_instance_props() translation. We just need
to set 'has_node_id = true' in possible_cpus for all cpus.

But I'd like to do this cleanup on top of this series.

> 
> > +        if (!mc->cpu_index_to_instance_props) {
> > +            error_report("default CPUs to NUMA node mapping isn't supported");
> > +            exit(1);
> > +        }
> >          if (i == nb_numa_nodes) {
> >              for (i = 0; i < max_cpus; i++) {
> > -                unsigned node_id = i % nb_numa_nodes;
> > -                if (mc->cpu_index_to_socket_id) {
> > -                    node_id = mc->cpu_index_to_socket_id(i) % nb_numa_nodes;
> > -                }
> > +                CpuInstanceProperties props;
> > +                props = mc->cpu_index_to_instance_props(ms, i);
> >  
> > -                set_bit(i, numa_info[node_id].node_cpu);
> > +                set_bit(i, numa_info[props.node_id].node_cpu);
> >              }
> >          }
> >    
> [...]
> 

Re: [Qemu-devel] [PATCH v2 02/24] hw/arm/virt: extract mp-affinity calculation in separate function

[Andrew Jones]

On Wed, May 03, 2017 at 02:56:56PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>  (Drew)
>    - s/virt_idx2mp_affinity/virt_cpu_mp_affinity/
>    - add arm_cpu_mp_affinity() to reduce code duplication
> ---
>  target/arm/cpu.h |  2 ++
>  hw/arm/virt.c    | 43 ++++++++++++++++++++++++++-----------------
>  target/arm/cpu.c | 12 +++++++++---
>  3 files changed, 37 insertions(+), 20 deletions(-)
> 
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index 1055bfe..048faed 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -710,6 +710,8 @@ static inline ARMCPU *arm_env_get_cpu(CPUARMState *env)
>      return container_of(env, ARMCPU, env);
>  }
>  
> +uint64_t arm_cpu_mp_affinity(int idx, uint8_t clustersz);
> +
>  #define ENV_GET_CPU(e) CPU(arm_env_get_cpu(e))
>  
>  #define ENV_OFFSET offsetof(ARMCPU, env)
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 5f62a03..61ae437 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1194,6 +1194,29 @@ void virt_machine_done(Notifier *notifier, void *data)
>      virt_build_smbios(vms);
>  }
>  
> +static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
> +{
> +    uint8_t clustersz = ARM_DEFAULT_CPUS_PER_CLUSTER;
> +    VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
> +
> +    if (!vmc->disallow_affinity_adjustment) {
> +        /* Adjust MPIDR like 64-bit KVM hosts, which incorporate the
> +         * GIC's target-list limitations. 32-bit KVM hosts currently
> +         * always create clusters of 4 CPUs, but that is expected to
> +         * change when they gain support for gicv3. When KVM is enabled
> +         * it will override the changes we make here, therefore our
> +         * purposes are to make TCG consistent (with 64-bit KVM hosts)
> +         * and to improve SGI efficiency.
> +         */
> +        if (vms->gic_version == 3) {
> +            clustersz = GICV3_TARGETLIST_BITS;
> +        } else {
> +            clustersz = GIC_TARGETLIST_BITS;
> +        }
> +    }
> +    return arm_cpu_mp_affinity(idx, clustersz);
> +}
> +
>  static void machvirt_init(MachineState *machine)
>  {
>      VirtMachineState *vms = VIRT_MACHINE(machine);
> @@ -1210,7 +1233,6 @@ static void machvirt_init(MachineState *machine)
>      CPUClass *cc;
>      Error *err = NULL;
>      bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
> -    uint8_t clustersz;
>  
>      if (!cpu_model) {
>          cpu_model = "cortex-a15";
> @@ -1263,10 +1285,8 @@ static void machvirt_init(MachineState *machine)
>       */
>      if (vms->gic_version == 3) {
>          virt_max_cpus = vms->memmap[VIRT_GIC_REDIST].size / 0x20000;
> -        clustersz = GICV3_TARGETLIST_BITS;
>      } else {
>          virt_max_cpus = GIC_NCPU;
> -        clustersz = GIC_TARGETLIST_BITS;
>      }
>  
>      if (max_cpus > virt_max_cpus) {
> @@ -1326,20 +1346,9 @@ static void machvirt_init(MachineState *machine)
>  
>      for (n = 0; n < smp_cpus; n++) {
>          Object *cpuobj = object_new(typename);
> -        if (!vmc->disallow_affinity_adjustment) {
> -            /* Adjust MPIDR like 64-bit KVM hosts, which incorporate the
> -             * GIC's target-list limitations. 32-bit KVM hosts currently
> -             * always create clusters of 4 CPUs, but that is expected to
> -             * change when they gain support for gicv3. When KVM is enabled
> -             * it will override the changes we make here, therefore our
> -             * purposes are to make TCG consistent (with 64-bit KVM hosts)
> -             * and to improve SGI efficiency.
> -             */
> -            uint8_t aff1 = n / clustersz;
> -            uint8_t aff0 = n % clustersz;
> -            object_property_set_int(cpuobj, (aff1 << ARM_AFF1_SHIFT) | aff0,
> -                                    "mp-affinity", NULL);
> -        }
> +
> +        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
> +                                "mp-affinity", NULL);
>  
>          if (!vms->secure) {
>              object_property_set_bool(cpuobj, false, "has_el3", NULL);
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index b357aee..ee1406d 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -458,6 +458,13 @@ static void arm_disas_set_info(CPUState *cpu, disassemble_info *info)
>      }
>  }
>  
> +uint64_t arm_cpu_mp_affinity(int idx, uint8_t clustersz)
> +{
> +    uint32_t Aff1 = idx / clustersz;
> +    uint32_t Aff0 = idx % clustersz;
> +    return (Aff1 << ARM_AFF1_SHIFT) | Aff0;
> +}
> +

This function obviously assumes we only care about Aff1 and Aff0, which is
true right now. Eventually we'll want to teach qemu/arm about more complex
topologies that will need to set Aff{2,3} as well, but we can revisit this
function then.

>  static void arm_cpu_initfn(Object *obj)
>  {
>      CPUState *cs = CPU(obj);
> @@ -709,9 +716,8 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>       * so these bits always RAZ.
>       */
>      if (cpu->mp_affinity == ARM64_AFFINITY_INVALID) {
> -        uint32_t Aff1 = cs->cpu_index / ARM_DEFAULT_CPUS_PER_CLUSTER;
> -        uint32_t Aff0 = cs->cpu_index % ARM_DEFAULT_CPUS_PER_CLUSTER;
> -        cpu->mp_affinity = (Aff1 << ARM_AFF1_SHIFT) | Aff0;
> +        cpu->mp_affinity = arm_cpu_mp_affinity(cs->cpu_index,
> +                                               ARM_DEFAULT_CPUS_PER_CLUSTER);
>      }
>  
>      if (cpu->reset_hivecs) {
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [PATCH v2 03/24] hw/arm/virt: use machine->possible_cpus for storing possible topology info

[Andrew Jones]

On Wed, May 03, 2017 at 02:56:57PM +0200, Igor Mammedov wrote:
> for now precalculate and store mp_afinity in possible_cpus
> as ARM cpus don't have socket/core/thread-id properties yet.
> In follow patches possible_cpus will be used for storing
> and setting NUMA node mapping and replace legacy bitmap
> based numa_info[node_id].node_cpu/numa_get_node_for_cpu()
> 
> For the lack of better idea, this patch cannibalizes
> possible_cpus.cpus[x].props.thread_id so that
> *_cpu_index_to_props() callback could return addressable
> by props CPU which will be used by machine_set_cpu_numa_node()
> in follow up patches to assign a CPU to node. But
> cannibalizing is fine for now as that thread_id isn't exposed
> to users (no hotpluggable_cpus callback support for ARM yet)
> and it will be used only internally until 'device_add cpu'
> is supported where we can decide on which properties to use.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   (Drew)
>     - discarding result of possible_cpu_arch_ids() makes
>       call not obvious and is confusing. Instead assign
>       possible_cpu_arch_ids() result to local var and use
>       it instead of direct access to machine->possible_cpus
>       field, as it's done in pc.c
> ---
>  hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++---
>  1 file changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 61ae437..e2c5626 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1221,6 +1221,8 @@ static void machvirt_init(MachineState *machine)
>  {
>      VirtMachineState *vms = VIRT_MACHINE(machine);
>      VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine);
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +    const CPUArchIdList *possible_cpus;
>      qemu_irq pic[NUM_IRQS];
>      MemoryRegion *sysmem = get_system_memory();
>      MemoryRegion *secure_sysmem = NULL;
> @@ -1344,10 +1346,16 @@ static void machvirt_init(MachineState *machine)
>          exit(1);
>      }
>  
> -    for (n = 0; n < smp_cpus; n++) {
> -        Object *cpuobj = object_new(typename);
> +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> +    for (n = 0; n < possible_cpus->len; n++) {
> +        Object *cpuobj;
>  
> -        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
> +        if (n >= smp_cpus) {
> +            break;
> +        }

Why the break instead of just looping 'n < smp_cpus' like x86 does? Is
there some future work where looping up to possible_cpus->len (aka
max_cpus) is what we'll eventually want? If so, then we need a TODO
comment here. If not, then we should clean this up by removing the break.

Thanks,
drew

> +
> +        cpuobj = object_new(typename);
> +        object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
>                                  "mp-affinity", NULL);
>  
>          if (!vms->secure) {
> @@ -1527,6 +1535,31 @@ static void virt_set_gic_version(Object *obj, const char *value, Error **errp)
>      }
>  }
>  
> +static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
> +{
> +    int n;
> +    VirtMachineState *vms = VIRT_MACHINE(ms);
> +
> +    if (ms->possible_cpus) {
> +        assert(ms->possible_cpus->len == max_cpus);
> +        return ms->possible_cpus;
> +    }
> +
> +    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
> +                                  sizeof(CPUArchId) * max_cpus);
> +    ms->possible_cpus->len = max_cpus;
> +    for (n = 0; n < ms->possible_cpus->len; n++) {
> +        ms->possible_cpus->cpus[n].arch_id =
> +            virt_cpu_mp_affinity(vms, n);
> +        ms->possible_cpus->cpus[n].props.has_thread_id = true;
> +        ms->possible_cpus->cpus[n].props.thread_id = n;
> +
> +        /* TODO: add 'has_node/node' here to describe
> +           to which node core belongs */
> +    }
> +    return ms->possible_cpus;
> +}
> +
>  static void virt_machine_class_init(ObjectClass *oc, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(oc);
> @@ -1543,6 +1576,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
>      mc->pci_allow_0_address = true;
>      /* We know we will never create a pre-ARMv7 CPU which needs 1K pages */
>      mc->minimum_page_bits = 12;
> +    mc->possible_cpu_arch_ids = virt_possible_cpu_arch_ids;
>  }
>  
>  static const TypeInfo virt_machine_info = {
> -- 
> 2.7.4
> 

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Igor Mammedov]

On Wed, 3 May 2017 14:58:02 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:38:41PM -0300, Eduardo Habkost wrote:
> > On Wed, May 03, 2017 at 11:39:10AM -0500, Eric Blake wrote:  
> > > On 05/03/2017 11:35 AM, Eduardo Habkost wrote:
> > >   
> > > >> +
> > > >> +        memset(&cpu, 0, sizeof(cpu));
> > > >> +        cpu.has_node_id = object->u.cpu.has_node_id;
> > > >> +        cpu.node_id = object->u.cpu.node_id;
> > > >> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> > > >> +        cpu.socket_id = object->u.cpu.socket_id;
> > > >> +        cpu.has_core_id = object->u.cpu.has_core_id;
> > > >> +        cpu.core_id = object->u.cpu.core_id;
> > > >> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> > > >> +        cpu.thread_id = object->u.cpu.thread_id;  
> > > > 
> > > > We don't have a way to avoid copying each field individually?
> > > > Some visitor trick, maybe?
> > > > 
> > > > Eric, Markus, Michael, do you have any suggestions?  
> > > 
> > > Markus just added QAPI_CLONE_MEMBERS(), which sounds like what you want:
> > > 
> > > https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg04867.html  
> > 
> > Not sure if it would work in this case. Note that cpu and
> > object->u.cpu have different types. 'cpu' is CpuInstanceProps,
> > but object->u.cpu is NodeCpuOptions.
> > 
> > NodeCpuOptions has { 'base': 'CpuInstanceProps' }, and
> > CpuInstanceProps field declarations are duplicated in struct
> > NodeCpuOptions. Do you know why struct inheritance is implemented
> > by duplicating the fields of the base struct instead of embedding
> > the base struct?  
> 
> Nevermind, I just found out that QAPI generates a
> qapi_NumaCpuOptions_base() upcast helper. So this can be solved
> with no data copying at all:
> 
>  machine_set_cpu_numa_nodes(ms, qapi_NumaCpuOptions_base(&object->u.cpu), &err);
> 

Thanks for finding out, I'll try it this way

Re: [Qemu-devel] [PATCH v2 08/24] virt-arm: add node-id property to CPU

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:02PM +0200, Igor Mammedov wrote:
> it will allow switching from cpu_index to property based
> numa mapping in follow up patches.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  hw/arm/virt.c    | 15 +++++++++++++++
>  target/arm/cpu.c |  1 +
>  2 files changed, 16 insertions(+)
> 
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 3e19b5f..05a2822 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1350,6 +1350,7 @@ static void machvirt_init(MachineState *machine)
>      for (n = 0; n < possible_cpus->len; n++) {
>          Object *cpuobj;
>          CPUState *cs;
> +        int node_id;
>  
>          if (n >= smp_cpus) {
>              break;
> @@ -1362,6 +1363,20 @@ static void machvirt_init(MachineState *machine)
>          cs = CPU(cpuobj);
>          cs->cpu_index = n;
>  
> +        node_id = numa_get_node_for_cpu(cs->cpu_index);
> +        if (node_id == nb_numa_nodes) {
> +            /* by default CPUState::numa_node was 0 if it's not set via CLI
> +             * keep it this way for now but in future we probably should
> +             * refuse to start up with incomplete numa mapping */
> +             node_id = 0;
> +        }
> +        if (cs->numa_node == CPU_UNSET_NUMA_NODE_ID) {
> +            cs->numa_node = node_id;
> +        } else {
> +            /* CPU isn't device_add compatible yet, this shouldn't happen */
> +            error_setg(&error_abort, "user set node-id not implemented");
> +        }
> +
>          if (!vms->secure) {
>              object_property_set_bool(cpuobj, false, "has_el3", NULL);
>          }
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index ee1406d..c185eb1 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -1573,6 +1573,7 @@ static Property arm_cpu_properties[] = {
>      DEFINE_PROP_UINT32("midr", ARMCPU, midr, 0),
>      DEFINE_PROP_UINT64("mp-affinity", ARMCPU,
>                          mp_affinity, ARM64_AFFINITY_INVALID),
> +    DEFINE_PROP_INT32("node-id", CPUState, numa_node, CPU_UNSET_NUMA_NODE_ID),
>      DEFINE_PROP_END_OF_LIST()
>  };
>  
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:
> Default node mapping initialization already checks that board
> supports cpu_index to node mapping and refuses to start if
> it's not supported. Do the same for explicitly provided
> mapping "-numa node,cpus=..."
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  numa.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/numa.c b/numa.c
> index ab1661d..b517870 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
>      return -1;
>  }
>  
> -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> +                            QemuOpts *opts, Error **errp)
>  {
>      uint16_t nodenr;
>      uint16List *cpus = NULL;
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
>      if (node->has_nodeid) {
>          nodenr = node->nodeid;
> @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
>          return;
>      }
>  
> +    if (!mc->cpu_index_to_instance_props) {
> +        error_report("CPUs to NUMA node mapping isn't supported");
> +        exit(1);
> +    }
>      for (cpus = node->cpus; cpus; cpus = cpus->next) {
>          if (cpus->value >= max_cpus) {
>              error_setg(errp,
> @@ -215,6 +221,7 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
>  static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>  {
>      NumaOptions *object = NULL;
> +    MachineState *ms = opaque;
>      Error *err = NULL;
>  
>      {
> @@ -229,7 +236,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>  
>      switch (object->type) {
>      case NUMA_OPTIONS_TYPE_NODE:
> -        numa_node_parse(&object->u.node, opts, &err);
> +        numa_node_parse(ms, &object->u.node, opts, &err);
>          if (err) {
>              goto end;
>          }
> @@ -303,7 +310,7 @@ void parse_numa_opts(MachineState *ms)
>          numa_info[i].node_cpu = bitmap_new(max_cpus);
>      }
>  
> -    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, NULL, NULL)) {
> +    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, NULL)) {
>          exit(1);
>      }
>  
> -- 
> 2.7.4
>

With Eduardo's error message change

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> Introduce machine_set_cpu_numa_node() helper that stores
> node mapping for CPU in MachineState::possible_cpus.
> CPU and node it belongs to is specified by 'props' argument.
> 
> Patch doesn't remove old way of storing mapping in
> numa_info[X].node_cpu as removing it at the same time
> makes patch rather big. Instead it just mirrors mapping
> in possible_cpus and follow up per target patches will
> switch to possible_cpus and numa_info[X].node_cpu will
> be removed once there isn't any users left.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  include/hw/boards.h |  2 ++
>  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  numa.c              |  8 +++++++
>  3 files changed, 78 insertions(+)
> 
> diff --git a/include/hw/boards.h b/include/hw/boards.h
> index 5d6af21..1f518a1 100644
> --- a/include/hw/boards.h
> +++ b/include/hw/boards.h
> @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
>  bool machine_mem_merge(MachineState *machine);
>  void machine_register_compat_props(MachineState *machine);
>  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> +void machine_set_cpu_numa_node(MachineState *machine,
> +                               CpuInstanceProperties *props, Error **errp);
>  
>  /**
>   * CPUArchId:
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index ada9eea..a63f17b 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
>      return head;
>  }
>  
> +void machine_set_cpu_numa_node(MachineState *machine,
> +                               CpuInstanceProperties *props, Error **errp)
> +{
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +    bool match = false;
> +    int i;
> +
> +    if (!mc->possible_cpu_arch_ids) {
> +        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
> +        return;
> +    }
> +
> +    /* force board to initialize possible_cpus if it hasn't been done yet */
> +    mc->possible_cpu_arch_ids(machine);
> +
> +    for (i = 0; i < machine->possible_cpus->len; i++) {
> +        CPUArchId *slot = &machine->possible_cpus->cpus[i];
> +
> +        /* reject unsupported by board properties */
> +        if (props->has_thread_id && !slot->props.has_thread_id) {
> +            error_setg(errp, "thread-id is not supported");
> +            return;
> +        }
> +
> +        if (props->has_core_id && !slot->props.has_core_id) {
> +            error_setg(errp, "core-id is not supported");
> +            return;
> +        }
> +
> +        if (props->has_socket_id && !slot->props.has_socket_id) {
> +            error_setg(errp, "socket-id is not supported");
> +            return;
> +        }
> +
> +        /* skip slots with explicit mismatch */
> +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> +                continue;
> +        }
> +
> +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> +                continue;
> +        }
> +
> +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> +                continue;
> +        }

nit: above 3 if-conditions could be condensed into 1 compound condition

> +
> +        /* reject assignment if slot is already assigned, for compatibility
> +         * of legacy cpu_index mapping with SPAPR core based mapping do not
> +         * error out if cpu thread and matched core have the same node-id */
> +        if (slot->props.has_node_id &&
> +            slot->props.node_id != props->node_id) {
> +            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
> +                       slot->props.node_id);
> +            return;
> +        }
> +
> +        /* assign slot to node as it's matched '-numa cpu' key */
> +        match = true;
> +        slot->props.node_id = props->node_id;
> +        slot->props.has_node_id = props->has_node_id;
> +    }
> +
> +    if (!match) {
> +        error_setg(errp, "no match found");
> +    }
> +}
> +
>  static void machine_class_init(ObjectClass *oc, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(oc);
> diff --git a/numa.c b/numa.c
> index b517870..5ff1212 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
>          exit(1);
>      }
>      for (cpus = node->cpus; cpus; cpus = cpus->next) {
> +        CpuInstanceProperties props;
>          if (cpus->value >= max_cpus) {
>              error_setg(errp,
>                         "CPU index (%" PRIu16 ")"
> @@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
>              return;
>          }
>          bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
> +        props = mc->cpu_index_to_instance_props(ms, cpus->value);
> +        props.node_id = nodenr;
> +        props.has_node_id = true;
> +        machine_set_cpu_numa_node(ms, &props, &error_fatal);
>      }
>  
>      if (node->has_mem && node->has_memdev) {
> @@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
>          if (i == nb_numa_nodes) {
>              for (i = 0; i < max_cpus; i++) {
>                  CpuInstanceProperties props;
> +                /* fetch default mapping from board and enable it */
>                  props = mc->cpu_index_to_instance_props(ms, i);
> +                props.has_node_id = true;
>  
>                  set_bit(i, numa_info[props.node_id].node_cpu);
> +                machine_set_cpu_numa_node(ms, &props, &error_fatal);
>              }
>          }
>  
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 11/24] numa: do default mapping based on possible_cpus instead of node_cpu bitmaps

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:05PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  numa.c | 19 ++++++++++++-------
>  1 file changed, 12 insertions(+), 7 deletions(-)
> 
> diff --git a/numa.c b/numa.c
> index 5ff1212..c7e3e0a 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -309,6 +309,7 @@ static void validate_numa_cpus(void)
>  void parse_numa_opts(MachineState *ms)
>  {
>      int i;
> +    const CPUArchIdList *possible_cpus;
>      MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
>      for (i = 0; i < MAX_NODES; i++) {
> @@ -379,11 +380,6 @@ void parse_numa_opts(MachineState *ms)
>  
>          numa_set_mem_ranges();
>  
> -        for (i = 0; i < nb_numa_nodes; i++) {
> -            if (!bitmap_empty(numa_info[i].node_cpu, max_cpus)) {
> -                break;
> -            }
> -        }
>          /* Historically VCPUs were assigned in round-robin order to NUMA
>           * nodes. However it causes issues with guest not handling it nice
>           * in case where cores/threads from a multicore CPU appear on
> @@ -391,11 +387,20 @@ void parse_numa_opts(MachineState *ms)
>           * rule grouping VCPUs by socket so that VCPUs from the same socket
>           * would be on the same node.
>           */
> -        if (!mc->cpu_index_to_instance_props) {
> +        if (!mc->cpu_index_to_instance_props || !mc->possible_cpu_arch_ids) {
>              error_report("default CPUs to NUMA node mapping isn't supported");
>              exit(1);
>          }
> -        if (i == nb_numa_nodes) {
> +
> +        possible_cpus = mc->possible_cpu_arch_ids(ms);
> +        for (i = 0; i < possible_cpus->len; i++) {
> +            if (possible_cpus->cpus[i].props.has_node_id) {
> +                break;
> +            }
> +        }
> +
> +        /* no CPUs are assigned to NUMA nodes */
> +        if (i == possible_cpus->len) {
>              for (i = 0; i < max_cpus; i++) {
>                  CpuInstanceProperties props;
>                  /* fetch default mapping from board and enable it */
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> wrappers should make access to [has]node_id fields more readable
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  follow up patches will use this wrappers
>  v2:
>     - add wrappers (Drew)
> ---
>  include/sysemu/numa.h | 10 ++++++++++
>  numa.c                |  2 +-
>  2 files changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index 46ea6c7..98d01e6 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -35,4 +35,14 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp);
>  /* on success returns node index in numa_info,
>   * on failure returns nb_numa_nodes */
>  int numa_get_node_for_cpu(int idx);
> +
> +static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
> +{
> +    return possible_cpus->cpus[idx].props.has_node_id;
> +}
> +
> +static inline int numa_node_id(const CPUArchIdList *possible_cpus, int idx)
> +{
> +    return possible_cpus->cpus[idx].props.node_id;
> +}
>  #endif
> diff --git a/numa.c b/numa.c
> index c7e3e0a..872ee0d 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -394,7 +394,7 @@ void parse_numa_opts(MachineState *ms)
>  
>          possible_cpus = mc->possible_cpu_arch_ids(ms);
>          for (i = 0; i < possible_cpus->len; i++) {
> -            if (possible_cpus->cpus[i].props.has_node_id) {
> +            if (numa_has_node_id(possible_cpus, i)) {
>                  break;
>              }
>          }
> -- 
> 2.7.4
> 
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [PATCH v2 13/24] pc: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:07PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>    use numa_[has_]node_id() wrappers (Drew)
> ---
>  hw/acpi/cpu.c        |  7 +++----
>  hw/i386/acpi-build.c | 11 ++++-------
>  hw/i386/pc.c         | 18 ++++++++++--------
>  3 files changed, 17 insertions(+), 19 deletions(-)
> 
> diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
> index 8c719d3..2002198 100644
> --- a/hw/acpi/cpu.c
> +++ b/hw/acpi/cpu.c
> @@ -503,7 +503,6 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>  
>          /* build Processor object for each processor */
>          for (i = 0; i < arch_ids->len; i++) {
> -            int j;
>              Aml *dev;
>              Aml *uid = aml_int(i);
>              GArray *madt_buf = g_array_new(0, 1, 1);
> @@ -557,9 +556,9 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts,
>               * as a result _PXM is required for all CPUs which might
>               * be hot-plugged. For simplicity, add it for all CPUs.
>               */
> -            j = numa_get_node_for_cpu(i);
> -            if (j < nb_numa_nodes) {
> -                aml_append(dev, aml_name_decl("_PXM", aml_int(j)));
> +            if (numa_has_node_id(arch_ids, i)) {
> +                aml_append(dev,
> +                    aml_name_decl("_PXM", aml_int(numa_node_id(arch_ids, i))));
>              }
>  
>              aml_append(cpus_dev, dev);
> diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
> index 2073108..a66a968 100644
> --- a/hw/i386/acpi-build.c
> +++ b/hw/i386/acpi-build.c
> @@ -2306,7 +2306,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
>      srat->reserved1 = cpu_to_le32(1);
>  
>      for (i = 0; i < apic_ids->len; i++) {
> -        int j = numa_get_node_for_cpu(i);
> +        int node_id = numa_has_node_id(apic_ids, i) ?
> +            numa_node_id(apic_ids, i) : 0;
>          uint32_t apic_id = apic_ids->cpus[i].arch_id;
>  
>          if (apic_id < 255) {
> @@ -2316,9 +2317,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
>              core->type = ACPI_SRAT_PROCESSOR_APIC;
>              core->length = sizeof(*core);
>              core->local_apic_id = apic_id;
> -            if (j < nb_numa_nodes) {
> -                core->proximity_lo = j;
> -            }
> +            core->proximity_lo = node_id;
>              memset(core->proximity_hi, 0, 3);
>              core->local_sapic_eid = 0;
>              core->flags = cpu_to_le32(1);
> @@ -2329,9 +2328,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
>              core->type = ACPI_SRAT_PROCESSOR_x2APIC;
>              core->length = sizeof(*core);
>              core->x2apic_id = cpu_to_le32(apic_id);
> -            if (j < nb_numa_nodes) {
> -                core->proximity_domain = cpu_to_le32(j);
> -            }
> +            core->proximity_domain = cpu_to_le32(node_id);
>              core->flags = cpu_to_le32(1);
>          }
>      }
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 4a4fb1c..aeecf4b 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -747,7 +747,9 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
>  {
>      FWCfgState *fw_cfg;
>      uint64_t *numa_fw_cfg;
> -    int i, j;
> +    int i;
> +    const CPUArchIdList *cpus;
> +    MachineClass *mc = MACHINE_GET_CLASS(pcms);
>  
>      fw_cfg = fw_cfg_init_io_dma(FW_CFG_IO_BASE, FW_CFG_IO_BASE + 4, as);
>      fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, pcms->boot_cpus);
> @@ -782,12 +784,12 @@ static FWCfgState *bochs_bios_init(AddressSpace *as, PCMachineState *pcms)
>       */
>      numa_fw_cfg = g_new0(uint64_t, 1 + pcms->apic_id_limit + nb_numa_nodes);
>      numa_fw_cfg[0] = cpu_to_le64(nb_numa_nodes);
> -    for (i = 0; i < max_cpus; i++) {
> -        unsigned int apic_id = x86_cpu_apic_id_from_index(i);
> +    cpus = mc->possible_cpu_arch_ids(MACHINE(pcms));
> +    for (i = 0; i < cpus->len; i++) {
> +        unsigned int apic_id = cpus->cpus[i].arch_id;
>          assert(apic_id < pcms->apic_id_limit);
> -        j = numa_get_node_for_cpu(i);
> -        if (j < nb_numa_nodes) {
> -            numa_fw_cfg[apic_id + 1] = cpu_to_le64(j);
> +        if (numa_has_node_id(cpus, i)) {
> +            numa_fw_cfg[apic_id + 1] = cpu_to_le64(numa_node_id(cpus, i));
>          }
>      }
>      for (i = 0; i < nb_numa_nodes; i++) {
> @@ -1984,8 +1986,8 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
>      cs = CPU(cpu);
>      cs->cpu_index = idx;
>  
> -    node_id = numa_get_node_for_cpu(cs->cpu_index);
> -    if (node_id == nb_numa_nodes) {
> +    node_id = cpu_slot->props.node_id;
> +    if (!cpu_slot->props.has_node_id) {
>          /* by default CPUState::numa_node was 0 if it's not set via CLI
>           * keep it this way for now but in future we probably should
>           * refuse to start up with incomplete numa mapping */
> -- 
> 2.7.4
> 
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 15/24] virt-arm: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:09PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>    use numa_[has_]node_id() wrappers (Drew)
> ---
>  hw/arm/virt-acpi-build.c | 19 +++++++------------
>  hw/arm/virt.c            | 13 +++++++------
>  2 files changed, 14 insertions(+), 18 deletions(-)
> 
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 0835e59..2404bb1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -486,30 +486,25 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>      AcpiSystemResourceAffinityTable *srat;
>      AcpiSratProcessorGiccAffinity *core;
>      AcpiSratMemoryAffinity *numamem;
> -    int i, j, srat_start;
> +    int i, srat_start;
>      uint64_t mem_base;
> -    uint32_t *cpu_node = g_malloc0(vms->smp_cpus * sizeof(uint32_t));
> -
> -    for (i = 0; i < vms->smp_cpus; i++) {
> -        j = numa_get_node_for_cpu(i);
> -        if (j < nb_numa_nodes) {
> -                cpu_node[i] = j;
> -        }
> -    }
> +    MachineClass *mc = MACHINE_GET_CLASS(vms);
> +    const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids(MACHINE(vms));
>  
>      srat_start = table_data->len;
>      srat = acpi_data_push(table_data, sizeof(*srat));
>      srat->reserved1 = cpu_to_le32(1);
>  
> -    for (i = 0; i < vms->smp_cpus; ++i) {
> +    for (i = 0; i < cpu_list->len; ++i) {
> +        int node_id = numa_has_node_id(cpu_list, i) ?
> +            numa_node_id(cpu_list, i) : 0;
>          core = acpi_data_push(table_data, sizeof(*core));
>          core->type = ACPI_SRAT_PROCESSOR_GICC;
>          core->length = sizeof(*core);
> -        core->proximity = cpu_to_le32(cpu_node[i]);
> +        core->proximity = cpu_to_le32(node_id);
>          core->acpi_processor_uid = cpu_to_le32(i);
>          core->flags = cpu_to_le32(1);
>      }
> -    g_free(cpu_node);
>  
>      mem_base = vms->memmap[VIRT_MEM].base;
>      for (i = 0; i < nb_numa_nodes; ++i) {
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 05a2822..233e8a7 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -338,7 +338,7 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
>  {
>      int cpu;
>      int addr_cells = 1;
> -    unsigned int i;
> +    const MachineState *ms = MACHINE(vms);
>  
>      /*
>       * From Documentation/devicetree/bindings/arm/cpus.txt
> @@ -369,6 +369,7 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
>      for (cpu = vms->smp_cpus - 1; cpu >= 0; cpu--) {
>          char *nodename = g_strdup_printf("/cpus/cpu@%d", cpu);
>          ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(cpu));
> +        CPUState *cs = CPU(armcpu);
>  
>          qemu_fdt_add_subnode(vms->fdt, nodename);
>          qemu_fdt_setprop_string(vms->fdt, nodename, "device_type", "cpu");
> @@ -389,9 +390,9 @@ static void fdt_add_cpu_nodes(const VirtMachineState *vms)
>                                    armcpu->mp_affinity);
>          }
>  
> -        i = numa_get_node_for_cpu(cpu);
> -        if (i < nb_numa_nodes) {
> -            qemu_fdt_setprop_cell(vms->fdt, nodename, "numa-node-id", i);
> +        if (numa_has_node_id(ms->possible_cpus, cs->cpu_index)) {
> +            qemu_fdt_setprop_cell(vms->fdt, nodename, "numa-node-id",
> +                numa_node_id(ms->possible_cpus, cs->cpu_index));
>          }
>  
>          g_free(nodename);
> @@ -1363,8 +1364,8 @@ static void machvirt_init(MachineState *machine)
>          cs = CPU(cpuobj);
>          cs->cpu_index = n;
>  
> -        node_id = numa_get_node_for_cpu(cs->cpu_index);
> -        if (node_id == nb_numa_nodes) {
> +        node_id = numa_node_id(possible_cpus, cs->cpu_index);
> +        if (!numa_has_node_id(possible_cpus, cs->cpu_index)) {
>              /* by default CPUState::numa_node was 0 if it's not set via CLI
>               * keep it this way for now but in future we probably should
>               * refuse to start up with incomplete numa mapping */
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 18/24] numa: remove no longer used numa_get_node_for_cpu()

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:12PM +0200, Igor Mammedov wrote:
> it's been replaced by fetching mapping info from possible_cpus
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  include/sysemu/numa.h |  4 ----
>  numa.c                | 14 --------------
>  2 files changed, 18 deletions(-)
> 
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index 98d01e6..9077bb2 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -32,10 +32,6 @@ void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  uint32_t numa_get_node(ram_addr_t addr, Error **errp);
>  
> -/* on success returns node index in numa_info,
> - * on failure returns nb_numa_nodes */
> -int numa_get_node_for_cpu(int idx);
> -
>  static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
>  {
>      return possible_cpus->cpus[idx].props.has_node_id;
> diff --git a/numa.c b/numa.c
> index 872ee0d..06c42e9 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -583,20 +583,6 @@ MemdevList *qmp_query_memdev(Error **errp)
>      return list;
>  }
>  
> -int numa_get_node_for_cpu(int idx)
> -{
> -    int i;
> -
> -    assert(idx < max_cpus);
> -
> -    for (i = 0; i < nb_numa_nodes; i++) {
> -        if (test_bit(idx, numa_info[i].node_cpu)) {
> -            break;
> -        }
> -    }
> -    return i;
> -}
> -
>  void ram_block_notifier_add(RAMBlockNotifier *n)
>  {
>      QLIST_INSERT_HEAD(&ram_list.ramblock_notifiers, n, next);
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 19/24] numa: remove no longer need numa_post_machine_init()

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:13PM +0200, Igor Mammedov wrote:
> CPUState::numa_node is still in use but now it's set by
> board when it creates CPU objects. So there isn't any
> need to set it again after all CPU's are created,
> since it's been already set.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  include/sysemu/numa.h |  1 -
>  numa.c                | 15 ---------------
>  vl.c                  |  2 --
>  3 files changed, 18 deletions(-)
> 
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index 9077bb2..0b0a044 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -25,7 +25,6 @@ typedef struct node_info {
>  
>  extern NodeInfo numa_info[MAX_NODES];
>  void parse_numa_opts(MachineState *ms);
> -void numa_post_machine_init(void);
>  void query_numa_node_mem(uint64_t node_mem[]);
>  extern QemuOptsList qemu_numa_opts;
>  void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
> diff --git a/numa.c b/numa.c
> index 06c42e9..f8c849d 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -418,21 +418,6 @@ void parse_numa_opts(MachineState *ms)
>      }
>  }
>  
> -void numa_post_machine_init(void)
> -{
> -    CPUState *cpu;
> -    int i;
> -
> -    CPU_FOREACH(cpu) {
> -        for (i = 0; i < nb_numa_nodes; i++) {
> -            assert(cpu->cpu_index < max_cpus);
> -            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> -                cpu->numa_node = i;
> -            }
> -        }
> -    }
> -}
> -
>  static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
>                                             const char *name,
>                                             uint64_t ram_size)
> diff --git a/vl.c b/vl.c
> index c63f4d5..fe4741d 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4595,8 +4595,6 @@ int main(int argc, char **argv, char **envp)
>  
>      cpu_synchronize_all_post_init();
>  
> -    numa_post_machine_init();
> -
>      rom_reset_order_override();
>  
>      /*
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 21/24] numa: use possible_cpus for not mapped CPUs check

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:15PM +0200, Igor Mammedov wrote:
> and remove corresponding part in numa.c that uses
> node_cpu bitmaps.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> It's one more less user of node_cpu bitmpas, following
> commit will remove the last user along with
> node_cpu itself.
> ---
>  hw/core/machine.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  numa.c            | 10 ----------
>  2 files changed, 58 insertions(+), 10 deletions(-)
> 
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index 217b4f4..4229054 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -19,6 +19,7 @@
>  #include "sysemu/sysemu.h"
>  #include "qemu/error-report.h"
>  #include "qemu/cutils.h"
> +#include "sysemu/numa.h"
>  
>  static char *machine_get_accel(Object *obj, Error **errp)
>  {
> @@ -648,9 +649,66 @@ bool machine_mem_merge(MachineState *machine)
>      return machine->mem_merge;
>  }
>  
> +static char *cpu_slot_to_string(const CPUArchId *cpu)
> +{
> +    GString *s = g_string_new(NULL);
> +    if (cpu->props.has_socket_id) {
> +        g_string_append_printf(s, "socket-id: %"PRId64, cpu->props.socket_id);
> +    }
> +    if (cpu->props.has_core_id) {
> +        if (s->len) {
> +            g_string_append_printf(s, ", ");
> +        }
> +        g_string_append_printf(s, "core-id: %"PRId64, cpu->props.core_id);
> +    }
> +    if (cpu->props.has_thread_id) {
> +        if (s->len) {
> +            g_string_append_printf(s, ", ");
> +        }
> +        g_string_append_printf(s, "thread-id: %"PRId64, cpu->props.thread_id);
> +    }
> +    return g_string_free(s, false);
> +}
> +
> +static void machine_numa_validate(MachineState *machine)
> +{
> +    int i;
> +    GString *s = g_string_new(NULL);
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(machine);
> +
> +    assert(nb_numa_nodes);

Can't this be

 if (!nb_numa_nodes)
   return;

instead? And then..

> +    for (i = 0; i < possible_cpus->len; i++) {
> +        const CPUArchId *cpu_slot = &possible_cpus->cpus[i];
> +
> +        /* at this point numa mappings are initilized by CLI options
> +         * or with default mappings so it's sufficient to list
> +         * all not yet mapped CPUs here */
> +        /* TODO: make it hard error in future */
> +        if (!cpu_slot->props.has_node_id) {
> +            char *cpu_str = cpu_slot_to_string(cpu_slot);
> +            g_string_append_printf(s, "%sCPU %d [%s]", s->len ? ", " : "", i,
> +                                   cpu_str);
> +            g_free(cpu_str);
> +        }
> +    }
> +    if (s->len) {
> +        error_report("warning: CPU(s) not present in any NUMA nodes: %s",
> +                     s->str);
> +        error_report("warning: All CPU(s) up to maxcpus should be described "
> +                     "in NUMA config, ability to start up with partial NUMA "
> +                     "mappings is obsoleted and will be removed in future");
> +    }
> +    g_string_free(s, true);
> +}
> +
>  void machine_run_board_init(MachineState *machine)
>  {
>      MachineClass *machine_class = MACHINE_GET_CLASS(machine);
> +
> +    if (nb_numa_nodes) {
> +        machine_numa_validate(machine);
> +    }

...this condition wouldn't be necessary.

>      machine_class->init(machine);
>  }
>  
> diff --git a/numa.c b/numa.c
> index f8c849d..d7aa662 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -293,16 +293,6 @@ static void validate_numa_cpus(void)
>          bitmap_or(seen_cpus, seen_cpus,
>                    numa_info[i].node_cpu, max_cpus);
>      }
> -
> -    if (!bitmap_full(seen_cpus, max_cpus)) {
> -        char *msg;
> -        bitmap_complement(seen_cpus, seen_cpus, max_cpus);
> -        msg = enumerate_cpus(seen_cpus, max_cpus);
> -        error_report("warning: CPU(s) not present in any NUMA nodes: %s", msg);
> -        error_report("warning: All CPU(s) up to maxcpus should be described "
> -                     "in NUMA config");
> -        g_free(msg);
> -    }
>      g_free(seen_cpus);
>  }
>  
> -- 
> 2.7.4
>

Otherwise
Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 20/24] machine: call machine init from wrapper

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:14PM +0200, Igor Mammedov wrote:
> add machine_run_board_init() wrapper that calls machine
> init for now but in follow up patches it will be used
> to run generic machine code that should run before
> machine init.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  include/hw/boards.h | 1 +
>  hw/core/machine.c   | 6 ++++++
>  vl.c                | 2 +-
>  3 files changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/include/hw/boards.h b/include/hw/boards.h
> index 1f518a1..0e74484 100644
> --- a/include/hw/boards.h
> +++ b/include/hw/boards.h
> @@ -32,6 +32,7 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
>  MachineClass *find_default_machine(void);
>  extern MachineState *current_machine;
>  
> +void machine_run_board_init(MachineState *machine);
>  bool machine_usb(MachineState *machine);
>  bool machine_kernel_irqchip_allowed(MachineState *machine);
>  bool machine_kernel_irqchip_required(MachineState *machine);
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index a63f17b..217b4f4 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -648,6 +648,12 @@ bool machine_mem_merge(MachineState *machine)
>      return machine->mem_merge;
>  }
>  
> +void machine_run_board_init(MachineState *machine)
> +{
> +    MachineClass *machine_class = MACHINE_GET_CLASS(machine);
> +    machine_class->init(machine);
> +}
> +
>  static void machine_class_finalize(ObjectClass *klass, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(klass);
> diff --git a/vl.c b/vl.c
> index fe4741d..ac46d6e 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4562,7 +4562,7 @@ int main(int argc, char **argv, char **envp)
>      current_machine->boot_order = boot_order;
>      current_machine->cpu_model = cpu_model;
>  
> -    machine_class->init(current_machine);
> +    machine_run_board_init(current_machine);
>  
>      realtime_init();
>  
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 22/24] numa: remove node_cpu bitmaps as they are no longer used

[Andrew Jones]

On Wed, May 03, 2017 at 02:57:16PM +0200, Igor Mammedov wrote:
> Postfactum "CPU(s) present in multiple NUMA nodes" check
> was the last user of node_cpu bitmaps, but it's not need
> as machine_set_cpu_numa_node() does the similar check at
> the time mapping is set for cpus (i.e. when -numa cpus=
> is parsed) and ensures that cpu can be mapped only to
> one node.
> 
> Remove duplicate check based on node_cpu bitmaps and
> since the last user is gone remove node_cpu as well,
> which completes internal transition from legacy bitmap
> based mapping storage to possible_cpus storage.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  include/sysemu/numa.h |  1 -
>  numa.c                | 42 ------------------------------------------
>  2 files changed, 43 deletions(-)
> 
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index 0b0a044..7f2d397 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -17,7 +17,6 @@ struct numa_addr_range {
>  
>  typedef struct node_info {
>      uint64_t node_mem;
> -    unsigned long *node_cpu;
>      struct HostMemoryBackend *node_memdev;
>      bool present;
>      QLIST_HEAD(, numa_addr_range) addr; /* List to store address ranges */
> diff --git a/numa.c b/numa.c
> index d7aa662..40e9f44 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -177,7 +177,6 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
>                         cpus->value, max_cpus);
>              return;
>          }
> -        bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
>          props = mc->cpu_index_to_instance_props(ms, cpus->value);
>          props.node_id = nodenr;
>          props.has_node_id = true;
> @@ -261,51 +260,12 @@ end:
>      return 0;
>  }
>  
> -static char *enumerate_cpus(unsigned long *cpus, int max_cpus)
> -{
> -    int cpu;
> -    bool first = true;
> -    GString *s = g_string_new(NULL);
> -
> -    for (cpu = find_first_bit(cpus, max_cpus);
> -        cpu < max_cpus;
> -        cpu = find_next_bit(cpus, max_cpus, cpu + 1)) {
> -        g_string_append_printf(s, "%s%d", first ? "" : " ", cpu);
> -        first = false;
> -    }
> -    return g_string_free(s, FALSE);
> -}
> -
> -static void validate_numa_cpus(void)
> -{
> -    int i;
> -    unsigned long *seen_cpus = bitmap_new(max_cpus);
> -
> -    for (i = 0; i < nb_numa_nodes; i++) {
> -        if (bitmap_intersects(seen_cpus, numa_info[i].node_cpu, max_cpus)) {
> -            bitmap_and(seen_cpus, seen_cpus,
> -                       numa_info[i].node_cpu, max_cpus);
> -            error_report("CPU(s) present in multiple NUMA nodes: %s",
> -                         enumerate_cpus(seen_cpus, max_cpus));
> -            g_free(seen_cpus);
> -            exit(EXIT_FAILURE);
> -        }
> -        bitmap_or(seen_cpus, seen_cpus,
> -                  numa_info[i].node_cpu, max_cpus);
> -    }
> -    g_free(seen_cpus);
> -}
> -
>  void parse_numa_opts(MachineState *ms)
>  {
>      int i;
>      const CPUArchIdList *possible_cpus;
>      MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
> -    for (i = 0; i < MAX_NODES; i++) {
> -        numa_info[i].node_cpu = bitmap_new(max_cpus);
> -    }
> -
>      if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, NULL)) {
>          exit(1);
>      }
> @@ -397,12 +357,10 @@ void parse_numa_opts(MachineState *ms)
>                  props = mc->cpu_index_to_instance_props(ms, i);
>                  props.has_node_id = true;
>  
> -                set_bit(i, numa_info[props.node_id].node_cpu);
>                  machine_set_cpu_numa_node(ms, &props, &error_fatal);
>              }
>          }
>  
> -        validate_numa_cpus();
>      } else {
>          numa_set_mem_node_id(0, ram_size, 0);
>      }
> -- 
> 2.7.4
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 03/24] hw/arm/virt: use machine->possible_cpus for storing possible topology info

[Igor Mammedov]

On Thu, 4 May 2017 11:38:22 +0200
Andrew Jones <drjones@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:56:57PM +0200, Igor Mammedov wrote:
> > for now precalculate and store mp_afinity in possible_cpus
> > as ARM cpus don't have socket/core/thread-id properties yet.
> > In follow patches possible_cpus will be used for storing
> > and setting NUMA node mapping and replace legacy bitmap
> > based numa_info[node_id].node_cpu/numa_get_node_for_cpu()
> > 
> > For the lack of better idea, this patch cannibalizes
> > possible_cpus.cpus[x].props.thread_id so that
> > *_cpu_index_to_props() callback could return addressable
> > by props CPU which will be used by machine_set_cpu_numa_node()
> > in follow up patches to assign a CPU to node. But
> > cannibalizing is fine for now as that thread_id isn't exposed
> > to users (no hotpluggable_cpus callback support for ARM yet)
> > and it will be used only internally until 'device_add cpu'
> > is supported where we can decide on which properties to use.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> > v2:
> >   (Drew)
> >     - discarding result of possible_cpu_arch_ids() makes
> >       call not obvious and is confusing. Instead assign
> >       possible_cpu_arch_ids() result to local var and use
> >       it instead of direct access to machine->possible_cpus
> >       field, as it's done in pc.c
> > ---
> >  hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 37 insertions(+), 3 deletions(-)
> > 
> > diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> > index 61ae437..e2c5626 100644
> > --- a/hw/arm/virt.c
> > +++ b/hw/arm/virt.c
> > @@ -1221,6 +1221,8 @@ static void machvirt_init(MachineState *machine)
> >  {
> >      VirtMachineState *vms = VIRT_MACHINE(machine);
> >      VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine);
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    const CPUArchIdList *possible_cpus;
> >      qemu_irq pic[NUM_IRQS];
> >      MemoryRegion *sysmem = get_system_memory();
> >      MemoryRegion *secure_sysmem = NULL;
> > @@ -1344,10 +1346,16 @@ static void machvirt_init(MachineState *machine)
> >          exit(1);
> >      }
> >  
> > -    for (n = 0; n < smp_cpus; n++) {
> > -        Object *cpuobj = object_new(typename);
> > +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> > +    for (n = 0; n < possible_cpus->len; n++) {
> > +        Object *cpuobj;
> >  
> > -        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
> > +        if (n >= smp_cpus) {
> > +            break;
> > +        }  
> 
> Why the break instead of just looping 'n < smp_cpus' like x86 does? Is
> there some future work where looping up to possible_cpus->len (aka
> max_cpus) is what we'll eventually want? If so, then we need a TODO
> comment here. If not, then we should clean this up by removing the break.
There is no plans to loop here upto possible_cpus->len.

It seemed to me more consistent/safer to use index limited
by possible_cpus->len to index possible_cpus->cpus[n] array
than index limited by smp_cpus though the former currently is
always less than smp_cpus.

If you prefer 'n < smp_cpus' loop, then I can switch to it.

> 
> Thanks,
> drew
> 
> > +
> > +        cpuobj = object_new(typename);
> > +        object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
> >                                  "mp-affinity", NULL);
> >  
> >          if (!vms->secure) {
> > @@ -1527,6 +1535,31 @@ static void virt_set_gic_version(Object *obj, const char *value, Error **errp)
> >      }
> >  }
> >  
> > +static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
> > +{
> > +    int n;
> > +    VirtMachineState *vms = VIRT_MACHINE(ms);
> > +
> > +    if (ms->possible_cpus) {
> > +        assert(ms->possible_cpus->len == max_cpus);
> > +        return ms->possible_cpus;
> > +    }
> > +
> > +    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
> > +                                  sizeof(CPUArchId) * max_cpus);
> > +    ms->possible_cpus->len = max_cpus;
> > +    for (n = 0; n < ms->possible_cpus->len; n++) {
> > +        ms->possible_cpus->cpus[n].arch_id =
> > +            virt_cpu_mp_affinity(vms, n);
> > +        ms->possible_cpus->cpus[n].props.has_thread_id = true;
> > +        ms->possible_cpus->cpus[n].props.thread_id = n;
> > +
> > +        /* TODO: add 'has_node/node' here to describe
> > +           to which node core belongs */
> > +    }
> > +    return ms->possible_cpus;
> > +}
> > +
> >  static void virt_machine_class_init(ObjectClass *oc, void *data)
> >  {
> >      MachineClass *mc = MACHINE_CLASS(oc);
> > @@ -1543,6 +1576,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
> >      mc->pci_allow_0_address = true;
> >      /* We know we will never create a pre-ARMv7 CPU which needs 1K pages */
> >      mc->minimum_page_bits = 12;
> > +    mc->possible_cpu_arch_ids = virt_possible_cpu_arch_ids;
> >  }
> >  
> >  static const TypeInfo virt_machine_info = {
> > -- 
> > 2.7.4
> >   
> 

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

On Thu, 4 May 2017 13:40:18 +0200
Andrew Jones <drjones@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> > Introduce machine_set_cpu_numa_node() helper that stores
> > node mapping for CPU in MachineState::possible_cpus.
> > CPU and node it belongs to is specified by 'props' argument.
> > 
> > Patch doesn't remove old way of storing mapping in
> > numa_info[X].node_cpu as removing it at the same time
> > makes patch rather big. Instead it just mirrors mapping
> > in possible_cpus and follow up per target patches will
> > switch to possible_cpus and numa_info[X].node_cpu will
> > be removed once there isn't any users left.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > ---
> >  include/hw/boards.h |  2 ++
> >  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  numa.c              |  8 +++++++
> >  3 files changed, 78 insertions(+)
> > 
> > diff --git a/include/hw/boards.h b/include/hw/boards.h
> > index 5d6af21..1f518a1 100644
> > --- a/include/hw/boards.h
> > +++ b/include/hw/boards.h
> > @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
> >  bool machine_mem_merge(MachineState *machine);
> >  void machine_register_compat_props(MachineState *machine);
> >  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp);
> >  
> >  /**
> >   * CPUArchId:
> > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > index ada9eea..a63f17b 100644
> > --- a/hw/core/machine.c
> > +++ b/hw/core/machine.c
> > @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
> >      return head;
> >  }
> >  
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp)
> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    bool match = false;
> > +    int i;
> > +
> > +    if (!mc->possible_cpu_arch_ids) {
> > +        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
> > +        return;
> > +    }
> > +
> > +    /* force board to initialize possible_cpus if it hasn't been done yet */
> > +    mc->possible_cpu_arch_ids(machine);
> > +
> > +    for (i = 0; i < machine->possible_cpus->len; i++) {
> > +        CPUArchId *slot = &machine->possible_cpus->cpus[i];
> > +
> > +        /* reject unsupported by board properties */
> > +        if (props->has_thread_id && !slot->props.has_thread_id) {
> > +            error_setg(errp, "thread-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_core_id && !slot->props.has_core_id) {
> > +            error_setg(errp, "core-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_socket_id && !slot->props.has_socket_id) {
> > +            error_setg(errp, "socket-id is not supported");
> > +            return;
> > +        }
> > +
> > +        /* skip slots with explicit mismatch */
> > +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> > +                continue;
> > +        }  
> 
> nit: above 3 if-conditions could be condensed into 1 compound condition
I'll merge them on the respin

> 
> > +
> > +        /* reject assignment if slot is already assigned, for compatibility
> > +         * of legacy cpu_index mapping with SPAPR core based mapping do not
> > +         * error out if cpu thread and matched core have the same node-id */
> > +        if (slot->props.has_node_id &&
> > +            slot->props.node_id != props->node_id) {
> > +            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
> > +                       slot->props.node_id);
> > +            return;
> > +        }
> > +
> > +        /* assign slot to node as it's matched '-numa cpu' key */
> > +        match = true;
> > +        slot->props.node_id = props->node_id;
> > +        slot->props.has_node_id = props->has_node_id;
> > +    }
> > +
> > +    if (!match) {
> > +        error_setg(errp, "no match found");
> > +    }
> > +}
> > +
> >  static void machine_class_init(ObjectClass *oc, void *data)
> >  {
> >      MachineClass *mc = MACHINE_CLASS(oc);
> > diff --git a/numa.c b/numa.c
> > index b517870..5ff1212 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >          exit(1);
> >      }
> >      for (cpus = node->cpus; cpus; cpus = cpus->next) {
> > +        CpuInstanceProperties props;
> >          if (cpus->value >= max_cpus) {
> >              error_setg(errp,
> >                         "CPU index (%" PRIu16 ")"
> > @@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >              return;
> >          }
> >          bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
> > +        props = mc->cpu_index_to_instance_props(ms, cpus->value);
> > +        props.node_id = nodenr;
> > +        props.has_node_id = true;
> > +        machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >      }
> >  
> >      if (node->has_mem && node->has_memdev) {
> > @@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
> >          if (i == nb_numa_nodes) {
> >              for (i = 0; i < max_cpus; i++) {
> >                  CpuInstanceProperties props;
> > +                /* fetch default mapping from board and enable it */
> >                  props = mc->cpu_index_to_instance_props(ms, i);
> > +                props.has_node_id = true;
> >  
> >                  set_bit(i, numa_info[props.node_id].node_cpu);
> > +                machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >              }
> >          }
> >  
> > -- 
> > 2.7.4
> >  
> 
> Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 21/24] numa: use possible_cpus for not mapped CPUs check

[Igor Mammedov]

On Thu, 4 May 2017 14:43:12 +0200
Andrew Jones <drjones@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:15PM +0200, Igor Mammedov wrote:
> > and remove corresponding part in numa.c that uses
> > node_cpu bitmaps.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > ---
> > It's one more less user of node_cpu bitmpas, following
> > commit will remove the last user along with
> > node_cpu itself.
> > ---
> >  hw/core/machine.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  numa.c            | 10 ----------
> >  2 files changed, 58 insertions(+), 10 deletions(-)
> > 
> > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > index 217b4f4..4229054 100644
> > --- a/hw/core/machine.c
> > +++ b/hw/core/machine.c
> > @@ -19,6 +19,7 @@
> >  #include "sysemu/sysemu.h"
> >  #include "qemu/error-report.h"
> >  #include "qemu/cutils.h"
> > +#include "sysemu/numa.h"
> >  
> >  static char *machine_get_accel(Object *obj, Error **errp)
> >  {
> > @@ -648,9 +649,66 @@ bool machine_mem_merge(MachineState *machine)
> >      return machine->mem_merge;
> >  }
> >  
> > +static char *cpu_slot_to_string(const CPUArchId *cpu)
> > +{
> > +    GString *s = g_string_new(NULL);
> > +    if (cpu->props.has_socket_id) {
> > +        g_string_append_printf(s, "socket-id: %"PRId64, cpu->props.socket_id);
> > +    }
> > +    if (cpu->props.has_core_id) {
> > +        if (s->len) {
> > +            g_string_append_printf(s, ", ");
> > +        }
> > +        g_string_append_printf(s, "core-id: %"PRId64, cpu->props.core_id);
> > +    }
> > +    if (cpu->props.has_thread_id) {
> > +        if (s->len) {
> > +            g_string_append_printf(s, ", ");
> > +        }
> > +        g_string_append_printf(s, "thread-id: %"PRId64, cpu->props.thread_id);
> > +    }
> > +    return g_string_free(s, false);
> > +}
> > +
> > +static void machine_numa_validate(MachineState *machine)
> > +{
> > +    int i;
> > +    GString *s = g_string_new(NULL);
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(machine);
> > +
> > +    assert(nb_numa_nodes);  
> 
> Can't this be
> 
>  if (!nb_numa_nodes)
>    return;
> 
> instead? And then..
> 
> > +    for (i = 0; i < possible_cpus->len; i++) {
> > +        const CPUArchId *cpu_slot = &possible_cpus->cpus[i];
> > +
> > +        /* at this point numa mappings are initilized by CLI options
> > +         * or with default mappings so it's sufficient to list
> > +         * all not yet mapped CPUs here */
> > +        /* TODO: make it hard error in future */
> > +        if (!cpu_slot->props.has_node_id) {
> > +            char *cpu_str = cpu_slot_to_string(cpu_slot);
> > +            g_string_append_printf(s, "%sCPU %d [%s]", s->len ? ", " : "", i,
> > +                                   cpu_str);
> > +            g_free(cpu_str);
> > +        }
> > +    }
> > +    if (s->len) {
> > +        error_report("warning: CPU(s) not present in any NUMA nodes: %s",
> > +                     s->str);
> > +        error_report("warning: All CPU(s) up to maxcpus should be described "
> > +                     "in NUMA config, ability to start up with partial NUMA "
> > +                     "mappings is obsoleted and will be removed in future");
> > +    }
> > +    g_string_free(s, true);
> > +}
> > +
> >  void machine_run_board_init(MachineState *machine)
> >  {
> >      MachineClass *machine_class = MACHINE_GET_CLASS(machine);
> > +
> > +    if (nb_numa_nodes) {
> > +        machine_numa_validate(machine);
> > +    }  
> 
> ...this condition wouldn't be necessary.
I prefer to keep such conditions in the caller as during review/reading
it allows to exclude jumping into called function if condition is not
relevant for considered usecase.

> 
> >      machine_class->init(machine);
> >  }
> >  
> > diff --git a/numa.c b/numa.c
> > index f8c849d..d7aa662 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -293,16 +293,6 @@ static void validate_numa_cpus(void)
> >          bitmap_or(seen_cpus, seen_cpus,
> >                    numa_info[i].node_cpu, max_cpus);
> >      }
> > -
> > -    if (!bitmap_full(seen_cpus, max_cpus)) {
> > -        char *msg;
> > -        bitmap_complement(seen_cpus, seen_cpus, max_cpus);
> > -        msg = enumerate_cpus(seen_cpus, max_cpus);
> > -        error_report("warning: CPU(s) not present in any NUMA nodes: %s", msg);
> > -        error_report("warning: All CPU(s) up to maxcpus should be described "
> > -                     "in NUMA config");
> > -        g_free(msg);
> > -    }
> >      g_free(seen_cpus);
> >  }
> >  
> > -- 
> > 2.7.4
> >  
> 
> Otherwise
> Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [PATCH v2 03/24] hw/arm/virt: use machine->possible_cpus for storing possible topology info

[Andrew Jones]

On Thu, May 04, 2017 at 02:55:09PM +0200, Igor Mammedov wrote:
> On Thu, 4 May 2017 11:38:22 +0200
> Andrew Jones <drjones@redhat.com> wrote:
> 
> > On Wed, May 03, 2017 at 02:56:57PM +0200, Igor Mammedov wrote:
> > > for now precalculate and store mp_afinity in possible_cpus
> > > as ARM cpus don't have socket/core/thread-id properties yet.
> > > In follow patches possible_cpus will be used for storing
> > > and setting NUMA node mapping and replace legacy bitmap
> > > based numa_info[node_id].node_cpu/numa_get_node_for_cpu()
> > > 
> > > For the lack of better idea, this patch cannibalizes
> > > possible_cpus.cpus[x].props.thread_id so that
> > > *_cpu_index_to_props() callback could return addressable
> > > by props CPU which will be used by machine_set_cpu_numa_node()
> > > in follow up patches to assign a CPU to node. But
> > > cannibalizing is fine for now as that thread_id isn't exposed
> > > to users (no hotpluggable_cpus callback support for ARM yet)
> > > and it will be used only internally until 'device_add cpu'
> > > is supported where we can decide on which properties to use.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > > v2:
> > >   (Drew)
> > >     - discarding result of possible_cpu_arch_ids() makes
> > >       call not obvious and is confusing. Instead assign
> > >       possible_cpu_arch_ids() result to local var and use
> > >       it instead of direct access to machine->possible_cpus
> > >       field, as it's done in pc.c
> > > ---
> > >  hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 37 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> > > index 61ae437..e2c5626 100644
> > > --- a/hw/arm/virt.c
> > > +++ b/hw/arm/virt.c
> > > @@ -1221,6 +1221,8 @@ static void machvirt_init(MachineState *machine)
> > >  {
> > >      VirtMachineState *vms = VIRT_MACHINE(machine);
> > >      VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine);
> > > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > > +    const CPUArchIdList *possible_cpus;
> > >      qemu_irq pic[NUM_IRQS];
> > >      MemoryRegion *sysmem = get_system_memory();
> > >      MemoryRegion *secure_sysmem = NULL;
> > > @@ -1344,10 +1346,16 @@ static void machvirt_init(MachineState *machine)
> > >          exit(1);
> > >      }
> > >  
> > > -    for (n = 0; n < smp_cpus; n++) {
> > > -        Object *cpuobj = object_new(typename);
> > > +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> > > +    for (n = 0; n < possible_cpus->len; n++) {
> > > +        Object *cpuobj;
> > >  
> > > -        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
> > > +        if (n >= smp_cpus) {
> > > +            break;
> > > +        }  
> > 
> > Why the break instead of just looping 'n < smp_cpus' like x86 does? Is
> > there some future work where looping up to possible_cpus->len (aka
> > max_cpus) is what we'll eventually want? If so, then we need a TODO
> > comment here. If not, then we should clean this up by removing the break.
> There is no plans to loop here upto possible_cpus->len.
> 
> It seemed to me more consistent/safer to use index limited
> by possible_cpus->len to index possible_cpus->cpus[n] array
> than index limited by smp_cpus though the former currently is
> always less than smp_cpus.
         ^ greater than or equal to
> 
> If you prefer 'n < smp_cpus' loop, then I can switch to it.

I just don't like the 'if (n >= smp_cpus) { break; }' - the whole thing
would look much nicer without it. And, if there's a valid concern that
possible_cpus->len can be < smp_cpus, then we should check it in x86
too. Anyway we can check both conditions in the 'for', which would
look a bit more pleasing to me...

 for (n = 0; n < possible_cpus->len && n < smp_cpus; n++) {
     Object *cpuobj = object_new(typename);
     object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
                             "mp-affinity", NULL);
     ...

All that said, it's just a nit in the end, so

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [PATCH v2 01/24] tests: add CPUs to numa node mapping test

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:55PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

Queued on numa-next. Thanks!

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 02/24] hw/arm/virt: extract mp-affinity calculation in separate function

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:56PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Queued on machine-next, thanks!

But I would still like to get an Acked-by from Peter Maydell on
the ARM patches, before including them on a pull request.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 00/24] numa: add '-numa cpu' option

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:56:54PM +0200, Igor Mammedov wrote:
[...]
> Igor Mammedov (24):
>   tests: add CPUs to numa node mapping test
>   hw/arm/virt: extract mp-affinity calculation in separate function
>   hw/arm/virt: use machine->possible_cpus for storing possible topology
>     info
>   hw/arm/virt: explicitly allocate cpu_index for cpus

Patches 1-4 (above) queued on numa-next. Thanks!

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 03/24] hw/arm/virt: use machine->possible_cpus for storing possible topology info

[Igor Mammedov]

On Thu, 4 May 2017 15:16:02 +0200
Andrew Jones <drjones@redhat.com> wrote:

> On Thu, May 04, 2017 at 02:55:09PM +0200, Igor Mammedov wrote:
> > On Thu, 4 May 2017 11:38:22 +0200
> > Andrew Jones <drjones@redhat.com> wrote:
> >   
> > > On Wed, May 03, 2017 at 02:56:57PM +0200, Igor Mammedov wrote:  
> > > > for now precalculate and store mp_afinity in possible_cpus
> > > > as ARM cpus don't have socket/core/thread-id properties yet.
> > > > In follow patches possible_cpus will be used for storing
> > > > and setting NUMA node mapping and replace legacy bitmap
> > > > based numa_info[node_id].node_cpu/numa_get_node_for_cpu()
> > > > 
> > > > For the lack of better idea, this patch cannibalizes
> > > > possible_cpus.cpus[x].props.thread_id so that
> > > > *_cpu_index_to_props() callback could return addressable
> > > > by props CPU which will be used by machine_set_cpu_numa_node()
> > > > in follow up patches to assign a CPU to node. But
> > > > cannibalizing is fine for now as that thread_id isn't exposed
> > > > to users (no hotpluggable_cpus callback support for ARM yet)
> > > > and it will be used only internally until 'device_add cpu'
> > > > is supported where we can decide on which properties to use.
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > ---
> > > > v2:
> > > >   (Drew)
> > > >     - discarding result of possible_cpu_arch_ids() makes
> > > >       call not obvious and is confusing. Instead assign
> > > >       possible_cpu_arch_ids() result to local var and use
> > > >       it instead of direct access to machine->possible_cpus
> > > >       field, as it's done in pc.c
> > > > ---
> > > >  hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 37 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> > > > index 61ae437..e2c5626 100644
> > > > --- a/hw/arm/virt.c
> > > > +++ b/hw/arm/virt.c
> > > > @@ -1221,6 +1221,8 @@ static void machvirt_init(MachineState *machine)
> > > >  {
> > > >      VirtMachineState *vms = VIRT_MACHINE(machine);
> > > >      VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine);
> > > > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > > > +    const CPUArchIdList *possible_cpus;
> > > >      qemu_irq pic[NUM_IRQS];
> > > >      MemoryRegion *sysmem = get_system_memory();
> > > >      MemoryRegion *secure_sysmem = NULL;
> > > > @@ -1344,10 +1346,16 @@ static void machvirt_init(MachineState *machine)
> > > >          exit(1);
> > > >      }
> > > >  
> > > > -    for (n = 0; n < smp_cpus; n++) {
> > > > -        Object *cpuobj = object_new(typename);
> > > > +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> > > > +    for (n = 0; n < possible_cpus->len; n++) {
> > > > +        Object *cpuobj;
> > > >  
> > > > -        object_property_set_int(cpuobj, virt_cpu_mp_affinity(vms, n),
> > > > +        if (n >= smp_cpus) {
> > > > +            break;
> > > > +        }    
> > > 
> > > Why the break instead of just looping 'n < smp_cpus' like x86 does? Is
> > > there some future work where looping up to possible_cpus->len (aka
> > > max_cpus) is what we'll eventually want? If so, then we need a TODO
> > > comment here. If not, then we should clean this up by removing the break.  
> > There is no plans to loop here upto possible_cpus->len.
> > 
> > It seemed to me more consistent/safer to use index limited
> > by possible_cpus->len to index possible_cpus->cpus[n] array
> > than index limited by smp_cpus though the former currently is
> > always less than smp_cpus.  
>          ^ greater than or equal to
> > 
> > If you prefer 'n < smp_cpus' loop, then I can switch to it.  
> 
> I just don't like the 'if (n >= smp_cpus) { break; }' - the whole thing
> would look much nicer without it. And, if there's a valid concern that
> possible_cpus->len can be < smp_cpus, then we should check it in x86
> too. Anyway we can check both conditions in the 'for', which would
> look a bit more pleasing to me...
> 
>  for (n = 0; n < possible_cpus->len && n < smp_cpus; n++) {
nice, I'll do it this way on respin.

>      Object *cpuobj = object_new(typename);
>      object_property_set_int(cpuobj, possible_cpus->cpus[n].arch_id,
>                              "mp-affinity", NULL);
>      ...
> 
> All that said, it's just a nit in the end, so
> 
> Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [PATCH v2 00/24] numa: add '-numa cpu' option

[Igor Mammedov]

On Thu, 4 May 2017 10:41:55 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:56:54PM +0200, Igor Mammedov wrote:
> [...]
> > Igor Mammedov (24):
> >   tests: add CPUs to numa node mapping test
> >   hw/arm/virt: extract mp-affinity calculation in separate function
> >   hw/arm/virt: use machine->possible_cpus for storing possible topology
> >     info
> >   hw/arm/virt: explicitly allocate cpu_index for cpus  
> 
> Patches 1-4 (above) queued on numa-next. Thanks!
> 

Thanks, I'll rebase on top of it and re-post remaining patches.

Re: [Qemu-devel] [PATCH v2 06/24] spapr: add node-id property to sPAPR core

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6302 bytes --]

On Wed, May 03, 2017 at 06:12:17PM +0200, Igor Mammedov wrote:
> On Wed, 3 May 2017 11:46:44 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Wed, May 03, 2017 at 02:57:00PM +0200, Igor Mammedov wrote:
> > > it will allow switching from cpu_index to core based numa
> > > mapping in follow up patches.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>  
> > 
> > Have you considered adding this to TYPE_CPU instead of
> > duplicating the same code on multiple architectures/boards?
> it was in TYPE_CPU at RFC time, but it adds public node-id
> property to every CPU.
> So I've rewrote it they way it would affect only necessary CPUs,
> 
> it might be possible to generalize node_id sanity checks at
> pre_plug time into a common wrapper, I can do it on top so
> it would be visible what it's generalized (or an extra patch on
> respin)

Note that the spapr cpu core type here is *not* a descendent of
TYPE_CPU (the threads under it are).

> 
> > 
> > > ---
> > >  include/hw/ppc/spapr_cpu_core.h |  1 +
> > >  include/qom/cpu.h               |  2 ++
> > >  hw/ppc/spapr.c                  | 17 +++++++++++++++++
> > >  hw/ppc/spapr_cpu_core.c         | 11 ++++++++---
> > >  4 files changed, 28 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
> > > index 3c35665..93051e9 100644
> > > --- a/include/hw/ppc/spapr_cpu_core.h
> > > +++ b/include/hw/ppc/spapr_cpu_core.h
> > > @@ -27,6 +27,7 @@ typedef struct sPAPRCPUCore {
> > >  
> > >      /*< public >*/
> > >      void *threads;
> > > +    int node_id;
> > >  } sPAPRCPUCore;
> > >  
> > >  typedef struct sPAPRCPUCoreClass {
> > > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > > index 5d10359..55214ce 100644
> > > --- a/include/qom/cpu.h
> > > +++ b/include/qom/cpu.h
> > > @@ -258,6 +258,8 @@ typedef void (*run_on_cpu_func)(CPUState *cpu, run_on_cpu_data data);
> > >  
> > >  struct qemu_work_item;
> > >  
> > > +#define CPU_UNSET_NUMA_NODE_ID -1
> > > +
> > >  /**
> > >   * CPUState:
> > >   * @cpu_index: CPU index (informative).
> > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > index 33405a0..7f58ee4 100644
> > > --- a/hw/ppc/spapr.c
> > > +++ b/hw/ppc/spapr.c
> > > @@ -2824,9 +2824,11 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > >      MachineClass *mc = MACHINE_GET_CLASS(hotplug_dev);
> > >      Error *local_err = NULL;
> > >      CPUCore *cc = CPU_CORE(dev);
> > > +    sPAPRCPUCore *sc = SPAPR_CPU_CORE(dev);
> > >      char *base_core_type = spapr_get_cpu_core_type(machine->cpu_model);
> > >      const char *type = object_get_typename(OBJECT(dev));
> > >      CPUArchId *core_slot;
> > > +    int node_id;
> > >      int index;
> > >  
> > >      if (dev->hotplugged && !mc->has_hotpluggable_cpus) {
> > > @@ -2861,6 +2863,21 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > >          goto out;
> > >      }
> > >  
> > > +    node_id = numa_get_node_for_cpu(cc->core_id);
> > > +    if (node_id == nb_numa_nodes) {
> > > +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> > > +         * keep it this way for now but in future we probably should
> > > +         * refuse to start up with incomplete numa mapping */
> > > +        node_id = 0;
> > > +    }
> > > +    if (sc->node_id == CPU_UNSET_NUMA_NODE_ID) {
> > > +        sc->node_id = node_id;
> > > +    } else if (sc->node_id != node_id) {
> > > +        error_setg(&local_err, "node-id %d must match numa node specified"
> > > +            "with -numa option for cpu-index %d", sc->node_id, cc->core_id);
> > > +        goto out;
> > > +    }
> > > +
> > >  out:
> > >      g_free(base_core_type);
> > >      error_propagate(errp, local_err);
> > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > index 4389ef4..9de7a56 100644
> > > --- a/hw/ppc/spapr_cpu_core.c
> > > +++ b/hw/ppc/spapr_cpu_core.c
> > > @@ -176,7 +176,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> > >      const char *typename = object_class_get_name(scc->cpu_class);
> > >      size_t size = object_type_get_instance_size(typename);
> > >      Error *local_err = NULL;
> > > -    int core_node_id = numa_get_node_for_cpu(cc->core_id);;
> > >      void *obj;
> > >      int i, j;
> > >  
> > > @@ -194,10 +193,10 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> > >  
> > >          /* Set NUMA node for the added CPUs  */
> > >          node_id = numa_get_node_for_cpu(cs->cpu_index);
> > > -        if (node_id != core_node_id) {
> > > +        if (node_id != sc->node_id) {
> > >              error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
> > >                  " on CPU[core-id: %d, node-id: %d], node-id must be the same",
> > > -                 node_id, cs->cpu_index, cc->core_id, core_node_id);
> > > +                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
> > >              goto err;
> > >          }
> > >          if (node_id < nb_numa_nodes) {
> > > @@ -263,6 +262,11 @@ static const char *spapr_core_models[] = {
> > >      "POWER9_v1.0",
> > >  };
> > >  
> > > +static Property spapr_cpu_core_properties[] = {
> > > +    DEFINE_PROP_INT32("node-id", sPAPRCPUCore, node_id, CPU_UNSET_NUMA_NODE_ID),
> > > +    DEFINE_PROP_END_OF_LIST()
> > > +};
> > > +
> > >  void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> > >  {
> > >      DeviceClass *dc = DEVICE_CLASS(oc);
> > > @@ -270,6 +274,7 @@ void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> > >  
> > >      dc->realize = spapr_cpu_core_realize;
> > >      dc->unrealize = spapr_cpu_core_unrealizefn;
> > > +    dc->props = spapr_cpu_core_properties;
> > >      scc->cpu_class = cpu_class_by_name(TYPE_POWERPC_CPU, data);
> > >      g_assert(scc->cpu_class);
> > >  }
> > 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 09/24] numa: add check that board supports cpu_index to node mapping

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

On Wed, May 03, 2017 at 02:57:03PM +0200, Igor Mammedov wrote:
> Default node mapping initialization already checks that board
> supports cpu_index to node mapping and refuses to start if
> it's not supported. Do the same for explicitly provided
> mapping "-numa node,cpus=..."
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

Makes sense in the series at present, although it might be obsoleted
by other suggested changes.

> ---
>  numa.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/numa.c b/numa.c
> index ab1661d..b517870 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -140,10 +140,12 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp)
>      return -1;
>  }
>  
> -static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
> +static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> +                            QemuOpts *opts, Error **errp)
>  {
>      uint16_t nodenr;
>      uint16List *cpus = NULL;
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
>  
>      if (node->has_nodeid) {
>          nodenr = node->nodeid;
> @@ -162,6 +164,10 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
>          return;
>      }
>  
> +    if (!mc->cpu_index_to_instance_props) {
> +        error_report("CPUs to NUMA node mapping isn't supported");
> +        exit(1);
> +    }
>      for (cpus = node->cpus; cpus; cpus = cpus->next) {
>          if (cpus->value >= max_cpus) {
>              error_setg(errp,
> @@ -215,6 +221,7 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
>  static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>  {
>      NumaOptions *object = NULL;
> +    MachineState *ms = opaque;
>      Error *err = NULL;
>  
>      {
> @@ -229,7 +236,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>  
>      switch (object->type) {
>      case NUMA_OPTIONS_TYPE_NODE:
> -        numa_node_parse(&object->u.node, opts, &err);
> +        numa_node_parse(ms, &object->u.node, opts, &err);
>          if (err) {
>              goto end;
>          }
> @@ -303,7 +310,7 @@ void parse_numa_opts(MachineState *ms)
>          numa_info[i].node_cpu = bitmap_new(max_cpus);
>      }
>  
> -    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, NULL, NULL)) {
> +    if (qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, NULL)) {
>          exit(1);
>      }
>  

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> wrappers should make access to [has]node_id fields more readable
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

Correct, though I'm not sure it actually simplifies things that much.
Maybe more in future patches, though.

> ---
>  follow up patches will use this wrappers
>  v2:
>     - add wrappers (Drew)
> ---
>  include/sysemu/numa.h | 10 ++++++++++
>  numa.c                |  2 +-
>  2 files changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index 46ea6c7..98d01e6 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -35,4 +35,14 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp);
>  /* on success returns node index in numa_info,
>   * on failure returns nb_numa_nodes */
>  int numa_get_node_for_cpu(int idx);
> +
> +static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
> +{
> +    return possible_cpus->cpus[idx].props.has_node_id;
> +}
> +
> +static inline int numa_node_id(const CPUArchIdList *possible_cpus, int idx)
> +{
> +    return possible_cpus->cpus[idx].props.node_id;
> +}
>  #endif
> diff --git a/numa.c b/numa.c
> index c7e3e0a..872ee0d 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -394,7 +394,7 @@ void parse_numa_opts(MachineState *ms)
>  
>          possible_cpus = mc->possible_cpu_arch_ids(ms);
>          for (i = 0; i < possible_cpus->len; i++) {
> -            if (possible_cpus->cpus[i].props.has_node_id) {
> +            if (numa_has_node_id(possible_cpus, i)) {
>                  break;
>              }
>          }

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 14/24] spapr: get numa node mapping from possible_cpus instead of numa_get_node_for_cpu()

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2753 bytes --]

On Wed, May 03, 2017 at 02:57:08PM +0200, Igor Mammedov wrote:
> it's safe to remove thread node_id != core node_id error
> branch as machine_set_cpu_numa_node() also does mismatch
> check and is called even before any CPU is created.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Acked-by: David Gibson <david@gibson.dropbear.id.au>

> ---
>  hw/ppc/spapr.c          |  4 ++--
>  hw/ppc/spapr_cpu_core.c | 14 ++------------
>  2 files changed, 4 insertions(+), 14 deletions(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 7f58ee4..bcb91e7 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -2863,8 +2863,8 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
>          goto out;
>      }
>  
> -    node_id = numa_get_node_for_cpu(cc->core_id);
> -    if (node_id == nb_numa_nodes) {
> +    node_id = core_slot->props.node_id;
> +    if (!core_slot->props.has_node_id) {
>          /* by default CPUState::numa_node was 0 if it's not set via CLI
>           * keep it this way for now but in future we probably should
>           * refuse to start up with incomplete numa mapping */
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index 9de7a56..a17ea07 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -181,7 +181,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
>  
>      sc->threads = g_malloc0(size * cc->nr_threads);
>      for (i = 0; i < cc->nr_threads; i++) {
> -        int node_id;
>          char id[32];
>          CPUState *cs;
>  
> @@ -191,17 +190,8 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
>          cs = CPU(obj);
>          cs->cpu_index = cc->core_id + i;
>  
> -        /* Set NUMA node for the added CPUs  */
> -        node_id = numa_get_node_for_cpu(cs->cpu_index);
> -        if (node_id != sc->node_id) {
> -            error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
> -                " on CPU[core-id: %d, node-id: %d], node-id must be the same",
> -                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
> -            goto err;
> -        }
> -        if (node_id < nb_numa_nodes) {
> -            cs->numa_node = node_id;
> -        }
> +        /* Set NUMA node for the threads belonged to core  */
> +        cs->numa_node = sc->node_id;
>  
>          snprintf(id, sizeof(id), "thread[%d]", i);
>          object_property_add_child(OBJECT(sc), id, obj, &local_err);

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 17/24] tests: numa: add case for QMP command query-cpus

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

On Wed, May 03, 2017 at 02:57:11PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

> ---
> v2:
>  * fix checkpatch error, move { to newline
> ---
>  tests/numa-test.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 45 insertions(+)
> 
> diff --git a/tests/numa-test.c b/tests/numa-test.c
> index f5da0c8..2722687 100644
> --- a/tests/numa-test.c
> +++ b/tests/numa-test.c
> @@ -87,6 +87,50 @@ static void test_mon_partial(const void *data)
>      g_free(cli);
>  }
>  
> +static QList *get_cpus(QDict **resp)
> +{
> +    *resp = qmp("{ 'execute': 'query-cpus' }");
> +    g_assert(*resp);
> +    g_assert(qdict_haskey(*resp, "return"));
> +    return  qdict_get_qlist(*resp, "return");
> +}
> +
> +static void test_query_cpus(const void *data)
> +{
> +    char *cli;
> +    QDict *resp;
> +    QList *cpus;
> +    const QObject *e;
> +
> +    cli = make_cli(data, "-smp 8 -numa node,cpus=0-3 -numa node,cpus=4-7");
> +    qtest_start(cli);
> +    cpus = get_cpus(&resp);
> +    g_assert(cpus);
> +
> +    while ((e = qlist_pop(cpus))) {
> +        QDict *cpu, *props;
> +        int64_t cpu_idx, node;
> +
> +        cpu = qobject_to_qdict(e);
> +        g_assert(qdict_haskey(cpu, "CPU"));
> +        g_assert(qdict_haskey(cpu, "props"));
> +
> +        cpu_idx = qdict_get_int(cpu, "CPU");
> +        props = qdict_get_qdict(cpu, "props");
> +        g_assert(qdict_haskey(props, "node-id"));
> +        node = qdict_get_int(props, "node-id");
> +        if (cpu_idx >= 0 && cpu_idx < 4) {
> +            g_assert_cmpint(node, ==, 0);
> +        } else {
> +            g_assert_cmpint(node, ==, 1);
> +        }
> +    }
> +
> +    QDECREF(resp);
> +    qtest_end();
> +    g_free(cli);
> +}
> +
>  int main(int argc, char **argv)
>  {
>      const char *args = NULL;
> @@ -101,6 +145,7 @@ int main(int argc, char **argv)
>      qtest_add_data_func("/numa/mon/default", args, test_mon_default);
>      qtest_add_data_func("/numa/mon/cpus/explicit", args, test_mon_explicit);
>      qtest_add_data_func("/numa/mon/cpus/partial", args, test_mon_partial);
> +    qtest_add_data_func("/numa/qmp/cpus/query-cpus", args, test_query_cpus);
>  
>      return g_test_run();
>  }

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 20/24] machine: call machine init from wrapper

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

On Wed, May 03, 2017 at 02:57:14PM +0200, Igor Mammedov wrote:
> add machine_run_board_init() wrapper that calls machine
> init for now but in follow up patches it will be used
> to run generic machine code that should run before
> machine init.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

> ---
>  include/hw/boards.h | 1 +
>  hw/core/machine.c   | 6 ++++++
>  vl.c                | 2 +-
>  3 files changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/include/hw/boards.h b/include/hw/boards.h
> index 1f518a1..0e74484 100644
> --- a/include/hw/boards.h
> +++ b/include/hw/boards.h
> @@ -32,6 +32,7 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,
>  MachineClass *find_default_machine(void);
>  extern MachineState *current_machine;
>  
> +void machine_run_board_init(MachineState *machine);
>  bool machine_usb(MachineState *machine);
>  bool machine_kernel_irqchip_allowed(MachineState *machine);
>  bool machine_kernel_irqchip_required(MachineState *machine);
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index a63f17b..217b4f4 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -648,6 +648,12 @@ bool machine_mem_merge(MachineState *machine)
>      return machine->mem_merge;
>  }
>  
> +void machine_run_board_init(MachineState *machine)
> +{
> +    MachineClass *machine_class = MACHINE_GET_CLASS(machine);
> +    machine_class->init(machine);
> +}
> +
>  static void machine_class_finalize(ObjectClass *klass, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(klass);
> diff --git a/vl.c b/vl.c
> index fe4741d..ac46d6e 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4562,7 +4562,7 @@ int main(int argc, char **argv, char **envp)
>      current_machine->boot_order = boot_order;
>      current_machine->cpu_model = cpu_model;
>  
> -    machine_class->init(current_machine);
> +    machine_run_board_init(current_machine);
>  
>      realtime_init();
>  

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 05/24] numa: move source of default CPUs to NUMA node mapping into boards

[Igor Mammedov]

On Thu, 4 May 2017 17:32:13 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, May 03, 2017 at 11:42:40AM -0300, Eduardo Habkost wrote:
> > On Wed, May 03, 2017 at 02:56:59PM +0200, Igor Mammedov wrote:  
> > > Originally CPU threads were by default assigned in
> > > round-robin fashion. However it was causing issues in
> > > guest since CPU threads from the same socket/core could
> > > be placed on different NUMA nodes.
> > > Commit fb43b73b (pc: fix default VCPU to NUMA node mapping)
> > > fixed it by grouping threads within a socket on the same node
> > > introducing cpu_index_to_socket_id() callback and commit
> > > 20bb648d (spapr: Fix default NUMA node allocation for threads)
> > > reused callback to fix similar issues for SPAPR machine
> > > even though socket doesn't make much sense there.
> > > 
> > > As result QEMU ended up having 3 default distribution rules
> > > used by 3 targets /virt-arm, spapr, pc/.
> > > 
> > > In effort of moving NUMA mapping for CPUs into possible_cpus,
> > > generalize default mapping in numa.c by making boards decide
> > > on default mapping and let them explicitly tell generic
> > > numa code to which node a CPU thread belongs to by replacing
> > > cpu_index_to_socket_id() with @cpu_index_to_instance_props()
> > > which provides default node_id assigned by board to specified
> > > cpu_index.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> > 
> > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> > 
> > Just two extra comments below:
> > 
> > [...]  
> > > +static CpuInstanceProperties
> > > +virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > > +{
> > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > > +
> > > +    assert(cpu_index < possible_cpus->len);
> > > +    return possible_cpus->cpus[cpu_index].props;;
> > > +}
> > > +  
> > [...]  
> > > +static CpuInstanceProperties
> > > +pc_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > >  {
> > > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > > +
> > > +    assert(cpu_index < possible_cpus->len);
> > > +    return possible_cpus->cpus[cpu_index].props;;
> > >  }  
> > 
> > The fact that these two implementations look exactly the same
> > made me wonder:
> > 
> > 1) Why this isn't the default implementation;
> > 2) Why exactly spapr needs a different implementation.
> > 
> > Then I noticed that there's nothing in the common machine code
> > that specifies that possible_cpus->cpus[] is indexed by
> > cpu_index. This means it is indeed safer to require each machine
> > to provide its own cpu_index_to_props implementation than having
> > a default implementation that can unexpectedly break (e.g. if
> > granularity at possible_cpus is not at VCPU/thread level).
> > 
> > I would still like to have an abstraction that wouldn't require
> > writing machine-specific code (e.g. cpu_index ranges to
> > possible_cpus like David suggested), but that's for a follow-up
> > series.  
> 
> Yeah, that similarity bothered me to, but like you I realised the
> problem is that spapr simply doesn't have the same granularity of
> information as x86 and ARM - there's only one entry per core for PAPR
> instead of one per thread.
> 
> So, we do need a machine specific mapping of cpu_index to location
> properties, which is what the callback is for.
> 
> It does occur to me that another way of accomplishing that would be
> for possible_cpu_arch_ids() to create a cpu_index->props mapping as a
> simple array ofpointers, in addition to the list of possiblee props
> structures.
> 
> Not sure if that would end up looking better or not.
I'd like to remove foo_cpu_index_to_props() callbacks in future,
to do so we would need to obsolete and remove cpu_index based
'-numa node,cpus='. Lets give a year for new interface to settle
in and then remove old interface.

so I'd rather do not extend possible_cpus with more complex structure
and related mgmt routines.



> 
> > [...]  
> > > +static CpuInstanceProperties
> > > +spapr_cpu_index_to_props(MachineState *machine, unsigned cpu_index)
> > >  {
> > > +    CPUArchId *core_slot;
> > > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > > +
> > > +    /* make sure possible_cpu are intialized */
> > > +    mc->possible_cpu_arch_ids(machine);
> > > +    core_slot = spapr_find_cpu_slot(machine, cpu_index, NULL);
> > > +    assert(core_slot);
> > > +    return core_slot->props;
> > >  }  
> > 
> > If you need to submit v3, maybe a comment here explaining why
> > spapr needs a different cpu_index_to_props implementation would
> > be helpful. I took a while to figure it out.
> >   
> 

Re: [Qemu-devel] [PATCH v2 06/24] spapr: add node-id property to sPAPR core

[Igor Mammedov]

On Fri, 5 May 2017 02:49:52 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, May 03, 2017 at 06:12:17PM +0200, Igor Mammedov wrote:
> > On Wed, 3 May 2017 11:46:44 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> >   
> > > On Wed, May 03, 2017 at 02:57:00PM +0200, Igor Mammedov wrote:  
> > > > it will allow switching from cpu_index to core based numa
> > > > mapping in follow up patches.
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>    
> > > 
> > > Have you considered adding this to TYPE_CPU instead of
> > > duplicating the same code on multiple architectures/boards?  
> > it was in TYPE_CPU at RFC time, but it adds public node-id
> > property to every CPU.
> > So I've rewrote it they way it would affect only necessary CPUs,
> > 
> > it might be possible to generalize node_id sanity checks at
> > pre_plug time into a common wrapper, I can do it on top so
> > it would be visible what it's generalized (or an extra patch on
> > respin)  
> 
> Note that the spapr cpu core type here is *not* a descendent of
> TYPE_CPU (the threads under it are).
yep, what I was saying is that numa checks in pre_plug callbacks
are very similar and it might be possible to replaces them with
common function.

> >   
> > >   
> > > > ---
> > > >  include/hw/ppc/spapr_cpu_core.h |  1 +
> > > >  include/qom/cpu.h               |  2 ++
> > > >  hw/ppc/spapr.c                  | 17 +++++++++++++++++
> > > >  hw/ppc/spapr_cpu_core.c         | 11 ++++++++---
> > > >  4 files changed, 28 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
> > > > index 3c35665..93051e9 100644
> > > > --- a/include/hw/ppc/spapr_cpu_core.h
> > > > +++ b/include/hw/ppc/spapr_cpu_core.h
> > > > @@ -27,6 +27,7 @@ typedef struct sPAPRCPUCore {
> > > >  
> > > >      /*< public >*/
> > > >      void *threads;
> > > > +    int node_id;
> > > >  } sPAPRCPUCore;
> > > >  
> > > >  typedef struct sPAPRCPUCoreClass {
> > > > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > > > index 5d10359..55214ce 100644
> > > > --- a/include/qom/cpu.h
> > > > +++ b/include/qom/cpu.h
> > > > @@ -258,6 +258,8 @@ typedef void (*run_on_cpu_func)(CPUState *cpu, run_on_cpu_data data);
> > > >  
> > > >  struct qemu_work_item;
> > > >  
> > > > +#define CPU_UNSET_NUMA_NODE_ID -1
> > > > +
> > > >  /**
> > > >   * CPUState:
> > > >   * @cpu_index: CPU index (informative).
> > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > index 33405a0..7f58ee4 100644
> > > > --- a/hw/ppc/spapr.c
> > > > +++ b/hw/ppc/spapr.c
> > > > @@ -2824,9 +2824,11 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > > >      MachineClass *mc = MACHINE_GET_CLASS(hotplug_dev);
> > > >      Error *local_err = NULL;
> > > >      CPUCore *cc = CPU_CORE(dev);
> > > > +    sPAPRCPUCore *sc = SPAPR_CPU_CORE(dev);
> > > >      char *base_core_type = spapr_get_cpu_core_type(machine->cpu_model);
> > > >      const char *type = object_get_typename(OBJECT(dev));
> > > >      CPUArchId *core_slot;
> > > > +    int node_id;
> > > >      int index;
> > > >  
> > > >      if (dev->hotplugged && !mc->has_hotpluggable_cpus) {
> > > > @@ -2861,6 +2863,21 @@ static void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > > >          goto out;
> > > >      }
> > > >  
> > > > +    node_id = numa_get_node_for_cpu(cc->core_id);
> > > > +    if (node_id == nb_numa_nodes) {
> > > > +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> > > > +         * keep it this way for now but in future we probably should
> > > > +         * refuse to start up with incomplete numa mapping */
> > > > +        node_id = 0;
> > > > +    }
> > > > +    if (sc->node_id == CPU_UNSET_NUMA_NODE_ID) {
> > > > +        sc->node_id = node_id;
> > > > +    } else if (sc->node_id != node_id) {
> > > > +        error_setg(&local_err, "node-id %d must match numa node specified"
> > > > +            "with -numa option for cpu-index %d", sc->node_id, cc->core_id);
> > > > +        goto out;
> > > > +    }
> > > > +
> > > >  out:
> > > >      g_free(base_core_type);
> > > >      error_propagate(errp, local_err);
> > > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > > index 4389ef4..9de7a56 100644
> > > > --- a/hw/ppc/spapr_cpu_core.c
> > > > +++ b/hw/ppc/spapr_cpu_core.c
> > > > @@ -176,7 +176,6 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> > > >      const char *typename = object_class_get_name(scc->cpu_class);
> > > >      size_t size = object_type_get_instance_size(typename);
> > > >      Error *local_err = NULL;
> > > > -    int core_node_id = numa_get_node_for_cpu(cc->core_id);;
> > > >      void *obj;
> > > >      int i, j;
> > > >  
> > > > @@ -194,10 +193,10 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
> > > >  
> > > >          /* Set NUMA node for the added CPUs  */
> > > >          node_id = numa_get_node_for_cpu(cs->cpu_index);
> > > > -        if (node_id != core_node_id) {
> > > > +        if (node_id != sc->node_id) {
> > > >              error_setg(&local_err, "Invalid node-id=%d of thread[cpu-index: %d]"
> > > >                  " on CPU[core-id: %d, node-id: %d], node-id must be the same",
> > > > -                 node_id, cs->cpu_index, cc->core_id, core_node_id);
> > > > +                 node_id, cs->cpu_index, cc->core_id, sc->node_id);
> > > >              goto err;
> > > >          }
> > > >          if (node_id < nb_numa_nodes) {
> > > > @@ -263,6 +262,11 @@ static const char *spapr_core_models[] = {
> > > >      "POWER9_v1.0",
> > > >  };
> > > >  
> > > > +static Property spapr_cpu_core_properties[] = {
> > > > +    DEFINE_PROP_INT32("node-id", sPAPRCPUCore, node_id, CPU_UNSET_NUMA_NODE_ID),
> > > > +    DEFINE_PROP_END_OF_LIST()
> > > > +};
> > > > +
> > > >  void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> > > >  {
> > > >      DeviceClass *dc = DEVICE_CLASS(oc);
> > > > @@ -270,6 +274,7 @@ void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
> > > >  
> > > >      dc->realize = spapr_cpu_core_realize;
> > > >      dc->unrealize = spapr_cpu_core_unrealizefn;
> > > > +    dc->props = spapr_cpu_core_properties;
> > > >      scc->cpu_class = cpu_class_by_name(TYPE_POWERPC_CPU, data);
> > > >      g_assert(scc->cpu_class);
> > > >  }  
> > >   
> >   
> 

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Igor Mammedov]

On Fri, 5 May 2017 11:45:22 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> > wrappers should make access to [has]node_id fields more readable
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> 
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> 
> Correct, though I'm not sure it actually simplifies things that much.
> Maybe more in future patches, though.
that's what Drew insisted on, and even though I prefer other way around
I won't stall series arguing about styling issues,
so here this patch goes.

> 
> > ---
> >  follow up patches will use this wrappers
> >  v2:
> >     - add wrappers (Drew)
> > ---
> >  include/sysemu/numa.h | 10 ++++++++++
> >  numa.c                |  2 +-
> >  2 files changed, 11 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> > index 46ea6c7..98d01e6 100644
> > --- a/include/sysemu/numa.h
> > +++ b/include/sysemu/numa.h
> > @@ -35,4 +35,14 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp);
> >  /* on success returns node index in numa_info,
> >   * on failure returns nb_numa_nodes */
> >  int numa_get_node_for_cpu(int idx);
> > +
> > +static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
> > +{
> > +    return possible_cpus->cpus[idx].props.has_node_id;
> > +}
> > +
> > +static inline int numa_node_id(const CPUArchIdList *possible_cpus, int idx)
> > +{
> > +    return possible_cpus->cpus[idx].props.node_id;
> > +}
> >  #endif
> > diff --git a/numa.c b/numa.c
> > index c7e3e0a..872ee0d 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -394,7 +394,7 @@ void parse_numa_opts(MachineState *ms)
> >  
> >          possible_cpus = mc->possible_cpu_arch_ids(ms);
> >          for (i = 0; i < possible_cpus->len; i++) {
> > -            if (possible_cpus->cpus[i].props.has_node_id) {
> > +            if (numa_has_node_id(possible_cpus, i)) {
> >                  break;
> >              }
> >          }  
> 

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Andrew Jones]

On Fri, May 05, 2017 at 10:09:18AM +0200, Igor Mammedov wrote:
> On Fri, 5 May 2017 11:45:22 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> > > wrappers should make access to [has]node_id fields more readable
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> > 
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > 
> > Correct, though I'm not sure it actually simplifies things that much.
> > Maybe more in future patches, though.
> that's what Drew insisted on, and even though I prefer other way around
> I won't stall series arguing about styling issues,
> so here this patch goes.

My argument in the last review of this series was that references like

 machine->possible_cpus->cpus[cs->cpu_index].props.has_node_id
and
 machine->possible_cpus->cpus[cs->cpu_index].props.node_id

are quite long, and only differ by 'has_', making it tough to
easily recognize. But, if nobody, but me, sees value in changing
them to

 numa_has_node_id(machine->possible_cpus, cs->cpu_index)
and
 numa_node_id(machine->possible_cpus, cs->cpu_index)

then I won't insist.

Thanks,
drew

> 
> > 
> > > ---
> > >  follow up patches will use this wrappers
> > >  v2:
> > >     - add wrappers (Drew)
> > > ---
> > >  include/sysemu/numa.h | 10 ++++++++++
> > >  numa.c                |  2 +-
> > >  2 files changed, 11 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> > > index 46ea6c7..98d01e6 100644
> > > --- a/include/sysemu/numa.h
> > > +++ b/include/sysemu/numa.h
> > > @@ -35,4 +35,14 @@ uint32_t numa_get_node(ram_addr_t addr, Error **errp);
> > >  /* on success returns node index in numa_info,
> > >   * on failure returns nb_numa_nodes */
> > >  int numa_get_node_for_cpu(int idx);
> > > +
> > > +static inline bool numa_has_node_id(const CPUArchIdList *possible_cpus, int idx)
> > > +{
> > > +    return possible_cpus->cpus[idx].props.has_node_id;
> > > +}
> > > +
> > > +static inline int numa_node_id(const CPUArchIdList *possible_cpus, int idx)
> > > +{
> > > +    return possible_cpus->cpus[idx].props.node_id;
> > > +}
> > >  #endif
> > > diff --git a/numa.c b/numa.c
> > > index c7e3e0a..872ee0d 100644
> > > --- a/numa.c
> > > +++ b/numa.c
> > > @@ -394,7 +394,7 @@ void parse_numa_opts(MachineState *ms)
> > >  
> > >          possible_cpus = mc->possible_cpu_arch_ids(ms);
> > >          for (i = 0; i < possible_cpus->len; i++) {
> > > -            if (possible_cpus->cpus[i].props.has_node_id) {
> > > +            if (numa_has_node_id(possible_cpus, i)) {
> > >                  break;
> > >              }
> > >          }  
> > 
> 
> 

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

On Thu, 4 May 2017 13:40:18 +0200
Andrew Jones <drjones@redhat.com> wrote:

[...]
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp)
> > +{
[...]

> > +        }
> > +
> > +        /* skip slots with explicit mismatch */
> > +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> > +                continue;
> > +        }  
> 
> nit: above 3 if-conditions could be condensed into 1 compound condition
this reduces number of lines but result is a bit ugly due to 80chr/line limit:

        /* skip slots with explicit mismatch */                                  
        if ((props->has_thread_id &&                                             
             props->thread_id != slot->props.thread_id) ||                       
            (props->has_core_id && props->core_id != slot->props.core_id) ||     
            (props->has_socket_id && props->socket_id != slot->props.socket_id)  
        ) {                                                                      
                continue;                                                        
        }

do you still want the change?

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Andrew Jones]

On Fri, May 05, 2017 at 01:28:26PM +0200, Igor Mammedov wrote:
> On Thu, 4 May 2017 13:40:18 +0200
> Andrew Jones <drjones@redhat.com> wrote:
> 
> [...]
> > > +void machine_set_cpu_numa_node(MachineState *machine,
> > > +                               CpuInstanceProperties *props, Error **errp)
> > > +{
> [...]
> 
> > > +        }
> > > +
> > > +        /* skip slots with explicit mismatch */
> > > +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> > > +                continue;
> > > +        }
> > > +
> > > +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> > > +                continue;
> > > +        }
> > > +
> > > +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> > > +                continue;
> > > +        }  
> > 
> > nit: above 3 if-conditions could be condensed into 1 compound condition
> this reduces number of lines but result is a bit ugly due to 80chr/line limit:
> 
>         /* skip slots with explicit mismatch */                                  
>         if ((props->has_thread_id &&                                             
>              props->thread_id != slot->props.thread_id) ||                       
>             (props->has_core_id && props->core_id != slot->props.core_id) ||     
>             (props->has_socket_id && props->socket_id != slot->props.socket_id)  
>         ) {                                                                      
>                 continue;                                                        
>         }
> 
> do you still want the change?

If only we could kill the 80 char limit! I don't feel strongly about any
of this stuff, and you've already got my R-b either way.

Thanks,
drew

> 
> 
> 
> 

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

On Wed, 3 May 2017 12:20:22 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> > Introduce machine_set_cpu_numa_node() helper that stores
> > node mapping for CPU in MachineState::possible_cpus.
> > CPU and node it belongs to is specified by 'props' argument.
> > 
> > Patch doesn't remove old way of storing mapping in
> > numa_info[X].node_cpu as removing it at the same time
> > makes patch rather big. Instead it just mirrors mapping
> > in possible_cpus and follow up per target patches will
> > switch to possible_cpus and numa_info[X].node_cpu will
> > be removed once there isn't any users left.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > ---
> >  include/hw/boards.h |  2 ++
> >  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  numa.c              |  8 +++++++
> >  3 files changed, 78 insertions(+)
> > 
> > diff --git a/include/hw/boards.h b/include/hw/boards.h
> > index 5d6af21..1f518a1 100644
> > --- a/include/hw/boards.h
> > +++ b/include/hw/boards.h
> > @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
> >  bool machine_mem_merge(MachineState *machine);
> >  void machine_register_compat_props(MachineState *machine);
> >  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp);
> >  
> >  /**
> >   * CPUArchId:
> > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > index ada9eea..a63f17b 100644
> > --- a/hw/core/machine.c
> > +++ b/hw/core/machine.c
> > @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
> >      return head;
> >  }
> >  
> > +void machine_set_cpu_numa_node(MachineState *machine,
> > +                               CpuInstanceProperties *props, Error **errp)  
> 
> As the semantics of this function aren't trivial, it would be
> nice to have a comment explaining what exactly this function do.
> 
> e.g.:
> * make it clear that it could affect multiple CPU slots;
> * make it clear what does it mean to have props->has_node_id=false as
>   argument (is it really valid?);
> * make it clear that it will refuse to change an existing mapping.
Will be following comment sufficient?

+/**
+ * machine_set_cpu_numa_node:
+ * @machine: machine object to modify
+ * @props: specifies which cpu objects to assign to
+ *         numa node specified by @props.node_id
+ * @errp: if an error occurs, a pointer to an area to store the error
+ *
+ * Associate NUMA node specified by @props.node_id with cpu slots that
+ * match socket/core/thread-ids specified by @props. It's recommended to use
+ * query-hotpluggable-cpus.props values to specify affected cpu slots,
+ * which would lead to exact 1:1 mapping of cpu slots to NUMA node.
+ *
+ * However for CLI convenience it's possible to pass in subset of properties,
+ * which would affect all cpu slots that match it.
+ * Ex for pc machine:
+ *    -smp 4,cores=2,sockets=2 -numa node,nodeid=0 -numa node,nodeid=1 \
+ *    -numa cpu,node-id=0,socket_id=0 \
+ *    -numa cpu,node-id=1,socket_id=1
+ * will assign all child cores of socket 0 to node 0 and
+ * of socket 1 to node 1.
+ *
+ * Empty subset is disallowed and function will return with error in this case.
+ */
 void machine_set_cpu_numa_node(MachineState *machine,
                                CpuInstanceProperties *props, Error **errp)
 {
@@ -400,6 +423,8 @@ void machine_set_cpu_numa_node(MachineState *machine,
         error_setg(errp, "mapping of CPUs to NUMA node is not supported");
         return;
     }
+    /* disabling numa mapping is not supported, forbid it */
+    assert(props->has_node_id);

> 
> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    bool match = false;
> > +    int i;
> > +
> > +    if (!mc->possible_cpu_arch_ids) {
> > +        error_setg(errp, "mapping of CPUs to NUMA node is not supported");
> > +        return;
> > +    }
> > +
> > +    /* force board to initialize possible_cpus if it hasn't been done yet */
> > +    mc->possible_cpu_arch_ids(machine);
> > +
> > +    for (i = 0; i < machine->possible_cpus->len; i++) {
> > +        CPUArchId *slot = &machine->possible_cpus->cpus[i];
> > +
> > +        /* reject unsupported by board properties */
> > +        if (props->has_thread_id && !slot->props.has_thread_id) {
> > +            error_setg(errp, "thread-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_core_id && !slot->props.has_core_id) {
> > +            error_setg(errp, "core-id is not supported");
> > +            return;
> > +        }
> > +
> > +        if (props->has_socket_id && !slot->props.has_socket_id) {
> > +            error_setg(errp, "socket-id is not supported");
> > +            return;
> > +        }
> > +
> > +        /* skip slots with explicit mismatch */
> > +        if (props->has_thread_id && props->thread_id != slot->props.thread_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_core_id && props->core_id != slot->props.core_id) {
> > +                continue;
> > +        }
> > +
> > +        if (props->has_socket_id && props->socket_id != slot->props.socket_id) {
> > +                continue;
> > +        }
> > +
> > +        /* reject assignment if slot is already assigned, for compatibility
> > +         * of legacy cpu_index mapping with SPAPR core based mapping do not
> > +         * error out if cpu thread and matched core have the same node-id */
> > +        if (slot->props.has_node_id &&
> > +            slot->props.node_id != props->node_id) {
> > +            error_setg(errp, "CPU is already assigned to node-id: %" PRId64,
> > +                       slot->props.node_id);
> > +            return;
> > +        }
> > +
> > +        /* assign slot to node as it's matched '-numa cpu' key */
> > +        match = true;
> > +        slot->props.node_id = props->node_id;
> > +        slot->props.has_node_id = props->has_node_id;  
> 
> Is it really valid to have has_node_id=false? Maybe an
> assert(props->has_node_id) at the beginning of the function would
> be useful.
> 
> > +    }
> > +
> > +    if (!match) {
> > +        error_setg(errp, "no match found");
> > +    }
> > +}
> > +
> >  static void machine_class_init(ObjectClass *oc, void *data)
> >  {
> >      MachineClass *mc = MACHINE_CLASS(oc);
> > diff --git a/numa.c b/numa.c
> > index b517870..5ff1212 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -169,6 +169,7 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >          exit(1);
> >      }
> >      for (cpus = node->cpus; cpus; cpus = cpus->next) {
> > +        CpuInstanceProperties props;
> >          if (cpus->value >= max_cpus) {
> >              error_setg(errp,
> >                         "CPU index (%" PRIu16 ")"
> > @@ -177,6 +178,10 @@ static void numa_node_parse(MachineState *ms, NumaNodeOptions *node,
> >              return;
> >          }
> >          bitmap_set(numa_info[nodenr].node_cpu, cpus->value, 1);
> > +        props = mc->cpu_index_to_instance_props(ms, cpus->value);
> > +        props.node_id = nodenr;
> > +        props.has_node_id = true;
> > +        machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >      }
> >  
> >      if (node->has_mem && node->has_memdev) {
> > @@ -393,9 +398,12 @@ void parse_numa_opts(MachineState *ms)
> >          if (i == nb_numa_nodes) {
> >              for (i = 0; i < max_cpus; i++) {
> >                  CpuInstanceProperties props;
> > +                /* fetch default mapping from board and enable it */
> >                  props = mc->cpu_index_to_instance_props(ms, i);
> > +                props.has_node_id = true;
> >  
> >                  set_bit(i, numa_info[props.node_id].node_cpu);
> > +                machine_set_cpu_numa_node(ms, &props, &error_fatal);
> >              }
> >          }
> >  
> > -- 
> > 2.7.4
> >   
> 

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Eduardo Habkost]

On Fri, May 05, 2017 at 02:16:48PM +0200, Igor Mammedov wrote:
> On Wed, 3 May 2017 12:20:22 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> > > Introduce machine_set_cpu_numa_node() helper that stores
> > > node mapping for CPU in MachineState::possible_cpus.
> > > CPU and node it belongs to is specified by 'props' argument.
> > > 
> > > Patch doesn't remove old way of storing mapping in
> > > numa_info[X].node_cpu as removing it at the same time
> > > makes patch rather big. Instead it just mirrors mapping
> > > in possible_cpus and follow up per target patches will
> > > switch to possible_cpus and numa_info[X].node_cpu will
> > > be removed once there isn't any users left.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > > ---
> > >  include/hw/boards.h |  2 ++
> > >  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  numa.c              |  8 +++++++
> > >  3 files changed, 78 insertions(+)
> > > 
> > > diff --git a/include/hw/boards.h b/include/hw/boards.h
> > > index 5d6af21..1f518a1 100644
> > > --- a/include/hw/boards.h
> > > +++ b/include/hw/boards.h
> > > @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
> > >  bool machine_mem_merge(MachineState *machine);
> > >  void machine_register_compat_props(MachineState *machine);
> > >  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> > > +void machine_set_cpu_numa_node(MachineState *machine,
> > > +                               CpuInstanceProperties *props, Error **errp);
> > >  
> > >  /**
> > >   * CPUArchId:
> > > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > > index ada9eea..a63f17b 100644
> > > --- a/hw/core/machine.c
> > > +++ b/hw/core/machine.c
> > > @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
> > >      return head;
> > >  }
> > >  
> > > +void machine_set_cpu_numa_node(MachineState *machine,
> > > +                               CpuInstanceProperties *props, Error **errp)  
> > 
> > As the semantics of this function aren't trivial, it would be
> > nice to have a comment explaining what exactly this function do.
> > 
> > e.g.:
> > * make it clear that it could affect multiple CPU slots;
> > * make it clear what does it mean to have props->has_node_id=false as
> >   argument (is it really valid?);
> > * make it clear that it will refuse to change an existing mapping.
> Will be following comment sufficient?
> 
> +/**
> + * machine_set_cpu_numa_node:
> + * @machine: machine object to modify
> + * @props: specifies which cpu objects to assign to
> + *         numa node specified by @props.node_id
> + * @errp: if an error occurs, a pointer to an area to store the error
> + *
> + * Associate NUMA node specified by @props.node_id with cpu slots that
> + * match socket/core/thread-ids specified by @props. It's recommended to use
> + * query-hotpluggable-cpus.props values to specify affected cpu slots,
> + * which would lead to exact 1:1 mapping of cpu slots to NUMA node.
> + *
> + * However for CLI convenience it's possible to pass in subset of properties,
> + * which would affect all cpu slots that match it.
> + * Ex for pc machine:
> + *    -smp 4,cores=2,sockets=2 -numa node,nodeid=0 -numa node,nodeid=1 \
> + *    -numa cpu,node-id=0,socket_id=0 \
> + *    -numa cpu,node-id=1,socket_id=1
> + * will assign all child cores of socket 0 to node 0 and
> + * of socket 1 to node 1.
> + *
> + * Empty subset is disallowed and function will return with error in this case.
> + */
>  void machine_set_cpu_numa_node(MachineState *machine,
>                                 CpuInstanceProperties *props, Error **errp)

Sounds good to me.

While at it, we could make 'props' const, as it's not going to be
touched by the function.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Eduardo Habkost]

On Fri, May 05, 2017 at 11:06:02AM +0200, Andrew Jones wrote:
> On Fri, May 05, 2017 at 10:09:18AM +0200, Igor Mammedov wrote:
> > On Fri, 5 May 2017 11:45:22 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> > 
> > > On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> > > > wrappers should make access to [has]node_id fields more readable
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> > > 
> > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > > 
> > > Correct, though I'm not sure it actually simplifies things that much.
> > > Maybe more in future patches, though.
> > that's what Drew insisted on, and even though I prefer other way around
> > I won't stall series arguing about styling issues,
> > so here this patch goes.
> 
> My argument in the last review of this series was that references like
> 
>  machine->possible_cpus->cpus[cs->cpu_index].props.has_node_id
> and
>  machine->possible_cpus->cpus[cs->cpu_index].props.node_id
> 
> are quite long, and only differ by 'has_', making it tough to
> easily recognize.

If expression length is a problem, we can just use an extra
variable:

 CPUArchId *slot = &machine->possible_cpus->cpus[cs->cpu_index];
 if (slot->props.has_node_id && slot->props.node_id == FOO) ...


>                   But, if nobody, but me, sees value in changing
> them to
> 
>  numa_has_node_id(machine->possible_cpus, cs->cpu_index)
> and
>  numa_node_id(machine->possible_cpus, cs->cpu_index)
> 
> then I won't insist.

I don't mind that much, but I still prefer the version without
the wrappers.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 10/24] numa: mirror cpu to node mapping in MachineState::possible_cpus

[Igor Mammedov]

On Fri, 5 May 2017 14:04:15 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, May 05, 2017 at 02:16:48PM +0200, Igor Mammedov wrote:
> > On Wed, 3 May 2017 12:20:22 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Wed, May 03, 2017 at 02:57:04PM +0200, Igor Mammedov wrote:
> > > > Introduce machine_set_cpu_numa_node() helper that stores
> > > > node mapping for CPU in MachineState::possible_cpus.
> > > > CPU and node it belongs to is specified by 'props' argument.
> > > > 
> > > > Patch doesn't remove old way of storing mapping in
> > > > numa_info[X].node_cpu as removing it at the same time
> > > > makes patch rather big. Instead it just mirrors mapping
> > > > in possible_cpus and follow up per target patches will
> > > > switch to possible_cpus and numa_info[X].node_cpu will
> > > > be removed once there isn't any users left.
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > > > ---
> > > >  include/hw/boards.h |  2 ++
> > > >  hw/core/machine.c   | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >  numa.c              |  8 +++++++
> > > >  3 files changed, 78 insertions(+)
> > > > 
> > > > diff --git a/include/hw/boards.h b/include/hw/boards.h
> > > > index 5d6af21..1f518a1 100644
> > > > --- a/include/hw/boards.h
> > > > +++ b/include/hw/boards.h
> > > > @@ -42,6 +42,8 @@ bool machine_dump_guest_core(MachineState *machine);
> > > >  bool machine_mem_merge(MachineState *machine);
> > > >  void machine_register_compat_props(MachineState *machine);
> > > >  HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine);
> > > > +void machine_set_cpu_numa_node(MachineState *machine,
> > > > +                               CpuInstanceProperties *props, Error **errp);
> > > >  
> > > >  /**
> > > >   * CPUArchId:
> > > > diff --git a/hw/core/machine.c b/hw/core/machine.c
> > > > index ada9eea..a63f17b 100644
> > > > --- a/hw/core/machine.c
> > > > +++ b/hw/core/machine.c
> > > > @@ -388,6 +388,74 @@ HotpluggableCPUList *machine_query_hotpluggable_cpus(MachineState *machine)
> > > >      return head;
> > > >  }
> > > >  
> > > > +void machine_set_cpu_numa_node(MachineState *machine,
> > > > +                               CpuInstanceProperties *props, Error **errp)  
> > > 
> > > As the semantics of this function aren't trivial, it would be
> > > nice to have a comment explaining what exactly this function do.
> > > 
> > > e.g.:
> > > * make it clear that it could affect multiple CPU slots;
> > > * make it clear what does it mean to have props->has_node_id=false as
> > >   argument (is it really valid?);
> > > * make it clear that it will refuse to change an existing mapping.
> > Will be following comment sufficient?
> > 
> > +/**
> > + * machine_set_cpu_numa_node:
> > + * @machine: machine object to modify
> > + * @props: specifies which cpu objects to assign to
> > + *         numa node specified by @props.node_id
> > + * @errp: if an error occurs, a pointer to an area to store the error
> > + *
> > + * Associate NUMA node specified by @props.node_id with cpu slots that
> > + * match socket/core/thread-ids specified by @props. It's recommended to use
> > + * query-hotpluggable-cpus.props values to specify affected cpu slots,
> > + * which would lead to exact 1:1 mapping of cpu slots to NUMA node.
> > + *
> > + * However for CLI convenience it's possible to pass in subset of properties,
> > + * which would affect all cpu slots that match it.
> > + * Ex for pc machine:
> > + *    -smp 4,cores=2,sockets=2 -numa node,nodeid=0 -numa node,nodeid=1 \
> > + *    -numa cpu,node-id=0,socket_id=0 \
> > + *    -numa cpu,node-id=1,socket_id=1
> > + * will assign all child cores of socket 0 to node 0 and
> > + * of socket 1 to node 1.
> > + *
> > + * Empty subset is disallowed and function will return with error in this case.
> > + */
> >  void machine_set_cpu_numa_node(MachineState *machine,
> >                                 CpuInstanceProperties *props, Error **errp)
> 
> Sounds good to me.
> 
> While at it, we could make 'props' const, as it's not going to be
> touched by the function.
sure

Re: [Qemu-devel] [PATCH v2 12/24] numa: add numa_[has_]node_id() wrappers

[Igor Mammedov]

On Fri, 5 May 2017 14:12:26 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, May 05, 2017 at 11:06:02AM +0200, Andrew Jones wrote:
> > On Fri, May 05, 2017 at 10:09:18AM +0200, Igor Mammedov wrote:
> > > On Fri, 5 May 2017 11:45:22 +1000
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > > 
> > > > On Wed, May 03, 2017 at 02:57:06PM +0200, Igor Mammedov wrote:
> > > > > wrappers should make access to [has]node_id fields more readable
> > > > > 
> > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>  
> > > > 
> > > > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > > > 
> > > > Correct, though I'm not sure it actually simplifies things that much.
> > > > Maybe more in future patches, though.
> > > that's what Drew insisted on, and even though I prefer other way around
> > > I won't stall series arguing about styling issues,
> > > so here this patch goes.
> > 
> > My argument in the last review of this series was that references like
> > 
> >  machine->possible_cpus->cpus[cs->cpu_index].props.has_node_id
> > and
> >  machine->possible_cpus->cpus[cs->cpu_index].props.node_id
> > 
> > are quite long, and only differ by 'has_', making it tough to
> > easily recognize.
> 
> If expression length is a problem, we can just use an extra
> variable:
> 
>  CPUArchId *slot = &machine->possible_cpus->cpus[cs->cpu_index];
>  if (slot->props.has_node_id && slot->props.node_id == FOO) ...
> 
> 
> >                   But, if nobody, but me, sees value in changing
> > them to
> > 
> >  numa_has_node_id(machine->possible_cpus, cs->cpu_index)
> > and
> >  numa_node_id(machine->possible_cpus, cs->cpu_index)
> > 
> > then I won't insist.
> 
> I don't mind that much, but I still prefer the version without
> the wrappers.
considering 3vs1 for wrapper less variant, I'll return it back.
but respin will have to wait till Tuesday due to holiday over here.

Re: [Qemu-devel] [PATCH v2 07/24] pc: add node-id property to CPU

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:01PM +0200, Igor Mammedov wrote:
> it will allow switching from cpu_index to property based
> numa mapping in follow up patches.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  hw/i386/pc.c      | 17 +++++++++++++++++
>  target/i386/cpu.c |  1 +
>  2 files changed, 18 insertions(+)
> 
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 9cec0c1..4a4fb1c 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1893,6 +1893,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
>                              DeviceState *dev, Error **errp)
>  {
>      int idx;
> +    int node_id;
>      CPUState *cs;
>      CPUArchId *cpu_slot;
>      X86CPUTopoInfo topo;
> @@ -1982,6 +1983,22 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
>  
>      cs = CPU(cpu);
>      cs->cpu_index = idx;
> +
> +    node_id = numa_get_node_for_cpu(cs->cpu_index);
> +    if (node_id == nb_numa_nodes) {
> +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> +         * keep it this way for now but in future we probably should
> +         * refuse to start up with incomplete numa mapping */
> +        node_id = 0;
> +    }
> +    if (cs->numa_node == CPU_UNSET_NUMA_NODE_ID) {
> +        cs->numa_node = node_id;

I suggest moving the above lines to cpu_common_realizefn(), and
deleting the code in numa_post_machine_init().

If you want to do this in v3 of this series, or as a follow-up
patch, it's up to you.


> +    } else if (cs->numa_node != node_id) {
> +            error_setg(errp, "node-id %d must match numa node specified"
> +                "with -numa option for cpu-index %d",
> +                cs->numa_node, cs->cpu_index);
> +            return;
> +    }
>  }
>  
>  static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 13c0985..007a5bd 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -3982,6 +3982,7 @@ static Property x86_cpu_properties[] = {
>      DEFINE_PROP_INT32("core-id", X86CPU, core_id, -1),
>      DEFINE_PROP_INT32("socket-id", X86CPU, socket_id, -1),
>  #endif
> +    DEFINE_PROP_INT32("node-id", CPUState, numa_node, CPU_UNSET_NUMA_NODE_ID),

This changes the default from 0 to CPU_UNSET_NUMA_NODE_ID, but it
looks like there's no existing i386 or PC code that ever looks at
CPUState::numa_node. Maybe worth mentioning in the commit
message.

>      DEFINE_PROP_BOOL("pmu", X86CPU, enable_pmu, false),
>      { .name  = "hv-spinlocks", .info  = &qdev_prop_spinlocks },
>      DEFINE_PROP_BOOL("hv-relaxed", X86CPU, hyperv_relaxed_timing, false),
> -- 
> 2.7.4
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 7781 bytes --]

On Wed, May 03, 2017 at 02:57:17PM +0200, Igor Mammedov wrote:
> legacy cpu to node mapping is using cpu index values to map
> VCPU to node with help of '-numa node,nodeid=node,cpus=x[-y]'
> option. However cpu index is internal concept and QEMU users
> have to guess /reimplement qemu's logic/ to map it to
> a concrete cpu socket/core/thread to make sane CPUs
> placement across numa nodes.
> 
> This patch allows to map cpu objects to numa nodes using
> the same properties as used for cpus with -device/device_add
> (socket-id/core-id/thread-id/node-id).
> 
> At present valid properties/values to address CPUs could be
> fetched using hotpluggable-cpus monitor/qmp command, it will
> require user to start qemu twice when creating domain to fetch
> possible CPUs for a machine type/-smp layout first and
> then the second time with numa explicit mapping for actual
> usage. The first step results could be saved and reused to
> set/change mapping later as far as machine type/-smp stays
> the same.
> 
> Proposed impl. supports exact and wildcard matching to
> simplify CLI and allow to set mapping for a specific cpu
> or group of cpu objects specified by matched properties.
> 
> For example:
> 
>    # exact mapping x86
>    -numa cpu,node-id=x,socket-id=y,core-id=z,thread-id=n
> 
>    # exact mapping SPAPR
>    -numa cpu,node-id=x,core-id=y
> 
>    # wildcard mapping, all cpu objects that match socket-id=y
>    # are mapped to node-id=x
>    -numa cpu,node-id=x,socket-id=y
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   - use new NumaCpuOptions instead of CpuInstanceProperties in
>     NumaOptions, so that in future we could decouple both
>     if needed. (Eduardo Habkost <ehabkost@redhat.com>)
>   - clarify effect of NumaCpuOptions.node-id in qapi-schema.json
> ---
>  numa.c           | 25 +++++++++++++++++++++++++
>  qapi-schema.json | 21 +++++++++++++++++++--
>  qemu-options.hx  | 23 ++++++++++++++++++++++-
>  3 files changed, 66 insertions(+), 3 deletions(-)
> 
> diff --git a/numa.c b/numa.c
> index 40e9f44..61521f5 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>      NumaOptions *object = NULL;
>      MachineState *ms = opaque;
>      Error *err = NULL;
> +    CpuInstanceProperties cpu;
>  
>      {
>          Visitor *v = opts_visitor_new(opts);
> @@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
>          }
>          nb_numa_nodes++;
>          break;
> +    case NUMA_OPTIONS_TYPE_CPU:
> +        if (!object->u.cpu.has_node_id) {
> +            error_setg(&err, "Missing mandatory node-id property");
> +            goto end;
> +        }
> +        if (!numa_info[object->u.cpu.node_id].present) {
> +            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
> +                "defined with -numa node,nodeid=ID before it's used with "
> +                "-numa cpu,node-id=ID", object->u.cpu.node_id);
> +            goto end;
> +        }
> +
> +        memset(&cpu, 0, sizeof(cpu));
> +        cpu.has_node_id = object->u.cpu.has_node_id;
> +        cpu.node_id = object->u.cpu.node_id;
> +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> +        cpu.socket_id = object->u.cpu.socket_id;
> +        cpu.has_core_id = object->u.cpu.has_core_id;
> +        cpu.core_id = object->u.cpu.core_id;
> +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> +        cpu.thread_id = object->u.cpu.thread_id;
> +
> +        machine_set_cpu_numa_node(ms, &cpu, &err);

It's possible I've confused myself by not looking at this whole series
at once.

But, would it be possible to make a single machine hook which maps a
constructed cpu property set to a "canonical" cpu property set from
the table of CPU slots (or errors, of course).  That would let you do
what you need here, and I suspect in other places, without multiple
hooks.

> +        break;
>      default:
>          abort();
>      }
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 76d137d..5baf3a4 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -5680,10 +5680,12 @@
>  ##
>  # @NumaOptionsType:
>  #
> +# @cpu: property based CPU(s) to node mapping (Since: 2.10)
> +#
>  # Since: 2.1
>  ##
>  { 'enum': 'NumaOptionsType',
> -  'data': [ 'node' ] }
> +  'data': [ 'node', 'cpu' ] }
>  
>  ##
>  # @NumaOptions:
> @@ -5696,7 +5698,8 @@
>    'base': { 'type': 'NumaOptionsType' },
>    'discriminator': 'type',
>    'data': {
> -    'node': 'NumaNodeOptions' }}
> +    'node': 'NumaNodeOptions',
> +    'cpu': 'NumaCpuOptions' }}
>  
>  ##
>  # @NumaNodeOptions:
> @@ -5725,6 +5728,20 @@
>     '*memdev': 'str' }}
>  
>  ##
> +# @NumaCpuOptions:
> +#
> +# Option "-numa cpu" overrides default cpu to node mapping.
> +# It accepts the same set of cpu properties as returned by
> +# query-hotpluggable-cpus[].props, where node-id could be used to
> +# override default node mapping.
> +#
> +# Since: 2.10
> +##
> +{ 'struct': 'NumaCpuOptions',
> +   'base': 'CpuInstanceProperties',
> +   'data' : {} }
> +
> +##
>  # @HostMemPolicy:
>  #
>  # Host memory policy types
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 787b9c3..e88f534 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -139,13 +139,16 @@ ETEXI
>  
>  DEF("numa", HAS_ARG, QEMU_OPTION_numa,
>      "-numa node[,mem=size][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n"
> -    "-numa node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n", QEMU_ARCH_ALL)
> +    "-numa node[,memdev=id][,cpus=firstcpu[-lastcpu]][,nodeid=node]\n"
> +    "-numa cpu,node-id=node[,socket-id=x][,core-id=y][,thread-id=z]\n", QEMU_ARCH_ALL)
>  STEXI
>  @item -numa node[,mem=@var{size}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}]
>  @itemx -numa node[,memdev=@var{id}][,cpus=@var{firstcpu}[-@var{lastcpu}]][,nodeid=@var{node}]
> +@itemx -numa cpu,node-id=@var{node}[,socket-id=@var{x}][,core-id=@var{y}][,thread-id=@var{z}]
>  @findex -numa
>  Define a NUMA node and assign RAM and VCPUs to it.
>  
> +Legacy VCPU assignment uses @samp{cpus} option where
>  @var{firstcpu} and @var{lastcpu} are CPU indexes. Each
>  @samp{cpus} option represent a contiguous range of CPU indexes
>  (or a single VCPU if @var{lastcpu} is omitted). A non-contiguous
> @@ -159,6 +162,24 @@ a NUMA node:
>  -numa node,cpus=0-2,cpus=5
>  @end example
>  
> +@samp{cpu} option is a new alternative to @samp{cpus} option
> +which uses @samp{socket-id|core-id|thread-id} properties to assign
> +CPU objects to a @var{node} using topology layout properties of CPU.
> +The set of properties is machine specific, and depends on used
> +machine type/@samp{smp} options. It could be queried with
> +@samp{hotpluggable-cpus} monitor command.
> +@samp{node-id} property specifies @var{node} to which CPU object
> +will be assigned, it's required for @var{node} to be declared
> +with @samp{node} option before it's used with @samp{cpu} option.
> +
> +For example:
> +@example
> +-M pc \
> +-smp 1,sockets=2,maxcpus=2 \
> +-numa node,nodeid=0 -numa node,nodeid=1 \
> +-numa cpu,node-id=0,socket-id=0 -numa cpu,node-id=1,socket-id=1
> +@end example
> +
>  @samp{mem} assigns a given RAM amount to a node. @samp{memdev}
>  assigns RAM from a given memory backend device to a node. If
>  @samp{mem} and @samp{memdev} are omitted in all nodes, RAM is

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 24/24] tests: check -numa node, cpu=props_list usecase

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5965 bytes --]

On Wed, May 03, 2017 at 02:57:18PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

> ---
>  tests/numa-test.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 151 insertions(+)
> 
> diff --git a/tests/numa-test.c b/tests/numa-test.c
> index 2722687..c3475d6 100644
> --- a/tests/numa-test.c
> +++ b/tests/numa-test.c
> @@ -131,6 +131,144 @@ static void test_query_cpus(const void *data)
>      g_free(cli);
>  }
>  
> +static void pc_numa_cpu(const void *data)
> +{
> +    char *cli;
> +    QDict *resp;
> +    QList *cpus;
> +    const QObject *e;
> +
> +    cli = make_cli(data, "-cpu pentium -smp 8,sockets=2,cores=2,threads=2 "
> +        "-numa node,nodeid=0 -numa node,nodeid=1 "
> +        "-numa cpu,node-id=1,socket-id=0 "
> +        "-numa cpu,node-id=0,socket-id=1,core-id=0 "
> +        "-numa cpu,node-id=0,socket-id=1,core-id=1,thread-id=0 "
> +        "-numa cpu,node-id=1,socket-id=1,core-id=1,thread-id=1");
> +    qtest_start(cli);
> +    cpus = get_cpus(&resp);
> +    g_assert(cpus);
> +
> +    while ((e = qlist_pop(cpus))) {
> +        QDict *cpu, *props;
> +        int64_t socket, core, thread, node;
> +
> +        cpu = qobject_to_qdict(e);
> +        g_assert(qdict_haskey(cpu, "props"));
> +        props = qdict_get_qdict(cpu, "props");
> +
> +        g_assert(qdict_haskey(props, "node-id"));
> +        node = qdict_get_int(props, "node-id");
> +        g_assert(qdict_haskey(props, "socket-id"));
> +        socket = qdict_get_int(props, "socket-id");
> +        g_assert(qdict_haskey(props, "core-id"));
> +        core = qdict_get_int(props, "core-id");
> +        g_assert(qdict_haskey(props, "thread-id"));
> +        thread = qdict_get_int(props, "thread-id");
> +
> +        if (socket == 0) {
> +            g_assert_cmpint(node, ==, 1);
> +        } else if (socket == 1 && core == 0) {
> +            g_assert_cmpint(node, ==, 0);
> +        } else if (socket == 1 && core == 1 && thread == 0) {
> +            g_assert_cmpint(node, ==, 0);
> +        } else if (socket == 1 && core == 1 && thread == 1) {
> +            g_assert_cmpint(node, ==, 1);
> +        } else {
> +            g_assert(false);
> +        }
> +    }
> +
> +    QDECREF(resp);
> +    qtest_end();
> +    g_free(cli);
> +}
> +
> +static void spapr_numa_cpu(const void *data)
> +{
> +    char *cli;
> +    QDict *resp;
> +    QList *cpus;
> +    const QObject *e;
> +
> +    cli = make_cli(data, "-smp 4,cores=4 "
> +        "-numa node,nodeid=0 -numa node,nodeid=1 "
> +        "-numa cpu,node-id=0,core-id=0 "
> +        "-numa cpu,node-id=0,core-id=1 "
> +        "-numa cpu,node-id=0,core-id=2 "
> +        "-numa cpu,node-id=1,core-id=3");
> +    qtest_start(cli);
> +    cpus = get_cpus(&resp);
> +    g_assert(cpus);
> +
> +    while ((e = qlist_pop(cpus))) {
> +        QDict *cpu, *props;
> +        int64_t core, node;
> +
> +        cpu = qobject_to_qdict(e);
> +        g_assert(qdict_haskey(cpu, "props"));
> +        props = qdict_get_qdict(cpu, "props");
> +
> +        g_assert(qdict_haskey(props, "node-id"));
> +        node = qdict_get_int(props, "node-id");
> +        g_assert(qdict_haskey(props, "core-id"));
> +        core = qdict_get_int(props, "core-id");
> +
> +        if (core >= 0 && core < 3) {
> +            g_assert_cmpint(node, ==, 0);
> +        } else if (core == 3) {
> +            g_assert_cmpint(node, ==, 1);
> +        } else {
> +            g_assert(false);
> +        }
> +    }
> +
> +    QDECREF(resp);
> +    qtest_end();
> +    g_free(cli);
> +}
> +
> +static void aarch64_numa_cpu(const void *data)
> +{
> +    char *cli;
> +    QDict *resp;
> +    QList *cpus;
> +    const QObject *e;
> +
> +    cli = make_cli(data, "-smp 2 "
> +        "-numa node,nodeid=0 -numa node,nodeid=1 "
> +        "-numa cpu,node-id=1,thread-id=0 "
> +        "-numa cpu,node-id=0,thread-id=1");
> +    qtest_start(cli);
> +    cpus = get_cpus(&resp);
> +    g_assert(cpus);
> +
> +    while ((e = qlist_pop(cpus))) {
> +        QDict *cpu, *props;
> +        int64_t thread, node;
> +
> +        cpu = qobject_to_qdict(e);
> +        g_assert(qdict_haskey(cpu, "props"));
> +        props = qdict_get_qdict(cpu, "props");
> +
> +        g_assert(qdict_haskey(props, "node-id"));
> +        node = qdict_get_int(props, "node-id");
> +        g_assert(qdict_haskey(props, "thread-id"));
> +        thread = qdict_get_int(props, "thread-id");
> +
> +        if (thread == 0) {
> +            g_assert_cmpint(node, ==, 1);
> +        } else if (thread == 1) {
> +            g_assert_cmpint(node, ==, 0);
> +        } else {
> +            g_assert(false);
> +        }
> +    }
> +
> +    QDECREF(resp);
> +    qtest_end();
> +    g_free(cli);
> +}
> +
>  int main(int argc, char **argv)
>  {
>      const char *args = NULL;
> @@ -147,5 +285,18 @@ int main(int argc, char **argv)
>      qtest_add_data_func("/numa/mon/cpus/partial", args, test_mon_partial);
>      qtest_add_data_func("/numa/qmp/cpus/query-cpus", args, test_query_cpus);
>  
> +    if (!strcmp(arch, "i386") || !strcmp(arch, "x86_64")) {
> +        qtest_add_data_func("/numa/pc/cpu/explicit", args, pc_numa_cpu);
> +    }
> +
> +    if (!strcmp(arch, "ppc64")) {
> +        qtest_add_data_func("/numa/spapr/cpu/explicit", args, spapr_numa_cpu);
> +    }
> +
> +    if (!strcmp(arch, "aarch64")) {
> +        qtest_add_data_func("/numa/aarch64/cpu/explicit", args,
> +                            aarch64_numa_cpu);
> +    }
> +
>      return g_test_run();
>  }

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v2 20/24] machine: call machine init from wrapper

[Eduardo Habkost]

On Wed, May 03, 2017 at 02:57:14PM +0200, Igor Mammedov wrote:
> add machine_run_board_init() wrapper that calls machine
> init for now but in follow up patches it will be used
> to run generic machine code that should run before
> machine init.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

I like the name "board init". It becomes clear that it's
something different from instance_init.

Maybe we could rename MachineClass::init to
MachineClass::board_init in the future.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Eduardo Habkost]

On Mon, May 08, 2017 at 03:40:04PM +1000, David Gibson wrote:
> On Wed, May 03, 2017 at 02:57:17PM +0200, Igor Mammedov wrote:
> > legacy cpu to node mapping is using cpu index values to map
> > VCPU to node with help of '-numa node,nodeid=node,cpus=x[-y]'
> > option. However cpu index is internal concept and QEMU users
> > have to guess /reimplement qemu's logic/ to map it to
> > a concrete cpu socket/core/thread to make sane CPUs
> > placement across numa nodes.
> > 
> > This patch allows to map cpu objects to numa nodes using
> > the same properties as used for cpus with -device/device_add
> > (socket-id/core-id/thread-id/node-id).
> > 
> > At present valid properties/values to address CPUs could be
> > fetched using hotpluggable-cpus monitor/qmp command, it will
> > require user to start qemu twice when creating domain to fetch
> > possible CPUs for a machine type/-smp layout first and
> > then the second time with numa explicit mapping for actual
> > usage. The first step results could be saved and reused to
> > set/change mapping later as far as machine type/-smp stays
> > the same.
> > 
> > Proposed impl. supports exact and wildcard matching to
> > simplify CLI and allow to set mapping for a specific cpu
> > or group of cpu objects specified by matched properties.
> > 
> > For example:
> > 
> >    # exact mapping x86
> >    -numa cpu,node-id=x,socket-id=y,core-id=z,thread-id=n
> > 
> >    # exact mapping SPAPR
> >    -numa cpu,node-id=x,core-id=y
> > 
> >    # wildcard mapping, all cpu objects that match socket-id=y
> >    # are mapped to node-id=x
> >    -numa cpu,node-id=x,socket-id=y
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> > v2:
> >   - use new NumaCpuOptions instead of CpuInstanceProperties in
> >     NumaOptions, so that in future we could decouple both
> >     if needed. (Eduardo Habkost <ehabkost@redhat.com>)
> >   - clarify effect of NumaCpuOptions.node-id in qapi-schema.json
> > ---
> >  numa.c           | 25 +++++++++++++++++++++++++
> >  qapi-schema.json | 21 +++++++++++++++++++--
> >  qemu-options.hx  | 23 ++++++++++++++++++++++-
> >  3 files changed, 66 insertions(+), 3 deletions(-)
> > 
> > diff --git a/numa.c b/numa.c
> > index 40e9f44..61521f5 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> >      NumaOptions *object = NULL;
> >      MachineState *ms = opaque;
> >      Error *err = NULL;
> > +    CpuInstanceProperties cpu;
> >  
> >      {
> >          Visitor *v = opts_visitor_new(opts);
> > @@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> >          }
> >          nb_numa_nodes++;
> >          break;
> > +    case NUMA_OPTIONS_TYPE_CPU:
> > +        if (!object->u.cpu.has_node_id) {
> > +            error_setg(&err, "Missing mandatory node-id property");
> > +            goto end;
> > +        }
> > +        if (!numa_info[object->u.cpu.node_id].present) {
> > +            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
> > +                "defined with -numa node,nodeid=ID before it's used with "
> > +                "-numa cpu,node-id=ID", object->u.cpu.node_id);
> > +            goto end;
> > +        }
> > +
> > +        memset(&cpu, 0, sizeof(cpu));
> > +        cpu.has_node_id = object->u.cpu.has_node_id;
> > +        cpu.node_id = object->u.cpu.node_id;
> > +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> > +        cpu.socket_id = object->u.cpu.socket_id;
> > +        cpu.has_core_id = object->u.cpu.has_core_id;
> > +        cpu.core_id = object->u.cpu.core_id;
> > +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> > +        cpu.thread_id = object->u.cpu.thread_id;
> > +
> > +        machine_set_cpu_numa_node(ms, &cpu, &err);
> 
> It's possible I've confused myself by not looking at this whole series
> at once.
> 
> But, would it be possible to make a single machine hook which maps a
> constructed cpu property set to a "canonical" cpu property set from
> the table of CPU slots (or errors, of course).  That would let you do
> what you need here, and I suspect in other places, without multiple
> hooks.

I think I understand your point, but I don't see where any
opportunities to reuse a more generic hook in the current series.

I see only two new hooks added by this series:
1) MachineClass::cpu_index_to_instance_props()
2) machine_set_cpu_numa_node()

The former takes only one cpu_index argument as input. As it
needs to be a machine-specific hook, I think it's good to keep
its input as simple and specific as possible.

The latter could use a generic lookup function to map
CpuInstanceProperties to a set of CPU slots. But I don't see any
case where that logic would be reused by now, so I don't think
it's worth the extra complexity.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v2 07/24] pc: add node-id property to CPU

[Igor Mammedov]

On Fri, 5 May 2017 17:29:23 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Wed, May 03, 2017 at 02:57:01PM +0200, Igor Mammedov wrote:
> > it will allow switching from cpu_index to property based
> > numa mapping in follow up patches.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  hw/i386/pc.c      | 17 +++++++++++++++++
> >  target/i386/cpu.c |  1 +
> >  2 files changed, 18 insertions(+)
> > 
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 9cec0c1..4a4fb1c 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -1893,6 +1893,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> >                              DeviceState *dev, Error **errp)
> >  {
> >      int idx;
> > +    int node_id;
> >      CPUState *cs;
> >      CPUArchId *cpu_slot;
> >      X86CPUTopoInfo topo;
> > @@ -1982,6 +1983,22 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> >  
> >      cs = CPU(cpu);
> >      cs->cpu_index = idx;
> > +
> > +    node_id = numa_get_node_for_cpu(cs->cpu_index);
> > +    if (node_id == nb_numa_nodes) {
> > +        /* by default CPUState::numa_node was 0 if it's not set via CLI
> > +         * keep it this way for now but in future we probably should
> > +         * refuse to start up with incomplete numa mapping */
> > +        node_id = 0;
> > +    }
> > +    if (cs->numa_node == CPU_UNSET_NUMA_NODE_ID) {
> > +        cs->numa_node = node_id;  
> 
> I suggest moving the above lines to cpu_common_realizefn(), and
> deleting the code in numa_post_machine_init().
> 
> If you want to do this in v3 of this series, or as a follow-up
> patch, it's up to you.
maybe not to realize time, but yep I'm thinking about moving
this whole hunk into a helper function that could be reused
by pc/arm/spapr without duplicating.
I'd do it as follow-up.


> > +    } else if (cs->numa_node != node_id) {
> > +            error_setg(errp, "node-id %d must match numa node specified"
> > +                "with -numa option for cpu-index %d",
> > +                cs->numa_node, cs->cpu_index);
> > +            return;
> > +    }
> >  }
> >  
> >  static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
> > diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> > index 13c0985..007a5bd 100644
> > --- a/target/i386/cpu.c
> > +++ b/target/i386/cpu.c
> > @@ -3982,6 +3982,7 @@ static Property x86_cpu_properties[] = {
> >      DEFINE_PROP_INT32("core-id", X86CPU, core_id, -1),
> >      DEFINE_PROP_INT32("socket-id", X86CPU, socket_id, -1),
> >  #endif
> > +    DEFINE_PROP_INT32("node-id", CPUState, numa_node, CPU_UNSET_NUMA_NODE_ID),  
> 
> This changes the default from 0 to CPU_UNSET_NUMA_NODE_ID, but it
> looks like there's no existing i386 or PC code that ever looks at
> CPUState::numa_node. Maybe worth mentioning in the commit
> message.
In preceding hunk there is code which still effectively keeps it 0 by default
with comment explaining that it's so for legacy reasons.
I can copy comment in commit message as well if that's what you've meant.

for PC it would affect only HMP 'info numa' output,
but pc_cpu_pre_plug() in this patch takes care of it
to ensure that old behavior is preserved.

> 
> >      DEFINE_PROP_BOOL("pmu", X86CPU, enable_pmu, false),
> >      { .name  = "hv-spinlocks", .info  = &qdev_prop_spinlocks },
> >      DEFINE_PROP_BOOL("hv-relaxed", X86CPU, hyperv_relaxed_timing, false),
> > -- 
> > 2.7.4
> >   
> 

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[Igor Mammedov]

On Mon, 8 May 2017 15:40:04 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, May 03, 2017 at 02:57:17PM +0200, Igor Mammedov wrote:
> > legacy cpu to node mapping is using cpu index values to map
> > VCPU to node with help of '-numa node,nodeid=node,cpus=x[-y]'
> > option. However cpu index is internal concept and QEMU users
> > have to guess /reimplement qemu's logic/ to map it to
> > a concrete cpu socket/core/thread to make sane CPUs
> > placement across numa nodes.
> > 
> > This patch allows to map cpu objects to numa nodes using
> > the same properties as used for cpus with -device/device_add
> > (socket-id/core-id/thread-id/node-id).
> > 
> > At present valid properties/values to address CPUs could be
> > fetched using hotpluggable-cpus monitor/qmp command, it will
> > require user to start qemu twice when creating domain to fetch
> > possible CPUs for a machine type/-smp layout first and
> > then the second time with numa explicit mapping for actual
> > usage. The first step results could be saved and reused to
> > set/change mapping later as far as machine type/-smp stays
> > the same.
> > 
> > Proposed impl. supports exact and wildcard matching to
> > simplify CLI and allow to set mapping for a specific cpu
> > or group of cpu objects specified by matched properties.
> > 
> > For example:
> > 
> >    # exact mapping x86
> >    -numa cpu,node-id=x,socket-id=y,core-id=z,thread-id=n
> > 
> >    # exact mapping SPAPR
> >    -numa cpu,node-id=x,core-id=y
> > 
> >    # wildcard mapping, all cpu objects that match socket-id=y
> >    # are mapped to node-id=x
> >    -numa cpu,node-id=x,socket-id=y
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> > v2:
> >   - use new NumaCpuOptions instead of CpuInstanceProperties in
> >     NumaOptions, so that in future we could decouple both
> >     if needed. (Eduardo Habkost <ehabkost@redhat.com>)
> >   - clarify effect of NumaCpuOptions.node-id in qapi-schema.json
> > ---
> >  numa.c           | 25 +++++++++++++++++++++++++
> >  qapi-schema.json | 21 +++++++++++++++++++--
> >  qemu-options.hx  | 23 ++++++++++++++++++++++-
> >  3 files changed, 66 insertions(+), 3 deletions(-)
> > 
> > diff --git a/numa.c b/numa.c
> > index 40e9f44..61521f5 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> >      NumaOptions *object = NULL;
> >      MachineState *ms = opaque;
> >      Error *err = NULL;
> > +    CpuInstanceProperties cpu;
> >  
> >      {
> >          Visitor *v = opts_visitor_new(opts);
> > @@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> >          }
> >          nb_numa_nodes++;
> >          break;
> > +    case NUMA_OPTIONS_TYPE_CPU:
> > +        if (!object->u.cpu.has_node_id) {
> > +            error_setg(&err, "Missing mandatory node-id property");
> > +            goto end;
> > +        }
> > +        if (!numa_info[object->u.cpu.node_id].present) {
> > +            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
> > +                "defined with -numa node,nodeid=ID before it's used with "
> > +                "-numa cpu,node-id=ID", object->u.cpu.node_id);
> > +            goto end;
> > +        }
> > +
> > +        memset(&cpu, 0, sizeof(cpu));
> > +        cpu.has_node_id = object->u.cpu.has_node_id;
> > +        cpu.node_id = object->u.cpu.node_id;
> > +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> > +        cpu.socket_id = object->u.cpu.socket_id;
> > +        cpu.has_core_id = object->u.cpu.has_core_id;
> > +        cpu.core_id = object->u.cpu.core_id;
> > +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> > +        cpu.thread_id = object->u.cpu.thread_id;
> > +
> > +        machine_set_cpu_numa_node(ms, &cpu, &err);  
> 
> It's possible I've confused myself by not looking at this whole series
> at once.
> 
> But, would it be possible to make a single machine hook which maps a
> constructed cpu property set to a "canonical" cpu property set from
> the table of CPU slots (or errors, of course).  That would let you do
> what you need here, and I suspect in other places, without multiple
> hooks.
Not sure that get question.

if it's about machine_set_cpu_numa_node() than it's a single generic
setter of MachineState::possible_cpus(node-id) from non machine code
and the only place it's supposed to be used is from numa configuration
code. I don't see any need to reuse it for something/somewhere else
so far.

More over it's not 1:1 mapping wrt input:affected slots,
it's 1:[1-n] where input could affect multiple slots.

I think I've tried to get machine return a list of cpu slots
and then deal with them in numa.c but result looked messy and
not to my liking.
Eventually I stopped on machine_set_cpu_numa_node() variant
as the cleanest for now.

Re: [Qemu-devel] [PATCH v2 23/24] numa: add '-numa cpu, ...' option for property based node mapping

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5529 bytes --]

On Tue, May 09, 2017 at 05:58:29PM +0200, Igor Mammedov wrote:
> On Mon, 8 May 2017 15:40:04 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Wed, May 03, 2017 at 02:57:17PM +0200, Igor Mammedov wrote:
> > > legacy cpu to node mapping is using cpu index values to map
> > > VCPU to node with help of '-numa node,nodeid=node,cpus=x[-y]'
> > > option. However cpu index is internal concept and QEMU users
> > > have to guess /reimplement qemu's logic/ to map it to
> > > a concrete cpu socket/core/thread to make sane CPUs
> > > placement across numa nodes.
> > > 
> > > This patch allows to map cpu objects to numa nodes using
> > > the same properties as used for cpus with -device/device_add
> > > (socket-id/core-id/thread-id/node-id).
> > > 
> > > At present valid properties/values to address CPUs could be
> > > fetched using hotpluggable-cpus monitor/qmp command, it will
> > > require user to start qemu twice when creating domain to fetch
> > > possible CPUs for a machine type/-smp layout first and
> > > then the second time with numa explicit mapping for actual
> > > usage. The first step results could be saved and reused to
> > > set/change mapping later as far as machine type/-smp stays
> > > the same.
> > > 
> > > Proposed impl. supports exact and wildcard matching to
> > > simplify CLI and allow to set mapping for a specific cpu
> > > or group of cpu objects specified by matched properties.
> > > 
> > > For example:
> > > 
> > >    # exact mapping x86
> > >    -numa cpu,node-id=x,socket-id=y,core-id=z,thread-id=n
> > > 
> > >    # exact mapping SPAPR
> > >    -numa cpu,node-id=x,core-id=y
> > > 
> > >    # wildcard mapping, all cpu objects that match socket-id=y
> > >    # are mapped to node-id=x
> > >    -numa cpu,node-id=x,socket-id=y
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > > v2:
> > >   - use new NumaCpuOptions instead of CpuInstanceProperties in
> > >     NumaOptions, so that in future we could decouple both
> > >     if needed. (Eduardo Habkost <ehabkost@redhat.com>)
> > >   - clarify effect of NumaCpuOptions.node-id in qapi-schema.json
> > > ---
> > >  numa.c           | 25 +++++++++++++++++++++++++
> > >  qapi-schema.json | 21 +++++++++++++++++++--
> > >  qemu-options.hx  | 23 ++++++++++++++++++++++-
> > >  3 files changed, 66 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/numa.c b/numa.c
> > > index 40e9f44..61521f5 100644
> > > --- a/numa.c
> > > +++ b/numa.c
> > > @@ -227,6 +227,7 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> > >      NumaOptions *object = NULL;
> > >      MachineState *ms = opaque;
> > >      Error *err = NULL;
> > > +    CpuInstanceProperties cpu;
> > >  
> > >      {
> > >          Visitor *v = opts_visitor_new(opts);
> > > @@ -246,6 +247,30 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
> > >          }
> > >          nb_numa_nodes++;
> > >          break;
> > > +    case NUMA_OPTIONS_TYPE_CPU:
> > > +        if (!object->u.cpu.has_node_id) {
> > > +            error_setg(&err, "Missing mandatory node-id property");
> > > +            goto end;
> > > +        }
> > > +        if (!numa_info[object->u.cpu.node_id].present) {
> > > +            error_setg(&err, "Invalid node-id=%" PRId64 ", NUMA node must be "
> > > +                "defined with -numa node,nodeid=ID before it's used with "
> > > +                "-numa cpu,node-id=ID", object->u.cpu.node_id);
> > > +            goto end;
> > > +        }
> > > +
> > > +        memset(&cpu, 0, sizeof(cpu));
> > > +        cpu.has_node_id = object->u.cpu.has_node_id;
> > > +        cpu.node_id = object->u.cpu.node_id;
> > > +        cpu.has_socket_id = object->u.cpu.has_socket_id;
> > > +        cpu.socket_id = object->u.cpu.socket_id;
> > > +        cpu.has_core_id = object->u.cpu.has_core_id;
> > > +        cpu.core_id = object->u.cpu.core_id;
> > > +        cpu.has_thread_id = object->u.cpu.has_thread_id;
> > > +        cpu.thread_id = object->u.cpu.thread_id;
> > > +
> > > +        machine_set_cpu_numa_node(ms, &cpu, &err);  
> > 
> > It's possible I've confused myself by not looking at this whole series
> > at once.
> > 
> > But, would it be possible to make a single machine hook which maps a
> > constructed cpu property set to a "canonical" cpu property set from
> > the table of CPU slots (or errors, of course).  That would let you do
> > what you need here, and I suspect in other places, without multiple
> > hooks.
> Not sure that get question.
> 
> if it's about machine_set_cpu_numa_node() than it's a single generic
> setter of MachineState::possible_cpus(node-id) from non machine code
> and the only place it's supposed to be used is from numa configuration
> code. I don't see any need to reuse it for something/somewhere else
> so far.
> 
> More over it's not 1:1 mapping wrt input:affected slots,
> it's 1:[1-n] where input could affect multiple slots.

Ah, good point.

> I think I've tried to get machine return a list of cpu slots
> and then deal with them in numa.c but result looked messy and
> not to my liking.
> Eventually I stopped on machine_set_cpu_numa_node() variant
> as the cleanest for now.

Ok, makes sense.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]