[PATCH v3 0/6] hw/loongarch: Refine numa memory map

[PATCH v3 0/6] hw/loongarch: Refine numa memory map

[Bibo Mao]

One LoongArch virt machine platform, there is limitation for memory
map information. The minimum memory size is 256M and minimum memory
size for numa node0 is 256M also. With qemu numa qtest, it is possible
that memory size of numa node0 is 128M.

Limitations for minimum memory size for both total memory and numa
node0 is removed here, including acpi srat table, fadt memory map table
and fw_cfg memory map table.

Also remove numa node about memory region, there is only low memory
region and how memory region.

---
v2 ... v3:
  1. Modify numa test case for loongarch system.
  2. Rebase the patch based on the latest version.

v1 ... v2:
  1. Refresh the patch based on the latest qemu version, solve the
confliction issue.
  2. Add numa test case for loongarch system.
---
Bibo Mao (6):
  hw/loongarch: Refine acpi srat table for numa memory
  hw/loongarch: Refine fadt memory table for numa memory
  hw/loongarch: Refine fwcfg memory map
  hw/loongarch: Refine system dram memory region
  hw/loongarch: Remove minimum and default memory size
  tests/qtest: Add numa test for loongarch system

 hw/loongarch/acpi-build.c |  58 +++++++------
 hw/loongarch/virt.c       | 166 +++++++++++++++++++++++++++-----------
 tests/qtest/meson.build   |   2 +-
 tests/qtest/numa-test.c   |  53 ++++++++++++
 4 files changed, 206 insertions(+), 73 deletions(-)


base-commit: dafec285bdbfe415ac6823abdc510e0b92c3f094
-- 
2.39.3

[PATCH v3 1/6] hw/loongarch: Refine acpi srat table for numa memory

[Bibo Mao]

One LoongArch virt machine platform, there is limitation for memory
map information. The minimum memory size is 256M and minimum memory
size for numa node0 is 256M also. With qemu numa qtest, it is possible
that memory size of numa node0 is 128M.

Limitations for minimum memory size for both total memory and numa
node0 is removed for acpi srat table creation.

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 hw/loongarch/acpi-build.c | 58 +++++++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 24 deletions(-)

diff --git a/hw/loongarch/acpi-build.c b/hw/loongarch/acpi-build.c
index 5ef010d4da..af45ce526d 100644
--- a/hw/loongarch/acpi-build.c
+++ b/hw/loongarch/acpi-build.c
@@ -166,8 +166,9 @@ static void
 build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
 {
     int i, arch_id, node_id;
-    uint64_t mem_len, mem_base;
-    int nb_numa_nodes = machine->numa_state->num_nodes;
+    hwaddr len, base, gap;
+    NodeInfo *numa_info;
+    int nodes, nb_numa_nodes = machine->numa_state->num_nodes;
     LoongArchVirtMachineState *lvms = LOONGARCH_VIRT_MACHINE(machine);
     MachineClass *mc = MACHINE_GET_CLASS(lvms);
     const CPUArchIdList *arch_ids = mc->possible_cpu_arch_ids(machine);
@@ -196,35 +197,44 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
         build_append_int_noprefix(table_data, 0, 4); /* Reserved */
     }
 
-    /* Node0 */
-    build_srat_memory(table_data, VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE,
-                      0, MEM_AFFINITY_ENABLED);
-    mem_base = VIRT_HIGHMEM_BASE;
-    if (!nb_numa_nodes) {
-        mem_len = machine->ram_size - VIRT_LOWMEM_SIZE;
-    } else {
-        mem_len = machine->numa_state->nodes[0].node_mem - VIRT_LOWMEM_SIZE;
+    base = VIRT_LOWMEM_BASE;
+    gap = VIRT_LOWMEM_SIZE;
+    numa_info = machine->numa_state->nodes;
+    nodes = nb_numa_nodes;
+    if (!nodes) {
+        nodes = 1;
     }
-    if (mem_len)
-        build_srat_memory(table_data, mem_base, mem_len, 0, MEM_AFFINITY_ENABLED);
-
-    /* Node1 - Nodemax */
-    if (nb_numa_nodes) {
-        mem_base += mem_len;
-        for (i = 1; i < nb_numa_nodes; ++i) {
-            if (machine->numa_state->nodes[i].node_mem > 0) {
-                build_srat_memory(table_data, mem_base,
-                                  machine->numa_state->nodes[i].node_mem, i,
-                                  MEM_AFFINITY_ENABLED);
-                mem_base += machine->numa_state->nodes[i].node_mem;
-            }
+
+    for (i = 0; i < nodes; i++) {
+        if (nb_numa_nodes) {
+            len = numa_info[i].node_mem;
+        } else {
+            len = machine->ram_size;
+        }
+
+        /*
+         * memory for the node splited into two part
+         *   lowram:  [base, +gap)
+         *   highram: [VIRT_HIGHMEM_BASE, +(len - gap))
+         */
+        if (len >= gap) {
+            build_srat_memory(table_data, base, len, i, MEM_AFFINITY_ENABLED);
+            len -= gap;
+            base = VIRT_HIGHMEM_BASE;
+            gap = machine->ram_size - VIRT_LOWMEM_SIZE;
+        }
+
+        if (len) {
+            build_srat_memory(table_data, base, len, i, MEM_AFFINITY_ENABLED);
+            base += len;
+            gap  -= len;
         }
     }
 
     if (machine->device_memory) {
         build_srat_memory(table_data, machine->device_memory->base,
                           memory_region_size(&machine->device_memory->mr),
-                          nb_numa_nodes - 1,
+                          nodes - 1,
                           MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);
     }
 
-- 
2.39.3

[PATCH v3 2/6] hw/loongarch: Refine fadt memory table for numa memory

[Bibo Mao]

One LoongArch virt machine platform, there is limitation for memory
map information. The minimum memory size is 256M and minimum memory
size for numa node0 is 256M also. With qemu numa qtest, it is possible
that memory size of numa node0 is 128M.

Limitations for minimum memory size for both total memory and numa
node0 is removed for fadt numa memory table creation.

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 hw/loongarch/virt.c | 46 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)

diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index f0640d2d80..c996305d87 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -473,6 +473,48 @@ static void fdt_add_memory_node(MachineState *ms,
     g_free(nodename);
 }
 
+static void fdt_add_memory_nodes(MachineState *ms)
+{
+    hwaddr base, size, ram_size, gap;
+    int i, nb_numa_nodes, nodes;
+    NodeInfo *numa_info;
+
+    ram_size = ms->ram_size;
+    base = VIRT_LOWMEM_BASE;
+    gap = VIRT_LOWMEM_SIZE;
+    nodes = nb_numa_nodes = ms->numa_state->num_nodes;
+    numa_info = ms->numa_state->nodes;
+    if (!nodes) {
+        nodes = 1;
+    }
+
+    for (i = 0; i < nodes; i++) {
+        if (nb_numa_nodes) {
+            size = numa_info[i].node_mem;
+        } else {
+            size = ram_size;
+        }
+
+        /*
+         * memory for the node splited into two part
+         *   lowram:  [base, +gap)
+         *   highram: [VIRT_HIGHMEM_BASE, +(len - gap))
+         */
+        if (size >= gap) {
+            fdt_add_memory_node(ms, base, gap, i);
+            size -= gap;
+            base = VIRT_HIGHMEM_BASE;
+            gap = ram_size - VIRT_LOWMEM_SIZE;
+        }
+
+        if (size) {
+            fdt_add_memory_node(ms, base, size, i);
+            base += size;
+            gap -= size;
+        }
+    }
+}
+
 static void virt_build_smbios(LoongArchVirtMachineState *lvms)
 {
     MachineState *ms = MACHINE(lvms);
@@ -915,10 +957,10 @@ static void virt_init(MachineState *machine)
         lacpu->phy_id = machine->possible_cpus->cpus[i].arch_id;
     }
     fdt_add_cpu_nodes(lvms);
+    fdt_add_memory_nodes(machine);
 
     /* Node0 memory */
     memmap_add_entry(VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 1);
-    fdt_add_memory_node(machine, VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 0);
     memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
                              machine->ram, offset, VIRT_LOWMEM_SIZE);
     memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
@@ -932,7 +974,6 @@ static void virt_init(MachineState *machine)
     }
     phyAddr = VIRT_HIGHMEM_BASE;
     memmap_add_entry(phyAddr, highram_size, 1);
-    fdt_add_memory_node(machine, phyAddr, highram_size, 0);
     memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
                               machine->ram, offset, highram_size);
     memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
@@ -948,7 +989,6 @@ static void virt_init(MachineState *machine)
                                  offset,  numa_info[i].node_mem);
         memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
         memmap_add_entry(phyAddr, numa_info[i].node_mem, 1);
-        fdt_add_memory_node(machine, phyAddr, numa_info[i].node_mem, i);
         offset += numa_info[i].node_mem;
         phyAddr += numa_info[i].node_mem;
     }
-- 
2.39.3

[PATCH v3 3/6] hw/loongarch: Refine fwcfg memory map

[Bibo Mao]

Memory map table for fwcfg is used for UEFI BIOS, UEFI BIOS uses the first
entry from fwcfg memory map as the first memory HOB, the second memory HOB
will be used if the first memory HOB is used up.

Memory map table for fwcfg does not care about numa node, however in
generic the first memory HOB is part of numa node0, so that runtime
memory of UEFI which is allocated from the first memory HOB is located
at numa node0.

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 hw/loongarch/virt.c | 60 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 57 insertions(+), 3 deletions(-)

diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index c996305d87..b67d691fa5 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -912,6 +912,62 @@ static const MemoryRegionOps virt_iocsr_misc_ops = {
     },
 };
 
+static void fw_cfg_add_memory(MachineState *ms)
+{
+    hwaddr base, size, ram_size, gap;
+    int nb_numa_nodes, nodes;
+    NodeInfo *numa_info;
+
+    ram_size = ms->ram_size;
+    base = VIRT_LOWMEM_BASE;
+    gap = VIRT_LOWMEM_SIZE;
+    nodes = nb_numa_nodes = ms->numa_state->num_nodes;
+    numa_info = ms->numa_state->nodes;
+    if (!nodes) {
+        nodes = 1;
+    }
+
+    /* add fw_cfg memory map of node0 */
+    if (nb_numa_nodes) {
+        size = numa_info[0].node_mem;
+    } else {
+        size = ram_size;
+    }
+
+    if (size >= gap) {
+        memmap_add_entry(base, gap, 1);
+        size -= gap;
+        base = VIRT_HIGHMEM_BASE;
+        gap = ram_size - VIRT_LOWMEM_SIZE;
+    }
+
+    if (size) {
+        memmap_add_entry(base, size, 1);
+        base += size;
+    }
+
+    if (nodes < 2) {
+        return;
+    }
+
+    /* add fw_cfg memory map of other nodes */
+    size = ram_size - numa_info[0].node_mem;
+    gap  = VIRT_LOWMEM_BASE + VIRT_LOWMEM_SIZE;
+    if (base < gap && (base + size) > gap) {
+        /*
+         * memory map for the maining nodes splited into two part
+         *   lowram:  [base, +(gap - base))
+         *   highram: [VIRT_HIGHMEM_BASE, +(size - (gap - base)))
+         */
+        memmap_add_entry(base, gap - base, 1);
+        size -= gap - base;
+        base = VIRT_HIGHMEM_BASE;
+    }
+
+   if (size)
+        memmap_add_entry(base, size, 1);
+}
+
 static void virt_init(MachineState *machine)
 {
     LoongArchCPU *lacpu;
@@ -958,9 +1014,9 @@ static void virt_init(MachineState *machine)
     }
     fdt_add_cpu_nodes(lvms);
     fdt_add_memory_nodes(machine);
+    fw_cfg_add_memory(machine);
 
     /* Node0 memory */
-    memmap_add_entry(VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 1);
     memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
                              machine->ram, offset, VIRT_LOWMEM_SIZE);
     memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
@@ -973,7 +1029,6 @@ static void virt_init(MachineState *machine)
         highram_size = ram_size - VIRT_LOWMEM_SIZE;
     }
     phyAddr = VIRT_HIGHMEM_BASE;
-    memmap_add_entry(phyAddr, highram_size, 1);
     memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
                               machine->ram, offset, highram_size);
     memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
@@ -988,7 +1043,6 @@ static void virt_init(MachineState *machine)
         memory_region_init_alias(nodemem, NULL, ramName, machine->ram,
                                  offset,  numa_info[i].node_mem);
         memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
-        memmap_add_entry(phyAddr, numa_info[i].node_mem, 1);
         offset += numa_info[i].node_mem;
         phyAddr += numa_info[i].node_mem;
     }
-- 
2.39.3

[PATCH v3 4/6] hw/loongarch: Refine system dram memory region

[Bibo Mao]

For system dram memory region, it is not necessary to use numa node
information. There is only low memory region and high memory region.

Remove numa node information for ddr memory region here, it can reduce
memory region number on LoongArch virt machine.

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 hw/loongarch/virt.c | 55 +++++++++++++++------------------------------
 1 file changed, 18 insertions(+), 37 deletions(-)

diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index b67d691fa5..ac980aec8e 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -972,14 +972,10 @@ static void virt_init(MachineState *machine)
 {
     LoongArchCPU *lacpu;
     const char *cpu_model = machine->cpu_type;
-    ram_addr_t offset = 0;
-    ram_addr_t ram_size = machine->ram_size;
-    uint64_t highram_size = 0, phyAddr = 0;
     MemoryRegion *address_space_mem = get_system_memory();
     LoongArchVirtMachineState *lvms = LOONGARCH_VIRT_MACHINE(machine);
-    int nb_numa_nodes = machine->numa_state->num_nodes;
-    NodeInfo *numa_info = machine->numa_state->nodes;
     int i;
+    hwaddr base, size, ram_size = machine->ram_size;
     const CPUArchIdList *possible_cpus;
     MachineClass *mc = MACHINE_GET_CLASS(machine);
     CPUState *cpu;
@@ -1017,40 +1013,27 @@ static void virt_init(MachineState *machine)
     fw_cfg_add_memory(machine);
 
     /* Node0 memory */
-    memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
-                             machine->ram, offset, VIRT_LOWMEM_SIZE);
-    memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
-
-    offset += VIRT_LOWMEM_SIZE;
-    if (nb_numa_nodes > 0) {
-        assert(numa_info[0].node_mem > VIRT_LOWMEM_SIZE);
-        highram_size = numa_info[0].node_mem - VIRT_LOWMEM_SIZE;
-    } else {
-        highram_size = ram_size - VIRT_LOWMEM_SIZE;
-    }
-    phyAddr = VIRT_HIGHMEM_BASE;
-    memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
-                              machine->ram, offset, highram_size);
-    memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
-
-    /* Node1 - Nodemax memory */
-    offset += highram_size;
-    phyAddr += highram_size;
-
-    for (i = 1; i < nb_numa_nodes; i++) {
-        MemoryRegion *nodemem = g_new(MemoryRegion, 1);
-        g_autofree char *ramName = g_strdup_printf("loongarch.node%d.ram", i);
-        memory_region_init_alias(nodemem, NULL, ramName, machine->ram,
-                                 offset,  numa_info[i].node_mem);
-        memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
-        offset += numa_info[i].node_mem;
-        phyAddr += numa_info[i].node_mem;
+    size = ram_size;
+    base = VIRT_LOWMEM_BASE;
+    if (size > VIRT_LOWMEM_SIZE) {
+        size = VIRT_LOWMEM_SIZE;
     }
 
+    memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.lowram",
+                              machine->ram, base, size);
+    memory_region_add_subregion(address_space_mem, base, &lvms->lowmem);
+    base += size;
+    if (ram_size - size) {
+        base = VIRT_HIGHMEM_BASE;
+        memory_region_init_alias(&lvms->highmem, NULL, "loongarch.highram",
+                machine->ram, VIRT_LOWMEM_BASE + size, ram_size - size);
+        memory_region_add_subregion(address_space_mem, base, &lvms->highmem);
+        base += ram_size - size;
+     }
+
     /* initialize device memory address space */
     if (machine->ram_size < machine->maxram_size) {
         ram_addr_t device_mem_size = machine->maxram_size - machine->ram_size;
-        hwaddr device_mem_base;
 
         if (machine->ram_slots > ACPI_MAX_RAM_SLOTS) {
             error_report("unsupported amount of memory slots: %"PRIu64,
@@ -1064,9 +1047,7 @@ static void virt_init(MachineState *machine)
                          "%d bytes", TARGET_PAGE_SIZE);
             exit(EXIT_FAILURE);
         }
-        /* device memory base is the top of high memory address. */
-        device_mem_base = ROUND_UP(VIRT_HIGHMEM_BASE + highram_size, 1 * GiB);
-        machine_memory_devices_init(machine, device_mem_base, device_mem_size);
+        machine_memory_devices_init(machine, base, device_mem_size);
     }
 
     /* load the BIOS image. */
-- 
2.39.3

[PATCH v3 5/6] hw/loongarch: Remove minimum and default memory size

[Bibo Mao]

Some qtest test cases such as numa use default memory size of generic
machine class, which is 128M by fault.

Here generic default memory size is used, and also remove minimum memory
size which is 1G originally.

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 hw/loongarch/virt.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index ac980aec8e..7d3d1d1689 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -984,10 +984,6 @@ static void virt_init(MachineState *machine)
         cpu_model = LOONGARCH_CPU_TYPE_NAME("la464");
     }
 
-    if (ram_size < 1 * GiB) {
-        error_report("ram_size must be greater than 1G.");
-        exit(1);
-    }
     create_fdt(lvms);
 
     /* Create IOCSR space */
@@ -1273,7 +1269,6 @@ static void virt_class_init(ObjectClass *oc, void *data)
     HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
     mc->init = virt_init;
-    mc->default_ram_size = 1 * GiB;
     mc->default_cpu_type = LOONGARCH_CPU_TYPE_NAME("la464");
     mc->default_ram_id = "loongarch.ram";
     mc->max_cpus = LOONGARCH_MAX_CPUS;
-- 
2.39.3

[PATCH v3 6/6] tests/qtest: Add numa test for loongarch system

[Bibo Mao]

Add numa test case for loongarch system, it passes to run
with command "make check-qtest".

Signed-off-by: Bibo Mao <maobibo@loongson.cn>
---
 tests/qtest/meson.build |  2 +-
 tests/qtest/numa-test.c | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 86293051dc..8df7b85d72 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -140,7 +140,7 @@ qtests_hppa = ['boot-serial-test'] + \
   (config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : [])
 
 qtests_loongarch64 = qtests_filter + \
-  ['boot-serial-test']
+  ['boot-serial-test', 'numa-test']
 
 qtests_m68k = ['boot-serial-test'] + \
   qtests_filter
diff --git a/tests/qtest/numa-test.c b/tests/qtest/numa-test.c
index 4f4404a4b1..2f71d57a8d 100644
--- a/tests/qtest/numa-test.c
+++ b/tests/qtest/numa-test.c
@@ -265,6 +265,54 @@ static void aarch64_numa_cpu(const void *data)
     qtest_quit(qts);
 }
 
+static void loongarch64_numa_cpu(const void *data)
+{
+    QDict *resp;
+    QList *cpus;
+    QObject *e;
+    QTestState *qts;
+    g_autofree char *cli = NULL;
+
+    cli = make_cli(data, "-machine "
+        "smp.cpus=2,smp.sockets=2,smp.cores=1,smp.threads=1 "
+        "-numa node,nodeid=0,memdev=ram -numa node,nodeid=1 "
+        "-numa cpu,node-id=0,socket-id=1,core-id=0,thread-id=0 "
+        "-numa cpu,node-id=1,socket-id=0,core-id=0,thread-id=0");
+    qts = qtest_init(cli);
+    cpus = get_cpus(qts, &resp);
+    g_assert(cpus);
+
+    while ((e = qlist_pop(cpus))) {
+        QDict *cpu, *props;
+        int64_t socket, core, thread, node;
+
+        cpu = qobject_to(QDict, e);
+        g_assert(qdict_haskey(cpu, "props"));
+        props = qdict_get_qdict(cpu, "props");
+
+        g_assert(qdict_haskey(props, "node-id"));
+        node = qdict_get_int(props, "node-id");
+        g_assert(qdict_haskey(props, "socket-id"));
+        socket = qdict_get_int(props, "socket-id");
+        g_assert(qdict_haskey(props, "core-id"));
+        core = qdict_get_int(props, "core-id");
+        g_assert(qdict_haskey(props, "thread-id"));
+        thread = qdict_get_int(props, "thread-id");
+
+        if (socket == 0 && core == 0 && thread == 0) {
+            g_assert_cmpint(node, ==, 1);
+        } else if (socket == 1 && core == 0 && thread == 0) {
+            g_assert_cmpint(node, ==, 0);
+        } else {
+            g_assert(false);
+        }
+        qobject_unref(e);
+    }
+
+    qobject_unref(resp);
+    qtest_quit(qts);
+}
+
 static void pc_dynamic_cpu_cfg(const void *data)
 {
     QObject *e;
@@ -590,5 +638,10 @@ int main(int argc, char **argv)
                             aarch64_numa_cpu);
     }
 
+    if (!strcmp(arch, "loongarch64")) {
+        qtest_add_data_func("/numa/loongarch64/cpu/explicit", args,
+                            loongarch64_numa_cpu);
+    }
+
     return g_test_run();
 }
-- 
2.39.3

Re: [PATCH v3 1/6] hw/loongarch: Refine acpi srat table for numa memory

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> One LoongArch virt machine platform, there is limitation for memory
> map information. The minimum memory size is 256M and minimum memory
> size for numa node0 is 256M also. With qemu numa qtest, it is possible
> that memory size of numa node0 is 128M.
>
> Limitations for minimum memory size for both total memory and numa
> node0 is removed for acpi srat table creation.
>
> Signed-off-by: Bibo Mao <maobibo@loongson.cn>
> ---
Reviewed-by: Song Gao <gaosong@loongson.cn>

Thanks.
Song Gao
>   hw/loongarch/acpi-build.c | 58 +++++++++++++++++++++++----------------
>   1 file changed, 34 insertions(+), 24 deletions(-)
>
> diff --git a/hw/loongarch/acpi-build.c b/hw/loongarch/acpi-build.c
> index 5ef010d4da..af45ce526d 100644
> --- a/hw/loongarch/acpi-build.c
> +++ b/hw/loongarch/acpi-build.c
> @@ -166,8 +166,9 @@ static void
>   build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
>   {
>       int i, arch_id, node_id;
> -    uint64_t mem_len, mem_base;
> -    int nb_numa_nodes = machine->numa_state->num_nodes;
> +    hwaddr len, base, gap;
> +    NodeInfo *numa_info;
> +    int nodes, nb_numa_nodes = machine->numa_state->num_nodes;
>       LoongArchVirtMachineState *lvms = LOONGARCH_VIRT_MACHINE(machine);
>       MachineClass *mc = MACHINE_GET_CLASS(lvms);
>       const CPUArchIdList *arch_ids = mc->possible_cpu_arch_ids(machine);
> @@ -196,35 +197,44 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine)
>           build_append_int_noprefix(table_data, 0, 4); /* Reserved */
>       }
>   
> -    /* Node0 */
> -    build_srat_memory(table_data, VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE,
> -                      0, MEM_AFFINITY_ENABLED);
> -    mem_base = VIRT_HIGHMEM_BASE;
> -    if (!nb_numa_nodes) {
> -        mem_len = machine->ram_size - VIRT_LOWMEM_SIZE;
> -    } else {
> -        mem_len = machine->numa_state->nodes[0].node_mem - VIRT_LOWMEM_SIZE;
> +    base = VIRT_LOWMEM_BASE;
> +    gap = VIRT_LOWMEM_SIZE;
> +    numa_info = machine->numa_state->nodes;
> +    nodes = nb_numa_nodes;
> +    if (!nodes) {
> +        nodes = 1;
>       }
> -    if (mem_len)
> -        build_srat_memory(table_data, mem_base, mem_len, 0, MEM_AFFINITY_ENABLED);
> -
> -    /* Node1 - Nodemax */
> -    if (nb_numa_nodes) {
> -        mem_base += mem_len;
> -        for (i = 1; i < nb_numa_nodes; ++i) {
> -            if (machine->numa_state->nodes[i].node_mem > 0) {
> -                build_srat_memory(table_data, mem_base,
> -                                  machine->numa_state->nodes[i].node_mem, i,
> -                                  MEM_AFFINITY_ENABLED);
> -                mem_base += machine->numa_state->nodes[i].node_mem;
> -            }
> +
> +    for (i = 0; i < nodes; i++) {
> +        if (nb_numa_nodes) {
> +            len = numa_info[i].node_mem;
> +        } else {
> +            len = machine->ram_size;
> +        }
> +
> +        /*
> +         * memory for the node splited into two part
> +         *   lowram:  [base, +gap)
> +         *   highram: [VIRT_HIGHMEM_BASE, +(len - gap))
> +         */
> +        if (len >= gap) {
> +            build_srat_memory(table_data, base, len, i, MEM_AFFINITY_ENABLED);
> +            len -= gap;
> +            base = VIRT_HIGHMEM_BASE;
> +            gap = machine->ram_size - VIRT_LOWMEM_SIZE;
> +        }
> +
> +        if (len) {
> +            build_srat_memory(table_data, base, len, i, MEM_AFFINITY_ENABLED);
> +            base += len;
> +            gap  -= len;
>           }
>       }
>   
>       if (machine->device_memory) {
>           build_srat_memory(table_data, machine->device_memory->base,
>                             memory_region_size(&machine->device_memory->mr),
> -                          nb_numa_nodes - 1,
> +                          nodes - 1,
>                             MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);
>       }
>   

Re: [PATCH v3 2/6] hw/loongarch: Refine fadt memory table for numa memory

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> One LoongArch virt machine platform, there is limitation for memory
> map information. The minimum memory size is 256M and minimum memory
> size for numa node0 is 256M also. With qemu numa qtest, it is possible
> that memory size of numa node0 is 128M.
>
> Limitations for minimum memory size for both total memory and numa
> node0 is removed for fadt numa memory table creation.
>
> Signed-off-by: Bibo Mao <maobibo@loongson.cn>
> ---
Reviewed-by: Song Gao <gaosong@loongson.cn>

Thanks.
Song Gao
>   hw/loongarch/virt.c | 46 ++++++++++++++++++++++++++++++++++++++++++---
>   1 file changed, 43 insertions(+), 3 deletions(-)
>
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index f0640d2d80..c996305d87 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -473,6 +473,48 @@ static void fdt_add_memory_node(MachineState *ms,
>       g_free(nodename);
>   }
>   
> +static void fdt_add_memory_nodes(MachineState *ms)
> +{
> +    hwaddr base, size, ram_size, gap;
> +    int i, nb_numa_nodes, nodes;
> +    NodeInfo *numa_info;
> +
> +    ram_size = ms->ram_size;
> +    base = VIRT_LOWMEM_BASE;
> +    gap = VIRT_LOWMEM_SIZE;
> +    nodes = nb_numa_nodes = ms->numa_state->num_nodes;
> +    numa_info = ms->numa_state->nodes;
> +    if (!nodes) {
> +        nodes = 1;
> +    }
> +
> +    for (i = 0; i < nodes; i++) {
> +        if (nb_numa_nodes) {
> +            size = numa_info[i].node_mem;
> +        } else {
> +            size = ram_size;
> +        }
> +
> +        /*
> +         * memory for the node splited into two part
> +         *   lowram:  [base, +gap)
> +         *   highram: [VIRT_HIGHMEM_BASE, +(len - gap))
> +         */
> +        if (size >= gap) {
> +            fdt_add_memory_node(ms, base, gap, i);
> +            size -= gap;
> +            base = VIRT_HIGHMEM_BASE;
> +            gap = ram_size - VIRT_LOWMEM_SIZE;
> +        }
> +
> +        if (size) {
> +            fdt_add_memory_node(ms, base, size, i);
> +            base += size;
> +            gap -= size;
> +        }
> +    }
> +}
> +
>   static void virt_build_smbios(LoongArchVirtMachineState *lvms)
>   {
>       MachineState *ms = MACHINE(lvms);
> @@ -915,10 +957,10 @@ static void virt_init(MachineState *machine)
>           lacpu->phy_id = machine->possible_cpus->cpus[i].arch_id;
>       }
>       fdt_add_cpu_nodes(lvms);
> +    fdt_add_memory_nodes(machine);
>   
>       /* Node0 memory */
>       memmap_add_entry(VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 1);
> -    fdt_add_memory_node(machine, VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 0);
>       memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
>                                machine->ram, offset, VIRT_LOWMEM_SIZE);
>       memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
> @@ -932,7 +974,6 @@ static void virt_init(MachineState *machine)
>       }
>       phyAddr = VIRT_HIGHMEM_BASE;
>       memmap_add_entry(phyAddr, highram_size, 1);
> -    fdt_add_memory_node(machine, phyAddr, highram_size, 0);
>       memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
>                                 machine->ram, offset, highram_size);
>       memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
> @@ -948,7 +989,6 @@ static void virt_init(MachineState *machine)
>                                    offset,  numa_info[i].node_mem);
>           memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
>           memmap_add_entry(phyAddr, numa_info[i].node_mem, 1);
> -        fdt_add_memory_node(machine, phyAddr, numa_info[i].node_mem, i);
>           offset += numa_info[i].node_mem;
>           phyAddr += numa_info[i].node_mem;
>       }

Re: [PATCH v3 3/6] hw/loongarch: Refine fwcfg memory map

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> Memory map table for fwcfg is used for UEFI BIOS, UEFI BIOS uses the first
> entry from fwcfg memory map as the first memory HOB, the second memory HOB
> will be used if the first memory HOB is used up.
>
> Memory map table for fwcfg does not care about numa node, however in
> generic the first memory HOB is part of numa node0, so that runtime
> memory of UEFI which is allocated from the first memory HOB is located
> at numa node0.
>
> Signed-off-by: Bibo Mao <maobibo@loongson.cn>
> ---
Reviewed-by: Song Gao <gaosong@loongson.cn>

Thanks.
Song Gao
>   hw/loongarch/virt.c | 60 ++++++++++++++++++++++++++++++++++++++++++---
>   1 file changed, 57 insertions(+), 3 deletions(-)
>
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index c996305d87..b67d691fa5 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -912,6 +912,62 @@ static const MemoryRegionOps virt_iocsr_misc_ops = {
>       },
>   };
>   
> +static void fw_cfg_add_memory(MachineState *ms)
> +{
> +    hwaddr base, size, ram_size, gap;
> +    int nb_numa_nodes, nodes;
> +    NodeInfo *numa_info;
> +
> +    ram_size = ms->ram_size;
> +    base = VIRT_LOWMEM_BASE;
> +    gap = VIRT_LOWMEM_SIZE;
> +    nodes = nb_numa_nodes = ms->numa_state->num_nodes;
> +    numa_info = ms->numa_state->nodes;
> +    if (!nodes) {
> +        nodes = 1;
> +    }
> +
> +    /* add fw_cfg memory map of node0 */
> +    if (nb_numa_nodes) {
> +        size = numa_info[0].node_mem;
> +    } else {
> +        size = ram_size;
> +    }
> +
> +    if (size >= gap) {
> +        memmap_add_entry(base, gap, 1);
> +        size -= gap;
> +        base = VIRT_HIGHMEM_BASE;
> +        gap = ram_size - VIRT_LOWMEM_SIZE;
> +    }
> +
> +    if (size) {
> +        memmap_add_entry(base, size, 1);
> +        base += size;
> +    }
> +
> +    if (nodes < 2) {
> +        return;
> +    }
> +
> +    /* add fw_cfg memory map of other nodes */
> +    size = ram_size - numa_info[0].node_mem;
> +    gap  = VIRT_LOWMEM_BASE + VIRT_LOWMEM_SIZE;
> +    if (base < gap && (base + size) > gap) {
> +        /*
> +         * memory map for the maining nodes splited into two part
> +         *   lowram:  [base, +(gap - base))
> +         *   highram: [VIRT_HIGHMEM_BASE, +(size - (gap - base)))
> +         */
> +        memmap_add_entry(base, gap - base, 1);
> +        size -= gap - base;
> +        base = VIRT_HIGHMEM_BASE;
> +    }
> +
> +   if (size)
> +        memmap_add_entry(base, size, 1);
> +}
> +
>   static void virt_init(MachineState *machine)
>   {
>       LoongArchCPU *lacpu;
> @@ -958,9 +1014,9 @@ static void virt_init(MachineState *machine)
>       }
>       fdt_add_cpu_nodes(lvms);
>       fdt_add_memory_nodes(machine);
> +    fw_cfg_add_memory(machine);
>   
>       /* Node0 memory */
> -    memmap_add_entry(VIRT_LOWMEM_BASE, VIRT_LOWMEM_SIZE, 1);
>       memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
>                                machine->ram, offset, VIRT_LOWMEM_SIZE);
>       memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
> @@ -973,7 +1029,6 @@ static void virt_init(MachineState *machine)
>           highram_size = ram_size - VIRT_LOWMEM_SIZE;
>       }
>       phyAddr = VIRT_HIGHMEM_BASE;
> -    memmap_add_entry(phyAddr, highram_size, 1);
>       memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
>                                 machine->ram, offset, highram_size);
>       memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
> @@ -988,7 +1043,6 @@ static void virt_init(MachineState *machine)
>           memory_region_init_alias(nodemem, NULL, ramName, machine->ram,
>                                    offset,  numa_info[i].node_mem);
>           memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
> -        memmap_add_entry(phyAddr, numa_info[i].node_mem, 1);
>           offset += numa_info[i].node_mem;
>           phyAddr += numa_info[i].node_mem;
>       }

Re: [PATCH v3 4/6] hw/loongarch: Refine system dram memory region

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> For system dram memory region, it is not necessary to use numa node
> information. There is only low memory region and high memory region.
>
> Remove numa node information for ddr memory region here, it can reduce
> memory region number on LoongArch virt machine.
>
> Signed-off-by: Bibo Mao <maobibo@loongson.cn>
> ---
>   hw/loongarch/virt.c | 55 +++++++++++++++------------------------------
>   1 file changed, 18 insertions(+), 37 deletions(-)
>
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index b67d691fa5..ac980aec8e 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -972,14 +972,10 @@ static void virt_init(MachineState *machine)
>   {
>       LoongArchCPU *lacpu;
>       const char *cpu_model = machine->cpu_type;
> -    ram_addr_t offset = 0;
> -    ram_addr_t ram_size = machine->ram_size;
> -    uint64_t highram_size = 0, phyAddr = 0;
>       MemoryRegion *address_space_mem = get_system_memory();
>       LoongArchVirtMachineState *lvms = LOONGARCH_VIRT_MACHINE(machine);
> -    int nb_numa_nodes = machine->numa_state->num_nodes;
> -    NodeInfo *numa_info = machine->numa_state->nodes;
>       int i;
> +    hwaddr base, size, ram_size = machine->ram_size;
>       const CPUArchIdList *possible_cpus;
>       MachineClass *mc = MACHINE_GET_CLASS(machine);
>       CPUState *cpu;
> @@ -1017,40 +1013,27 @@ static void virt_init(MachineState *machine)
>       fw_cfg_add_memory(machine);
>   
>       /* Node0 memory */
> -    memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.node0.lowram",
> -                             machine->ram, offset, VIRT_LOWMEM_SIZE);
> -    memory_region_add_subregion(address_space_mem, phyAddr, &lvms->lowmem);
> -
> -    offset += VIRT_LOWMEM_SIZE;
> -    if (nb_numa_nodes > 0) {
> -        assert(numa_info[0].node_mem > VIRT_LOWMEM_SIZE);
> -        highram_size = numa_info[0].node_mem - VIRT_LOWMEM_SIZE;
> -    } else {
> -        highram_size = ram_size - VIRT_LOWMEM_SIZE;
> -    }
> -    phyAddr = VIRT_HIGHMEM_BASE;
> -    memory_region_init_alias(&lvms->highmem, NULL, "loongarch.node0.highram",
> -                              machine->ram, offset, highram_size);
> -    memory_region_add_subregion(address_space_mem, phyAddr, &lvms->highmem);
> -
> -    /* Node1 - Nodemax memory */
> -    offset += highram_size;
> -    phyAddr += highram_size;
> -
> -    for (i = 1; i < nb_numa_nodes; i++) {
> -        MemoryRegion *nodemem = g_new(MemoryRegion, 1);
> -        g_autofree char *ramName = g_strdup_printf("loongarch.node%d.ram", i);
> -        memory_region_init_alias(nodemem, NULL, ramName, machine->ram,
> -                                 offset,  numa_info[i].node_mem);
> -        memory_region_add_subregion(address_space_mem, phyAddr, nodemem);
> -        offset += numa_info[i].node_mem;
> -        phyAddr += numa_info[i].node_mem;
> +    size = ram_size;
> +    base = VIRT_LOWMEM_BASE;
> +    if (size > VIRT_LOWMEM_SIZE) {
> +        size = VIRT_LOWMEM_SIZE;
>       }
>   
> +    memory_region_init_alias(&lvms->lowmem, NULL, "loongarch.lowram",
> +                              machine->ram, base, size);
> +    memory_region_add_subregion(address_space_mem, base, &lvms->lowmem);
> +    base += size;
> +    if (ram_size - size) {
> +        base = VIRT_HIGHMEM_BASE;
> +        memory_region_init_alias(&lvms->highmem, NULL, "loongarch.highram",
> +                machine->ram, VIRT_LOWMEM_BASE + size, ram_size - size);
> +        memory_region_add_subregion(address_space_mem, base, &lvms->highmem);
> +        base += ram_size - size;
> +     }
an extra space before '}'

Reviewed-by: Song Gao <gaosong@loongson.cn>

Thanks.
Song Gao
> +
>       /* initialize device memory address space */
>       if (machine->ram_size < machine->maxram_size) {
>           ram_addr_t device_mem_size = machine->maxram_size - machine->ram_size;
> -        hwaddr device_mem_base;
>   
>           if (machine->ram_slots > ACPI_MAX_RAM_SLOTS) {
>               error_report("unsupported amount of memory slots: %"PRIu64,
> @@ -1064,9 +1047,7 @@ static void virt_init(MachineState *machine)
>                            "%d bytes", TARGET_PAGE_SIZE);
>               exit(EXIT_FAILURE);
>           }
> -        /* device memory base is the top of high memory address. */
> -        device_mem_base = ROUND_UP(VIRT_HIGHMEM_BASE + highram_size, 1 * GiB);
> -        machine_memory_devices_init(machine, device_mem_base, device_mem_size);
> +        machine_memory_devices_init(machine, base, device_mem_size);
>       }
>   
>       /* load the BIOS image. */

Re: [PATCH v3 5/6] hw/loongarch: Remove minimum and default memory size

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> Some qtest test cases such as numa use default memory size of generic
> machine class, which is 128M by fault.
>
> Here generic default memory size is used, and also remove minimum memory
> size which is 1G originally.
>
> Signed-off-by: Bibo Mao <maobibo@loongson.cn>
> ---
Reviewed-by: Song Gao <gaosong@loongson.cn>

Thanks.
Song Gao
>   hw/loongarch/virt.c | 5 -----
>   1 file changed, 5 deletions(-)
>
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index ac980aec8e..7d3d1d1689 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -984,10 +984,6 @@ static void virt_init(MachineState *machine)
>           cpu_model = LOONGARCH_CPU_TYPE_NAME("la464");
>       }
>   
> -    if (ram_size < 1 * GiB) {
> -        error_report("ram_size must be greater than 1G.");
> -        exit(1);
> -    }
>       create_fdt(lvms);
>   
>       /* Create IOCSR space */
> @@ -1273,7 +1269,6 @@ static void virt_class_init(ObjectClass *oc, void *data)
>       HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
>   
>       mc->init = virt_init;
> -    mc->default_ram_size = 1 * GiB;
>       mc->default_cpu_type = LOONGARCH_CPU_TYPE_NAME("la464");
>       mc->default_ram_id = "loongarch.ram";
>       mc->max_cpus = LOONGARCH_MAX_CPUS;

Re: [PATCH v3 0/6] hw/loongarch: Refine numa memory map

[gaosong]

在 2024/5/15 下午5:39, Bibo Mao 写道:
> One LoongArch virt machine platform, there is limitation for memory
> map information. The minimum memory size is 256M and minimum memory
> size for numa node0 is 256M also. With qemu numa qtest, it is possible
> that memory size of numa node0 is 128M.
>
> Limitations for minimum memory size for both total memory and numa
> node0 is removed here, including acpi srat table, fadt memory map table
> and fw_cfg memory map table.
>
> Also remove numa node about memory region, there is only low memory
> region and how memory region.
>
> ---
> v2 ... v3:
>    1. Modify numa test case for loongarch system.
>    2. Rebase the patch based on the latest version.
>
> v1 ... v2:
>    1. Refresh the patch based on the latest qemu version, solve the
> confliction issue.
>    2. Add numa test case for loongarch system.
> ---
> Bibo Mao (6):
>    hw/loongarch: Refine acpi srat table for numa memory
>    hw/loongarch: Refine fadt memory table for numa memory
>    hw/loongarch: Refine fwcfg memory map
>    hw/loongarch: Refine system dram memory region
>    hw/loongarch: Remove minimum and default memory size
>    tests/qtest: Add numa test for loongarch system

Patches 1 -5  queued to loongarch-next, thanks.

Thanks.
Song Gao
>   hw/loongarch/acpi-build.c |  58 +++++++------
>   hw/loongarch/virt.c       | 166 +++++++++++++++++++++++++++-----------
>   tests/qtest/meson.build   |   2 +-
>   tests/qtest/numa-test.c   |  53 ++++++++++++
>   4 files changed, 206 insertions(+), 73 deletions(-)
>
>
> base-commit: dafec285bdbfe415ac6823abdc510e0b92c3f094