From: David Gibson <david@gibson.dropbear.id.au>
To: peter.maydell@linaro.org
Cc: danielhb413@gmail.com, qemu-devel@nongnu.org, groug@kaod.org,
qemu-ppc@nongnu.org, bauerman@linux.ibm.com,
David Gibson <david@gibson.dropbear.id.au>
Subject: [PULL 30/30] spapr_numa: move NVLink2 associativity handling to spapr_numa.c
Date: Fri, 4 Sep 2020 13:47:19 +1000 [thread overview]
Message-ID: <20200904034719.673626-31-david@gibson.dropbear.id.au> (raw)
In-Reply-To: <20200904034719.673626-1-david@gibson.dropbear.id.au>
From: Daniel Henrique Barboza <danielhb413@gmail.com>
The NVLink2 GPUs works like a regular NUMA node with its
own associativity values, regardless of user input.
This can be handled inside spapr_numa_associativity_init(),
initializing NVGPU_MAX_NUM associativity arrays that can
be used by the GPUs.
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Message-Id: <20200903220639.563090-5-danielhb413@gmail.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
hw/ppc/spapr_numa.c | 28 +++++++++++++++++++++++++++-
hw/ppc/spapr_pci_nvlink2.c | 20 +++-----------------
2 files changed, 30 insertions(+), 18 deletions(-)
diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c
index 5a82a84438..93a000b729 100644
--- a/hw/ppc/spapr_numa.c
+++ b/hw/ppc/spapr_numa.c
@@ -13,14 +13,18 @@
#include "qemu/osdep.h"
#include "qemu-common.h"
#include "hw/ppc/spapr_numa.h"
+#include "hw/pci-host/spapr.h"
#include "hw/ppc/fdt.h"
+/* Moved from hw/ppc/spapr_pci_nvlink2.c */
+#define SPAPR_GPU_NUMA_ID (cpu_to_be32(1))
void spapr_numa_associativity_init(SpaprMachineState *spapr,
MachineState *machine)
{
+ SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr);
int nb_numa_nodes = machine->numa_state->num_nodes;
- int i;
+ int i, j, max_nodes_with_gpus;
/*
* For all associativity arrays: first position is the size,
@@ -35,6 +39,28 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr,
spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS);
spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i);
}
+
+ /*
+ * Initialize NVLink GPU associativity arrays. We know that
+ * the first GPU will take the first available NUMA id, and
+ * we'll have a maximum of NVGPU_MAX_NUM GPUs in the machine.
+ * At this point we're not sure if there are GPUs or not, but
+ * let's initialize the associativity arrays and allow NVLink
+ * GPUs to be handled like regular NUMA nodes later on.
+ */
+ max_nodes_with_gpus = nb_numa_nodes + NVGPU_MAX_NUM;
+
+ for (i = nb_numa_nodes; i < max_nodes_with_gpus; i++) {
+ spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS);
+
+ for (j = 1; j < MAX_DISTANCE_REF_POINTS; j++) {
+ uint32_t gpu_assoc = smc->pre_5_1_assoc_refpoints ?
+ SPAPR_GPU_NUMA_ID : cpu_to_be32(i);
+ spapr->numa_assoc_array[i][j] = gpu_assoc;
+ }
+
+ spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i);
+ }
}
void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt,
diff --git a/hw/ppc/spapr_pci_nvlink2.c b/hw/ppc/spapr_pci_nvlink2.c
index 76ae77ebc8..8ef9b40a18 100644
--- a/hw/ppc/spapr_pci_nvlink2.c
+++ b/hw/ppc/spapr_pci_nvlink2.c
@@ -26,6 +26,7 @@
#include "qemu-common.h"
#include "hw/pci/pci.h"
#include "hw/pci-host/spapr.h"
+#include "hw/ppc/spapr_numa.h"
#include "qemu/error-report.h"
#include "hw/ppc/fdt.h"
#include "hw/pci/pci_bridge.h"
@@ -37,8 +38,6 @@
#define PHANDLE_NVLINK(phb, gn, nn) (0x00130000 | (((phb)->index) << 8) | \
((gn) << 4) | (nn))
-#define SPAPR_GPU_NUMA_ID (cpu_to_be32(1))
-
typedef struct SpaprPhbPciNvGpuSlot {
uint64_t tgt;
uint64_t gpa;
@@ -360,13 +359,6 @@ void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState *sphb, void *fdt)
Object *nv_mrobj = object_property_get_link(OBJECT(nvslot->gpdev),
"nvlink2-mr[0]",
&error_abort);
- uint32_t associativity[] = {
- cpu_to_be32(0x4),
- cpu_to_be32(nvslot->numa_id),
- cpu_to_be32(nvslot->numa_id),
- cpu_to_be32(nvslot->numa_id),
- cpu_to_be32(nvslot->numa_id)
- };
uint64_t size = object_property_get_uint(nv_mrobj, "size", NULL);
uint64_t mem_reg[2] = { cpu_to_be64(nvslot->gpa), cpu_to_be64(size) };
char *mem_name = g_strdup_printf("memory@%"PRIx64, nvslot->gpa);
@@ -376,14 +368,8 @@ void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState *sphb, void *fdt)
_FDT((fdt_setprop_string(fdt, off, "device_type", "memory")));
_FDT((fdt_setprop(fdt, off, "reg", mem_reg, sizeof(mem_reg))));
- if (sphb->pre_5_1_assoc) {
- associativity[1] = SPAPR_GPU_NUMA_ID;
- associativity[2] = SPAPR_GPU_NUMA_ID;
- associativity[3] = SPAPR_GPU_NUMA_ID;
- }
-
- _FDT((fdt_setprop(fdt, off, "ibm,associativity", associativity,
- sizeof(associativity))));
+ spapr_numa_write_associativity_dt(SPAPR_MACHINE(qdev_get_machine()),
+ fdt, off, nvslot->numa_id);
_FDT((fdt_setprop_string(fdt, off, "compatible",
"ibm,coherent-device-memory")));
--
2.26.2
next prev parent reply other threads:[~2020-09-04 4:04 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-04 3:46 [PULL 00/30] ppc-for-5.2 queue 20200904 David Gibson
2020-09-04 3:46 ` [PULL 01/30] adb: Correct class size on TYPE_ADB_DEVICE David Gibson
2020-09-04 3:46 ` [PULL 02/30] ppc/pnv: Fix TypeInfo of PnvLpcController abstract class David Gibson
2020-09-04 3:46 ` [PULL 03/30] spapr: Remove unnecessary DRC type-checker macros David Gibson
2020-09-04 3:46 ` [PULL 04/30] spapr/xive: Add a 'hv-prio' property to represent the KVM escalation priority David Gibson
2020-09-04 3:46 ` [PULL 05/30] ppc/pnv: Add a HIOMAP erase command David Gibson
2020-09-04 3:46 ` [PULL 06/30] spapr_vscsi: do not allow device hotplug David Gibson
2020-09-04 3:46 ` [PULL 07/30] spapr/xive: Use the xics flag to check for XIVE-only IRQ backends David Gibson
2020-09-04 3:46 ` [PULL 08/30] spapr/xive: Modify kvm_cpu_is_enabled() interface David Gibson
2020-09-04 3:46 ` [PULL 09/30] spapr/xive: Use kvmppc_xive_source_reset() in post_load David Gibson
2020-09-04 3:46 ` [PULL 10/30] spapr/xive: Allocate IPIs independently from the other sources David Gibson
2020-09-04 3:47 ` [PULL 11/30] spapr/xive: Allocate vCPU IPIs from the vCPU contexts David Gibson
2020-09-04 3:47 ` [PULL 12/30] ppc/spapr_nvdimm: use g_autofree in spapr_nvdimm_validate_opts() David Gibson
2020-09-04 3:47 ` [PULL 13/30] spapr, spapr_nvdimm: fold NVDIMM validation in the same place David Gibson
2020-09-04 3:47 ` [PULL 14/30] ppc/spapr_nvdimm: do not enable support with 'nvdimm=off' David Gibson
2020-09-04 3:47 ` [PULL 15/30] target/arm: Move start-powered-off property to generic CPUState David Gibson
2020-09-04 3:47 ` [PULL 16/30] target/arm: Move setting of CPU halted state to generic code David Gibson
2020-09-04 3:47 ` [PULL 17/30] ppc/spapr: Use start-powered-off CPUState property David Gibson
2020-09-04 3:47 ` [PULL 18/30] ppc/e500: " David Gibson
2020-09-04 3:47 ` [PULL 19/30] mips/cps: " David Gibson
2020-09-04 3:47 ` [PULL 20/30] sparc/sun4m: Don't set cs->halted = 0 in main_cpu_reset() David Gibson
2020-09-04 3:47 ` [PULL 21/30] sparc/sun4m: Use start-powered-off CPUState property David Gibson
2020-09-04 3:47 ` [PULL 22/30] target/s390x: " David Gibson
2020-09-04 3:47 ` [PULL 23/30] hw/ppc/ppc4xx_pci: Use ARRAY_SIZE() instead of magic value David Gibson
2020-09-04 3:47 ` [PULL 24/30] hw/ppc/ppc4xx_pci: Replace pointless warning by assert() David Gibson
2020-09-04 3:47 ` [PULL 25/30] ppc: introducing spapr_numa.c NUMA code helper David Gibson
2020-09-04 3:47 ` [PULL 26/30] ppc/spapr_nvdimm: turn spapr_dt_nvdimm() static David Gibson
2020-09-04 3:47 ` [PULL 27/30] spapr: introduce SpaprMachineState::numa_assoc_array David Gibson
2020-09-04 3:47 ` [PULL 28/30] spapr, spapr_numa: handle vcpu ibm,associativity David Gibson
2020-09-04 3:47 ` [PULL 29/30] spapr, spapr_numa: move lookup-arrays handling to spapr_numa.c David Gibson
2020-09-04 3:47 ` David Gibson [this message]
2020-09-06 15:20 ` [PULL 00/30] ppc-for-5.2 queue 20200904 Peter Maydell
2020-09-07 2:38 ` David Gibson
2020-09-07 13:29 ` Laurent Vivier
2020-09-07 14:05 ` Philippe Mathieu-Daudé
2020-09-07 14:31 ` Laurent Vivier
2020-09-07 14:51 ` Cornelia Huck
2020-09-07 16:29 ` Laurent Vivier
2020-09-07 17:26 ` Laurent Vivier
2020-09-07 19:46 ` Philippe Mathieu-Daudé
2020-09-07 23:50 ` David Gibson
2020-09-08 6:11 ` Cornelia Huck
2020-09-08 15:12 ` Thiago Jung Bauermann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200904034719.673626-31-david@gibson.dropbear.id.au \
--to=david@gibson.dropbear.id.au \
--cc=bauerman@linux.ibm.com \
--cc=danielhb413@gmail.com \
--cc=groug@kaod.org \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
--cc=qemu-ppc@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).