* [PATCH v2 1/8] powerpc/numa: Look up device node in of_get_assoc_arrays()
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
@ 2017-10-20 13:21 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 2/8] powerpc/numa: Look up device node in of_get_usable_memory() Nathan Fontenot
` (7 subsequent siblings)
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:21 UTC (permalink / raw)
To: linuxppc-dev
Look up the device node for the associativity array property instead
of having it passed in as a parameter. This changes precedes an update
in which the calling routines for of_get_assoc_arrays() will not have
the device node pointer to pass in.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
arch/powerpc/mm/numa.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index b95c584ce19d..ca5cc1d4d387 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -466,19 +466,27 @@ struct assoc_arrays {
* indicating the size of each associativity array, followed by a list
* of N associativity arrays.
*/
-static int of_get_assoc_arrays(struct device_node *memory,
- struct assoc_arrays *aa)
+static int of_get_assoc_arrays(struct assoc_arrays *aa)
{
+ struct device_node *memory;
const __be32 *prop;
u32 len;
+ memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!memory)
+ return -1;
+
prop = of_get_property(memory, "ibm,associativity-lookup-arrays", &len);
- if (!prop || len < 2 * sizeof(unsigned int))
+ if (!prop || len < 2 * sizeof(unsigned int)) {
+ of_node_put(memory);
return -1;
+ }
aa->n_arrays = of_read_number(prop++, 1);
aa->array_sz = of_read_number(prop++, 1);
+ of_node_put(memory);
+
/* Now that we know the number of arrays and size of each array,
* revalidate the size of the property read in.
*/
@@ -661,7 +669,7 @@ static void __init parse_drconf_memory(struct device_node *memory)
if (!lmb_size)
return;
- rc = of_get_assoc_arrays(memory, &aa);
+ rc = of_get_assoc_arrays(&aa);
if (rc)
return;
@@ -996,7 +1004,7 @@ static int hot_add_drconf_scn_to_nid(struct device_node *memory,
if (!lmb_size)
return -1;
- rc = of_get_assoc_arrays(memory, &aa);
+ rc = of_get_assoc_arrays(&aa);
if (rc)
return -1;
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 2/8] powerpc/numa: Look up device node in of_get_usable_memory()
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
2017-10-20 13:21 ` [PATCH v2 1/8] powerpc/numa: Look up device node in of_get_assoc_arrays() Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format Nathan Fontenot
` (6 subsequent siblings)
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
Look up the device node for the usable memory property instead
of having it passed in as a parameter. This changes precedes an update
in which the calling routines for of_get_usable_memory() will not have
the device node pointer to pass in.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
arch/powerpc/mm/numa.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index ca5cc1d4d387..a9aa353d41cd 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -184,11 +184,19 @@ static const __be32 *of_get_associativity(struct device_node *dev)
* it exists (the property exists only in kexec/kdump kernels,
* added by kexec-tools)
*/
-static const __be32 *of_get_usable_memory(struct device_node *memory)
+static const __be32 *of_get_usable_memory(void)
{
+ struct device_node *memory;
const __be32 *prop;
u32 len;
+
+ memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!memory)
+ return NULL;
+
prop = of_get_property(memory, "linux,drconf-usable-memory", &len);
+ of_node_put(memory);
+
if (!prop || len < sizeof(unsigned int))
return NULL;
return prop;
@@ -674,7 +682,7 @@ static void __init parse_drconf_memory(struct device_node *memory)
return;
/* check if this is a kexec/kdump kernel */
- usm = of_get_usable_memory(memory);
+ usm = of_get_usable_memory();
if (usm != NULL)
is_kexec_kdump = 1;
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
2017-10-20 13:21 ` [PATCH v2 1/8] powerpc/numa: Look up device node in of_get_assoc_arrays() Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 2/8] powerpc/numa: Look up device node in of_get_usable_memory() Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-24 6:08 ` Michael Ellerman
2017-11-12 12:43 ` Michael Ellerman
2017-10-20 13:22 ` [PATCH v2 4/8] powerpc/numa: Update numa code use drmem LMB array Nathan Fontenot
` (5 subsequent siblings)
8 siblings, 2 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
We currently have code to parse the dynamic reconfiguration LMB
information from the ibm,dynamic-meory device tree property in
multiple locations (numa.c, prom.c, and pseries/hotplug-memory.c).
In anticipation of adding support for a version 2 of the
ibm,dynamic-memory property this patch aims to separate the device
tree information from the device tree format.
To do this an array of per-LMB information is created at boot time
and any pieces of code that need to look at LMB information are then
updated to use this array instead of parsing the device tree directly.
Doing this provides several benefits. The device tree property
parsing is located in a single place, it makes walking through the
possible LMBs on a system easier, and provides a common data structure
so that any piece of the kernel needing LMB information does not
have to provide multiple parsing routines for the supported device
tree formats.
This patch consolidates the device tree parsing into a common
location and builds the LMB array at boot time in a new file
powerpc/mm/drmem.c This patch also updates the prom
code to use the new LMB array.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
Updates for V2: Correct build break for non-pseries builds.
---
arch/powerpc/include/asm/drmem.h | 45 +++++++++++++
arch/powerpc/kernel/prom.c | 128 +++++++++++++++++++-------------------
arch/powerpc/mm/Makefile | 2 -
arch/powerpc/mm/drmem.c | 84 +++++++++++++++++++++++++
4 files changed, 195 insertions(+), 64 deletions(-)
create mode 100644 arch/powerpc/include/asm/drmem.h
create mode 100644 arch/powerpc/mm/drmem.c
diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
new file mode 100644
index 000000000000..cafe1a3b7da6
--- /dev/null
+++ b/arch/powerpc/include/asm/drmem.h
@@ -0,0 +1,45 @@
+/*
+ * drmem.h: Power specific logical memory block representation
+ *
+ * Copyright 2017 IBM Corporation
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _ASM_POWERPC_LMB_H
+#define _ASM_POWERPC_LMB_H
+
+struct drmem_lmb {
+ u64 base_addr;
+ u32 drc_index;
+ u32 aa_index;
+ u32 flags;
+};
+
+struct drmem_lmb_info {
+ struct drmem_lmb *lmbs;
+ int n_lmbs;
+ u32 lmb_size;
+};
+
+extern struct drmem_lmb_info *drmem_info;
+
+#define for_each_drmem_lmb_in_range(lmb, start, end) \
+ for ((lmb) = (start); (lmb) <= (end); (lmb)++)
+
+#define for_each_drmem_lmb(lmb) \
+ for_each_drmem_lmb_in_range((lmb), \
+ &drmem_info->lmbs[0], \
+ &drmem_info->lmbs[drmem_info->n_lmbs - 1])
+
+static inline u32 drmem_lmb_size(void)
+{
+ return drmem_info->lmb_size;
+}
+
+extern int __init init_drmem_lmbs(unsigned long node);
+
+#endif
diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
index f83056297441..917184c13890 100644
--- a/arch/powerpc/kernel/prom.c
+++ b/arch/powerpc/kernel/prom.c
@@ -58,6 +58,7 @@
#include <asm/epapr_hcalls.h>
#include <asm/firmware.h>
#include <asm/dt_cpu_ftrs.h>
+#include <asm/drmem.h>
#include <mm/mmu_decl.h>
@@ -454,92 +455,93 @@ static int __init early_init_dt_scan_chosen_ppc(unsigned long node,
#ifdef CONFIG_PPC_PSERIES
/*
- * Interpret the ibm,dynamic-memory property in the
- * /ibm,dynamic-reconfiguration-memory node.
+ * Interpret the ibm dynamic reconfiguration memory LMBs.
* This contains a list of memory blocks along with NUMA affinity
* information.
*/
-static int __init early_init_dt_scan_drconf_memory(unsigned long node)
+
+static void __init add_drmem_lmb(struct drmem_lmb *lmb, const __be32 **usm)
{
- const __be32 *dm, *ls, *usm;
- int l;
- unsigned long n, flags;
- u64 base, size, memblock_size;
- unsigned int is_kexec_kdump = 0, rngs;
+ u64 base, size;
+ int is_kexec_kdump = 0, rngs;
- ls = of_get_flat_dt_prop(node, "ibm,lmb-size", &l);
- if (ls == NULL || l < dt_root_size_cells * sizeof(__be32))
- return 0;
- memblock_size = dt_mem_next_cell(dt_root_size_cells, &ls);
+ base = lmb->base_addr;
+ size = drmem_lmb_size();
+ rngs = 1;
- dm = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &l);
- if (dm == NULL || l < sizeof(__be32))
- return 0;
+ if (*usm)
+ is_kexec_kdump = 1;
- n = of_read_number(dm++, 1); /* number of entries */
- if (l < (n * (dt_root_addr_cells + 4) + 1) * sizeof(__be32))
- return 0;
+ if (is_kexec_kdump) {
+ /*
+ * For each memblock in ibm,dynamic-memory, a
+ * corresponding entry in linux,drconf-usable-memory
+ * property contains a counter 'p' followed by 'p'
+ * (base, size) duple. Now read the counter from
+ * linux,drconf-usable-memory property
+ */
+ rngs = dt_mem_next_cell(dt_root_size_cells, usm);
+ if (!rngs) /* there are no (base, size) duple */
+ return;
+ }
+
+ do {
+ if (is_kexec_kdump) {
+ base = dt_mem_next_cell(dt_root_addr_cells, usm);
+ size = dt_mem_next_cell(dt_root_size_cells, usm);
+ }
+
+ if (iommu_is_off) {
+ if (base >= 0x80000000ul)
+ continue;
+ if ((base + size) > 0x80000000ul)
+ size = 0x80000000ul - base;
+ }
+
+ memblock_add(base, size);
+ } while (--rngs);
+}
+
+static void __init early_init_dt_scan_drmem_lmbs(unsigned long node)
+{
+ struct drmem_lmb *lmb;
+ const __be32 *usm;
+ int l;
/* check if this is a kexec/kdump kernel. */
- usm = of_get_flat_dt_prop(node, "linux,drconf-usable-memory",
- &l);
- if (usm != NULL)
- is_kexec_kdump = 1;
+ usm = of_get_flat_dt_prop(node, "linux,drconf-usable-memory", &l);
- for (; n != 0; --n) {
- base = dt_mem_next_cell(dt_root_addr_cells, &dm);
- flags = of_read_number(&dm[3], 1);
- /* skip DRC index, pad, assoc. list index, flags */
- dm += 4;
+ for_each_drmem_lmb(lmb) {
/* skip this block if the reserved bit is set in flags
- or if the block is not assigned to this partition */
- if ((flags & DRCONF_MEM_RESERVED) ||
- !(flags & DRCONF_MEM_ASSIGNED))
+ * or if the block is not assigned to this partition.
+ */
+ if ((lmb->flags & DRCONF_MEM_RESERVED) ||
+ !(lmb->flags & DRCONF_MEM_ASSIGNED))
continue;
- size = memblock_size;
- rngs = 1;
- if (is_kexec_kdump) {
- /*
- * For each memblock in ibm,dynamic-memory, a corresponding
- * entry in linux,drconf-usable-memory property contains
- * a counter 'p' followed by 'p' (base, size) duple.
- * Now read the counter from
- * linux,drconf-usable-memory property
- */
- rngs = dt_mem_next_cell(dt_root_size_cells, &usm);
- if (!rngs) /* there are no (base, size) duple */
- continue;
- }
- do {
- if (is_kexec_kdump) {
- base = dt_mem_next_cell(dt_root_addr_cells,
- &usm);
- size = dt_mem_next_cell(dt_root_size_cells,
- &usm);
- }
- if (iommu_is_off) {
- if (base >= 0x80000000ul)
- continue;
- if ((base + size) > 0x80000000ul)
- size = 0x80000000ul - base;
- }
- memblock_add(base, size);
- } while (--rngs);
+
+ add_drmem_lmb(lmb, &usm);
}
+
memblock_dump_all();
- return 0;
}
#else
-#define early_init_dt_scan_drconf_memory(node) 0
+static void __init early_init_dt_scan_drmem_lmbs(unsigned long node) {}
#endif /* CONFIG_PPC_PSERIES */
static int __init early_init_dt_scan_memory_ppc(unsigned long node,
const char *uname,
int depth, void *data)
{
+ int rc;
+
if (depth == 1 &&
- strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0)
- return early_init_dt_scan_drconf_memory(node);
+ strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0) {
+ rc = init_drmem_lmbs(node);
+ if (!rc)
+ early_init_dt_scan_drmem_lmbs(node);
+
+ return rc;
+ }
return early_init_dt_scan_memory(node, uname, depth, data);
}
diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile
index fb844d2f266e..8186d5721bfe 100644
--- a/arch/powerpc/mm/Makefile
+++ b/arch/powerpc/mm/Makefile
@@ -8,7 +8,7 @@ ccflags-$(CONFIG_PPC64) := $(NO_MINIMAL_TOC)
obj-y := fault.o mem.o pgtable.o mmap.o \
init_$(BITS).o pgtable_$(BITS).o \
- init-common.o mmu_context.o
+ init-common.o mmu_context.o drmem.o
obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \
tlb_nohash_low.o
obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o
diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
new file mode 100644
index 000000000000..8ad7cf36b2c4
--- /dev/null
+++ b/arch/powerpc/mm/drmem.c
@@ -0,0 +1,84 @@
+/*
+ * Dynamic reconfiguration memory support
+ *
+ * Copyright 2017 IBM Corporation
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#define pr_fmt(fmt) "drmem: " fmt
+
+#include <linux/kernel.h>
+#include <linux/of.h>
+#include <linux/of_fdt.h>
+#include <linux/bootmem.h>
+#include <asm/prom.h>
+#include <asm/drmem.h>
+
+static struct drmem_lmb_info __drmem_info;
+struct drmem_lmb_info *drmem_info = &__drmem_info;
+
+int __init init_drmem_lmbs(unsigned long node)
+{
+ struct drmem_lmb *lmb;
+ const __be32 *prop;
+ int prop_sz;
+ u32 len;
+
+ prop = of_get_flat_dt_prop(node, "ibm,lmb-size", &len);
+ if (!prop || len < dt_root_size_cells * sizeof(__be32))
+ return -1;
+
+ drmem_info->lmb_size = dt_mem_next_cell(dt_root_size_cells, &prop);
+
+ prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &len);
+ if (!prop || len < dt_root_size_cells * sizeof(__be32))
+ return -1;
+
+ drmem_info->n_lmbs = of_read_number(prop++, 1);
+ prop_sz = drmem_info->n_lmbs * sizeof(struct of_drconf_cell)
+ + sizeof(__be32);
+ if (prop_sz < len)
+ return -1;
+
+ drmem_info->lmbs = alloc_bootmem(drmem_info->n_lmbs * sizeof(*lmb));
+ if (!drmem_info->lmbs)
+ return -1;
+
+ for_each_drmem_lmb(lmb) {
+ lmb->base_addr = dt_mem_next_cell(dt_root_addr_cells, &prop);
+ lmb->drc_index = of_read_number(prop++, 1);
+
+ prop++; /* skip reserved field */
+
+ lmb->aa_index = of_read_number(prop++, 1);
+ lmb->flags = of_read_number(prop++, 1);
+ }
+
+ return 0;
+}
+
+static int __init drmem_init(void)
+{
+ struct drmem_lmb *lmbs;
+ u32 lmbs_sz;
+
+ if (drmem_info->lmbs) {
+ lmbs_sz = drmem_info->n_lmbs * sizeof(*lmbs);
+ lmbs = kcalloc(drmem_info->n_lmbs, sizeof(*lmbs), GFP_KERNEL);
+
+ if (lmbs) {
+ memcpy(lmbs, drmem_info->lmbs, lmbs_sz);
+ free_bootmem(__pa(drmem_info->lmbs), lmbs_sz);
+ drmem_info->lmbs = lmbs;
+ }
+ }
+
+ return 0;
+}
+
+late_initcall(drmem_init);
+
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-10-20 13:22 ` [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format Nathan Fontenot
@ 2017-10-24 6:08 ` Michael Ellerman
2017-10-24 20:33 ` Nathan Fontenot
2017-11-12 12:43 ` Michael Ellerman
1 sibling, 1 reply; 19+ messages in thread
From: Michael Ellerman @ 2017-10-24 6:08 UTC (permalink / raw)
To: Nathan Fontenot, linuxppc-dev
Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
> new file mode 100644
> index 000000000000..8ad7cf36b2c4
> --- /dev/null
> +++ b/arch/powerpc/mm/drmem.c
> @@ -0,0 +1,84 @@
> +/*
> + * Dynamic reconfiguration memory support
> + *
> + * Copyright 2017 IBM Corporation
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#define pr_fmt(fmt) "drmem: " fmt
> +
> +#include <linux/kernel.h>
> +#include <linux/of.h>
> +#include <linux/of_fdt.h>
> +#include <linux/bootmem.h>
> +#include <asm/prom.h>
> +#include <asm/drmem.h>
> +
> +static struct drmem_lmb_info __drmem_info;
> +struct drmem_lmb_info *drmem_info = &__drmem_info;
> +
> +int __init init_drmem_lmbs(unsigned long node)
> +{
Something in here is blowing up for me.
This is a p8 LPAR, which uses petitboot so we kexec into the kernel,
gives me:
[ 29.020618] kexec_core: Starting new kernel
[ 0.000000] bootconsole [udbg0] enabled
-> early_setup(), dt_ptr: 0x1ec60000
[ 0.000000] bootmem alloc of 3024 bytes failed!
[ 0.000000] Kernel panic - not syncing: Out of memory
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-rc2-gcc6-gf7212f8 #1
[ 0.000000] Call Trace:
[ 0.000000] [c000000000f57b80] [c000000000a2bae0] dump_stack+0xb0/0xf0 (unreliable)
[ 0.000000] [c000000000f57bc0] [c0000000000fcdb0] panic+0x148/0x338
[ 0.000000] [c000000000f57c60] [c000000000d4bb08] ___alloc_bootmem.part.1+0x3c/0x40
[ 0.000000] [c000000000f57cc0] [c000000000d4bf18] __alloc_bootmem+0x3c/0x50
[ 0.000000] [c000000000f57cf0] [c000000000d226c4] init_drmem_lmbs+0xf8/0x31c
[ 0.000000] [c000000000f57d70] [c000000000d1a05c] early_init_dt_scan_memory_ppc+0x88/0x25c
[ 0.000000] [c000000000f57e10] [0000000000d78e78] 0xd78e78
[ 0.000000] [c000000000f57e70] [0000000000d1a92c] 0xd1a92c
[ 0.000000] [c000000000f57f10] [0000000000d1be3c] 0xd1be3c
[ 0.000000] [c000000000f57f90] [000000000000b13c] 0xb13c
[ 0.000000] Rebooting in 180 seconds..
[ 0.000000] System Halted, OK to turn off power
I'm at kernel summit so haven't had time to look any further sorry.
cheers
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-10-24 6:08 ` Michael Ellerman
@ 2017-10-24 20:33 ` Nathan Fontenot
2017-10-25 9:16 ` Michael Ellerman
0 siblings, 1 reply; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-24 20:33 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev
On 10/24/2017 01:08 AM, Michael Ellerman wrote:
> Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
>
>> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
>> new file mode 100644
>> index 000000000000..8ad7cf36b2c4
>> --- /dev/null
>> +++ b/arch/powerpc/mm/drmem.c
>> @@ -0,0 +1,84 @@
>> +/*
>> + * Dynamic reconfiguration memory support
>> + *
>> + * Copyright 2017 IBM Corporation
>> + *
>> + * This program is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU General Public License
>> + * as published by the Free Software Foundation; either version
>> + * 2 of the License, or (at your option) any later version.
>> + */
>> +
>> +#define pr_fmt(fmt) "drmem: " fmt
>> +
>> +#include <linux/kernel.h>
>> +#include <linux/of.h>
>> +#include <linux/of_fdt.h>
>> +#include <linux/bootmem.h>
>> +#include <asm/prom.h>
>> +#include <asm/drmem.h>
>> +
>> +static struct drmem_lmb_info __drmem_info;
>> +struct drmem_lmb_info *drmem_info = &__drmem_info;
>> +
>> +int __init init_drmem_lmbs(unsigned long node)
>> +{
>
> Something in here is blowing up for me.
>
> This is a p8 LPAR, which uses petitboot so we kexec into the kernel,
> gives me:
>
> [ 29.020618] kexec_core: Starting new kernel
> [ 0.000000] bootconsole [udbg0] enabled
> -> early_setup(), dt_ptr: 0x1ec60000
> [ 0.000000] bootmem alloc of 3024 bytes failed!
> [ 0.000000] Kernel panic - not syncing: Out of memory
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-rc2-gcc6-gf7212f8 #1
> [ 0.000000] Call Trace:
> [ 0.000000] [c000000000f57b80] [c000000000a2bae0] dump_stack+0xb0/0xf0 (unreliable)
> [ 0.000000] [c000000000f57bc0] [c0000000000fcdb0] panic+0x148/0x338
> [ 0.000000] [c000000000f57c60] [c000000000d4bb08] ___alloc_bootmem.part.1+0x3c/0x40
> [ 0.000000] [c000000000f57cc0] [c000000000d4bf18] __alloc_bootmem+0x3c/0x50
> [ 0.000000] [c000000000f57cf0] [c000000000d226c4] init_drmem_lmbs+0xf8/0x31c
> [ 0.000000] [c000000000f57d70] [c000000000d1a05c] early_init_dt_scan_memory_ppc+0x88/0x25c
> [ 0.000000] [c000000000f57e10] [0000000000d78e78] 0xd78e78
> [ 0.000000] [c000000000f57e70] [0000000000d1a92c] 0xd1a92c
> [ 0.000000] [c000000000f57f10] [0000000000d1be3c] 0xd1be3c
> [ 0.000000] [c000000000f57f90] [000000000000b13c] 0xb13c
> [ 0.000000] Rebooting in 180 seconds..
> [ 0.000000] System Halted, OK to turn off power
>
>
Looks like we're out of memory when trying to allocate the LMB array, not sure why.
I'll try to re-create this on one of my systems.
I found a patch where you tried to remove traces of bootmem from a couple of
yearts ago. Not sure if it make a difference but should I be using
memblock_virt_alloc() instead of alloc_bootmem()?
-Nathan
> I'm at kernel summit so haven't had time to look any further sorry.
>
> cheers
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-10-24 20:33 ` Nathan Fontenot
@ 2017-10-25 9:16 ` Michael Ellerman
0 siblings, 0 replies; 19+ messages in thread
From: Michael Ellerman @ 2017-10-25 9:16 UTC (permalink / raw)
To: Nathan Fontenot, linuxppc-dev
Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
> On 10/24/2017 01:08 AM, Michael Ellerman wrote:
>> Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
>>
>>> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
>>> new file mode 100644
>>> index 000000000000..8ad7cf36b2c4
>>> --- /dev/null
>>> +++ b/arch/powerpc/mm/drmem.c
>>> @@ -0,0 +1,84 @@
>>> +/*
>>> + * Dynamic reconfiguration memory support
>>> + *
>>> + * Copyright 2017 IBM Corporation
>>> + *
>>> + * This program is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU General Public License
>>> + * as published by the Free Software Foundation; either version
>>> + * 2 of the License, or (at your option) any later version.
>>> + */
>>> +
>>> +#define pr_fmt(fmt) "drmem: " fmt
>>> +
>>> +#include <linux/kernel.h>
>>> +#include <linux/of.h>
>>> +#include <linux/of_fdt.h>
>>> +#include <linux/bootmem.h>
>>> +#include <asm/prom.h>
>>> +#include <asm/drmem.h>
>>> +
>>> +static struct drmem_lmb_info __drmem_info;
>>> +struct drmem_lmb_info *drmem_info = &__drmem_info;
>>> +
>>> +int __init init_drmem_lmbs(unsigned long node)
>>> +{
>>
>> Something in here is blowing up for me.
>>
>> This is a p8 LPAR, which uses petitboot so we kexec into the kernel,
>> gives me:
>>
>> [ 29.020618] kexec_core: Starting new kernel
>> [ 0.000000] bootconsole [udbg0] enabled
>> -> early_setup(), dt_ptr: 0x1ec60000
>> [ 0.000000] bootmem alloc of 3024 bytes failed!
>> [ 0.000000] Kernel panic - not syncing: Out of memory
>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-rc2-gcc6-gf7212f8 #1
>> [ 0.000000] Call Trace:
>> [ 0.000000] [c000000000f57b80] [c000000000a2bae0] dump_stack+0xb0/0xf0 (unreliable)
>> [ 0.000000] [c000000000f57bc0] [c0000000000fcdb0] panic+0x148/0x338
>> [ 0.000000] [c000000000f57c60] [c000000000d4bb08] ___alloc_bootmem.part.1+0x3c/0x40
>> [ 0.000000] [c000000000f57cc0] [c000000000d4bf18] __alloc_bootmem+0x3c/0x50
>> [ 0.000000] [c000000000f57cf0] [c000000000d226c4] init_drmem_lmbs+0xf8/0x31c
>> [ 0.000000] [c000000000f57d70] [c000000000d1a05c] early_init_dt_scan_memory_ppc+0x88/0x25c
>> [ 0.000000] [c000000000f57e10] [0000000000d78e78] 0xd78e78
>> [ 0.000000] [c000000000f57e70] [0000000000d1a92c] 0xd1a92c
>> [ 0.000000] [c000000000f57f10] [0000000000d1be3c] 0xd1be3c
>> [ 0.000000] [c000000000f57f90] [000000000000b13c] 0xb13c
>> [ 0.000000] Rebooting in 180 seconds..
>> [ 0.000000] System Halted, OK to turn off power
>>
>>
>
> Looks like we're out of memory when trying to allocate the LMB array, not sure why.
> I'll try to re-create this on one of my systems.
Thanks.
> I found a patch where you tried to remove traces of bootmem from a couple of
> yearts ago. Not sure if it make a difference but should I be using
> memblock_virt_alloc() instead of alloc_bootmem()?
Yes.
Though it *shouldn't* matter, because alloc_bootmem() will use the
version in mm/nobootmem.c which eventually calls memblock anyway.
But we are trying to get rid of bootmem entirely so you should use
memblock directly in new code.
cheers
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-10-20 13:22 ` [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format Nathan Fontenot
2017-10-24 6:08 ` Michael Ellerman
@ 2017-11-12 12:43 ` Michael Ellerman
2017-11-13 14:51 ` Nathan Fontenot
1 sibling, 1 reply; 19+ messages in thread
From: Michael Ellerman @ 2017-11-12 12:43 UTC (permalink / raw)
To: Nathan Fontenot, linuxppc-dev
Hi Nathan,
Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
> diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
> index f83056297441..917184c13890 100644
> --- a/arch/powerpc/kernel/prom.c
> +++ b/arch/powerpc/kernel/prom.c
> @@ -454,92 +455,93 @@ static int __init early_init_dt_scan_chosen_ppc(unsigned long node,
...
>
> static int __init early_init_dt_scan_memory_ppc(unsigned long node,
> const char *uname,
> int depth, void *data)
> {
> + int rc;
> +
> if (depth == 1 &&
> - strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0)
> - return early_init_dt_scan_drconf_memory(node);
> + strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0) {
> + rc = init_drmem_lmbs(node);
> + if (!rc)
> + early_init_dt_scan_drmem_lmbs(node);
> +
> + return rc;
> + }
>
> return early_init_dt_scan_memory(node, uname, depth, data);
> }
There's one bug in here which is that you return rc as returned by
init_drmem_lmbs(). Returning non-zero from these scan routines
terminates the scan, which means if anything goes wrong in
init_drmem_lmbs() we may not call early_init_dt_scan_memory()
in which case we won't have any memory at all.
I say "may not" because it depends on the order of the nodes in the
device tree whether you hit the memory nodes or the dynamic reconfig mem
info first. And the order of the nodes in the device tree is arbitrary
so we can't rely on that.
> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
> new file mode 100644
> index 000000000000..8ad7cf36b2c4
> --- /dev/null
> +++ b/arch/powerpc/mm/drmem.c
> @@ -0,0 +1,84 @@
...
> +
> +int __init init_drmem_lmbs(unsigned long node)
> +{
> + struct drmem_lmb *lmb;
> + const __be32 *prop;
> + int prop_sz;
> + u32 len;
> +
> + prop = of_get_flat_dt_prop(node, "ibm,lmb-size", &len);
> + if (!prop || len < dt_root_size_cells * sizeof(__be32))
> + return -1;
> +
> + drmem_info->lmb_size = dt_mem_next_cell(dt_root_size_cells, &prop);
> +
> + prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &len);
> + if (!prop || len < dt_root_size_cells * sizeof(__be32))
> + return -1;
> +
> + drmem_info->n_lmbs = of_read_number(prop++, 1);
> + prop_sz = drmem_info->n_lmbs * sizeof(struct of_drconf_cell)
> + + sizeof(__be32);
> + if (prop_sz < len)
> + return -1;
> +
> + drmem_info->lmbs = alloc_bootmem(drmem_info->n_lmbs * sizeof(*lmb));
> + if (!drmem_info->lmbs)
> + return -1;
The bigger problem we have though is that you're trying to allocate
memory, in order to find out what memory we have :)
I suspect it works in some cases because you hit the memory@0 node first
in the device tree, and add that memory to memblock, which means
init_drmem_lmbs() *can* allocate memory, and everything's good.
But if we hit init_drmem_lmbs() first, or there's not enough space in
memory@0, then allocating memory in order to discover memory is not
going to work.
I'm not sure what the best solution is. One option would be to
statically allocate some space, so that we can discover some of the LMBs
without doing an allocation. But we wouldn't be able to guarantee that
we had enough space i nthat static allocation, so the code would need to
handle doing that and then potentially finding more LMBs later using a
dynamic alloc. So that could be a bit messy.
The other option would be for the early_init_dt_scan_drmem_lmbs() code
to still work on the device tree directly, rather than using the
drmem_info array. That would make for uglier code, but may be necessary.
cheers
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-11-12 12:43 ` Michael Ellerman
@ 2017-11-13 14:51 ` Nathan Fontenot
2017-11-14 10:25 ` Michael Ellerman
0 siblings, 1 reply; 19+ messages in thread
From: Nathan Fontenot @ 2017-11-13 14:51 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev
On 11/12/2017 06:43 AM, Michael Ellerman wrote:
> Hi Nathan,
>
> Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
>> diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
>> index f83056297441..917184c13890 100644
>> --- a/arch/powerpc/kernel/prom.c
>> +++ b/arch/powerpc/kernel/prom.c
>> @@ -454,92 +455,93 @@ static int __init early_init_dt_scan_chosen_ppc(unsigned long node,
> ...
>>
>> static int __init early_init_dt_scan_memory_ppc(unsigned long node,
>> const char *uname,
>> int depth, void *data)
>> {
>> + int rc;
>> +
>> if (depth == 1 &&
>> - strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0)
>> - return early_init_dt_scan_drconf_memory(node);
>> + strcmp(uname, "ibm,dynamic-reconfiguration-memory") == 0) {
>> + rc = init_drmem_lmbs(node);
>> + if (!rc)
>> + early_init_dt_scan_drmem_lmbs(node);
>> +
>> + return rc;
>> + }
>>
>> return early_init_dt_scan_memory(node, uname, depth, data);
>> }
>
> There's one bug in here which is that you return rc as returned by
> init_drmem_lmbs(). Returning non-zero from these scan routines
> terminates the scan, which means if anything goes wrong in
> init_drmem_lmbs() we may not call early_init_dt_scan_memory()
> in which case we won't have any memory at all.
>
I didn't know this would stop scanning the device tree, thanks for letting me know.
> I say "may not" because it depends on the order of the nodes in the
> device tree whether you hit the memory nodes or the dynamic reconfig mem
> info first. And the order of the nodes in the device tree is arbitrary
> so we can't rely on that.
>
>
>> diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
>> new file mode 100644
>> index 000000000000..8ad7cf36b2c4
>> --- /dev/null
>> +++ b/arch/powerpc/mm/drmem.c
>> @@ -0,0 +1,84 @@
> ...
>> +
>> +int __init init_drmem_lmbs(unsigned long node)
>> +{
>> + struct drmem_lmb *lmb;
>> + const __be32 *prop;
>> + int prop_sz;
>> + u32 len;
>> +
>> + prop = of_get_flat_dt_prop(node, "ibm,lmb-size", &len);
>> + if (!prop || len < dt_root_size_cells * sizeof(__be32))
>> + return -1;
>> +
>> + drmem_info->lmb_size = dt_mem_next_cell(dt_root_size_cells, &prop);
>> +
>> + prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &len);
>> + if (!prop || len < dt_root_size_cells * sizeof(__be32))
>> + return -1;
>> +
>> + drmem_info->n_lmbs = of_read_number(prop++, 1);
>> + prop_sz = drmem_info->n_lmbs * sizeof(struct of_drconf_cell)
>> + + sizeof(__be32);
>> + if (prop_sz < len)
>> + return -1;
>> +
>> + drmem_info->lmbs = alloc_bootmem(drmem_info->n_lmbs * sizeof(*lmb));
>> + if (!drmem_info->lmbs)
>> + return -1;
>
> The bigger problem we have though is that you're trying to allocate
> memory, in order to find out what memory we have :)
>
> I suspect it works in some cases because you hit the memory@0 node first
> in the device tree, and add that memory to memblock, which means
> init_drmem_lmbs() *can* allocate memory, and everything's good.
>
> But if we hit init_drmem_lmbs() first, or there's not enough space in
> memory@0, then allocating memory in order to discover memory is not
> going to work.
>
> I'm not sure what the best solution is. One option would be to
> statically allocate some space, so that we can discover some of the LMBs
> without doing an allocation. But we wouldn't be able to guarantee that
> we had enough space i nthat static allocation, so the code would need to
> handle doing that and then potentially finding more LMBs later using a
> dynamic alloc. So that could be a bit messy.
>
> The other option would be for the early_init_dt_scan_drmem_lmbs() code
> to still work on the device tree directly, rather than using the
> drmem_info array. That would make for uglier code, but may be necessary.
>
I have been thinking about my initial approach, and the more I look at it
the more I do not like trying to do the bootmem allocation. As you mention
there is just too much that could go wrong with that.
I have started looking at a design where an interface similar to
walk_memory_range() is used for the prom and numa code so we do not have to rely
on making the allocation for the lmb array early in boot. The lmb array
could then be allocated in the late_initcall in drmem.c at which point
the general kernel allocator is available.
I'm still working on getting this coded up and when send out a new patch set
once it's ready unless anyone has objections to this approach.
-Nathan
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format
2017-11-13 14:51 ` Nathan Fontenot
@ 2017-11-14 10:25 ` Michael Ellerman
0 siblings, 0 replies; 19+ messages in thread
From: Michael Ellerman @ 2017-11-14 10:25 UTC (permalink / raw)
To: Nathan Fontenot, linuxppc-dev
Nathan Fontenot <nfont@linux.vnet.ibm.com> writes:
> On 11/12/2017 06:43 AM, Michael Ellerman wrote:
...
>> The bigger problem we have though is that you're trying to allocate
>> memory, in order to find out what memory we have :)
>>
>> I suspect it works in some cases because you hit the memory@0 node first
>> in the device tree, and add that memory to memblock, which means
>> init_drmem_lmbs() *can* allocate memory, and everything's good.
>>
>> But if we hit init_drmem_lmbs() first, or there's not enough space in
>> memory@0, then allocating memory in order to discover memory is not
>> going to work.
>>
>> I'm not sure what the best solution is. One option would be to
>> statically allocate some space, so that we can discover some of the LMBs
>> without doing an allocation. But we wouldn't be able to guarantee that
>> we had enough space i nthat static allocation, so the code would need to
>> handle doing that and then potentially finding more LMBs later using a
>> dynamic alloc. So that could be a bit messy.
>>
>> The other option would be for the early_init_dt_scan_drmem_lmbs() code
>> to still work on the device tree directly, rather than using the
>> drmem_info array. That would make for uglier code, but may be necessary.
>
> I have been thinking about my initial approach, and the more I look at it
> the more I do not like trying to do the bootmem allocation. As you mention
> there is just too much that could go wrong with that.
>
> I have started looking at a design where an interface similar to
> walk_memory_range() is used for the prom and numa code so we do not have to rely
> on making the allocation for the lmb array early in boot. The lmb array
> could then be allocated in the late_initcall in drmem.c at which point
> the general kernel allocator is available.
>
> I'm still working on getting this coded up and when send out a new patch set
> once it's ready unless anyone has objections to this approach.
Thanks. That sounds like it could work.
I definitely liked this version in that it removed a lot of code that
was looking directly at the device tree and abstracted it to just
operate on the LMBs.
If we can end up with something similar but without needing to do the
allocation before we have memory, that'd be great.
cheers
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v2 4/8] powerpc/numa: Update numa code use drmem LMB array
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (2 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 3/8] powerpc/mm: Separate ibm, dynamic-memory data from DT format Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 5/8] powerpc/pseries: Update memory hotplug code to " Nathan Fontenot
` (4 subsequent siblings)
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
Update code in powerpc/numa.c to use the array of dynamic
reconfiguration memory LMBs instead of parsing the device tree
property directly. This allows for the removal of several
helper routines used to read dynamic reconfiguration memory
device tree property information and eases the gathering of
LMB information.
This patch also prepares the numa code for support of the
version 2 dynamic reconfiguration memory property.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
Updates for V2: Removed unused device node paramter to
numa_setup_drmem_lmbs() and hot_add_drconf_scn_to_nid().
---
arch/powerpc/include/asm/drmem.h | 1
arch/powerpc/mm/drmem.c | 9 ++
arch/powerpc/mm/numa.c | 158 ++++++++------------------------------
3 files changed, 42 insertions(+), 126 deletions(-)
diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
index cafe1a3b7da6..912712ecf6c6 100644
--- a/arch/powerpc/include/asm/drmem.h
+++ b/arch/powerpc/include/asm/drmem.h
@@ -41,5 +41,6 @@ static inline u32 drmem_lmb_size(void)
}
extern int __init init_drmem_lmbs(unsigned long node);
+extern u64 drmem_lmb_memory_max(void);
#endif
diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
index 8ad7cf36b2c4..96e453e1fdd7 100644
--- a/arch/powerpc/mm/drmem.c
+++ b/arch/powerpc/mm/drmem.c
@@ -61,6 +61,14 @@ int __init init_drmem_lmbs(unsigned long node)
return 0;
}
+u64 drmem_lmb_memory_max(void)
+{
+ struct drmem_lmb *last_lmb;
+
+ last_lmb = &drmem_info->lmbs[drmem_info->n_lmbs - 1];
+ return last_lmb->base_addr + drmem_lmb_size();
+}
+
static int __init drmem_init(void)
{
struct drmem_lmb *lmbs;
@@ -81,4 +89,3 @@ static int __init drmem_init(void)
}
late_initcall(drmem_init);
-
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index a9aa353d41cd..6cddc7e73b21 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -40,6 +40,7 @@
#include <asm/hvcall.h>
#include <asm/setup.h>
#include <asm/vdso.h>
+#include <asm/drmem.h>
static int numa_enabled = 1;
@@ -395,69 +396,6 @@ static unsigned long read_n_cells(int n, const __be32 **buf)
return result;
}
-/*
- * Read the next memblock list entry from the ibm,dynamic-memory property
- * and return the information in the provided of_drconf_cell structure.
- */
-static void read_drconf_cell(struct of_drconf_cell *drmem, const __be32 **cellp)
-{
- const __be32 *cp;
-
- drmem->base_addr = read_n_cells(n_mem_addr_cells, cellp);
-
- cp = *cellp;
- drmem->drc_index = of_read_number(cp, 1);
- drmem->reserved = of_read_number(&cp[1], 1);
- drmem->aa_index = of_read_number(&cp[2], 1);
- drmem->flags = of_read_number(&cp[3], 1);
-
- *cellp = cp + 4;
-}
-
-/*
- * Retrieve and validate the ibm,dynamic-memory property of the device tree.
- *
- * The layout of the ibm,dynamic-memory property is a number N of memblock
- * list entries followed by N memblock list entries. Each memblock list entry
- * contains information as laid out in the of_drconf_cell struct above.
- */
-static int of_get_drconf_memory(struct device_node *memory, const __be32 **dm)
-{
- const __be32 *prop;
- u32 len, entries;
-
- prop = of_get_property(memory, "ibm,dynamic-memory", &len);
- if (!prop || len < sizeof(unsigned int))
- return 0;
-
- entries = of_read_number(prop++, 1);
-
- /* Now that we know the number of entries, revalidate the size
- * of the property read in to ensure we have everything
- */
- if (len < (entries * (n_mem_addr_cells + 4) + 1) * sizeof(unsigned int))
- return 0;
-
- *dm = prop;
- return entries;
-}
-
-/*
- * Retrieve and validate the ibm,lmb-size property for drconf memory
- * from the device tree.
- */
-static u64 of_get_lmb_size(struct device_node *memory)
-{
- const __be32 *prop;
- u32 len;
-
- prop = of_get_property(memory, "ibm,lmb-size", &len);
- if (!prop || len < sizeof(unsigned int))
- return 0;
-
- return read_n_cells(n_mem_size_cells, &prop);
-}
-
struct assoc_arrays {
u32 n_arrays;
u32 array_sz;
@@ -509,7 +447,7 @@ static int of_get_assoc_arrays(struct assoc_arrays *aa)
* This is like of_node_to_nid_single() for memory represented in the
* ibm,dynamic-reconfiguration-memory node.
*/
-static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,
+static int of_drconf_to_nid_single(struct drmem_lmb *lmb,
struct assoc_arrays *aa)
{
int default_nid = 0;
@@ -517,16 +455,16 @@ static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,
int index;
if (min_common_depth > 0 && min_common_depth <= aa->array_sz &&
- !(drmem->flags & DRCONF_MEM_AI_INVALID) &&
- drmem->aa_index < aa->n_arrays) {
- index = drmem->aa_index * aa->array_sz + min_common_depth - 1;
+ !(lmb->flags & DRCONF_MEM_AI_INVALID) &&
+ lmb->aa_index < aa->n_arrays) {
+ index = lmb->aa_index * aa->array_sz + min_common_depth - 1;
nid = of_read_number(&aa->arrays[index], 1);
if (nid == 0xffff || nid >= MAX_NUMNODES)
nid = default_nid;
if (nid > 0) {
- index = drmem->aa_index * aa->array_sz;
+ index = lmb->aa_index * aa->array_sz;
initialize_distance_lookup_table(nid,
&aa->arrays[index]);
}
@@ -661,22 +599,15 @@ static inline int __init read_usm_ranges(const __be32 **usm)
* Extract NUMA information from the ibm,dynamic-reconfiguration-memory
* node. This assumes n_mem_{addr,size}_cells have been set.
*/
-static void __init parse_drconf_memory(struct device_node *memory)
+static void __init numa_setup_drmem_lmbs(void)
{
- const __be32 *uninitialized_var(dm), *usm;
- unsigned int n, rc, ranges, is_kexec_kdump = 0;
- unsigned long lmb_size, base, size, sz;
+ struct drmem_lmb *lmb;
+ const __be32 *usm;
+ unsigned int rc, ranges, is_kexec_kdump = 0;
+ unsigned long base, size, sz;
int nid;
struct assoc_arrays aa = { .arrays = NULL };
- n = of_get_drconf_memory(memory, &dm);
- if (!n)
- return;
-
- lmb_size = of_get_lmb_size(memory);
- if (!lmb_size)
- return;
-
rc = of_get_assoc_arrays(&aa);
if (rc)
return;
@@ -686,19 +617,15 @@ static void __init parse_drconf_memory(struct device_node *memory)
if (usm != NULL)
is_kexec_kdump = 1;
- for (; n != 0; --n) {
- struct of_drconf_cell drmem;
-
- read_drconf_cell(&drmem, &dm);
-
+ for_each_drmem_lmb(lmb) {
/* skip this block if the reserved bit is set in flags (0x80)
or if the block is not assigned to this partition (0x8) */
- if ((drmem.flags & DRCONF_MEM_RESERVED)
- || !(drmem.flags & DRCONF_MEM_ASSIGNED))
+ if ((lmb->flags & DRCONF_MEM_RESERVED)
+ || !(lmb->flags & DRCONF_MEM_ASSIGNED))
continue;
- base = drmem.base_addr;
- size = lmb_size;
+ base = lmb->base_addr;
+ size = drmem_lmb_size();
ranges = 1;
if (is_kexec_kdump) {
@@ -711,7 +638,7 @@ static void __init parse_drconf_memory(struct device_node *memory)
base = read_n_cells(n_mem_addr_cells, &usm);
size = read_n_cells(n_mem_size_cells, &usm);
}
- nid = of_drconf_to_nid_single(&drmem, &aa);
+ nid = of_drconf_to_nid_single(lmb, &aa);
fake_numa_create_new_node(
((base + size) >> PAGE_SHIFT),
&nid);
@@ -816,8 +743,10 @@ static int __init parse_numa_properties(void)
* ibm,dynamic-reconfiguration-memory node.
*/
memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
- if (memory)
- parse_drconf_memory(memory);
+ if (memory) {
+ numa_setup_drmem_lmbs();
+ of_node_put(memory);
+ }
return 0;
}
@@ -995,43 +924,32 @@ early_param("topology_updates", early_topology_updates);
* memory represented in the device tree by the property
* ibm,dynamic-reconfiguration-memory/ibm,dynamic-memory.
*/
-static int hot_add_drconf_scn_to_nid(struct device_node *memory,
- unsigned long scn_addr)
+static int hot_add_drconf_scn_to_nid(unsigned long scn_addr)
{
- const __be32 *dm;
- unsigned int drconf_cell_cnt, rc;
+ struct drmem_lmb *lmb;
+ unsigned int rc;
unsigned long lmb_size;
struct assoc_arrays aa;
int nid = -1;
- drconf_cell_cnt = of_get_drconf_memory(memory, &dm);
- if (!drconf_cell_cnt)
- return -1;
-
- lmb_size = of_get_lmb_size(memory);
- if (!lmb_size)
- return -1;
+ lmb_size = drmem_lmb_size();
rc = of_get_assoc_arrays(&aa);
if (rc)
return -1;
- for (; drconf_cell_cnt != 0; --drconf_cell_cnt) {
- struct of_drconf_cell drmem;
-
- read_drconf_cell(&drmem, &dm);
-
+ for_each_drmem_lmb(lmb) {
/* skip this block if it is reserved or not assigned to
* this partition */
- if ((drmem.flags & DRCONF_MEM_RESERVED)
- || !(drmem.flags & DRCONF_MEM_ASSIGNED))
+ if ((lmb->flags & DRCONF_MEM_RESERVED)
+ || !(lmb->flags & DRCONF_MEM_ASSIGNED))
continue;
- if ((scn_addr < drmem.base_addr)
- || (scn_addr >= (drmem.base_addr + lmb_size)))
+ if ((scn_addr < lmb->base_addr)
+ || (scn_addr >= (lmb->base_addr + lmb_size)))
continue;
- nid = of_drconf_to_nid_single(&drmem, &aa);
+ nid = of_drconf_to_nid_single(lmb, &aa);
break;
}
@@ -1096,7 +1014,7 @@ int hot_add_scn_to_nid(unsigned long scn_addr)
memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
if (memory) {
- nid = hot_add_drconf_scn_to_nid(memory, scn_addr);
+ nid = hot_add_drconf_scn_to_nid(scn_addr);
of_node_put(memory);
} else {
nid = hot_add_node_scn_to_nid(scn_addr);
@@ -1112,11 +1030,7 @@ static u64 hot_add_drconf_memory_max(void)
{
struct device_node *memory = NULL;
struct device_node *dn = NULL;
- unsigned int drconf_cell_cnt = 0;
- u64 lmb_size = 0;
- const __be32 *dm = NULL;
const __be64 *lrdr = NULL;
- struct of_drconf_cell drmem;
dn = of_find_node_by_path("/rtas");
if (dn) {
@@ -1128,14 +1042,8 @@ static u64 hot_add_drconf_memory_max(void)
memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
if (memory) {
- drconf_cell_cnt = of_get_drconf_memory(memory, &dm);
- lmb_size = of_get_lmb_size(memory);
-
- /* Advance to the last cell, each cell has 6 32 bit integers */
- dm += (drconf_cell_cnt - 1) * 6;
- read_drconf_cell(&drmem, &dm);
of_node_put(memory);
- return drmem.base_addr + lmb_size;
+ return drmem_lmb_memory_max();
}
return 0;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 5/8] powerpc/pseries: Update memory hotplug code to use drmem LMB array
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (3 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 4/8] powerpc/numa: Update numa code use drmem LMB array Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 6/8] powerpc: Move of_drconf_cell struct to asm/drmem.h Nathan Fontenot
` (3 subsequent siblings)
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
Update the pseries memory hotplug code to use the newly added
dynamic reconfiguration LMB array. Doing this is required for the
upcoming support of version 2 of the dynamic reconfiguration
device tree property.
In addition, making this change cleans up the code that parses the
LMB information as we no longer need to worry about device tree
format. This allows us to discard one of the first steps on memory
hotplug where we make a working copy of the device tree property and
convert the entire property to cpu format. Instead we just use the
LMB array directly while holding the memory hotplug lock.
This patch also moves the updating of the device tree property to
powerpc/mm/drmem.c. This allows to the hotplug code to work without
needing to know the device tree format and provides a single
routine for updating the device tree property. This new routine
will handle determination of the proper device tree format and
generate a properly formatted device tree property.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
Updates for V2: Correct build issues with uninitialized variables
---
arch/powerpc/include/asm/drmem.h | 18 +
arch/powerpc/mm/drmem.c | 81 ++++
arch/powerpc/platforms/pseries/hotplug-memory.c | 516 +++++++++--------------
3 files changed, 297 insertions(+), 318 deletions(-)
diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
index 912712ecf6c6..ccd8e1aa0cec 100644
--- a/arch/powerpc/include/asm/drmem.h
+++ b/arch/powerpc/include/asm/drmem.h
@@ -40,7 +40,25 @@ static inline u32 drmem_lmb_size(void)
return drmem_info->lmb_size;
}
+#define DRMEM_LMB_RESERVED 0x80000000
+
+static inline void drmem_mark_lmb_reserved(struct drmem_lmb *lmb)
+{
+ lmb->flags |= DRMEM_LMB_RESERVED;
+}
+
+static inline void drmem_remove_lmb_reservation(struct drmem_lmb *lmb)
+{
+ lmb->flags &= ~DRMEM_LMB_RESERVED;
+}
+
+static inline bool drmem_lmb_reserved(struct drmem_lmb *lmb)
+{
+ return lmb->flags & DRMEM_LMB_RESERVED;
+}
+
extern int __init init_drmem_lmbs(unsigned long node);
extern u64 drmem_lmb_memory_max(void);
+extern int drmem_update_dt(void);
#endif
diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
index 96e453e1fdd7..5aaee23b315c 100644
--- a/arch/powerpc/mm/drmem.c
+++ b/arch/powerpc/mm/drmem.c
@@ -69,6 +69,87 @@ u64 drmem_lmb_memory_max(void)
return last_lmb->base_addr + drmem_lmb_size();
}
+static u32 drmem_lmb_flags(struct drmem_lmb *lmb)
+{
+ return lmb->flags & ~DRMEM_LMB_RESERVED;
+}
+
+static struct property *clone_property(struct property *prop, u32 prop_sz)
+{
+ struct property *new_prop;
+
+ new_prop = kzalloc(sizeof(*new_prop), GFP_KERNEL);
+ if (!new_prop)
+ return NULL;
+
+ new_prop->name = kstrdup(prop->name, GFP_KERNEL);
+ new_prop->value = kzalloc(prop_sz, GFP_KERNEL);
+ if (!new_prop->name || !new_prop->value) {
+ kfree(new_prop->name);
+ kfree(new_prop->value);
+ kfree(new_prop);
+ return NULL;
+ }
+
+ new_prop->length = prop_sz;
+ of_property_set_flag(new_prop, OF_DYNAMIC);
+ return new_prop;
+}
+
+static int drmem_update_dt_v1(struct device_node *memory,
+ struct property *prop)
+{
+ struct property *new_prop;
+ struct of_drconf_cell *dr_cell;
+ struct drmem_lmb *lmb;
+ u32 *p;
+
+ new_prop = clone_property(prop, prop->length);
+ if (!new_prop)
+ return -1;
+
+ p = new_prop->value;
+ *p++ = cpu_to_be32(drmem_info->n_lmbs);
+
+ dr_cell = (struct of_drconf_cell *)p;
+
+ for_each_drmem_lmb(lmb) {
+ dr_cell->base_addr = cpu_to_be64(lmb->base_addr);
+ dr_cell->drc_index = cpu_to_be32(lmb->drc_index);
+ dr_cell->aa_index = cpu_to_be32(lmb->aa_index);
+
+ /* Do not copy out the bit we use internally to mark
+ * an lmb as reserved during hortplug processing.
+ */
+ dr_cell->flags = cpu_to_be32(drmem_lmb_flags(lmb));
+
+ dr_cell++;
+ }
+
+ of_update_property(memory, new_prop);
+ return 0;
+}
+
+int drmem_update_dt(void)
+{
+ struct device_node *memory;
+ struct property *prop;
+ int rc;
+
+ memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!memory)
+ return -1;
+
+ prop = of_find_property(memory, "ibm,dynamic-memory", NULL);
+ if (prop)
+ rc = drmem_update_dt_v1(memory, prop);
+ else
+ rc = -1;
+
+ of_node_put(memory);
+ return rc;
+}
+
static int __init drmem_init(void)
{
struct drmem_lmb *lmbs;
diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c
index 1d48ab424bd9..2043bc2b77b3 100644
--- a/arch/powerpc/platforms/pseries/hotplug-memory.c
+++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
@@ -23,6 +23,7 @@
#include <asm/prom.h>
#include <asm/sparsemem.h>
#include <asm/fadump.h>
+#include <asm/drmem.h>
#include "pseries.h"
static bool rtas_hp_event;
@@ -100,100 +101,6 @@ static struct property *dlpar_clone_property(struct property *prop,
return new_prop;
}
-static struct property *dlpar_clone_drconf_property(struct device_node *dn)
-{
- struct property *prop, *new_prop;
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
- int i;
-
- prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
- if (!prop)
- return NULL;
-
- new_prop = dlpar_clone_property(prop, prop->length);
- if (!new_prop)
- return NULL;
-
- /* Convert the property to cpu endian-ness */
- p = new_prop->value;
- *p = be32_to_cpu(*p);
-
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
- for (i = 0; i < num_lmbs; i++) {
- lmbs[i].base_addr = be64_to_cpu(lmbs[i].base_addr);
- lmbs[i].drc_index = be32_to_cpu(lmbs[i].drc_index);
- lmbs[i].aa_index = be32_to_cpu(lmbs[i].aa_index);
- lmbs[i].flags = be32_to_cpu(lmbs[i].flags);
- }
-
- return new_prop;
-}
-
-static void dlpar_update_drconf_property(struct device_node *dn,
- struct property *prop)
-{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
- int i;
-
- /* Convert the property back to BE */
- p = prop->value;
- num_lmbs = *p;
- *p = cpu_to_be32(*p);
- p++;
-
- lmbs = (struct of_drconf_cell *)p;
- for (i = 0; i < num_lmbs; i++) {
- lmbs[i].base_addr = cpu_to_be64(lmbs[i].base_addr);
- lmbs[i].drc_index = cpu_to_be32(lmbs[i].drc_index);
- lmbs[i].aa_index = cpu_to_be32(lmbs[i].aa_index);
- lmbs[i].flags = cpu_to_be32(lmbs[i].flags);
- }
-
- rtas_hp_event = true;
- of_update_property(dn, prop);
- rtas_hp_event = false;
-}
-
-static int dlpar_update_device_tree_lmb(struct of_drconf_cell *lmb)
-{
- struct device_node *dn;
- struct property *prop;
- struct of_drconf_cell *lmbs;
- u32 *p, num_lmbs;
- int i;
-
- dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
- if (!dn)
- return -ENODEV;
-
- prop = dlpar_clone_drconf_property(dn);
- if (!prop) {
- of_node_put(dn);
- return -ENODEV;
- }
-
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
- for (i = 0; i < num_lmbs; i++) {
- if (lmbs[i].drc_index == lmb->drc_index) {
- lmbs[i].flags = lmb->flags;
- lmbs[i].aa_index = lmb->aa_index;
-
- dlpar_update_drconf_property(dn, prop);
- break;
- }
- }
-
- of_node_put(dn);
- return 0;
-}
-
static u32 find_aa_index(struct device_node *dr_node,
struct property *ala_prop, const u32 *lmb_assoc)
{
@@ -256,7 +163,7 @@ static u32 find_aa_index(struct device_node *dr_node,
return aa_index;
}
-static u32 lookup_lmb_associativity_index(struct of_drconf_cell *lmb)
+static u32 lookup_lmb_associativity_index(struct drmem_lmb *lmb)
{
struct device_node *parent, *lmb_node, *dr_node;
struct property *ala_prop;
@@ -299,9 +206,9 @@ static u32 lookup_lmb_associativity_index(struct of_drconf_cell *lmb)
return aa_index;
}
-static int dlpar_add_device_tree_lmb(struct of_drconf_cell *lmb)
+static int dlpar_add_device_tree_lmb(struct drmem_lmb *lmb)
{
- int aa_index;
+ int rc, aa_index;
lmb->flags |= DRCONF_MEM_ASSIGNED;
@@ -313,17 +220,29 @@ static int dlpar_add_device_tree_lmb(struct of_drconf_cell *lmb)
}
lmb->aa_index = aa_index;
- return dlpar_update_device_tree_lmb(lmb);
+
+ rtas_hp_event = true;
+ rc = drmem_update_dt();
+ rtas_hp_event = false;
+
+ return rc;
}
-static int dlpar_remove_device_tree_lmb(struct of_drconf_cell *lmb)
+static int dlpar_remove_device_tree_lmb(struct drmem_lmb *lmb)
{
+ int rc;
+
lmb->flags &= ~DRCONF_MEM_ASSIGNED;
lmb->aa_index = 0xffffffff;
- return dlpar_update_device_tree_lmb(lmb);
+
+ rtas_hp_event = true;
+ rc = drmem_update_dt();
+ rtas_hp_event = false;
+
+ return rc;
}
-static struct memory_block *lmb_to_memblock(struct of_drconf_cell *lmb)
+static struct memory_block *lmb_to_memblock(struct drmem_lmb *lmb)
{
unsigned long section_nr;
struct mem_section *mem_sect;
@@ -336,7 +255,36 @@ static struct memory_block *lmb_to_memblock(struct of_drconf_cell *lmb)
return mem_block;
}
-static int dlpar_change_lmb_state(struct of_drconf_cell *lmb, bool online)
+static int get_lmb_range(u32 drc_index, int n_lmbs,
+ struct drmem_lmb **start_lmb,
+ struct drmem_lmb **end_lmb)
+{
+ struct drmem_lmb *lmb, *start, *end;
+ struct drmem_lmb *last_lmb;
+
+ start = NULL;
+ for_each_drmem_lmb(lmb) {
+ if (lmb->drc_index == drc_index) {
+ start = lmb;
+ break;
+ }
+ }
+
+ if (!start)
+ return -EINVAL;
+
+ end = &start[n_lmbs - 1];
+
+ last_lmb = &drmem_info->lmbs[drmem_info->n_lmbs - 1];
+ if (end > last_lmb)
+ return -EINVAL;
+
+ *start_lmb = start;
+ *end_lmb = end;
+ return 0;
+}
+
+static int dlpar_change_lmb_state(struct drmem_lmb *lmb, bool online)
{
struct memory_block *mem_block;
int rc;
@@ -357,13 +305,13 @@ static int dlpar_change_lmb_state(struct of_drconf_cell *lmb, bool online)
return rc;
}
-static int dlpar_online_lmb(struct of_drconf_cell *lmb)
+static int dlpar_online_lmb(struct drmem_lmb *lmb)
{
return dlpar_change_lmb_state(lmb, true);
}
#ifdef CONFIG_MEMORY_HOTREMOVE
-static int dlpar_offline_lmb(struct of_drconf_cell *lmb)
+static int dlpar_offline_lmb(struct drmem_lmb *lmb)
{
return dlpar_change_lmb_state(lmb, false);
}
@@ -426,7 +374,7 @@ static int pseries_remove_mem_node(struct device_node *np)
return 0;
}
-static bool lmb_is_removable(struct of_drconf_cell *lmb)
+static bool lmb_is_removable(struct drmem_lmb *lmb)
{
int i, scns_per_block;
int rc = 1;
@@ -458,9 +406,9 @@ static bool lmb_is_removable(struct of_drconf_cell *lmb)
return rc ? true : false;
}
-static int dlpar_add_lmb(struct of_drconf_cell *);
+static int dlpar_add_lmb(struct drmem_lmb *);
-static int dlpar_remove_lmb(struct of_drconf_cell *lmb)
+static int dlpar_remove_lmb(struct drmem_lmb *lmb)
{
unsigned long block_sz;
int nid, rc;
@@ -484,28 +432,25 @@ static int dlpar_remove_lmb(struct of_drconf_cell *lmb)
return 0;
}
-static int dlpar_memory_remove_by_count(u32 lmbs_to_remove,
- struct property *prop)
+static int dlpar_memory_remove_by_count(u32 lmbs_to_remove)
{
- struct of_drconf_cell *lmbs;
+ struct drmem_lmb *lmb;
int lmbs_removed = 0;
int lmbs_available = 0;
- u32 num_lmbs, *p;
- int i, rc;
+ int rc;
pr_info("Attempting to hot-remove %d LMB(s)\n", lmbs_to_remove);
if (lmbs_to_remove == 0)
return -EINVAL;
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
/* Validate that there are enough LMBs to satisfy the request */
- for (i = 0; i < num_lmbs; i++) {
- if (lmb_is_removable(&lmbs[i]))
+ for_each_drmem_lmb(lmb) {
+ if (lmb_is_removable(lmb))
lmbs_available++;
+
+ if (lmbs_available == lmbs_to_remove)
+ break;
}
if (lmbs_available < lmbs_to_remove) {
@@ -514,45 +459,47 @@ static int dlpar_memory_remove_by_count(u32 lmbs_to_remove,
return -EINVAL;
}
- for (i = 0; i < num_lmbs && lmbs_removed < lmbs_to_remove; i++) {
- rc = dlpar_remove_lmb(&lmbs[i]);
+ for_each_drmem_lmb(lmb) {
+ rc = dlpar_remove_lmb(lmb);
if (rc)
continue;
- lmbs_removed++;
-
/* Mark this lmb so we can add it later if all of the
* requested LMBs cannot be removed.
*/
- lmbs[i].reserved = 1;
+ drmem_mark_lmb_reserved(lmb);
+
+ lmbs_removed++;
+ if (lmbs_removed == lmbs_to_remove)
+ break;
}
if (lmbs_removed != lmbs_to_remove) {
pr_err("Memory hot-remove failed, adding LMB's back\n");
- for (i = 0; i < num_lmbs; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb(lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc)
pr_err("Failed to add LMB back, drc index %x\n",
- lmbs[i].drc_index);
+ lmb->drc_index);
- lmbs[i].reserved = 0;
+ drmem_remove_lmb_reservation(lmb);
}
rc = -EINVAL;
} else {
- for (i = 0; i < num_lmbs; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb(lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
pr_info("Memory at %llx was hot-removed\n",
- lmbs[i].base_addr);
+ lmb->base_addr);
- lmbs[i].reserved = 0;
+ drmem_remove_lmb_reservation(lmb);
}
rc = 0;
}
@@ -560,26 +507,21 @@ static int dlpar_memory_remove_by_count(u32 lmbs_to_remove,
return rc;
}
-static int dlpar_memory_remove_by_index(u32 drc_index, struct property *prop)
+static int dlpar_memory_remove_by_index(u32 drc_index)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
+ struct drmem_lmb *lmb;
int lmb_found;
- int i, rc;
+ int rc;
pr_info("Attempting to hot-remove LMB, drc index %x\n", drc_index);
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
lmb_found = 0;
- for (i = 0; i < num_lmbs; i++) {
- if (lmbs[i].drc_index == drc_index) {
+ for_each_drmem_lmb(lmb) {
+ if (lmb->drc_index == drc_index) {
lmb_found = 1;
- rc = dlpar_remove_lmb(&lmbs[i]);
+ rc = dlpar_remove_lmb(lmb);
if (!rc)
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
break;
}
@@ -590,35 +532,30 @@ static int dlpar_memory_remove_by_index(u32 drc_index, struct property *prop)
if (rc)
pr_info("Failed to hot-remove memory at %llx\n",
- lmbs[i].base_addr);
+ lmb->base_addr);
else
- pr_info("Memory at %llx was hot-removed\n", lmbs[i].base_addr);
+ pr_info("Memory at %llx was hot-removed\n", lmb->base_addr);
return rc;
}
-static int dlpar_memory_readd_by_index(u32 drc_index, struct property *prop)
+static int dlpar_memory_readd_by_index(u32 drc_index)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
+ struct drmem_lmb *lmb;
int lmb_found;
- int i, rc;
+ int rc;
pr_info("Attempting to update LMB, drc index %x\n", drc_index);
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
lmb_found = 0;
- for (i = 0; i < num_lmbs; i++) {
- if (lmbs[i].drc_index == drc_index) {
+ for_each_drmem_lmb(lmb) {
+ if (lmb->drc_index == drc_index) {
lmb_found = 1;
- rc = dlpar_remove_lmb(&lmbs[i]);
+ rc = dlpar_remove_lmb(lmb);
if (!rc) {
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc)
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
}
break;
}
@@ -629,20 +566,18 @@ static int dlpar_memory_readd_by_index(u32 drc_index, struct property *prop)
if (rc)
pr_info("Failed to update memory at %llx\n",
- lmbs[i].base_addr);
+ lmb->base_addr);
else
- pr_info("Memory at %llx was updated\n", lmbs[i].base_addr);
+ pr_info("Memory at %llx was updated\n", lmb->base_addr);
return rc;
}
-static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index,
- struct property *prop)
+static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
- int i, rc, start_lmb_found;
- int lmbs_available = 0, start_index = 0, end_index;
+ struct drmem_lmb *lmb, *start_lmb, *end_lmb;
+ int lmbs_available = 0;
+ int rc;
pr_info("Attempting to hot-remove %u LMB(s) at %x\n",
lmbs_to_remove, drc_index);
@@ -650,29 +585,13 @@ static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index,
if (lmbs_to_remove == 0)
return -EINVAL;
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
- start_lmb_found = 0;
-
- /* Navigate to drc_index */
- while (start_index < num_lmbs) {
- if (lmbs[start_index].drc_index == drc_index) {
- start_lmb_found = 1;
- break;
- }
-
- start_index++;
- }
-
- if (!start_lmb_found)
+ rc = get_lmb_range(drc_index, lmbs_to_remove, &start_lmb, &end_lmb);
+ if (rc)
return -EINVAL;
- end_index = start_index + lmbs_to_remove;
-
/* Validate that there are enough LMBs to satisfy the request */
- for (i = start_index; i < end_index; i++) {
- if (lmbs[i].flags & DRCONF_MEM_RESERVED)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (lmb->flags & DRCONF_MEM_RESERVED)
break;
lmbs_available++;
@@ -681,42 +600,43 @@ static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index,
if (lmbs_available < lmbs_to_remove)
return -EINVAL;
- for (i = start_index; i < end_index; i++) {
- if (!(lmbs[i].flags & DRCONF_MEM_ASSIGNED))
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (!(lmb->flags & DRCONF_MEM_ASSIGNED))
continue;
- rc = dlpar_remove_lmb(&lmbs[i]);
+ rc = dlpar_remove_lmb(lmb);
if (rc)
break;
- lmbs[i].reserved = 1;
+ drmem_mark_lmb_reserved(lmb);
}
if (rc) {
pr_err("Memory indexed-count-remove failed, adding any removed LMBs\n");
- for (i = start_index; i < end_index; i++) {
- if (!lmbs[i].reserved)
+
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc)
pr_err("Failed to add LMB, drc index %x\n",
- be32_to_cpu(lmbs[i].drc_index));
+ lmb->drc_index);
- lmbs[i].reserved = 0;
+ drmem_remove_lmb_reservation(lmb);
}
rc = -EINVAL;
} else {
- for (i = start_index; i < end_index; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
pr_info("Memory at %llx (drc index %x) was hot-removed\n",
- lmbs[i].base_addr, lmbs[i].drc_index);
+ lmb->base_addr, lmb->drc_index);
- lmbs[i].reserved = 0;
+ drmem_remove_lmb_reservation(lmb);
}
}
@@ -737,32 +657,30 @@ static inline int dlpar_memory_remove(struct pseries_hp_errorlog *hp_elog)
{
return -EOPNOTSUPP;
}
-static int dlpar_remove_lmb(struct of_drconf_cell *lmb)
+static int dlpar_remove_lmb(struct drmem_lmb *lmb)
{
return -EOPNOTSUPP;
}
-static int dlpar_memory_remove_by_count(u32 lmbs_to_remove,
- struct property *prop)
+static int dlpar_memory_remove_by_count(u32 lmbs_to_remove)
{
return -EOPNOTSUPP;
}
-static int dlpar_memory_remove_by_index(u32 drc_index, struct property *prop)
+static int dlpar_memory_remove_by_index(u32 drc_index)
{
return -EOPNOTSUPP;
}
-static int dlpar_memory_readd_by_index(u32 drc_index, struct property *prop)
+static int dlpar_memory_readd_by_index(u32 drc_index)
{
return -EOPNOTSUPP;
}
-static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index,
- struct property *prop)
+static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_MEMORY_HOTREMOVE */
-static int dlpar_add_lmb(struct of_drconf_cell *lmb)
+static int dlpar_add_lmb(struct drmem_lmb *lmb)
{
unsigned long block_sz;
int nid, rc;
@@ -801,77 +719,79 @@ static int dlpar_add_lmb(struct of_drconf_cell *lmb)
return rc;
}
-static int dlpar_memory_add_by_count(u32 lmbs_to_add, struct property *prop)
+static int dlpar_memory_add_by_count(u32 lmbs_to_add)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
+ struct drmem_lmb *lmb;
int lmbs_available = 0;
int lmbs_added = 0;
- int i, rc;
+ int rc;
pr_info("Attempting to hot-add %d LMB(s)\n", lmbs_to_add);
if (lmbs_to_add == 0)
return -EINVAL;
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
/* Validate that there are enough LMBs to satisfy the request */
- for (i = 0; i < num_lmbs; i++) {
- if (!(lmbs[i].flags & DRCONF_MEM_ASSIGNED))
+ for_each_drmem_lmb(lmb) {
+ if (!(lmb->flags & DRCONF_MEM_ASSIGNED))
lmbs_available++;
+
+ if (lmbs_available == lmbs_to_add)
+ break;
}
if (lmbs_available < lmbs_to_add)
return -EINVAL;
- for (i = 0; i < num_lmbs && lmbs_to_add != lmbs_added; i++) {
- if (lmbs[i].flags & DRCONF_MEM_ASSIGNED)
+ for_each_drmem_lmb(lmb) {
+ if (lmb->flags & DRCONF_MEM_ASSIGNED)
continue;
- rc = dlpar_acquire_drc(lmbs[i].drc_index);
+ rc = dlpar_acquire_drc(lmb->drc_index);
if (rc)
continue;
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc) {
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
continue;
}
- lmbs_added++;
-
/* Mark this lmb so we can remove it later if all of the
* requested LMBs cannot be added.
*/
- lmbs[i].reserved = 1;
+ drmem_mark_lmb_reserved(lmb);
+
+ lmbs_added++;
+ if (lmbs_added == lmbs_to_add)
+ break;
}
if (lmbs_added != lmbs_to_add) {
pr_err("Memory hot-add failed, removing any added LMBs\n");
- for (i = 0; i < num_lmbs; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb(lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- rc = dlpar_remove_lmb(&lmbs[i]);
+ rc = dlpar_remove_lmb(lmb);
if (rc)
pr_err("Failed to remove LMB, drc index %x\n",
- be32_to_cpu(lmbs[i].drc_index));
+ lmb->drc_index);
else
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
+
+ drmem_remove_lmb_reservation(lmb);
}
rc = -EINVAL;
} else {
- for (i = 0; i < num_lmbs; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb(lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
pr_info("Memory at %llx (drc index %x) was hot-added\n",
- lmbs[i].base_addr, lmbs[i].drc_index);
- lmbs[i].reserved = 0;
+ lmb->base_addr, lmb->drc_index);
+ drmem_remove_lmb_reservation(lmb);
}
rc = 0;
}
@@ -879,28 +799,22 @@ static int dlpar_memory_add_by_count(u32 lmbs_to_add, struct property *prop)
return rc;
}
-static int dlpar_memory_add_by_index(u32 drc_index, struct property *prop)
+static int dlpar_memory_add_by_index(u32 drc_index)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
- int i, lmb_found;
- int rc;
+ struct drmem_lmb *lmb;
+ int rc, lmb_found;
pr_info("Attempting to hot-add LMB, drc index %x\n", drc_index);
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
-
lmb_found = 0;
- for (i = 0; i < num_lmbs; i++) {
- if (lmbs[i].drc_index == drc_index) {
+ for_each_drmem_lmb(lmb) {
+ if (lmb->drc_index == drc_index) {
lmb_found = 1;
- rc = dlpar_acquire_drc(lmbs[i].drc_index);
+ rc = dlpar_acquire_drc(lmb->drc_index);
if (!rc) {
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc)
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
}
break;
@@ -914,18 +828,16 @@ static int dlpar_memory_add_by_index(u32 drc_index, struct property *prop)
pr_info("Failed to hot-add memory, drc index %x\n", drc_index);
else
pr_info("Memory at %llx (drc index %x) was hot-added\n",
- lmbs[i].base_addr, drc_index);
+ lmb->base_addr, drc_index);
return rc;
}
-static int dlpar_memory_add_by_ic(u32 lmbs_to_add, u32 drc_index,
- struct property *prop)
+static int dlpar_memory_add_by_ic(u32 lmbs_to_add, u32 drc_index)
{
- struct of_drconf_cell *lmbs;
- u32 num_lmbs, *p;
- int i, rc, start_lmb_found;
- int lmbs_available = 0, start_index = 0, end_index;
+ struct drmem_lmb *lmb, *start_lmb, *end_lmb;
+ int lmbs_available = 0;
+ int rc;
pr_info("Attempting to hot-add %u LMB(s) at index %x\n",
lmbs_to_add, drc_index);
@@ -933,29 +845,13 @@ static int dlpar_memory_add_by_ic(u32 lmbs_to_add, u32 drc_index,
if (lmbs_to_add == 0)
return -EINVAL;
- p = prop->value;
- num_lmbs = *p++;
- lmbs = (struct of_drconf_cell *)p;
- start_lmb_found = 0;
-
- /* Navigate to drc_index */
- while (start_index < num_lmbs) {
- if (lmbs[start_index].drc_index == drc_index) {
- start_lmb_found = 1;
- break;
- }
-
- start_index++;
- }
-
- if (!start_lmb_found)
+ rc = get_lmb_range(drc_index, lmbs_to_add, &start_lmb, &end_lmb);
+ if (rc)
return -EINVAL;
- end_index = start_index + lmbs_to_add;
-
/* Validate that the LMBs in this range are not reserved */
- for (i = start_index; i < end_index; i++) {
- if (lmbs[i].flags & DRCONF_MEM_RESERVED)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (lmb->flags & DRCONF_MEM_RESERVED)
break;
lmbs_available++;
@@ -964,46 +860,48 @@ static int dlpar_memory_add_by_ic(u32 lmbs_to_add, u32 drc_index,
if (lmbs_available < lmbs_to_add)
return -EINVAL;
- for (i = start_index; i < end_index; i++) {
- if (lmbs[i].flags & DRCONF_MEM_ASSIGNED)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (lmb->flags & DRCONF_MEM_ASSIGNED)
continue;
- rc = dlpar_acquire_drc(lmbs[i].drc_index);
+ rc = dlpar_acquire_drc(lmb->drc_index);
if (rc)
break;
- rc = dlpar_add_lmb(&lmbs[i]);
+ rc = dlpar_add_lmb(lmb);
if (rc) {
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
break;
}
- lmbs[i].reserved = 1;
+ drmem_mark_lmb_reserved(lmb);
}
if (rc) {
pr_err("Memory indexed-count-add failed, removing any added LMBs\n");
- for (i = start_index; i < end_index; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
- rc = dlpar_remove_lmb(&lmbs[i]);
+ rc = dlpar_remove_lmb(lmb);
if (rc)
pr_err("Failed to remove LMB, drc index %x\n",
- be32_to_cpu(lmbs[i].drc_index));
+ lmb->drc_index);
else
- dlpar_release_drc(lmbs[i].drc_index);
+ dlpar_release_drc(lmb->drc_index);
+
+ drmem_remove_lmb_reservation(lmb);
}
rc = -EINVAL;
} else {
- for (i = start_index; i < end_index; i++) {
- if (!lmbs[i].reserved)
+ for_each_drmem_lmb_in_range(lmb, start_lmb, end_lmb) {
+ if (!drmem_lmb_reserved(lmb))
continue;
pr_info("Memory at %llx (drc index %x) was hot-added\n",
- lmbs[i].base_addr, lmbs[i].drc_index);
- lmbs[i].reserved = 0;
+ lmb->base_addr, lmb->drc_index);
+ drmem_remove_lmb_reservation(lmb);
}
}
@@ -1012,37 +910,23 @@ static int dlpar_memory_add_by_ic(u32 lmbs_to_add, u32 drc_index,
int dlpar_memory(struct pseries_hp_errorlog *hp_elog)
{
- struct device_node *dn;
- struct property *prop;
u32 count, drc_index;
int rc;
lock_device_hotplug();
- dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
- if (!dn) {
- rc = -EINVAL;
- goto dlpar_memory_out;
- }
-
- prop = dlpar_clone_drconf_property(dn);
- if (!prop) {
- rc = -EINVAL;
- goto dlpar_memory_out;
- }
-
switch (hp_elog->action) {
case PSERIES_HP_ELOG_ACTION_ADD:
if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_COUNT) {
count = hp_elog->_drc_u.drc_count;
- rc = dlpar_memory_add_by_count(count, prop);
+ rc = dlpar_memory_add_by_count(count);
} else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_INDEX) {
drc_index = hp_elog->_drc_u.drc_index;
- rc = dlpar_memory_add_by_index(drc_index, prop);
+ rc = dlpar_memory_add_by_index(drc_index);
} else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_IC) {
count = hp_elog->_drc_u.ic.count;
drc_index = hp_elog->_drc_u.ic.index;
- rc = dlpar_memory_add_by_ic(count, drc_index, prop);
+ rc = dlpar_memory_add_by_ic(count, drc_index);
} else {
rc = -EINVAL;
}
@@ -1051,14 +935,14 @@ int dlpar_memory(struct pseries_hp_errorlog *hp_elog)
case PSERIES_HP_ELOG_ACTION_REMOVE:
if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_COUNT) {
count = hp_elog->_drc_u.drc_count;
- rc = dlpar_memory_remove_by_count(count, prop);
+ rc = dlpar_memory_remove_by_count(count);
} else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_INDEX) {
drc_index = hp_elog->_drc_u.drc_index;
- rc = dlpar_memory_remove_by_index(drc_index, prop);
+ rc = dlpar_memory_remove_by_index(drc_index);
} else if (hp_elog->id_type == PSERIES_HP_ELOG_ID_DRC_IC) {
count = hp_elog->_drc_u.ic.count;
drc_index = hp_elog->_drc_u.ic.index;
- rc = dlpar_memory_remove_by_ic(count, drc_index, prop);
+ rc = dlpar_memory_remove_by_ic(count, drc_index);
} else {
rc = -EINVAL;
}
@@ -1066,7 +950,7 @@ int dlpar_memory(struct pseries_hp_errorlog *hp_elog)
break;
case PSERIES_HP_ELOG_ACTION_READD:
drc_index = hp_elog->_drc_u.drc_index;
- rc = dlpar_memory_readd_by_index(drc_index, prop);
+ rc = dlpar_memory_readd_by_index(drc_index);
break;
default:
pr_err("Invalid action (%d) specified\n", hp_elog->action);
@@ -1074,10 +958,6 @@ int dlpar_memory(struct pseries_hp_errorlog *hp_elog)
break;
}
- dlpar_free_property(prop);
-
-dlpar_memory_out:
- of_node_put(dn);
unlock_device_hotplug();
return rc;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 6/8] powerpc: Move of_drconf_cell struct to asm/drmem.h
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (4 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 5/8] powerpc/pseries: Update memory hotplug code to " Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 7/8] powerpc/pseries: Add support for ibm, dynamic-memory-v2 property Nathan Fontenot
` (2 subsequent siblings)
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
Now that the powerpc code parses dynamic reconfiguration memory
LMB information from the LMB array and not the device tree
directly we can move the of_drconf_cell struct to drmem.h where
it fits better.
In addition, the struct is renamed to of_drconf_cell_v1 in
anticipation of upcoming support for version 2 of the dynamic
reconfiguration property.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/drmem.h | 18 ++++++++++++++++++
arch/powerpc/include/asm/prom.h | 16 ----------------
arch/powerpc/mm/drmem.c | 4 ++--
arch/powerpc/platforms/pseries/hotplug-memory.c | 6 +++---
4 files changed, 23 insertions(+), 21 deletions(-)
diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
index ccd8e1aa0cec..32d859c84202 100644
--- a/arch/powerpc/include/asm/drmem.h
+++ b/arch/powerpc/include/asm/drmem.h
@@ -35,6 +35,24 @@ extern struct drmem_lmb_info *drmem_info;
&drmem_info->lmbs[0], \
&drmem_info->lmbs[drmem_info->n_lmbs - 1])
+/* The of_drconf_cell struct defines the layout of the LMB array
+ * specified in the ibm,dynamic-memory device tree property.
+ * The property itself is a 32-bit value specifying the number of
+ * LMBs followed by an array of of_drconf_cell_v1 entries, one
+ * per LMB.
+ */
+struct of_drconf_cell_v1 {
+ u64 base_addr;
+ u32 drc_index;
+ u32 reserved;
+ u32 aa_index;
+ u32 flags;
+};
+
+#define DRCONF_MEM_ASSIGNED 0x00000008
+#define DRCONF_MEM_AI_INVALID 0x00000040
+#define DRCONF_MEM_RESERVED 0x00000080
+
static inline u32 drmem_lmb_size(void)
{
return drmem_info->lmb_size;
diff --git a/arch/powerpc/include/asm/prom.h b/arch/powerpc/include/asm/prom.h
index 825bd5998701..f0a30a003bd8 100644
--- a/arch/powerpc/include/asm/prom.h
+++ b/arch/powerpc/include/asm/prom.h
@@ -80,22 +80,6 @@ extern void of_instantiate_rtc(void);
extern int of_get_ibm_chip_id(struct device_node *np);
-/* The of_drconf_cell struct defines the layout of the LMB array
- * specified in the device tree property
- * ibm,dynamic-reconfiguration-memory/ibm,dynamic-memory
- */
-struct of_drconf_cell {
- u64 base_addr;
- u32 drc_index;
- u32 reserved;
- u32 aa_index;
- u32 flags;
-};
-
-#define DRCONF_MEM_ASSIGNED 0x00000008
-#define DRCONF_MEM_AI_INVALID 0x00000040
-#define DRCONF_MEM_RESERVED 0x00000080
-
/*
* There are two methods for telling firmware what our capabilities are.
* Newer machines have an "ibm,client-architecture-support" method on the
diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
index 5aaee23b315c..9cd9e680874e 100644
--- a/arch/powerpc/mm/drmem.c
+++ b/arch/powerpc/mm/drmem.c
@@ -100,7 +100,7 @@ static int drmem_update_dt_v1(struct device_node *memory,
struct property *prop)
{
struct property *new_prop;
- struct of_drconf_cell *dr_cell;
+ struct of_drconf_cell_v1 *dr_cell;
struct drmem_lmb *lmb;
u32 *p;
@@ -111,7 +111,7 @@ static int drmem_update_dt_v1(struct device_node *memory,
p = new_prop->value;
*p++ = cpu_to_be32(drmem_info->n_lmbs);
- dr_cell = (struct of_drconf_cell *)p;
+ dr_cell = (struct of_drconf_cell_v1 *)p;
for_each_drmem_lmb(lmb) {
dr_cell->base_addr = cpu_to_be64(lmb->base_addr);
diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c
index 2043bc2b77b3..c1578f54c626 100644
--- a/arch/powerpc/platforms/pseries/hotplug-memory.c
+++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
@@ -996,7 +996,7 @@ static int pseries_add_mem_node(struct device_node *np)
static int pseries_update_drconf_memory(struct of_reconfig_data *pr)
{
- struct of_drconf_cell *new_drmem, *old_drmem;
+ struct of_drconf_cell_v1 *new_drmem, *old_drmem;
unsigned long memblock_size;
u32 entries;
__be32 *p;
@@ -1019,11 +1019,11 @@ static int pseries_update_drconf_memory(struct of_reconfig_data *pr)
* of_drconf_cell's.
*/
entries = be32_to_cpu(*p++);
- old_drmem = (struct of_drconf_cell *)p;
+ old_drmem = (struct of_drconf_cell_v1 *)p;
p = (__be32 *)pr->prop->value;
p++;
- new_drmem = (struct of_drconf_cell *)p;
+ new_drmem = (struct of_drconf_cell_v1 *)p;
for (i = 0; i < entries; i++) {
if ((be32_to_cpu(old_drmem[i].flags) & DRCONF_MEM_ASSIGNED) &&
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 7/8] powerpc/pseries: Add support for ibm, dynamic-memory-v2 property
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (5 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 6/8] powerpc: Move of_drconf_cell struct to asm/drmem.h Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-10-20 13:22 ` [PATCH v2 8/8] powerpc: Enable support of ibm,dynamic-memory-v2 Nathan Fontenot
2017-11-16 5:37 ` [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Bharata B Rao
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
The Power Hypervisor has introduced a new device tree format for
the property describing the dynamic reconfiguration LMBs for a system.
This new format condenses the size of the property, especially
on large memory systems.
Instead of the current format that contains an entry for every
possible LMB, the new format contains an entry for every range
of LMBs that possess the same flags and associativity index.
This patch updates the powerpc/mm/drmem.c code to parse the new
property format at boot and create the LMB array. This also updates
the device tree updating routine to build a device tree property
in this format.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/drmem.h | 12 ++
arch/powerpc/mm/drmem.c | 188 ++++++++++++++++++++++++++++++++++----
2 files changed, 182 insertions(+), 18 deletions(-)
diff --git a/arch/powerpc/include/asm/drmem.h b/arch/powerpc/include/asm/drmem.h
index 32d859c84202..b7becafc528d 100644
--- a/arch/powerpc/include/asm/drmem.h
+++ b/arch/powerpc/include/asm/drmem.h
@@ -49,6 +49,18 @@ struct of_drconf_cell_v1 {
u32 flags;
};
+/* Version 2 of the ibm,dynamic-memory property is defined as a
+ * 32-bit value specifying the number of LMB sets followed by an
+ * array of of_drconf_cell_v2 entries, one per LMB set.
+ */
+struct of_drconf_cell_v2 {
+ u32 seq_lmbs;
+ u64 base_addr;
+ u32 drc_index;
+ u32 aa_index;
+ u32 flags;
+} __attribute__((packed));
+
#define DRCONF_MEM_ASSIGNED 0x00000008
#define DRCONF_MEM_AI_INVALID 0x00000040
#define DRCONF_MEM_RESERVED 0x00000080
diff --git a/arch/powerpc/mm/drmem.c b/arch/powerpc/mm/drmem.c
index 9cd9e680874e..b5b8b8f46292 100644
--- a/arch/powerpc/mm/drmem.c
+++ b/arch/powerpc/mm/drmem.c
@@ -21,25 +21,13 @@
static struct drmem_lmb_info __drmem_info;
struct drmem_lmb_info *drmem_info = &__drmem_info;
-int __init init_drmem_lmbs(unsigned long node)
+static int __init init_drmem_lmbs_v1(const __be32 *prop, u32 len)
{
struct drmem_lmb *lmb;
- const __be32 *prop;
- int prop_sz;
- u32 len;
-
- prop = of_get_flat_dt_prop(node, "ibm,lmb-size", &len);
- if (!prop || len < dt_root_size_cells * sizeof(__be32))
- return -1;
-
- drmem_info->lmb_size = dt_mem_next_cell(dt_root_size_cells, &prop);
-
- prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &len);
- if (!prop || len < dt_root_size_cells * sizeof(__be32))
- return -1;
+ u32 prop_sz;
drmem_info->n_lmbs = of_read_number(prop++, 1);
- prop_sz = drmem_info->n_lmbs * sizeof(struct of_drconf_cell)
+ prop_sz = drmem_info->n_lmbs * sizeof(struct of_drconf_cell_v1)
+ sizeof(__be32);
if (prop_sz < len)
return -1;
@@ -61,6 +49,89 @@ int __init init_drmem_lmbs(unsigned long node)
return 0;
}
+static void read_one_drconf_v2_cell(const __be32 **cell,
+ struct of_drconf_cell_v2 *dr_cell)
+{
+ const __be32 *p = *cell;
+
+ dr_cell->seq_lmbs = of_read_number(p++, 1);
+ dr_cell->base_addr = dt_mem_next_cell(dt_root_addr_cells, &p);
+ dr_cell->drc_index = of_read_number(p++, 1);
+ dr_cell->aa_index = of_read_number(p++, 1);
+ dr_cell->flags = of_read_number(p++, 1);
+
+ *cell = p;
+}
+
+static int __init init_drmem_lmbs_v2(const __be32 *prop, u32 len)
+{
+ struct drmem_lmb *lmb;
+ struct of_drconf_cell_v2 dr_cell;
+ const __be32 *p;
+ u32 lmb_sets, prop_sz;
+ int i, j, lmb_index;
+
+ lmb_sets = of_read_number(prop++, 1);
+ prop_sz = lmb_sets * sizeof(struct of_drconf_cell_v2)
+ + sizeof(__be32);
+ if (prop_sz < len)
+ return -1;
+
+ /* first pass, calculate the number of LMBs */
+ p = prop;
+ for (i = 0; i < lmb_sets; i++) {
+ read_one_drconf_v2_cell(&p, &dr_cell);
+ drmem_info->n_lmbs += dr_cell.seq_lmbs;
+ }
+
+ drmem_info->lmbs = alloc_bootmem(drmem_info->n_lmbs * sizeof(*lmb));
+ if (!drmem_info->lmbs)
+ return -1;
+
+ lmb_index = 0;
+ p = prop;
+ for (i = 0; i < lmb_sets; i++) {
+ read_one_drconf_v2_cell(&p, &dr_cell);
+
+ for (j = 0; j < dr_cell.seq_lmbs; j++) {
+ lmb = &drmem_info->lmbs[lmb_index++];
+
+ lmb->base_addr = dr_cell.base_addr;
+ dr_cell.base_addr += drmem_info->lmb_size;
+
+ lmb->drc_index = dr_cell.drc_index;
+ dr_cell.drc_index++;
+
+ lmb->aa_index = dr_cell.aa_index;
+ lmb->flags = dr_cell.flags;
+ }
+ }
+
+ return 0;
+}
+
+int __init init_drmem_lmbs(unsigned long node)
+{
+ const __be32 *prop;
+ u32 len;
+
+ prop = of_get_flat_dt_prop(node, "ibm,lmb-size", &len);
+ if (!prop || len < dt_root_size_cells * sizeof(__be32))
+ return -1;
+
+ drmem_info->lmb_size = dt_mem_next_cell(dt_root_size_cells, &prop);
+
+ prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory", &len);
+ if (prop)
+ return init_drmem_lmbs_v1(prop, len);
+
+ prop = of_get_flat_dt_prop(node, "ibm,dynamic-memory-v2", &len);
+ if (prop)
+ return init_drmem_lmbs_v2(prop, len);
+
+ return -1;
+}
+
u64 drmem_lmb_memory_max(void)
{
struct drmem_lmb *last_lmb;
@@ -130,6 +201,82 @@ static int drmem_update_dt_v1(struct device_node *memory,
return 0;
}
+static void init_drconf_v2_cell(struct of_drconf_cell_v2 *dr_cell,
+ struct drmem_lmb *lmb)
+{
+ dr_cell->base_addr = cpu_to_be64(lmb->base_addr);
+ dr_cell->drc_index = cpu_to_be32(lmb->drc_index);
+ dr_cell->aa_index = cpu_to_be32(lmb->aa_index);
+ dr_cell->flags = cpu_to_be32(lmb->flags);
+}
+
+static int drmem_update_dt_v2(struct device_node *memory,
+ struct property *prop)
+{
+ struct property *new_prop;
+ struct of_drconf_cell_v2 *dr_cell;
+ struct drmem_lmb *lmb, *prev_lmb;
+ u32 lmb_sets, prop_sz, seq_lmbs;
+ u32 *p;
+
+ lmb_sets = 0;
+ prev_lmb = NULL;
+ for_each_drmem_lmb(lmb) {
+ if (!prev_lmb) {
+ prev_lmb = lmb;
+ lmb_sets++;
+ continue;
+ }
+
+ if (prev_lmb->aa_index != lmb->aa_index ||
+ prev_lmb->flags != drmem_lmb_flags(lmb))
+ lmb_sets++;
+
+ prev_lmb = lmb;
+ }
+
+ prop_sz = lmb_sets * sizeof(*dr_cell) + sizeof(__be32);
+ new_prop = clone_property(prop, prop_sz);
+ if (!new_prop)
+ return -1;
+
+ p = new_prop->value;
+ *p++ = cpu_to_be32(lmb_sets);
+
+ dr_cell = (struct of_drconf_cell_v2 *)p;
+
+ prev_lmb = NULL;
+ seq_lmbs = 0;
+ for_each_drmem_lmb(lmb) {
+ if (prev_lmb == NULL) {
+ /* Start of first set */
+ prev_lmb = lmb;
+ init_drconf_v2_cell(dr_cell, lmb);
+ seq_lmbs++;
+ continue;
+ }
+
+ if (prev_lmb->aa_index != lmb->aa_index ||
+ prev_lmb->flags != drmem_lmb_flags(lmb)) {
+ /* end of one set, start of another */
+ dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs);
+ dr_cell++;
+
+ init_drconf_v2_cell(dr_cell, lmb);
+ seq_lmbs = 1;
+ } else {
+ seq_lmbs++;
+ }
+
+ prev_lmb = lmb;
+ }
+
+ /* close out last dr_cell */
+ dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs);
+ of_update_property(memory, new_prop);
+ return 0;
+}
+
int drmem_update_dt(void)
{
struct device_node *memory;
@@ -141,10 +288,15 @@ int drmem_update_dt(void)
return -1;
prop = of_find_property(memory, "ibm,dynamic-memory", NULL);
- if (prop)
+ if (prop) {
rc = drmem_update_dt_v1(memory, prop);
- else
- rc = -1;
+ } else {
+ prop = of_find_property(memory, "ibm,dynamic-memory-v2", NULL);
+ if (prop)
+ rc = drmem_update_dt_v2(memory, prop);
+ else
+ rc = -1;
+ }
of_node_put(memory);
return rc;
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v2 8/8] powerpc: Enable support of ibm,dynamic-memory-v2
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (6 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 7/8] powerpc/pseries: Add support for ibm, dynamic-memory-v2 property Nathan Fontenot
@ 2017-10-20 13:22 ` Nathan Fontenot
2017-11-16 5:37 ` [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Bharata B Rao
8 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-10-20 13:22 UTC (permalink / raw)
To: linuxppc-dev
Add required bits to the architecture vector to enable support
of the ibm,dynamic-memory-v2 device tree property.
Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/firmware.h | 3 ++-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 1 +
arch/powerpc/platforms/pseries/firmware.c | 1 +
4 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/firmware.h b/arch/powerpc/include/asm/firmware.h
index 8645897472b1..832df61f30ef 100644
--- a/arch/powerpc/include/asm/firmware.h
+++ b/arch/powerpc/include/asm/firmware.h
@@ -51,6 +51,7 @@
#define FW_FEATURE_BEST_ENERGY ASM_CONST(0x0000000080000000)
#define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000)
#define FW_FEATURE_PRRN ASM_CONST(0x0000000200000000)
+#define FW_FEATURE_DRMEM_V2 ASM_CONST(0x0000000400000000)
#ifndef __ASSEMBLY__
@@ -67,7 +68,7 @@ enum {
FW_FEATURE_CMO | FW_FEATURE_VPHN | FW_FEATURE_XCMO |
FW_FEATURE_SET_MODE | FW_FEATURE_BEST_ENERGY |
FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN |
- FW_FEATURE_HPT_RESIZE,
+ FW_FEATURE_HPT_RESIZE | FW_FEATURE_DRMEM_V2,
FW_FEATURE_PSERIES_ALWAYS = 0,
FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL,
FW_FEATURE_POWERNV_ALWAYS = 0,
diff --git a/arch/powerpc/include/asm/prom.h b/arch/powerpc/include/asm/prom.h
index f0a30a003bd8..9f27866e3126 100644
--- a/arch/powerpc/include/asm/prom.h
+++ b/arch/powerpc/include/asm/prom.h
@@ -143,6 +143,7 @@ extern int of_get_ibm_chip_id(struct device_node *np);
#define OV5_PFO_HW_842 0x1140 /* PFO Compression Accelerator */
#define OV5_PFO_HW_ENCR 0x1120 /* PFO Encryption Accelerator */
#define OV5_SUB_PROCESSORS 0x1501 /* 1,2,or 4 Sub-Processors supported */
+#define OV5_DRMEM_V2 0x1680 /* ibm,dynamic-reconfiguration-v2 */
#define OV5_XIVE_SUPPORT 0x17C0 /* XIVE Exploitation Support Mask */
#define OV5_XIVE_LEGACY 0x1700 /* XIVE legacy mode Only */
#define OV5_XIVE_EXPLOIT 0x1740 /* XIVE exploitation mode Only */
diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c
index 02190e90c7ae..acf4b2e0530c 100644
--- a/arch/powerpc/kernel/prom_init.c
+++ b/arch/powerpc/kernel/prom_init.c
@@ -869,6 +869,7 @@ struct ibm_arch_vec __cacheline_aligned ibm_architecture_vec = {
.reserved2 = 0,
.reserved3 = 0,
.subprocessors = 1,
+ .byte22 = OV5_FEAT(OV5_DRMEM_V2),
.intarch = 0,
.mmu = 0,
.hash_ext = 0,
diff --git a/arch/powerpc/platforms/pseries/firmware.c b/arch/powerpc/platforms/pseries/firmware.c
index 63cc82ad58ac..aac3ea2911b2 100644
--- a/arch/powerpc/platforms/pseries/firmware.c
+++ b/arch/powerpc/platforms/pseries/firmware.c
@@ -114,6 +114,7 @@ static __initdata struct vec5_fw_feature
vec5_fw_features_table[] = {
{FW_FEATURE_TYPE1_AFFINITY, OV5_TYPE1_AFFINITY},
{FW_FEATURE_PRRN, OV5_PRRN},
+ {FW_FEATURE_DRMEM_V2, OV5_DRMEM_V2},
};
static void __init fw_vec5_feature_init(const char *vec5, unsigned long len)
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property
2017-10-20 13:21 [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Nathan Fontenot
` (7 preceding siblings ...)
2017-10-20 13:22 ` [PATCH v2 8/8] powerpc: Enable support of ibm,dynamic-memory-v2 Nathan Fontenot
@ 2017-11-16 5:37 ` Bharata B Rao
2017-11-16 16:01 ` Nathan Fontenot
8 siblings, 1 reply; 19+ messages in thread
From: Bharata B Rao @ 2017-11-16 5:37 UTC (permalink / raw)
To: Nathan Fontenot; +Cc: linuxppc-dev, Aneesh Kumar
[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]
On Fri, Oct 20, 2017 at 6:51 PM, Nathan Fontenot <nfont@linux.vnet.ibm.com>
wrote:
> This patch set provides a set of updates to de-couple the LMB information
> provided in the ibm,dynamic-memory device tree property from the device
> tree property format. A part of this patch series introduces a new
> device tree property format for dynamic memory, ibm-dynamic-meory-v2.
> By separating the device tree format from the information provided by
> the device tree property consumers of this information need not know
> what format is currently being used and provide multiple parsing routines
> for the information.
>
> The first two patches update the of_get_assoc_arrays() and
> of_get_usable_memory() routines to look up the device node for the
> properties they parse. This is needed because the calling routines for
> these two functions will not have the device node to pass in in
> subsequent patches.
>
> The third patch adds a new kernel structure, struct drmem_lmb, that
> is used to represent each of the possible LMBs specified in the
> ibm,dynamic-memory* device tree properties. The patch adds code
> to parse the property and build the LMB array data, and updates prom.c
> to use this new data structure instead of parsing the device tree directly.
>
> The fourth and fifth patches update the numa and pseries hotplug code
> respectively to use the new LMB array data instead of parsing the
> device tree directly.
>
> The sixth patch moves the of_drconf_cell struct to drmem.h where it
> fits better than prom.h
>
> The seventh patch introduces support for the ibm,dynamic-memory-v2
> property format by updating the new drmem.c code to be able to parse
> and create this new device tree format.
>
> The last patch in the series updates the architecture vector to indicate
> support for ibm,dynamic-memory-v2.
Here we are consolidating LMBs into LMB sets but still end up working with
individual LMBs during hotplug. Can we instead start working with LMB sets
together during hotplug ? In other words
- The RTAS calls involved during DRC acquire stage can be done only once
per LMB set.
- One configure-connector call for the entire LMB set.
I think this should help hotplugging of large amounts of memory. Other than
that, if we choose to use LMB representation for PMEM, it will be useful
there too to handle all the LMBs of a PMEM range as one set.
Regards,
Bharata.
[-- Attachment #2: Type: text/html, Size: 3002 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property
2017-11-16 5:37 ` [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property Bharata B Rao
@ 2017-11-16 16:01 ` Nathan Fontenot
2017-11-17 4:51 ` Bharata B Rao
0 siblings, 1 reply; 19+ messages in thread
From: Nathan Fontenot @ 2017-11-16 16:01 UTC (permalink / raw)
To: Bharata B Rao; +Cc: linuxppc-dev, Aneesh Kumar
On 11/15/2017 11:37 PM, Bharata B Rao wrote:
> On Fri, Oct 20, 2017 at 6:51 PM, Nathan Fontenot <nfont@linux.vnet.ibm.com <mailto:nfont@linux.vnet.ibm.com>> wrote:
>
> This patch set provides a set of updates to de-couple the LMB information
> provided in the ibm,dynamic-memory device tree property from the device
> tree property format. A part of this patch series introduces a new
> device tree property format for dynamic memory, ibm-dynamic-meory-v2.
> By separating the device tree format from the information provided by
> the device tree property consumers of this information need not know
> what format is currently being used and provide multiple parsing routines
> for the information.
>
> The first two patches update the of_get_assoc_arrays() and
> of_get_usable_memory() routines to look up the device node for the
> properties they parse. This is needed because the calling routines for
> these two functions will not have the device node to pass in in
> subsequent patches.
>
> The third patch adds a new kernel structure, struct drmem_lmb, that
> is used to represent each of the possible LMBs specified in the
> ibm,dynamic-memory* device tree properties. The patch adds code
> to parse the property and build the LMB array data, and updates prom.c
> to use this new data structure instead of parsing the device tree directly.
>
> The fourth and fifth patches update the numa and pseries hotplug code
> respectively to use the new LMB array data instead of parsing the
> device tree directly.
>
> The sixth patch moves the of_drconf_cell struct to drmem.h where it
> fits better than prom.h
>
> The seventh patch introduces support for the ibm,dynamic-memory-v2
> property format by updating the new drmem.c code to be able to parse
> and create this new device tree format.
>
> The last patch in the series updates the architecture vector to indicate
> support for ibm,dynamic-memory-v2.
>
>
> Here we are consolidating LMBs into LMB sets but still end up working with individual LMBs during hotplug. Can we instead start working with LMB sets together during hotplug ? In other words
In a sense we do do this when handling memory DLPAR indexed-count requests. This takes a starting
drc index for a LMB and adds/removes the following <count> contiguous LMBs. This operation is
all-or-nothing, if any LMB fails to add/remove we revert back to the original state.
Thi isn't exactly what you're asking for but...
>
> - The RTAS calls involved during DRC acquire stage can be done only once per LMB set.
> - One configure-connector call for the entire LMB set.
these two interfaces work on a single drc index, not a set of drc indexes. Working on a set
of LMBs would require extending the current rtas calls or creating new ones.
One thing we can look into doing for indexed-count requests is to perform each of the
steps for all LMBs in the set at once, i.e. make the acquire call for LMBs, then make the
configure-connector calls for all the LMBs...
The only drawback is this approach would make handling failures and backing out of the
updates a bit messier, but I've never really thought that optimizing for the failure
case to be as important.
-Nathan
>
> I think this should help hotplugging of large amounts of memory. Other than that, if we choose to use LMB representation for PMEM, it will be useful there too to handle all the LMBs of a PMEM range as one set.
>
> Regards,
> Bharata.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property
2017-11-16 16:01 ` Nathan Fontenot
@ 2017-11-17 4:51 ` Bharata B Rao
2017-11-20 14:48 ` Nathan Fontenot
0 siblings, 1 reply; 19+ messages in thread
From: Bharata B Rao @ 2017-11-17 4:51 UTC (permalink / raw)
To: Nathan Fontenot; +Cc: linuxppc-dev, Aneesh Kumar
[-- Attachment #1: Type: text/plain, Size: 4408 bytes --]
On Thu, Nov 16, 2017 at 9:31 PM, Nathan Fontenot <nfont@linux.vnet.ibm.com>
wrote:
>
>
> On 11/15/2017 11:37 PM, Bharata B Rao wrote:
> > On Fri, Oct 20, 2017 at 6:51 PM, Nathan Fontenot <
> nfont@linux.vnet.ibm.com <mailto:nfont@linux.vnet.ibm.com>> wrote:
> >
> > This patch set provides a set of updates to de-couple the LMB
> information
> > provided in the ibm,dynamic-memory device tree property from the
> device
> > tree property format. A part of this patch series introduces a new
> > device tree property format for dynamic memory, ibm-dynamic-meory-v2.
> > By separating the device tree format from the information provided by
> > the device tree property consumers of this information need not know
> > what format is currently being used and provide multiple parsing
> routines
> > for the information.
> >
> > The first two patches update the of_get_assoc_arrays() and
> > of_get_usable_memory() routines to look up the device node for the
> > properties they parse. This is needed because the calling routines
> for
> > these two functions will not have the device node to pass in in
> > subsequent patches.
> >
> > The third patch adds a new kernel structure, struct drmem_lmb, that
> > is used to represent each of the possible LMBs specified in the
> > ibm,dynamic-memory* device tree properties. The patch adds code
> > to parse the property and build the LMB array data, and updates
> prom.c
> > to use this new data structure instead of parsing the device tree
> directly.
> >
> > The fourth and fifth patches update the numa and pseries hotplug code
> > respectively to use the new LMB array data instead of parsing the
> > device tree directly.
> >
> > The sixth patch moves the of_drconf_cell struct to drmem.h where it
> > fits better than prom.h
> >
> > The seventh patch introduces support for the ibm,dynamic-memory-v2
> > property format by updating the new drmem.c code to be able to parse
> > and create this new device tree format.
> >
> > The last patch in the series updates the architecture vector to
> indicate
> > support for ibm,dynamic-memory-v2.
> >
> >
> > Here we are consolidating LMBs into LMB sets but still end up working
> with individual LMBs during hotplug. Can we instead start working with LMB
> sets together during hotplug ? In other words
>
> In a sense we do do this when handling memory DLPAR indexed-count
> requests. This takes a starting
> drc index for a LMB and adds/removes the following <count> contiguous
> LMBs. This operation is
> all-or-nothing, if any LMB fails to add/remove we revert back to the
> original state.
>
I am aware of count-indexed and we do use it for memory hotplug/unplug for
KVM on Power. However the RTAS and configure-connector calls there are
still per-LMB.
> Thi isn't exactly what you're asking for but...
> >
> > - The RTAS calls involved during DRC acquire stage can be done only once
> per LMB set.
> > - One configure-connector call for the entire LMB set.
>
> these two interfaces work on a single drc index, not a set of drc indexes.
> Working on a set
> of LMBs would require extending the current rtas calls or creating new
> ones.
>
Yes.
>
> One thing we can look into doing for indexed-count requests is to perform
> each of the
> steps for all LMBs in the set at once, i.e. make the acquire call for
> LMBs, then make the
> configure-connector calls for all the LMBs...
>
That is what I am hinting at to check the feasibility of such a mechanism.
Given that all the LMBs of the set are supposed to have similar attributes
(like node associativity etc), it makes sense to have a single DRC acquire
call and single configure-connector call for the entire set.
> The only drawback is this approach would make handling failures and
> backing out of the
> updates a bit messier, but I've never really thought that optimizing for
> the failure
> case to be as important.
>
Yes, error recovery can be messy given that we have multiple calls under
DRC acquire call (get-sensor-state and set-indicator).
BTW I thought this reorganization involving ibm,drc-info and
ibm,dynamic-memory-v2 was for representing and hotplugging huge amounts of
memory efficiently and quickly. So you have not yet found the per-LMB calls
to be hurting when huge amount of memory is involved ?
Regards,
Bharata.
[-- Attachment #2: Type: text/html, Size: 5756 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v2 0/8] powerpc: Support ibm,dynamic-memory-v2 property
2017-11-17 4:51 ` Bharata B Rao
@ 2017-11-20 14:48 ` Nathan Fontenot
0 siblings, 0 replies; 19+ messages in thread
From: Nathan Fontenot @ 2017-11-20 14:48 UTC (permalink / raw)
To: Bharata B Rao; +Cc: linuxppc-dev, Aneesh Kumar
On 11/16/2017 10:51 PM, Bharata B Rao wrote:
>
> On Thu, Nov 16, 2017 at 9:31 PM, Nathan Fontenot <nfont@linux.vnet.ibm.com <mailto:nfont@linux.vnet.ibm.com>> wrote:
>
>
>
> On 11/15/2017 11:37 PM, Bharata B Rao wrote:
> > On Fri, Oct 20, 2017 at 6:51 PM, Nathan Fontenot <nfont@linux.vnet.ibm.com <mailto:nfont@linux.vnet.ibm.com> <mailto:nfont@linux.vnet.ibm.com <mailto:nfont@linux.vnet.ibm.com>>> wrote:
> >
> > This patch set provides a set of updates to de-couple the LMB information
> > provided in the ibm,dynamic-memory device tree property from the device
> > tree property format. A part of this patch series introduces a new
> > device tree property format for dynamic memory, ibm-dynamic-meory-v2.
> > By separating the device tree format from the information provided by
> > the device tree property consumers of this information need not know
> > what format is currently being used and provide multiple parsing routines
> > for the information.
> >
> > The first two patches update the of_get_assoc_arrays() and
> > of_get_usable_memory() routines to look up the device node for the
> > properties they parse. This is needed because the calling routines for
> > these two functions will not have the device node to pass in in
> > subsequent patches.
> >
> > The third patch adds a new kernel structure, struct drmem_lmb, that
> > is used to represent each of the possible LMBs specified in the
> > ibm,dynamic-memory* device tree properties. The patch adds code
> > to parse the property and build the LMB array data, and updates prom.c
> > to use this new data structure instead of parsing the device tree directly.
> >
> > The fourth and fifth patches update the numa and pseries hotplug code
> > respectively to use the new LMB array data instead of parsing the
> > device tree directly.
> >
> > The sixth patch moves the of_drconf_cell struct to drmem.h where it
> > fits better than prom.h
> >
> > The seventh patch introduces support for the ibm,dynamic-memory-v2
> > property format by updating the new drmem.c code to be able to parse
> > and create this new device tree format.
> >
> > The last patch in the series updates the architecture vector to indicate
> > support for ibm,dynamic-memory-v2.
> >
> >
> > Here we are consolidating LMBs into LMB sets but still end up working with individual LMBs during hotplug. Can we instead start working with LMB sets together during hotplug ? In other words
>
> In a sense we do do this when handling memory DLPAR indexed-count requests. This takes a starting
> drc index for a LMB and adds/removes the following <count> contiguous LMBs. This operation is
> all-or-nothing, if any LMB fails to add/remove we revert back to the original state.
>
>
> I am aware of count-indexed and we do use it for memory hotplug/unplug for KVM on Power. However the RTAS and configure-connector calls there are still per-LMB.
>
>
> Thi isn't exactly what you're asking for but...
> >
> > - The RTAS calls involved during DRC acquire stage can be done only once per LMB set.
> > - One configure-connector call for the entire LMB set.
>
> these two interfaces work on a single drc index, not a set of drc indexes. Working on a set
> of LMBs would require extending the current rtas calls or creating new ones.
>
>
> Yes.
>
>
>
> One thing we can look into doing for indexed-count requests is to perform each of the
> steps for all LMBs in the set at once, i.e. make the acquire call for LMBs, then make the
> configure-connector calls for all the LMBs...
>
>
> That is what I am hinting at to check the feasibility of such a mechanism. Given that all the LMBs of the set are supposed to have similar attributes (like node associativity etc), it makes sense to have a single DRC acquire call and single configure-connector call for the entire set.
I agree. I'll talk to pHyp development to see if this is something they are interested in pursuing.
If not we can submit updates to the PAPR to implement these new rtas calls even if they do not
support them in pHyp.
>
>
> The only drawback is this approach would make handling failures and backing out of the
> updates a bit messier, but I've never really thought that optimizing for the failure
> case to be as important.
>
>
> Yes, error recovery can be messy given that we have multiple calls under DRC acquire call (get-sensor-state and set-indicator).
>
> BTW I thought this reorganization involving ibm,drc-info and ibm,dynamic-memory-v2 was for representing and hotplugging huge amounts of memory efficiently and quickly. So you have not yet found the per-LMB calls to be hurting when huge amount of memory is involved ?
No. The updates for the new drc properties are for reducing the size of the properties, it does not change
the way we handle add and remove of memory.
-Nathan
>
> Regards,
> Bharata.
>
^ permalink raw reply [flat|nested] 19+ messages in thread