[PATCH v3 0/12] NUMA CPU Reconfiguration using PRRN

[PATCH v3 0/12] NUMA CPU Reconfiguration using PRRN

[Nathan Fontenot]

Newer firmware on Power systems can transparently reassign platform resources
(CPU and Memory) in use. For instance, if a processor or memory unit is
predicted to fail, the platform may transparently move the processing to an
equivalent unused processor or the memory state to an equivalent unused
memory unit. However, reassigning resources across NUMA boundaries may alter
the performance of the partition. When such reassignment is necessary, the
Platform Resource Reassignment Notification (PRRN) option provides a
mechanism to inform the Linux kernel of changes to the NUMA affinity of
its platform resources.

PRRN Events are RTAS events sent up through the event-scan mechanism on
Power. When these events are received the system needs can get the updated
device tree affinity information for the affected CPUs/memory via the
rtas update-nodes and update-properties calls. This information is then
used to update the NUMA affinity of the CPUs/Memory in the kernel.

This patch set adds the ability to recognize PRRN events, update the device
tree and kernel information for CPUs (memory will be handled in a later
patch), and add an interface to enable/disable toplogy updates from /proc.

Additionally, these updates solve an existing problem with the VPHN (Virtual
Processor Home Node) capability and allow us to re-enable this feature.

Nathan Fontenot

 arch/powerpc/include/asm/firmware.h               |    3 
 arch/powerpc/include/asm/prom.h                   |   46 ++--
 arch/powerpc/include/asm/rtas.h                   |    2 
 arch/powerpc/kernel/prom_init.c                   |   98 ++--------
 arch/powerpc/kernel/rtasd.c                       |   37 +++
 arch/powerpc/mm/numa.c                            |  214 +++++++++++++++-------
 arch/powerpc/platforms/pseries/firmware.c         |    1 
 arch/powerpc/platforms/pseries/mobility.c         |   24 +-
 powerpc/arch/powerpc/include/asm/firmware.h       |    4 
 powerpc/arch/powerpc/include/asm/machdep.h        |    2 
 powerpc/arch/powerpc/include/asm/prom.h           |   73 +++++++
 powerpc/arch/powerpc/include/asm/rtas.h           |    1 
 powerpc/arch/powerpc/include/asm/topology.h       |    5 
 powerpc/arch/powerpc/kernel/prom_init.c           |    2 
 powerpc/arch/powerpc/kernel/rtas.c                |   10 +
 powerpc/arch/powerpc/kernel/rtasd.c               |    7 
 powerpc/arch/powerpc/mm/numa.c                    |   62 ++++++
 powerpc/arch/powerpc/platforms/pseries/firmware.c |   49 ++++-
 powerpc/arch/powerpc/platforms/pseries/mobility.c |   20 +-
 powerpc/arch/powerpc/platforms/pseries/pseries.h  |    5 
 powerpc/arch/powerpc/platforms/pseries/setup.c    |   40 ++--
 21 files changed, 500 insertions(+), 205 deletions(-)

Updates for v3 of the patchset:

1/12 - Updated to use a ppc_md interface to invoke device tree updates, this
corrects the build break previously seen in patch 2/12 for non-pseries
platforms.

2/12 - New patch in the series to correct the parsing of the buffer returned
from ibm,update-properties rtas call.

5/12 - The parsing of architecture vector 5 has been made more efficient.

7/12 - Correct #define used in call the firmware_has_feature()

8/12 - Updated calling of stop_machine() to only call it once per PRRN event.

12/12 - Added inclusion of topology.h to rtasd.c to correct a build failure
on non-pseries platforms.

[PATCH v3 1/12] Create a powerpc update_devicetree interface

[Nathan Fontenot]

Newer firmware on Power systems can transparently reassign platform resources
(CPU and Memory) in use. For instance, if a processor or memory unit is
predicted to fail, the platform may transparently move the processing to an
equivalent unused processor or the memory state to an equivalent unused
memory unit. However, reassigning resources across NUMA boundaries may alter
the performance of the partition. When such reassignment is necessary, the
Platform Resource Reassignment Notification (PRRN) option provides a
mechanism to inform the Linux kernel of changes to the NUMA affinity of
its platform resources.

When rtasd receives a PRRN event, it needs to make a series of RTAS
calls (ibm,update-nodes and ibm,update-properties) to retrieve the
updated device tree information. These calls are already handled in the
pseries_devtree_update() routine used in partition migration.

This patch exposes a method for updating the device tree via
ppc_md.update_devicetree that takes a single 32-bit value as a parameter.
For pseries platforms this is the existing pseries_devicetree_update routine
which is updated to take the new parameter which is a scope value
to indicate the the reason for making the rtas calls. This parameter is
required by the ibm,update-nodes/ibm,update-properties RTAS calls, and
the appropriate value is contained within the RTAS event for PRRN
notifications. In pseries_devicetree_update() it was previously
hard-coded to 1, the scope value for partition migration.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/machdep.h        |    2 ++
 arch/powerpc/include/asm/rtas.h           |    1 +
 arch/powerpc/kernel/rtas.c                |   10 ++++++++++
 arch/powerpc/platforms/pseries/mobility.c |   24 +++++++++++++++---------
 4 files changed, 28 insertions(+), 9 deletions(-)

Index: powerpc/arch/powerpc/include/asm/rtas.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/rtas.h	2013-04-15 09:18:10.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/rtas.h	2013-04-17 12:58:33.000000000 -0500
@@ -276,6 +276,7 @@
 		const char *uname, int depth, void *data);
 
 extern void pSeries_log_error(char *buf, unsigned int err_type, int fatal);
+extern int update_devicetree(s32 scope);
 
 #ifdef CONFIG_PPC_RTAS_DAEMON
 extern void rtas_cancel_event_scan(void);
Index: powerpc/arch/powerpc/platforms/pseries/mobility.c
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/mobility.c	2013-04-15 09:18:10.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/mobility.c	2013-04-17 13:01:08.000000000 -0500
@@ -19,6 +19,7 @@
 #include <linux/slab.h>
 
 #include <asm/rtas.h>
+#include <asm/machdep.h>
 #include "pseries.h"
 
 static struct kobject *mobility_kobj;
@@ -37,14 +38,16 @@
 #define UPDATE_DT_NODE	0x02000000
 #define ADD_DT_NODE	0x03000000
 
-static int mobility_rtas_call(int token, char *buf)
+#define MIGRATION_SCOPE	(1)
+
+static int mobility_rtas_call(int token, char *buf, s32 scope)
 {
 	int rc;
 
 	spin_lock(&rtas_data_buf_lock);
 
 	memcpy(rtas_data_buf, buf, RTAS_DATA_BUF_SIZE);
-	rc = rtas_call(token, 2, 1, NULL, rtas_data_buf, 1);
+	rc = rtas_call(token, 2, 1, NULL, rtas_data_buf, scope);
 	memcpy(buf, rtas_data_buf, RTAS_DATA_BUF_SIZE);
 
 	spin_unlock(&rtas_data_buf_lock);
@@ -123,7 +126,7 @@
 	return 0;
 }
 
-static int update_dt_node(u32 phandle)
+static int update_dt_node(u32 phandle, s32 scope)
 {
 	struct update_props_workarea *upwa;
 	struct device_node *dn;
@@ -151,7 +154,8 @@
 	upwa->phandle = phandle;
 
 	do {
-		rc = mobility_rtas_call(update_properties_token, rtas_buf);
+		rc = mobility_rtas_call(update_properties_token, rtas_buf,
+					scope);
 		if (rc < 0)
 			break;
 
@@ -219,7 +223,7 @@
 	return rc;
 }
 
-static int pseries_devicetree_update(void)
+static int pseries_devicetree_update(s32 scope)
 {
 	char *rtas_buf;
 	u32 *data;
@@ -235,7 +239,7 @@
 		return -ENOMEM;
 
 	do {
-		rc = mobility_rtas_call(update_nodes_token, rtas_buf);
+		rc = mobility_rtas_call(update_nodes_token, rtas_buf, scope);
 		if (rc && rc != 1)
 			break;
 
@@ -256,7 +260,7 @@
 					delete_dt_node(phandle);
 					break;
 				case UPDATE_DT_NODE:
-					update_dt_node(phandle);
+					update_dt_node(phandle, scope);
 					break;
 				case ADD_DT_NODE:
 					drc_index = *data++;
@@ -276,7 +280,7 @@
 	int rc;
 	int activate_fw_token;
 
-	rc = pseries_devicetree_update();
+	rc = pseries_devicetree_update(MIGRATION_SCOPE);
 	if (rc) {
 		printk(KERN_ERR "Initial post-mobility device tree update "
 		       "failed: %d\n", rc);
@@ -292,7 +296,7 @@
 
 	rc = rtas_call(activate_fw_token, 0, 1, NULL);
 	if (!rc) {
-		rc = pseries_devicetree_update();
+		rc = pseries_devicetree_update(MIGRATION_SCOPE);
 		if (rc)
 			printk(KERN_ERR "Secondary post-mobility device tree "
 			       "update failed: %d\n", rc);
@@ -346,6 +350,8 @@
 {
 	int rc;
 
+	ppc_md.update_devicetree = pseries_devicetree_update;
+
 	mobility_kobj = kobject_create_and_add("mobility", kernel_kobj);
 	if (!mobility_kobj)
 		return -ENOMEM;
Index: powerpc/arch/powerpc/include/asm/machdep.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/machdep.h	2013-03-08 19:23:06.000000000 -0600
+++ powerpc/arch/powerpc/include/asm/machdep.h	2013-04-17 13:04:57.000000000 -0500
@@ -256,6 +256,8 @@
 	ssize_t (*cpu_probe)(const char *, size_t);
 	ssize_t (*cpu_release)(const char *, size_t);
 #endif
+
+	int (*update_devicetree)(s32);
 };
 
 extern void e500_idle(void);
Index: powerpc/arch/powerpc/kernel/rtas.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/rtas.c	2013-03-08 19:23:06.000000000 -0600
+++ powerpc/arch/powerpc/kernel/rtas.c	2013-04-17 13:02:29.000000000 -0500
@@ -1085,3 +1085,13 @@
 	timebase = 0;
 	arch_spin_unlock(&timebase_lock);
 }
+
+int update_devicetree(s32 scope)
+{
+	int rc = 0;
+
+	if (ppc_md.update_devicetree)
+		rc = ppc_md.update_devicetree(scope);
+
+	return rc;
+}

[PATCH v3 2/12] Correct buffer parsing in update-properties

[Nathan Fontenot]

Correct parsing of the buffer returned from ibm,update-properties. The first
element is a length and the path to the property which is slightly different
from the list of properties in the buffer so we need to specifically
handle this.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/pseries/mobility.c |   20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

Index: powerpc/arch/powerpc/platforms/pseries/mobility.c
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/mobility.c	2013-04-17 13:27:23.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/mobility.c	2013-04-17 13:28:58.000000000 -0500
@@ -135,6 +135,7 @@
 	char *prop_data;
 	char *rtas_buf;
 	int update_properties_token;
+	u32 vd;
 
 	update_properties_token = rtas_token("ibm,update-properties");
 	if (update_properties_token == RTAS_UNKNOWN_SERVICE)
@@ -161,13 +162,24 @@
 
 		prop_data = rtas_buf + sizeof(*upwa);
 
-		for (i = 0; i < upwa->nprops; i++) {
+		/* The first element of the buffer is the path of the node
+		 * being updated in the form of a 8 byte string length
+		 * followed by the string. Skip past this to get to the
+		 * properties being updated.
+		 */
+		vd = *prop_data++;
+		prop_data += vd;
+
+		/* The path we skipped over is counted as one of the elements
+		 * returned so start counting at one.
+		 */
+		for (i = 1; i < upwa->nprops; i++) {
 			char *prop_name;
-			u32 vd;
 
-			prop_name = prop_data + 1;
+			prop_name = prop_data;
 			prop_data += strlen(prop_name) + 1;
-			vd = *prop_data++;
+			vd = *(u32 *)prop_data;
+			prop_data += sizeof(vd);
 
 			switch (vd) {
 			case 0x00000000:

[PATCH v3 3/12] Add PRRN event handler

[Nathan Fontenot]

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

A PRRN event is signaled via the RTAS event-scan mechanism, which
returns a Hot Plug Event message "fixed part" indicating "Platform
Resource Reassignment". In response to the Hot Plug Event message,
we must call ibm,update-nodes to determine which resources were
reassigned and then ibm,update-properties to obtain the new affinity
information about those resources.

The PRRN event-scan RTAS message contains only the "fixed part" with
the "Type" field set to the value 160 and no Extended Event Log. The
four-byte Extended Event Log Length field is repurposed (since no
Extended Event Log message is included) to pass the "scope" parameter
that causes the ibm,update-nodes to return the nodes affected by the
specific resource reassignment.

This patch adds a handler for PRRN RTAS events. The function
pseries_devicetree_update() (from mobility.c) is used to make the
ibm,update-nodes/ibm,update-properties RTAS calls. Updating the NUMA maps
(handled by a subsequent patch) will require significant processing,
so pseries_devicetree_update() is called from an asynchronous workqueue
to allow event processing to continue. 

PRRN RTAS events on pseries systems are rare events that have to be
initiated from the HMC console for the system by an IBM tech. This allows
us to assume that these events are widely spaced. Additionally, all work
on the queue is flushed before handling any new work to ensure we only have
one event in flight being handled at a time.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/rtas.h |    2 ++
 arch/powerpc/kernel/rtasd.c     |   37 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 38 insertions(+), 1 deletion(-)

Index: powerpc/arch/powerpc/include/asm/rtas.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/rtas.h	2013-04-17 12:58:33.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/rtas.h	2013-04-17 13:24:06.000000000 -0500
@@ -143,6 +143,8 @@
 #define RTAS_TYPE_PMGM_TIME_ALARM	0x6f
 #define RTAS_TYPE_PMGM_CONFIG_CHANGE	0x70
 #define RTAS_TYPE_PMGM_SERVICE_PROC	0x71
+/* Platform Resource Reassignment Notification */
+#define RTAS_TYPE_PRRN			0xA0
 
 /* RTAS check-exception vector offset */
 #define RTAS_VECTOR_EXTERNAL_INTERRUPT	0x500
Index: powerpc/arch/powerpc/kernel/rtasd.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/rtasd.c	2013-04-17 12:55:11.000000000 -0500
+++ powerpc/arch/powerpc/kernel/rtasd.c	2013-04-17 13:27:00.000000000 -0500
@@ -87,6 +87,8 @@
 			return "Resource Deallocation Event";
 		case RTAS_TYPE_DUMP:
 			return "Dump Notification Event";
+		case RTAS_TYPE_PRRN:
+			return "Platform Resource Reassignment Event";
 	}
 
 	return rtas_type[0];
@@ -265,7 +267,40 @@
 		spin_unlock_irqrestore(&rtasd_log_lock, s);
 		return;
 	}
+}
+
+static s32 update_scope;
+
+static void prrn_work_fn(struct work_struct *work)
+{
+	/*
+	 * For PRRN, we must pass the negative of the scope value in
+	 * the RTAS event.
+	 */
+	if (ppc_md.update_devicetree)
+		ppc_md.update_devicetree(-update_scope);
+}
+
+static DECLARE_WORK(prrn_work, prrn_work_fn);
+
+void prrn_schedule_update(u32 scope)
+{
+	flush_work(&prrn_work);
+	update_scope = scope;
+	schedule_work(&prrn_work);
+}
+
+static void pseries_handle_event(const struct rtas_error_log *log)
+{
+	pSeries_log_error((char *)log, ERR_TYPE_RTAS_LOG, 0);
+
+	if (log->type == RTAS_TYPE_PRRN)
+		/* For PRRN Events the extended log length is used to denote
+		 * the scope for calling rtas update-nodes.
+		 */
+		prrn_schedule_update(log->extended_log_length);
 
+	return;
 }
 
 static int rtas_log_open(struct inode * inode, struct file * file)
@@ -389,7 +424,7 @@
 		}
 
 		if (error == 0)
-			pSeries_log_error(logdata, ERR_TYPE_RTAS_LOG, 0);
+			pseries_handle_event((struct rtas_error_log *)logdata);
 
 	} while(error == 0);
 }

[PATCH v3 4/12] Move architecture vector definitions to prom.h

[Nathan Fontenot]

As part of handling handling PRRN events we will need to check the
vector 5 portion of the architecture bits reported in the device tree
to ensure that PRRN event handling is enabled. In order to do this
firmware_has_feature is updated (in a subsequent patch) to
make this check.  To avoid having to re-define bits in the architecture
vector the bits are moved to prom.h.

This patch is the first step in updating firmware_has_feature
by simply moving the bit definitions from prom_init.c to asm/prom.h.
There are no functional changes.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>

---
 arch/powerpc/include/asm/prom.h |   73 ++++++++++++++++++++++++++++++++++++++
 arch/powerpc/kernel/prom_init.c |   75 +++-------------------------------------
 2 files changed, 79 insertions(+), 69 deletions(-)

Index: powerpc/arch/powerpc/include/asm/prom.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/prom.h	2013-04-16 21:25:16.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/prom.h	2013-04-17 13:43:13.000000000 -0500
@@ -74,6 +74,79 @@
 #define DRCONF_MEM_AI_INVALID	0x00000040
 #define DRCONF_MEM_RESERVED	0x00000080
 
+#if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV)
+/*
+ * There are two methods for telling firmware what our capabilities are.
+ * Newer machines have an "ibm,client-architecture-support" method on the
+ * root node.  For older machines, we have to call the "process-elf-header"
+ * method in the /packages/elf-loader node, passing it a fake 32-bit
+ * ELF header containing a couple of PT_NOTE sections that contain
+ * structures that contain various information.
+ */
+
+/* New method - extensible architecture description vector. */
+
+/* Option vector bits - generic bits in byte 1 */
+#define OV_IGNORE		0x80	/* ignore this vector */
+#define OV_CESSATION_POLICY	0x40	/* halt if unsupported option present*/
+
+/* Option vector 1: processor architectures supported */
+#define OV1_PPC_2_00		0x80	/* set if we support PowerPC 2.00 */
+#define OV1_PPC_2_01		0x40	/* set if we support PowerPC 2.01 */
+#define OV1_PPC_2_02		0x20	/* set if we support PowerPC 2.02 */
+#define OV1_PPC_2_03		0x10	/* set if we support PowerPC 2.03 */
+#define OV1_PPC_2_04		0x08	/* set if we support PowerPC 2.04 */
+#define OV1_PPC_2_05		0x04	/* set if we support PowerPC 2.05 */
+#define OV1_PPC_2_06		0x02	/* set if we support PowerPC 2.06 */
+#define OV1_PPC_2_07		0x01	/* set if we support PowerPC 2.07 */
+
+/* Option vector 2: Open Firmware options supported */
+#define OV2_REAL_MODE		0x20	/* set if we want OF in real mode */
+
+/* Option vector 3: processor options supported */
+#define OV3_FP			0x80	/* floating point */
+#define OV3_VMX			0x40	/* VMX/Altivec */
+#define OV3_DFP			0x20	/* decimal FP */
+
+/* Option vector 4: IBM PAPR implementation */
+#define OV4_MIN_ENT_CAP		0x01	/* minimum VP entitled capacity */
+
+/* Option vector 5: PAPR/OF options supported */
+#define OV5_LPAR		0x80	/* logical partitioning supported */
+#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
+/* ibm,dynamic-reconfiguration-memory property supported */
+#define OV5_DRCONF_MEMORY	0x20
+#define OV5_LARGE_PAGES		0x10	/* large pages supported */
+#define OV5_DONATE_DEDICATE_CPU	0x02	/* donate dedicated CPU support */
+/* PCIe/MSI support.  Without MSI full PCIe is not supported */
+#ifdef CONFIG_PCI_MSI
+#define OV5_MSI			0x01	/* PCIe/MSI support */
+#else
+#define OV5_MSI			0x00
+#endif /* CONFIG_PCI_MSI */
+#ifdef CONFIG_PPC_SMLPAR
+#define OV5_CMO			0x80	/* Cooperative Memory Overcommitment */
+#define OV5_XCMO		0x40	/* Page Coalescing */
+#else
+#define OV5_CMO			0x00
+#define OV5_XCMO		0x00
+#endif
+#define OV5_TYPE1_AFFINITY	0x80	/* Type 1 NUMA affinity */
+#define OV5_PFO_HW_RNG		0x80	/* PFO Random Number Generator */
+#define OV5_PFO_HW_842		0x40	/* PFO Compression Accelerator */
+#define OV5_PFO_HW_ENCR		0x20	/* PFO Encryption Accelerator */
+#define OV5_SUB_PROCESSORS	0x01	/* 1,2,or 4 Sub-Processors supported */
+
+/* Option Vector 6: IBM PAPR hints */
+#define OV6_LINUX		0x02	/* Linux is our OS */
+
+/*
+ * The architecture vector has an array of PVR mask/value pairs,
+ * followed by # option vectors - 1, followed by the option vectors.
+ */
+extern unsigned char ibm_architecture_vec[];
+#endif
+
 /* These includes are put at the bottom because they may contain things
  * that are overridden by this file.  Ideally they shouldn't be included
  * by this file, but there are a bunch of .c files that currently depend
Index: powerpc/arch/powerpc/kernel/prom_init.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/prom_init.c	2013-04-16 21:25:16.000000000 -0500
+++ powerpc/arch/powerpc/kernel/prom_init.c	2013-04-17 13:43:13.000000000 -0500
@@ -627,16 +627,11 @@
 
 #if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV)
 /*
- * There are two methods for telling firmware what our capabilities are.
- * Newer machines have an "ibm,client-architecture-support" method on the
- * root node.  For older machines, we have to call the "process-elf-header"
- * method in the /packages/elf-loader node, passing it a fake 32-bit
- * ELF header containing a couple of PT_NOTE sections that contain
- * structures that contain various information.
- */
-
-/*
- * New method - extensible architecture description vector.
+ * The architecture vector has an array of PVR mask/value pairs,
+ * followed by # option vectors - 1, followed by the option vectors.
+ *
+ * See prom.h for the definition of the bits specified in the
+ * achitecture vector.
  *
  * Because the description vector contains a mix of byte and word
  * values, we declare it as an unsigned char array, and use this
@@ -645,65 +640,7 @@
 #define W(x)	((x) >> 24) & 0xff, ((x) >> 16) & 0xff, \
 		((x) >> 8) & 0xff, (x) & 0xff
 
-/* Option vector bits - generic bits in byte 1 */
-#define OV_IGNORE		0x80	/* ignore this vector */
-#define OV_CESSATION_POLICY	0x40	/* halt if unsupported option present*/
-
-/* Option vector 1: processor architectures supported */
-#define OV1_PPC_2_00		0x80	/* set if we support PowerPC 2.00 */
-#define OV1_PPC_2_01		0x40	/* set if we support PowerPC 2.01 */
-#define OV1_PPC_2_02		0x20	/* set if we support PowerPC 2.02 */
-#define OV1_PPC_2_03		0x10	/* set if we support PowerPC 2.03 */
-#define OV1_PPC_2_04		0x08	/* set if we support PowerPC 2.04 */
-#define OV1_PPC_2_05		0x04	/* set if we support PowerPC 2.05 */
-#define OV1_PPC_2_06		0x02	/* set if we support PowerPC 2.06 */
-#define OV1_PPC_2_07		0x01	/* set if we support PowerPC 2.07 */
-
-/* Option vector 2: Open Firmware options supported */
-#define OV2_REAL_MODE		0x20	/* set if we want OF in real mode */
-
-/* Option vector 3: processor options supported */
-#define OV3_FP			0x80	/* floating point */
-#define OV3_VMX			0x40	/* VMX/Altivec */
-#define OV3_DFP			0x20	/* decimal FP */
-
-/* Option vector 4: IBM PAPR implementation */
-#define OV4_MIN_ENT_CAP		0x01	/* minimum VP entitled capacity */
-
-/* Option vector 5: PAPR/OF options supported */
-#define OV5_LPAR		0x80	/* logical partitioning supported */
-#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
-/* ibm,dynamic-reconfiguration-memory property supported */
-#define OV5_DRCONF_MEMORY	0x20
-#define OV5_LARGE_PAGES		0x10	/* large pages supported */
-#define OV5_DONATE_DEDICATE_CPU 0x02	/* donate dedicated CPU support */
-/* PCIe/MSI support.  Without MSI full PCIe is not supported */
-#ifdef CONFIG_PCI_MSI
-#define OV5_MSI			0x01	/* PCIe/MSI support */
-#else
-#define OV5_MSI			0x00
-#endif /* CONFIG_PCI_MSI */
-#ifdef CONFIG_PPC_SMLPAR
-#define OV5_CMO			0x80	/* Cooperative Memory Overcommitment */
-#define OV5_XCMO			0x40	/* Page Coalescing */
-#else
-#define OV5_CMO			0x00
-#define OV5_XCMO			0x00
-#endif
-#define OV5_TYPE1_AFFINITY	0x80	/* Type 1 NUMA affinity */
-#define OV5_PFO_HW_RNG		0x80	/* PFO Random Number Generator */
-#define OV5_PFO_HW_842		0x40	/* PFO Compression Accelerator */
-#define OV5_PFO_HW_ENCR		0x20	/* PFO Encryption Accelerator */
-#define OV5_SUB_PROCESSORS	0x01    /* 1,2,or 4 Sub-Processors supported */
-
-/* Option Vector 6: IBM PAPR hints */
-#define OV6_LINUX		0x02	/* Linux is our OS */
-
-/*
- * The architecture vector has an array of PVR mask/value pairs,
- * followed by # option vectors - 1, followed by the option vectors.
- */
-static unsigned char ibm_architecture_vec[] = {
+unsigned char ibm_architecture_vec[] = {
 	W(0xfffe0000), W(0x003a0000),	/* POWER5/POWER5+ */
 	W(0xffff0000), W(0x003e0000),	/* POWER6 */
 	W(0xffff0000), W(0x003f0000),	/* POWER7 */

[PATCH v3 5/12] Update firmware_has_feature() to check architecture bits

[Nathan Fontenot]

The firmware_has_feature() function makes it easy to check for supported
features of the hypervisor. This patch extends the capability of the
firmware_has_feature() function to include checking for specified bits
in vector 5 of the architecture vector as is reported in the device tree.

As part of this the #defines used for the architecture vector are
re-defined such that the vector 5 options have the vector
index and the feature bits encoded into them. This makes for a much
simpler design to update firmware_has_feature() to check for bits
in the architecture vector.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/firmware.h       |    4 +-
 arch/powerpc/include/asm/prom.h           |   45 ++++++++++++---------------
 arch/powerpc/kernel/prom_init.c           |   23 ++++++++++----
 arch/powerpc/platforms/pseries/firmware.c |   49 +++++++++++++++++++++++++-----
 arch/powerpc/platforms/pseries/pseries.h  |    5 ++-
 arch/powerpc/platforms/pseries/setup.c    |   40 ++++++++++++++++--------
 6 files changed, 113 insertions(+), 53 deletions(-)

Index: powerpc/arch/powerpc/include/asm/prom.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/prom.h	2013-04-17 13:43:13.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/prom.h	2013-04-17 13:51:46.000000000 -0500
@@ -111,31 +111,27 @@
 /* Option vector 4: IBM PAPR implementation */
 #define OV4_MIN_ENT_CAP		0x01	/* minimum VP entitled capacity */
 
-/* Option vector 5: PAPR/OF options supported */
-#define OV5_LPAR		0x80	/* logical partitioning supported */
-#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
+/* Option vector 5: PAPR/OF options supported
+ * Thses bits are also used for the platform_has_feature() call so
+ * we encode the vector index in the define and use the OV5_FEAT()
+ * and OV5_INDX() macros to extract the desired information.
+ */
+#define OV5_FEAT(x)	((x) & 0xff)
+#define OV5_INDX(x)	((x) >> 8)
+#define OV5_LPAR		0x0280	/* logical partitioning supported */
+#define OV5_SPLPAR		0x0240	/* shared-processor LPAR supported */
 /* ibm,dynamic-reconfiguration-memory property supported */
-#define OV5_DRCONF_MEMORY	0x20
-#define OV5_LARGE_PAGES		0x10	/* large pages supported */
-#define OV5_DONATE_DEDICATE_CPU	0x02	/* donate dedicated CPU support */
-/* PCIe/MSI support.  Without MSI full PCIe is not supported */
-#ifdef CONFIG_PCI_MSI
-#define OV5_MSI			0x01	/* PCIe/MSI support */
-#else
-#define OV5_MSI			0x00
-#endif /* CONFIG_PCI_MSI */
-#ifdef CONFIG_PPC_SMLPAR
-#define OV5_CMO			0x80	/* Cooperative Memory Overcommitment */
-#define OV5_XCMO		0x40	/* Page Coalescing */
-#else
-#define OV5_CMO			0x00
-#define OV5_XCMO		0x00
-#endif
-#define OV5_TYPE1_AFFINITY	0x80	/* Type 1 NUMA affinity */
-#define OV5_PFO_HW_RNG		0x80	/* PFO Random Number Generator */
-#define OV5_PFO_HW_842		0x40	/* PFO Compression Accelerator */
-#define OV5_PFO_HW_ENCR		0x20	/* PFO Encryption Accelerator */
-#define OV5_SUB_PROCESSORS	0x01	/* 1,2,or 4 Sub-Processors supported */
+#define OV5_DRCONF_MEMORY	0x0220
+#define OV5_LARGE_PAGES		0x0210	/* large pages supported */
+#define OV5_DONATE_DEDICATE_CPU	0x0202	/* donate dedicated CPU support */
+#define OV5_MSI			0x0201	/* PCIe/MSI support */
+#define OV5_CMO			0x0480	/* Cooperative Memory Overcommitment */
+#define OV5_XCMO		0x0440	/* Page Coalescing */
+#define OV5_TYPE1_AFFINITY	0x0580	/* Type 1 NUMA affinity */
+#define OV5_PFO_HW_RNG		0x0E80	/* PFO Random Number Generator */
+#define OV5_PFO_HW_842		0x0E40	/* PFO Compression Accelerator */
+#define OV5_PFO_HW_ENCR		0x0E20	/* PFO Encryption Accelerator */
+#define OV5_SUB_PROCESSORS	0x0F01	/* 1,2,or 4 Sub-Processors supported */
 
 /* Option Vector 6: IBM PAPR hints */
 #define OV6_LINUX		0x02	/* Linux is our OS */
@@ -145,6 +141,7 @@
  * followed by # option vectors - 1, followed by the option vectors.
  */
 extern unsigned char ibm_architecture_vec[];
+bool platform_has_feature(unsigned int);
 #endif
 
 /* These includes are put at the bottom because they may contain things
Index: powerpc/arch/powerpc/kernel/prom_init.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/prom_init.c	2013-04-17 13:43:13.000000000 -0500
+++ powerpc/arch/powerpc/kernel/prom_init.c	2013-04-17 13:51:46.000000000 -0500
@@ -684,11 +684,21 @@
 	/* option vector 5: PAPR/OF options */
 	19 - 2,				/* length */
 	0,				/* don't ignore, don't halt */
-	OV5_LPAR | OV5_SPLPAR | OV5_LARGE_PAGES | OV5_DRCONF_MEMORY |
-	OV5_DONATE_DEDICATE_CPU | OV5_MSI,
+	OV5_FEAT(OV5_LPAR) | OV5_FEAT(OV5_SPLPAR) | OV5_FEAT(OV5_LARGE_PAGES) |
+	OV5_FEAT(OV5_DRCONF_MEMORY) | OV5_FEAT(OV5_DONATE_DEDICATE_CPU) |
+#ifdef CONFIG_PCI_MSI
+	/* PCIe/MSI support.  Without MSI full PCIe is not supported */
+	OV5_FEAT(OV5_MSI),
+#else
+	0,
+#endif
+	0,
+#ifdef CONFIG_PPC_SMLPAR
+	OV5_FEAT(OV5_CMO) | OV5_FEAT(OV5_XCMO),
+#else
 	0,
-	OV5_CMO | OV5_XCMO,
-	OV5_TYPE1_AFFINITY,
+#endif
+	OV5_FEAT(OV5_TYPE1_AFFINITY),
 	0,
 	0,
 	0,
@@ -702,8 +712,9 @@
 	0,
 	0,
 	0,
-	OV5_PFO_HW_RNG | OV5_PFO_HW_ENCR | OV5_PFO_HW_842,
-	OV5_SUB_PROCESSORS,
+	OV5_FEAT(OV5_PFO_HW_RNG) | OV5_FEAT(OV5_PFO_HW_ENCR) |
+	OV5_FEAT(OV5_PFO_HW_842),
+	OV5_FEAT(OV5_SUB_PROCESSORS),
 	/* option vector 6: IBM PAPR hints */
 	4 - 2,				/* length */
 	0,
Index: powerpc/arch/powerpc/platforms/pseries/setup.c
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/setup.c	2013-04-16 21:25:15.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/setup.c	2013-04-17 13:51:46.000000000 -0500
@@ -628,25 +628,39 @@
  * Called very early, MMU is off, device-tree isn't unflattened
  */
 
-static int __init pSeries_probe_hypertas(unsigned long node,
-					 const char *uname, int depth,
-					 void *data)
+static int __init pseries_probe_fw_features(unsigned long node,
+					    const char *uname, int depth,
+					    void *data)
 {
-	const char *hypertas;
+	const char *prop;
 	unsigned long len;
+	static int hypertas_found;
+	static int vec5_found;
 
-	if (depth != 1 ||
-	    (strcmp(uname, "rtas") != 0 && strcmp(uname, "rtas@0") != 0))
+	if (depth != 1)
 		return 0;
 
-	hypertas = of_get_flat_dt_prop(node, "ibm,hypertas-functions", &len);
-	if (!hypertas)
-		return 1;
+	if (!strcmp(uname, "rtas") || !strcmp(uname, "rtas@0")) {
+		prop = of_get_flat_dt_prop(node, "ibm,hypertas-functions",
+					   &len);
+		if (prop) {
+			powerpc_firmware_features |= FW_FEATURE_LPAR;
+			fw_hypertas_feature_init(prop, len);
+		}
+
+		hypertas_found = 1;
+	}
+
+	if (!strcmp(uname, "chosen")) {
+		prop = of_get_flat_dt_prop(node, "ibm,architecture-vec-5",
+					   &len);
+		if (prop)
+			fw_vec5_feature_init(prop, len);
 
-	powerpc_firmware_features |= FW_FEATURE_LPAR;
-	fw_feature_init(hypertas, len);
+		vec5_found = 1;
+	}
 
-	return 1;
+	return hypertas_found && vec5_found;
 }
 
 static int __init pSeries_probe(void)
@@ -669,7 +683,7 @@
 	pr_debug("pSeries detected, looking for LPAR capability...\n");
 
 	/* Now try to figure out if we are running on LPAR */
-	of_scan_flat_dt(pSeries_probe_hypertas, NULL);
+	of_scan_flat_dt(pseries_probe_fw_features, NULL);
 
 	if (firmware_has_feature(FW_FEATURE_LPAR))
 		hpte_init_lpar();
Index: powerpc/arch/powerpc/platforms/pseries/firmware.c
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/firmware.c	2013-04-16 21:25:15.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/firmware.c	2013-04-17 13:51:46.000000000 -0500
@@ -31,15 +31,15 @@
 typedef struct {
     unsigned long val;
     char * name;
-} firmware_feature_t;
+} hypertas_fw_feature_t;
 
 /*
  * The names in this table match names in rtas/ibm,hypertas-functions.  If the
  * entry ends in a '*', only upto the '*' is matched.  Otherwise the entire
  * string must match.
  */
-static __initdata firmware_feature_t
-firmware_features_table[FIRMWARE_MAX_FEATURES] = {
+static __initdata hypertas_fw_feature_t
+hypertas_fw_features_table[FIRMWARE_MAX_FEATURES] = {
 	{FW_FEATURE_PFT,		"hcall-pft"},
 	{FW_FEATURE_TCE,		"hcall-tce"},
 	{FW_FEATURE_SPRG0,		"hcall-sprg0"},
@@ -69,16 +69,16 @@
  * device-tree/ibm,hypertas-functions.  Ultimately this functionality may
  * be moved into prom.c prom_init().
  */
-void __init fw_feature_init(const char *hypertas, unsigned long len)
+void __init fw_hypertas_feature_init(const char *hypertas, unsigned long len)
 {
 	const char *s;
 	int i;
 
-	pr_debug(" -> fw_feature_init()\n");
+	pr_debug(" -> fw_hypertas_feature_init()\n");
 
 	for (s = hypertas; s < hypertas + len; s += strlen(s) + 1) {
 		for (i = 0; i < FIRMWARE_MAX_FEATURES; i++) {
-			const char *name = firmware_features_table[i].name;
+			const char *name = hypertas_fw_features_table[i].name;
 			size_t size;
 			/* check value against table of strings */
 			if (!name)
@@ -96,10 +96,43 @@
 
 			/* we have a match */
 			powerpc_firmware_features |=
-				firmware_features_table[i].val;
+				hypertas_fw_features_table[i].val;
 			break;
 		}
 	}
 
-	pr_debug(" <- fw_feature_init()\n");
+	pr_debug(" <- fw_hypertas_feature_init()\n");
+}
+
+struct vec5_fw_feature {
+	unsigned long	val;
+	unsigned int	feature;
+
+};
+
+static __initdata struct vec5_fw_feature
+vec5_fw_features_table[FIRMWARE_MAX_FEATURES] = {
+	{FW_FEATURE_TYPE1_AFFINITY,	OV5_TYPE1_AFFINITY},
+};
+
+void __init fw_vec5_feature_init(const char *vec5, unsigned long len)
+{
+	unsigned int index, feat;
+	int i;
+
+	pr_debug(" -> fw_vec5_feature_init()\n");
+
+	for (i = 0; i < FIRMWARE_MAX_FEATURES; i++) {
+		if (!vec5_fw_features_table[i].feature)
+			continue;
+
+		index = OV5_INDX(vec5_fw_features_table[i].feature);
+		feat = OV5_FEAT(vec5_fw_features_table[i].feature);
+
+		if (vec5[index] & feat)
+			powerpc_firmware_features |=
+				vec5_fw_features_table[i].val;
+	}
+
+	pr_debug(" <- fw_vec5_feature_init()\n");
 }
Index: powerpc/arch/powerpc/include/asm/firmware.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/firmware.h	2013-04-16 21:25:15.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/firmware.h	2013-04-17 13:51:46.000000000 -0500
@@ -51,6 +51,7 @@
 #define FW_FEATURE_OPALv2	ASM_CONST(0x0000000020000000)
 #define FW_FEATURE_SET_MODE	ASM_CONST(0x0000000040000000)
 #define FW_FEATURE_BEST_ENERGY	ASM_CONST(0x0000000080000000)
+#define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000)
 
 #ifndef __ASSEMBLY__
 
@@ -65,7 +66,8 @@
 		FW_FEATURE_BULK_REMOVE | FW_FEATURE_XDABR |
 		FW_FEATURE_MULTITCE | FW_FEATURE_SPLPAR | FW_FEATURE_LPAR |
 		FW_FEATURE_CMO | FW_FEATURE_VPHN | FW_FEATURE_XCMO |
-		FW_FEATURE_SET_MODE | FW_FEATURE_BEST_ENERGY,
+		FW_FEATURE_SET_MODE | FW_FEATURE_BEST_ENERGY |
+		FW_FEATURE_TYPE1_AFFINITY,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
 	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | FW_FEATURE_OPALv2,
 	FW_FEATURE_POWERNV_ALWAYS = 0,
Index: powerpc/arch/powerpc/platforms/pseries/pseries.h
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/pseries.h	2013-04-16 21:25:15.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/pseries.h	2013-04-17 13:51:46.000000000 -0500
@@ -19,7 +19,10 @@
 
 #include <linux/of.h>
 
-extern void __init fw_feature_init(const char *hypertas, unsigned long len);
+extern void __init fw_hypertas_feature_init(const char *hypertas,
+					    unsigned long len);
+extern void __init fw_vec5_feature_init(const char *hypertas,
+					unsigned long len);
 
 struct pt_regs;
 

[PATCH v3 6/12] Update numa.c to use updated firmware_has_feature()

[Nathan Fontenot]

Update the numa code to use the updated firmware_has_feature() when checking
for type 1 affinity.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |   22 +++-------------------
 1 file changed, 3 insertions(+), 19 deletions(-)

Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-15 09:18:07.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-15 09:54:59.000000000 -0500
@@ -291,9 +291,7 @@
 static int __init find_min_common_depth(void)
 {
 	int depth;
-	struct device_node *chosen;
 	struct device_node *root;
-	const char *vec5;
 
 	if (firmware_has_feature(FW_FEATURE_OPAL))
 		root = of_find_node_by_path("/ibm,opal");
@@ -325,24 +323,10 @@
 
 	distance_ref_points_depth /= sizeof(int);
 
-#define VEC5_AFFINITY_BYTE	5
-#define VEC5_AFFINITY		0x80
-
-	if (firmware_has_feature(FW_FEATURE_OPAL))
+	if (firmware_has_feature(FW_FEATURE_OPAL) ||
+	    firmware_has_feature(FW_FEATURE_TYPE1_AFFINITY)) {
+		dbg("Using form 1 affinity\n");
 		form1_affinity = 1;
-	else {
-		chosen = of_find_node_by_path("/chosen");
-		if (chosen) {
-			vec5 = of_get_property(chosen,
-					       "ibm,architecture-vec-5", NULL);
-			if (vec5 && (vec5[VEC5_AFFINITY_BYTE] &
-							VEC5_AFFINITY)) {
-				dbg("Using form 1 affinity\n");
-				form1_affinity = 1;
-			}
-
-			of_node_put(chosen);
-		}
 	}
 
 	if (form1_affinity) {

[PATCH v3 7/12] Use stop machine to update cpu maps

[Nathan Fontenot]

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

Platform events such as partition migration or the new PRRN firmware
feature can cause the NUMA characteristics of a CPU to change, and these
changes will be reflected in the device tree nodes for the affected
CPUs.

This patch registers a handler for Open Firmware device tree updates
and reconfigures the CPU and node maps whenever the associativity
changes. Currently, this is accomplished by marking the affected CPUs in
the cpu_associativity_changes_mask and allowing
arch_update_cpu_topology() to retrieve the new associativity information
using hcall_vphn().

Protecting the NUMA cpu maps from concurrent access during an update
operation will be addressed in a subsequent patch in this series.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---

 arch/powerpc/include/asm/firmware.h       |    3 
 arch/powerpc/include/asm/prom.h           |    1 
 arch/powerpc/mm/numa.c                    |   99 ++++++++++++++++++++++--------
 arch/powerpc/platforms/pseries/firmware.c |    1 
 4 files changed, 79 insertions(+), 25 deletions(-)

Index: powerpc/arch/powerpc/include/asm/prom.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/prom.h	2013-04-15 14:03:52.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/prom.h	2013-04-15 14:04:47.000000000 -0500
@@ -128,6 +128,7 @@
 #define OV5_CMO			0x0480	/* Cooperative Memory Overcommitment */
 #define OV5_XCMO		0x0440	/* Page Coalescing */
 #define OV5_TYPE1_AFFINITY	0x0580	/* Type 1 NUMA affinity */
+#define OV5_PRRN		0x0540	/* Platform Resource Reassignment */
 #define OV5_PFO_HW_RNG		0x0E80	/* PFO Random Number Generator */
 #define OV5_PFO_HW_842		0x0E40	/* PFO Compression Accelerator */
 #define OV5_PFO_HW_ENCR		0x0E20	/* PFO Encryption Accelerator */
Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-15 14:04:46.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-15 14:06:20.000000000 -0500
@@ -1257,7 +1257,8 @@
 static u8 vphn_cpu_change_counts[NR_CPUS][MAX_DISTANCE_REF_POINTS];
 static cpumask_t cpu_associativity_changes_mask;
 static int vphn_enabled;
-static void set_topology_timer(void);
+static int prrn_enabled;
+static void reset_topology_timer(void);
 
 /*
  * Store the current values of the associativity change counters in the
@@ -1293,11 +1294,9 @@
  */
 static int update_cpu_associativity_changes_mask(void)
 {
-	int cpu, nr_cpus = 0;
+	int cpu;
 	cpumask_t *changes = &cpu_associativity_changes_mask;
 
-	cpumask_clear(changes);
-
 	for_each_possible_cpu(cpu) {
 		int i, changed = 0;
 		u8 *counts = vphn_cpu_change_counts[cpu];
@@ -1311,11 +1310,10 @@
 		}
 		if (changed) {
 			cpumask_set_cpu(cpu, changes);
-			nr_cpus++;
 		}
 	}
 
-	return nr_cpus;
+	return cpumask_weight(changes);
 }
 
 /*
@@ -1416,7 +1414,7 @@
 	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
 	struct device *dev;
 
-	for_each_cpu(cpu,&cpu_associativity_changes_mask) {
+	for_each_cpu(cpu, &cpu_associativity_changes_mask) {
 		vphn_get_associativity(cpu, associativity);
 		nid = associativity_to_nid(associativity);
 
@@ -1438,6 +1436,7 @@
 		dev = get_cpu_device(cpu);
 		if (dev)
 			kobject_uevent(&dev->kobj, KOBJ_CHANGE);
+		cpumask_clear_cpu(cpu, &cpu_associativity_changes_mask);
 		changed = 1;
 	}
 
@@ -1457,37 +1456,80 @@
 
 static void topology_timer_fn(unsigned long ignored)
 {
-	if (!vphn_enabled)
-		return;
-	if (update_cpu_associativity_changes_mask() > 0)
+	if (prrn_enabled && cpumask_weight(&cpu_associativity_changes_mask))
 		topology_schedule_update();
-	set_topology_timer();
+	else if (vphn_enabled) {
+		if (update_cpu_associativity_changes_mask() > 0)
+			topology_schedule_update();
+		reset_topology_timer();
+	}
 }
 static struct timer_list topology_timer =
 	TIMER_INITIALIZER(topology_timer_fn, 0, 0);
 
-static void set_topology_timer(void)
+static void reset_topology_timer(void)
 {
 	topology_timer.data = 0;
 	topology_timer.expires = jiffies + 60 * HZ;
-	add_timer(&topology_timer);
+	mod_timer(&topology_timer, topology_timer.expires);
+}
+
+static void stage_topology_update(int core_id)
+{
+	cpumask_or(&cpu_associativity_changes_mask,
+		&cpu_associativity_changes_mask, cpu_sibling_mask(core_id));
+	reset_topology_timer();
 }
 
+static int dt_update_callback(struct notifier_block *nb,
+				unsigned long action, void *data)
+{
+	struct of_prop_reconfig *update;
+	int rc = NOTIFY_DONE;
+
+	switch (action) {
+	case OF_RECONFIG_ADD_PROPERTY:
+	case OF_RECONFIG_UPDATE_PROPERTY:
+		update = (struct of_prop_reconfig *)data;
+		if (!of_prop_cmp(update->dn->type, "cpu")) {
+			u32 core_id;
+			of_property_read_u32(update->dn, "reg", &core_id);
+			stage_topology_update(core_id);
+			rc = NOTIFY_OK;
+		}
+		break;
+	}
+
+	return rc;
+}
+
+static struct notifier_block dt_update_nb = {
+	.notifier_call = dt_update_callback,
+};
+
 /*
- * Start polling for VPHN associativity changes.
+ * Start polling for associativity changes.
  */
 int start_topology_update(void)
 {
 	int rc = 0;
 
-	/* Disabled until races with load balancing are fixed */
-	if (0 && firmware_has_feature(FW_FEATURE_VPHN) &&
-	    get_lppaca()->shared_proc) {
-		vphn_enabled = 1;
-		setup_cpu_associativity_change_counters();
-		init_timer_deferrable(&topology_timer);
-		set_topology_timer();
-		rc = 1;
+	if (firmware_has_feature(FW_FEATURE_PRRN)) {
+		if (!prrn_enabled) {
+			prrn_enabled = 1;
+			vphn_enabled = 0;
+			rc = of_reconfig_notifier_register(&dt_update_nb);
+		}
+	} else if (0 && firmware_has_feature(FW_FEATURE_VPHN) &&
+		   get_lppaca()->shared_proc) {
+		/* Disabled until races with load balancing are fixed */
+		if (!vphn_enabled) {
+			prrn_enabled = 0;
+			vphn_enabled = 1;
+			setup_cpu_associativity_change_counters();
+			init_timer_deferrable(&topology_timer);
+			reset_topology_timer();
+		}
 	}
 
 	return rc;
@@ -1499,7 +1541,16 @@
  */
 int stop_topology_update(void)
 {
-	vphn_enabled = 0;
-	return del_timer_sync(&topology_timer);
+	int rc = 0;
+
+	if (prrn_enabled) {
+		prrn_enabled = 0;
+		rc = of_reconfig_notifier_unregister(&dt_update_nb);
+	} else if (vphn_enabled) {
+		vphn_enabled = 0;
+		rc = del_timer_sync(&topology_timer);
+	}
+
+	return rc;
 }
 #endif /* CONFIG_PPC_SPLPAR */
Index: powerpc/arch/powerpc/include/asm/firmware.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/firmware.h	2013-04-15 14:03:52.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/firmware.h	2013-04-15 14:04:47.000000000 -0500
@@ -52,6 +52,7 @@
 #define FW_FEATURE_SET_MODE	ASM_CONST(0x0000000040000000)
 #define FW_FEATURE_BEST_ENERGY	ASM_CONST(0x0000000080000000)
 #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000)
+#define FW_FEATURE_PRRN		ASM_CONST(0x0000000200000000)
 
 #ifndef __ASSEMBLY__
 
@@ -67,7 +68,7 @@
 		FW_FEATURE_MULTITCE | FW_FEATURE_SPLPAR | FW_FEATURE_LPAR |
 		FW_FEATURE_CMO | FW_FEATURE_VPHN | FW_FEATURE_XCMO |
 		FW_FEATURE_SET_MODE | FW_FEATURE_BEST_ENERGY |
-		FW_FEATURE_TYPE1_AFFINITY,
+		FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN,
 	FW_FEATURE_PSERIES_ALWAYS = 0,
 	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | FW_FEATURE_OPALv2,
 	FW_FEATURE_POWERNV_ALWAYS = 0,
Index: powerpc/arch/powerpc/platforms/pseries/firmware.c
===================================================================
--- powerpc.orig/arch/powerpc/platforms/pseries/firmware.c	2013-04-15 14:03:52.000000000 -0500
+++ powerpc/arch/powerpc/platforms/pseries/firmware.c	2013-04-15 14:04:47.000000000 -0500
@@ -113,6 +113,7 @@
 static __initdata struct vec5_fw_feature
 vec5_fw_features_table[FIRMWARE_MAX_FEATURES] = {
 	{FW_FEATURE_TYPE1_AFFINITY,	OV5_TYPE1_AFFINITY},
+	{FW_FEATURE_PRRN,		OV5_PRRN},
 };
 
 void __init fw_vec5_feature_init(const char *vec5, unsigned long len)

[PATCH v3 8/12] Use stop machine to update cpu maps

[Nathan Fontenot]

The new PRRN firmware feature allows CPU and memory resources to be
transparently reassigned across NUMA boundaries. When this happens, the
kernel must update the node maps to reflect the new affinity information.

Although the NUMA maps can be protected by locking primitives during the
update itself, this is insufficient to prevent concurrent accesses to these
structures. Since cpumask_of_node() hands out a pointer to these
structures, they can still be modified outside of the lock. Furthermore,
tracking down each usage of these pointers and adding locks would be quite
invasive and difficult to maintain.

The approach used is to make a list of affected cpus and call stop_machine
to have the update routine run on each of the affected cpus allowing them
to update themselves. Each cpu finds itself in the list of cpus and makes
the appropriate updates. We need to have each cpu do this for themselves to
handle calls to vdso_getcpu_init that is added in a subsequent patch.

Situations like these are best handled using stop_machine(). Since the NUMA
affinity updates are exceptionally rare events, this approach has the
benefit of not adding any overhead while accessing the NUMA maps during
normal operation.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |   82 ++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 64 insertions(+), 18 deletions(-)

Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-17 14:04:12.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-18 09:10:11.000000000 -0500
@@ -22,6 +22,7 @@
 #include <linux/pfn.h>
 #include <linux/cpuset.h>
 #include <linux/node.h>
+#include <linux/stop_machine.h>
 #include <asm/sparsemem.h>
 #include <asm/prom.h>
 #include <asm/smp.h>
@@ -1254,6 +1255,13 @@
 
 /* Virtual Processor Home Node (VPHN) support */
 #ifdef CONFIG_PPC_SPLPAR
+struct topology_update_data {
+	struct topology_update_data *next;
+	unsigned int cpu;
+	int old_nid;
+	int new_nid;
+};
+
 static u8 vphn_cpu_change_counts[NR_CPUS][MAX_DISTANCE_REF_POINTS];
 static cpumask_t cpu_associativity_changes_mask;
 static int vphn_enabled;
@@ -1405,41 +1413,79 @@
 }
 
 /*
+ * Update the CPU maps and sysfs entries for a single CPU when its NUMA
+ * characteristics change. This function doesn't perform any locking and is
+ * only safe to call from stop_machine().
+ */
+static int update_cpu_topology(void *data)
+{
+	struct topology_update_data *update;
+	unsigned long cpu;
+
+	if (!data)
+		return -EINVAL;
+
+	cpu = get_cpu();
+
+	for (update = data; update; update = update->next) {
+		if (cpu != update->cpu)
+			continue;
+
+		unregister_cpu_under_node(update->cpu, update->old_nid);
+		unmap_cpu_from_node(update->cpu);
+		map_cpu_to_node(update->cpu, update->new_nid);
+		register_cpu_under_node(update->cpu, update->new_nid);
+	}
+
+	return 0;
+}
+
+/*
  * Update the node maps and sysfs entries for each cpu whose home node
  * has changed. Returns 1 when the topology has changed, and 0 otherwise.
  */
 int arch_update_cpu_topology(void)
 {
-	int cpu, nid, old_nid, changed = 0;
+	unsigned int cpu, changed = 0;
+	struct topology_update_data *updates, *ud;
 	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
 	struct device *dev;
+	int weight, i = 0;
+
+	weight = cpumask_weight(&cpu_associativity_changes_mask);
+	if (!weight)
+		return 0;
+
+	updates = kzalloc(weight * (sizeof(*updates)), GFP_KERNEL);
+	if (!updates)
+		return 0;
 
 	for_each_cpu(cpu, &cpu_associativity_changes_mask) {
+		ud = &updates[i++];
+		ud->cpu = cpu;
 		vphn_get_associativity(cpu, associativity);
-		nid = associativity_to_nid(associativity);
+		ud->new_nid = associativity_to_nid(associativity);
 
-		if (nid < 0 || !node_online(nid))
-			nid = first_online_node;
+		if (ud->new_nid < 0 || !node_online(ud->new_nid))
+			ud->new_nid = first_online_node;
 
-		old_nid = numa_cpu_lookup_table[cpu];
+		ud->old_nid = numa_cpu_lookup_table[cpu];
 
-		/* Disable hotplug while we update the cpu
-		 * masks and sysfs.
-		 */
-		get_online_cpus();
-		unregister_cpu_under_node(cpu, old_nid);
-		unmap_cpu_from_node(cpu);
-		map_cpu_to_node(cpu, nid);
-		register_cpu_under_node(cpu, nid);
-		put_online_cpus();
+		if (i < weight)
+			ud->next = &updates[i];
+	}
+
+	stop_machine(update_cpu_topology, &updates[0], cpu_online_mask);
 
-		dev = get_cpu_device(cpu);
+	for (ud = &updates[0]; ud; ud = ud->next) {
+		dev = get_cpu_device(ud->cpu);
 		if (dev)
 			kobject_uevent(&dev->kobj, KOBJ_CHANGE);
-		cpumask_clear_cpu(cpu, &cpu_associativity_changes_mask);
+		cpumask_clear_cpu(ud->cpu, &cpu_associativity_changes_mask);
 		changed = 1;
 	}
 
+	kfree(updates);
 	return changed;
 }
 
@@ -1488,10 +1534,10 @@
 	int rc = NOTIFY_DONE;
 
 	switch (action) {
-	case OF_RECONFIG_ADD_PROPERTY:
 	case OF_RECONFIG_UPDATE_PROPERTY:
 		update = (struct of_prop_reconfig *)data;
-		if (!of_prop_cmp(update->dn->type, "cpu")) {
+		if (!of_prop_cmp(update->dn->type, "cpu") &&
+		    !of_prop_cmp(update->prop->name, "ibm,associativity")) {
 			u32 core_id;
 			of_property_read_u32(update->dn, "reg", &core_id);
 			stage_topology_update(core_id);

[PATCH v3 9/12] Update NUMA VDSO information

[Nathan Fontenot]

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

The following patch adds vdso_getcpu_init(), which stores the NUMA node for
a cpu in SPRG3:

Commit 18ad51dd34 ("powerpc: Add VDSO version of getcpu") adds
vdso_getcpu_init(), which stores the NUMA node for a cpu in SPRG3.

This patch ensures that this information is also updated when the NUMA
affinity of a cpu changes.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-18 09:10:11.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-22 09:39:02.000000000 -0500
@@ -30,6 +30,7 @@
 #include <asm/paca.h>
 #include <asm/hvcall.h>
 #include <asm/setup.h>
+#include <asm/vdso.h>
 
 static int numa_enabled = 1;
 
@@ -1434,6 +1435,7 @@
 		unregister_cpu_under_node(update->cpu, update->old_nid);
 		unmap_cpu_from_node(update->cpu);
 		map_cpu_to_node(update->cpu, update->new_nid);
+		vdso_getcpu_init();
 		register_cpu_under_node(update->cpu, update->new_nid);
 	}
 
@@ -1449,6 +1451,7 @@
 	unsigned int cpu, changed = 0;
 	struct topology_update_data *updates, *ud;
 	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
+	cpumask_t updated_cpus;
 	struct device *dev;
 	int weight, i = 0;
 
@@ -1460,6 +1463,8 @@
 	if (!updates)
 		return 0;
 
+	cpumask_clear(&updated_cpus);
+
 	for_each_cpu(cpu, &cpu_associativity_changes_mask) {
 		ud = &updates[i++];
 		ud->cpu = cpu;
@@ -1470,12 +1475,13 @@
 			ud->new_nid = first_online_node;
 
 		ud->old_nid = numa_cpu_lookup_table[cpu];
+		cpumask_set_cpu(cpu, &updated_cpus);
 
 		if (i < weight)
 			ud->next = &updates[i];
 	}
 
-	stop_machine(update_cpu_topology, &updates[0], cpu_online_mask);
+	stop_machine(update_cpu_topology, &updates[0], &updated_cpus);
 
 	for (ud = &updates[0]; ud; ud = ud->next) {
 		dev = get_cpu_device(ud->cpu);

[PATCH v3 10/12] Re-enable Virtual Private Home Node capabilities

[Nathan Fontenot]

From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>

The new PRRN firmware feature provides a more convenient and event-driven
interface than VPHN for notifying Linux of changes to the NUMA affinity of
platform resources. However, for practical reasons, it may not be feasible
for some customers to update to the latest firmware. For these customers,
the VPHN feature supported on previous firmware versions may still be the
best option.

The VPHN feature was previously disabled due to races with the load
balancing code when accessing the NUMA cpu maps, but the new stop_machine()
approach protects the NUMA cpu maps from these concurrent accesses. It
should be safe to re-enable this feature now.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/mm/numa.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-22 09:39:02.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-22 09:46:13.000000000 -0500
@@ -1572,9 +1572,8 @@
 			vphn_enabled = 0;
 			rc = of_reconfig_notifier_register(&dt_update_nb);
 		}
-	} else if (0 && firmware_has_feature(FW_FEATURE_VPHN) &&
+	} else if (firmware_has_feature(FW_FEATURE_VPHN) &&
 		   get_lppaca()->shared_proc) {
-		/* Disabled until races with load balancing are fixed */
 		if (!vphn_enabled) {
 			prrn_enabled = 0;
 			vphn_enabled = 1;

[PATCH v3 11/12] Enable PRRN Event handling

[Nathan Fontenot]

The Linux kernel and platform firmware negotiate their mutual support
of the PRRN option via the ibm,client-architecture-support interface.
This patch simply sets the appropriate fields in the client architecture
vector to indicate Linux support and will cause the firmware to begin
sending PRRN events via the RTAS event-scan mechanism.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/prom_init.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: powerpc/arch/powerpc/kernel/prom_init.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/prom_init.c	2013-04-18 09:09:22.000000000 -0500
+++ powerpc/arch/powerpc/kernel/prom_init.c	2013-04-22 09:49:28.000000000 -0500
@@ -698,7 +698,7 @@
 #else
 	0,
 #endif
-	OV5_FEAT(OV5_TYPE1_AFFINITY),
+	OV5_FEAT(OV5_TYPE1_AFFINITY) | OV5_FEAT(OV5_PRRN),
 	0,
 	0,
 	0,

[PATCH v3 12/12] Add /proc interface to control topology updates

[Nathan Fontenot]

There are instances in which we do not want topology updates to occur.
In order to allow this a /proc interface (/proc/powerpc/topology_updates)
is introduced so that topology updates can be enabled and disabled.

This patch also adds a prrn_is_enabled() call so that PRRN events are
handled in the kernel only if topology updating is enabled.

Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/topology.h |    5 ++
 arch/powerpc/kernel/rtasd.c         |    7 ++--
 arch/powerpc/mm/numa.c              |   62 +++++++++++++++++++++++++++++++++++-
 3 files changed, 71 insertions(+), 3 deletions(-)

Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
--- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-22 09:46:13.000000000 -0500
+++ powerpc/arch/powerpc/mm/numa.c	2013-04-22 09:51:10.000000000 -0500
@@ -23,6 +23,9 @@
 #include <linux/cpuset.h>
 #include <linux/node.h>
 #include <linux/stop_machine.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/uaccess.h>
 #include <asm/sparsemem.h>
 #include <asm/prom.h>
 #include <asm/smp.h>
@@ -1585,7 +1588,6 @@
 
 	return rc;
 }
-__initcall(start_topology_update);
 
 /*
  * Disable polling for VPHN associativity changes.
@@ -1604,4 +1606,62 @@
 
 	return rc;
 }
+
+inline int prrn_is_enabled(void)
+{
+	return prrn_enabled;
+}
+
+static int topology_read(struct seq_file *file, void *v)
+{
+	if (vphn_enabled || prrn_enabled)
+		seq_puts(file, "on\n");
+	else
+		seq_puts(file, "off\n");
+
+	return 0;
+}
+
+static int topology_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, topology_read, NULL);
+}
+
+static ssize_t topology_write(struct file *file, const char __user *buf,
+			      size_t count, loff_t *off)
+{
+	char kbuf[4]; /* "on" or "off" plus null. */
+	int read_len;
+
+	read_len = count < 3 ? count : 3;
+	if (copy_from_user(kbuf, buf, read_len))
+		return -EINVAL;
+
+	kbuf[read_len] = '\0';
+
+	if (!strncmp(kbuf, "on", 2))
+		start_topology_update();
+	else if (!strncmp(kbuf, "off", 3))
+		stop_topology_update();
+	else
+		return -EINVAL;
+
+	return count;
+}
+
+static const struct file_operations topology_ops = {
+	.read = seq_read,
+	.write = topology_write,
+	.open = topology_open,
+	.release = single_release
+};
+
+static int topology_update_init(void)
+{
+	start_topology_update();
+	proc_create("powerpc/topology_updates", 644, NULL, &topology_ops);
+
+	return 0;
+}
+device_initcall(topology_update_init);
 #endif /* CONFIG_PPC_SPLPAR */
Index: powerpc/arch/powerpc/include/asm/topology.h
===================================================================
--- powerpc.orig/arch/powerpc/include/asm/topology.h	2013-04-18 09:09:21.000000000 -0500
+++ powerpc/arch/powerpc/include/asm/topology.h	2013-04-22 09:51:10.000000000 -0500
@@ -71,6 +71,7 @@
 #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
 extern int start_topology_update(void);
 extern int stop_topology_update(void);
+extern inline int prrn_is_enabled(void);
 #else
 static inline int start_topology_update(void)
 {
@@ -80,6 +81,10 @@
 {
 	return 0;
 }
+static inline int prrn_is_enabled(void)
+{
+	return 0;
+}
 #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */
 
 #include <asm-generic/topology.h>
Index: powerpc/arch/powerpc/kernel/rtasd.c
===================================================================
--- powerpc.orig/arch/powerpc/kernel/rtasd.c	2013-04-18 09:09:21.000000000 -0500
+++ powerpc/arch/powerpc/kernel/rtasd.c	2013-04-22 09:51:10.000000000 -0500
@@ -29,6 +29,7 @@
 #include <asm/nvram.h>
 #include <linux/atomic.h>
 #include <asm/machdep.h>
+#include <asm/topology.h>
 
 
 static DEFINE_SPINLOCK(rtasd_log_lock);
@@ -294,11 +295,13 @@
 {
 	pSeries_log_error((char *)log, ERR_TYPE_RTAS_LOG, 0);
 
-	if (log->type == RTAS_TYPE_PRRN)
+	if (log->type == RTAS_TYPE_PRRN) {
 		/* For PRRN Events the extended log length is used to denote
 		 * the scope for calling rtas update-nodes.
 		 */
-		prrn_schedule_update(log->extended_log_length);
+		if (prrn_is_enabled())
+			prrn_schedule_update(log->extended_log_length);
+	}
 
 	return;
 }

Re: [PATCH v3 1/12] Create a powerpc update_devicetree interface

[Benjamin Herrenschmidt]

On Mon, 2013-04-22 at 13:30 -0500, Nathan Fontenot wrote:

> This patch exposes a method for updating the device tree via
> ppc_md.update_devicetree that takes a single 32-bit value as a parameter.
> For pseries platforms this is the existing pseries_devicetree_update routine
> which is updated to take the new parameter which is a scope value
> to indicate the the reason for making the rtas calls. This parameter is
> required by the ibm,update-nodes/ibm,update-properties RTAS calls, and
> the appropriate value is contained within the RTAS event for PRRN
> notifications. In pseries_devicetree_update() it was previously
> hard-coded to 1, the scope value for partition migration.

I think that's too much abstraction.... (see below)

Also you add this helper:

> Index: powerpc/arch/powerpc/kernel/rtas.c
> ===================================================================
> --- powerpc.orig/arch/powerpc/kernel/rtas.c	2013-03-08 19:23:06.000000000 -0600
> +++ powerpc/arch/powerpc/kernel/rtas.c	2013-04-17 13:02:29.000000000 -0500
> @@ -1085,3 +1085,13 @@
>  	timebase = 0;
>  	arch_spin_unlock(&timebase_lock);
>  }
> +
> +int update_devicetree(s32 scope)
> +{
> +	int rc = 0;
> +
> +	if (ppc_md.update_devicetree)
> +		rc = ppc_md.update_devicetree(scope);
> +
> +	return rc;
> +}

But then don't use it afaik (you call directly ppc_md.update_... from
prrn_work_fn().

In the end, the caller (PRRN stuff), while in rtasd, is really pseries
specific and the resulting update_device_tree() as well, so I don't
think we need the ppc_md. hook in the middle with that "oddball" scope
parameter which is not defined outside of pseries specific areas.

In this case, it might be better to make sure the PRRN related stuff in
rtasd is inside an ifdef CONFIG_PPC_PSERIES and have it call directly
into pseries_update_devicetree().

It makes the code somewhat easier to follow and I doubt anybody else
will ever use that specific hook, at least not in its current form. If
we need an abstraction later, we can add one then.

Cheers,
Ben.

Re: [PATCH v3 4/12] Move architecture vector definitions to prom.h

[Benjamin Herrenschmidt]

On Mon, 2013-04-22 at 13:35 -0500, Nathan Fontenot wrote:
> As part of handling handling PRRN events we will need to check the
> vector 5 portion of the architecture bits reported in the device tree
> to ensure that PRRN event handling is enabled. In order to do this
> firmware_has_feature is updated (in a subsequent patch) to
> make this check.  To avoid having to re-define bits in the architecture
> vector the bits are moved to prom.h.
> 
> This patch is the first step in updating firmware_has_feature
> by simply moving the bit definitions from prom_init.c to asm/prom.h.
> There are no functional changes.
> 
> Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
> 
> ---
>  arch/powerpc/include/asm/prom.h |   73 ++++++++++++++++++++++++++++++++++++++
>  arch/powerpc/kernel/prom_init.c |   75 +++-------------------------------------
>  2 files changed, 79 insertions(+), 69 deletions(-)
> 
> Index: powerpc/arch/powerpc/include/asm/prom.h
> ===================================================================
> --- powerpc.orig/arch/powerpc/include/asm/prom.h	2013-04-16 21:25:16.000000000 -0500
> +++ powerpc/arch/powerpc/include/asm/prom.h	2013-04-17 13:43:13.000000000 -0500
> @@ -74,6 +74,79 @@
>  #define DRCONF_MEM_AI_INVALID	0x00000040
>  #define DRCONF_MEM_RESERVED	0x00000080
>  
> +#if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV)
> +/

The ifdef is unnecessary

> + * There are two methods for telling firmware what our capabilities are.
> + * Newer machines have an "ibm,client-architecture-support" method on the
> + * root node.  For older machines, we have to call the "process-elf-header"
> + * method in the /packages/elf-loader node, passing it a fake 32-bit
> + * ELF header containing a couple of PT_NOTE sections that contain
> + * structures that contain various information.
> + */
> +
> +/* New method - extensible architecture description vector. */
> +
> +/* Option vector bits - generic bits in byte 1 */
> +#define OV_IGNORE		0x80	/* ignore this vector */
> +#define OV_CESSATION_POLICY	0x40	/* halt if unsupported option present*/
> +
> +/* Option vector 1: processor architectures supported */
> +#define OV1_PPC_2_00		0x80	/* set if we support PowerPC 2.00 */
> +#define OV1_PPC_2_01		0x40	/* set if we support PowerPC 2.01 */
> +#define OV1_PPC_2_02		0x20	/* set if we support PowerPC 2.02 */
> +#define OV1_PPC_2_03		0x10	/* set if we support PowerPC 2.03 */
> +#define OV1_PPC_2_04		0x08	/* set if we support PowerPC 2.04 */
> +#define OV1_PPC_2_05		0x04	/* set if we support PowerPC 2.05 */
> +#define OV1_PPC_2_06		0x02	/* set if we support PowerPC 2.06 */
> +#define OV1_PPC_2_07		0x01	/* set if we support PowerPC 2.07 */
> +
> +/* Option vector 2: Open Firmware options supported */
> +#define OV2_REAL_MODE		0x20	/* set if we want OF in real mode */
> +
> +/* Option vector 3: processor options supported */
> +#define OV3_FP			0x80	/* floating point */
> +#define OV3_VMX			0x40	/* VMX/Altivec */
> +#define OV3_DFP			0x20	/* decimal FP */
> +
> +/* Option vector 4: IBM PAPR implementation */
> +#define OV4_MIN_ENT_CAP		0x01	/* minimum VP entitled capacity */
> +
> +/* Option vector 5: PAPR/OF options supported */
> +#define OV5_LPAR		0x80	/* logical partitioning supported */
> +#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
> +/* ibm,dynamic-reconfiguration-memory property supported */
> +#define OV5_DRCONF_MEMORY	0x20
> +#define OV5_LARGE_PAGES		0x10	/* large pages supported */
> +#define OV5_DONATE_DEDICATE_CPU	0x02	/* donate dedicated CPU support */
> +/* PCIe/MSI support.  Without MSI full PCIe is not supported */
> +#ifdef CONFIG_PCI_MSI
> +#define OV5_MSI			0x01	/* PCIe/MSI support */
> +#else
> +#define OV5_MSI			0x00
> +#endif /* CONFIG_PCI_MSI */
> +#ifdef CONFIG_PPC_SMLPAR
> +#define OV5_CMO			0x80	/* Cooperative Memory Overcommitment */
> +#define OV5_XCMO		0x40	/* Page Coalescing */
> +#else
> +#define OV5_CMO			0x00
> +#define OV5_XCMO		0x00
> +#endif
> +#define OV5_TYPE1_AFFINITY	0x80	/* Type 1 NUMA affinity */
> +#define OV5_PFO_HW_RNG		0x80	/* PFO Random Number Generator */
> +#define OV5_PFO_HW_842		0x40	/* PFO Compression Accelerator */
> +#define OV5_PFO_HW_ENCR		0x20	/* PFO Encryption Accelerator */
> +#define OV5_SUB_PROCESSORS	0x01	/* 1,2,or 4 Sub-Processors supported */
> +
> +/* Option Vector 6: IBM PAPR hints */
> +#define OV6_LINUX		0x02	/* Linux is our OS */
> +
> +/*
> + * The architecture vector has an array of PVR mask/value pairs,
> + * followed by # option vectors - 1, followed by the option vectors.
> + */
> +extern unsigned char ibm_architecture_vec[];
> +#endif
> +
>  /* These includes are put at the bottom because they may contain things
>   * that are overridden by this file.  Ideally they shouldn't be included
>   * by this file, but there are a bunch of .c files that currently depend
> Index: powerpc/arch/powerpc/kernel/prom_init.c
> ===================================================================
> --- powerpc.orig/arch/powerpc/kernel/prom_init.c	2013-04-16 21:25:16.000000000 -0500
> +++ powerpc/arch/powerpc/kernel/prom_init.c	2013-04-17 13:43:13.000000000 -0500
> @@ -627,16 +627,11 @@
>  
>  #if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV)
>  /*
> - * There are two methods for telling firmware what our capabilities are.
> - * Newer machines have an "ibm,client-architecture-support" method on the
> - * root node.  For older machines, we have to call the "process-elf-header"
> - * method in the /packages/elf-loader node, passing it a fake 32-bit
> - * ELF header containing a couple of PT_NOTE sections that contain
> - * structures that contain various information.
> - */
> -
> -/*
> - * New method - extensible architecture description vector.
> + * The architecture vector has an array of PVR mask/value pairs,
> + * followed by # option vectors - 1, followed by the option vectors.
> + *
> + * See prom.h for the definition of the bits specified in the
> + * achitecture vector.
>   *
>   * Because the description vector contains a mix of byte and word
>   * values, we declare it as an unsigned char array, and use this
> @@ -645,65 +640,7 @@
>  #define W(x)	((x) >> 24) & 0xff, ((x) >> 16) & 0xff, \
>  		((x) >> 8) & 0xff, (x) & 0xff
>  
> -/* Option vector bits - generic bits in byte 1 */
> -#define OV_IGNORE		0x80	/* ignore this vector */
> -#define OV_CESSATION_POLICY	0x40	/* halt if unsupported option present*/
> -
> -/* Option vector 1: processor architectures supported */
> -#define OV1_PPC_2_00		0x80	/* set if we support PowerPC 2.00 */
> -#define OV1_PPC_2_01		0x40	/* set if we support PowerPC 2.01 */
> -#define OV1_PPC_2_02		0x20	/* set if we support PowerPC 2.02 */
> -#define OV1_PPC_2_03		0x10	/* set if we support PowerPC 2.03 */
> -#define OV1_PPC_2_04		0x08	/* set if we support PowerPC 2.04 */
> -#define OV1_PPC_2_05		0x04	/* set if we support PowerPC 2.05 */
> -#define OV1_PPC_2_06		0x02	/* set if we support PowerPC 2.06 */
> -#define OV1_PPC_2_07		0x01	/* set if we support PowerPC 2.07 */
> -
> -/* Option vector 2: Open Firmware options supported */
> -#define OV2_REAL_MODE		0x20	/* set if we want OF in real mode */
> -
> -/* Option vector 3: processor options supported */
> -#define OV3_FP			0x80	/* floating point */
> -#define OV3_VMX			0x40	/* VMX/Altivec */
> -#define OV3_DFP			0x20	/* decimal FP */
> -
> -/* Option vector 4: IBM PAPR implementation */
> -#define OV4_MIN_ENT_CAP		0x01	/* minimum VP entitled capacity */
> -
> -/* Option vector 5: PAPR/OF options supported */
> -#define OV5_LPAR		0x80	/* logical partitioning supported */
> -#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
> -/* ibm,dynamic-reconfiguration-memory property supported */
> -#define OV5_DRCONF_MEMORY	0x20
> -#define OV5_LARGE_PAGES		0x10	/* large pages supported */
> -#define OV5_DONATE_DEDICATE_CPU 0x02	/* donate dedicated CPU support */
> -/* PCIe/MSI support.  Without MSI full PCIe is not supported */
> -#ifdef CONFIG_PCI_MSI
> -#define OV5_MSI			0x01	/* PCIe/MSI support */
> -#else
> -#define OV5_MSI			0x00
> -#endif /* CONFIG_PCI_MSI */
> -#ifdef CONFIG_PPC_SMLPAR
> -#define OV5_CMO			0x80	/* Cooperative Memory Overcommitment */
> -#define OV5_XCMO			0x40	/* Page Coalescing */
> -#else
> -#define OV5_CMO			0x00
> -#define OV5_XCMO			0x00
> -#endif
> -#define OV5_TYPE1_AFFINITY	0x80	/* Type 1 NUMA affinity */
> -#define OV5_PFO_HW_RNG		0x80	/* PFO Random Number Generator */
> -#define OV5_PFO_HW_842		0x40	/* PFO Compression Accelerator */
> -#define OV5_PFO_HW_ENCR		0x20	/* PFO Encryption Accelerator */
> -#define OV5_SUB_PROCESSORS	0x01    /* 1,2,or 4 Sub-Processors supported */
> -
> -/* Option Vector 6: IBM PAPR hints */
> -#define OV6_LINUX		0x02	/* Linux is our OS */
> -
> -/*
> - * The architecture vector has an array of PVR mask/value pairs,
> - * followed by # option vectors - 1, followed by the option vectors.
> - */
> -static unsigned char ibm_architecture_vec[] = {
> +unsigned char ibm_architecture_vec[] = {
>  	W(0xfffe0000), W(0x003a0000),	/* POWER5/POWER5+ */
>  	W(0xffff0000), W(0x003e0000),	/* POWER6 */
>  	W(0xffff0000), W(0x003f0000),	/* POWER7 */
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [PATCH v3 7/12] Use stop machine to update cpu maps

[Benjamin Herrenschmidt]

On Mon, 2013-04-22 at 13:41 -0500, Nathan Fontenot wrote:
> From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
> 
> Platform events such as partition migration or the new PRRN firmware
> feature can cause the NUMA characteristics of a CPU to change, and these
> changes will be reflected in the device tree nodes for the affected
> CPUs.
> 
> This patch registers a handler for Open Firmware device tree updates
> and reconfigures the CPU and node maps whenever the associativity
> changes. Currently, this is accomplished by marking the affected CPUs in
> the cpu_associativity_changes_mask and allowing
> arch_update_cpu_topology() to retrieve the new associativity information
> using hcall_vphn().
> 
> Protecting the NUMA cpu maps from concurrent access during an update
> operation will be addressed in a subsequent patch in this series.

I see no more mention of stop_machine() ... is the patch subject stale ?

Cheers,
Ben.

> Signed-off-by: Nathan Fontenot <nfont@linux.vnet.ibm.com>
> ---
> 
>  arch/powerpc/include/asm/firmware.h       |    3 
>  arch/powerpc/include/asm/prom.h           |    1 
>  arch/powerpc/mm/numa.c                    |   99 ++++++++++++++++++++++--------
>  arch/powerpc/platforms/pseries/firmware.c |    1 
>  4 files changed, 79 insertions(+), 25 deletions(-)
> 
> Index: powerpc/arch/powerpc/include/asm/prom.h
> ===================================================================
> --- powerpc.orig/arch/powerpc/include/asm/prom.h	2013-04-15 14:03:52.000000000 -0500
> +++ powerpc/arch/powerpc/include/asm/prom.h	2013-04-15 14:04:47.000000000 -0500
> @@ -128,6 +128,7 @@
>  #define OV5_CMO			0x0480	/* Cooperative Memory Overcommitment */
>  #define OV5_XCMO		0x0440	/* Page Coalescing */
>  #define OV5_TYPE1_AFFINITY	0x0580	/* Type 1 NUMA affinity */
> +#define OV5_PRRN		0x0540	/* Platform Resource Reassignment */
>  #define OV5_PFO_HW_RNG		0x0E80	/* PFO Random Number Generator */
>  #define OV5_PFO_HW_842		0x0E40	/* PFO Compression Accelerator */
>  #define OV5_PFO_HW_ENCR		0x0E20	/* PFO Encryption Accelerator */
> Index: powerpc/arch/powerpc/mm/numa.c
> ===================================================================
> --- powerpc.orig/arch/powerpc/mm/numa.c	2013-04-15 14:04:46.000000000 -0500
> +++ powerpc/arch/powerpc/mm/numa.c	2013-04-15 14:06:20.000000000 -0500
> @@ -1257,7 +1257,8 @@
>  static u8 vphn_cpu_change_counts[NR_CPUS][MAX_DISTANCE_REF_POINTS];
>  static cpumask_t cpu_associativity_changes_mask;
>  static int vphn_enabled;
> -static void set_topology_timer(void);
> +static int prrn_enabled;
> +static void reset_topology_timer(void);
>  
>  /*
>   * Store the current values of the associativity change counters in the
> @@ -1293,11 +1294,9 @@
>   */
>  static int update_cpu_associativity_changes_mask(void)
>  {
> -	int cpu, nr_cpus = 0;
> +	int cpu;
>  	cpumask_t *changes = &cpu_associativity_changes_mask;
>  
> -	cpumask_clear(changes);
> -
>  	for_each_possible_cpu(cpu) {
>  		int i, changed = 0;
>  		u8 *counts = vphn_cpu_change_counts[cpu];
> @@ -1311,11 +1310,10 @@
>  		}
>  		if (changed) {
>  			cpumask_set_cpu(cpu, changes);
> -			nr_cpus++;
>  		}
>  	}
>  
> -	return nr_cpus;
> +	return cpumask_weight(changes);
>  }
>  
>  /*
> @@ -1416,7 +1414,7 @@
>  	unsigned int associativity[VPHN_ASSOC_BUFSIZE] = {0};
>  	struct device *dev;
>  
> -	for_each_cpu(cpu,&cpu_associativity_changes_mask) {
> +	for_each_cpu(cpu, &cpu_associativity_changes_mask) {
>  		vphn_get_associativity(cpu, associativity);
>  		nid = associativity_to_nid(associativity);
>  
> @@ -1438,6 +1436,7 @@
>  		dev = get_cpu_device(cpu);
>  		if (dev)
>  			kobject_uevent(&dev->kobj, KOBJ_CHANGE);
> +		cpumask_clear_cpu(cpu, &cpu_associativity_changes_mask);
>  		changed = 1;
>  	}
>  
> @@ -1457,37 +1456,80 @@
>  
>  static void topology_timer_fn(unsigned long ignored)
>  {
> -	if (!vphn_enabled)
> -		return;
> -	if (update_cpu_associativity_changes_mask() > 0)
> +	if (prrn_enabled && cpumask_weight(&cpu_associativity_changes_mask))
>  		topology_schedule_update();
> -	set_topology_timer();
> +	else if (vphn_enabled) {
> +		if (update_cpu_associativity_changes_mask() > 0)
> +			topology_schedule_update();
> +		reset_topology_timer();
> +	}
>  }
>  static struct timer_list topology_timer =
>  	TIMER_INITIALIZER(topology_timer_fn, 0, 0);
>  
> -static void set_topology_timer(void)
> +static void reset_topology_timer(void)
>  {
>  	topology_timer.data = 0;
>  	topology_timer.expires = jiffies + 60 * HZ;
> -	add_timer(&topology_timer);
> +	mod_timer(&topology_timer, topology_timer.expires);
> +}
> +
> +static void stage_topology_update(int core_id)
> +{
> +	cpumask_or(&cpu_associativity_changes_mask,
> +		&cpu_associativity_changes_mask, cpu_sibling_mask(core_id));
> +	reset_topology_timer();
>  }
>  
> +static int dt_update_callback(struct notifier_block *nb,
> +				unsigned long action, void *data)
> +{
> +	struct of_prop_reconfig *update;
> +	int rc = NOTIFY_DONE;
> +
> +	switch (action) {
> +	case OF_RECONFIG_ADD_PROPERTY:
> +	case OF_RECONFIG_UPDATE_PROPERTY:
> +		update = (struct of_prop_reconfig *)data;
> +		if (!of_prop_cmp(update->dn->type, "cpu")) {
> +			u32 core_id;
> +			of_property_read_u32(update->dn, "reg", &core_id);
> +			stage_topology_update(core_id);
> +			rc = NOTIFY_OK;
> +		}
> +		break;
> +	}
> +
> +	return rc;
> +}
> +
> +static struct notifier_block dt_update_nb = {
> +	.notifier_call = dt_update_callback,
> +};
> +
>  /*
> - * Start polling for VPHN associativity changes.
> + * Start polling for associativity changes.
>   */
>  int start_topology_update(void)
>  {
>  	int rc = 0;
>  
> -	/* Disabled until races with load balancing are fixed */
> -	if (0 && firmware_has_feature(FW_FEATURE_VPHN) &&
> -	    get_lppaca()->shared_proc) {
> -		vphn_enabled = 1;
> -		setup_cpu_associativity_change_counters();
> -		init_timer_deferrable(&topology_timer);
> -		set_topology_timer();
> -		rc = 1;
> +	if (firmware_has_feature(FW_FEATURE_PRRN)) {
> +		if (!prrn_enabled) {
> +			prrn_enabled = 1;
> +			vphn_enabled = 0;
> +			rc = of_reconfig_notifier_register(&dt_update_nb);
> +		}
> +	} else if (0 && firmware_has_feature(FW_FEATURE_VPHN) &&
> +		   get_lppaca()->shared_proc) {
> +		/* Disabled until races with load balancing are fixed */
> +		if (!vphn_enabled) {
> +			prrn_enabled = 0;
> +			vphn_enabled = 1;
> +			setup_cpu_associativity_change_counters();
> +			init_timer_deferrable(&topology_timer);
> +			reset_topology_timer();
> +		}
>  	}
>  
>  	return rc;
> @@ -1499,7 +1541,16 @@
>   */
>  int stop_topology_update(void)
>  {
> -	vphn_enabled = 0;
> -	return del_timer_sync(&topology_timer);
> +	int rc = 0;
> +
> +	if (prrn_enabled) {
> +		prrn_enabled = 0;
> +		rc = of_reconfig_notifier_unregister(&dt_update_nb);
> +	} else if (vphn_enabled) {
> +		vphn_enabled = 0;
> +		rc = del_timer_sync(&topology_timer);
> +	}
> +
> +	return rc;
>  }
>  #endif /* CONFIG_PPC_SPLPAR */
> Index: powerpc/arch/powerpc/include/asm/firmware.h
> ===================================================================
> --- powerpc.orig/arch/powerpc/include/asm/firmware.h	2013-04-15 14:03:52.000000000 -0500
> +++ powerpc/arch/powerpc/include/asm/firmware.h	2013-04-15 14:04:47.000000000 -0500
> @@ -52,6 +52,7 @@
>  #define FW_FEATURE_SET_MODE	ASM_CONST(0x0000000040000000)
>  #define FW_FEATURE_BEST_ENERGY	ASM_CONST(0x0000000080000000)
>  #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000)
> +#define FW_FEATURE_PRRN		ASM_CONST(0x0000000200000000)
>  
>  #ifndef __ASSEMBLY__
>  
> @@ -67,7 +68,7 @@
>  		FW_FEATURE_MULTITCE | FW_FEATURE_SPLPAR | FW_FEATURE_LPAR |
>  		FW_FEATURE_CMO | FW_FEATURE_VPHN | FW_FEATURE_XCMO |
>  		FW_FEATURE_SET_MODE | FW_FEATURE_BEST_ENERGY |
> -		FW_FEATURE_TYPE1_AFFINITY,
> +		FW_FEATURE_TYPE1_AFFINITY | FW_FEATURE_PRRN,
>  	FW_FEATURE_PSERIES_ALWAYS = 0,
>  	FW_FEATURE_POWERNV_POSSIBLE = FW_FEATURE_OPAL | FW_FEATURE_OPALv2,
>  	FW_FEATURE_POWERNV_ALWAYS = 0,
> Index: powerpc/arch/powerpc/platforms/pseries/firmware.c
> ===================================================================
> --- powerpc.orig/arch/powerpc/platforms/pseries/firmware.c	2013-04-15 14:03:52.000000000 -0500
> +++ powerpc/arch/powerpc/platforms/pseries/firmware.c	2013-04-15 14:04:47.000000000 -0500
> @@ -113,6 +113,7 @@
>  static __initdata struct vec5_fw_feature
>  vec5_fw_features_table[FIRMWARE_MAX_FEATURES] = {
>  	{FW_FEATURE_TYPE1_AFFINITY,	OV5_TYPE1_AFFINITY},
> +	{FW_FEATURE_PRRN,		OV5_PRRN},
>  };
>  
>  void __init fw_vec5_feature_init(const char *vec5, unsigned long len)
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [PATCH v3 5/12] Update firmware_has_feature() to check architecture bits

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

Hi Nathan,

On Mon, 22 Apr 2013 13:38:47 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>
> -/* Option vector 5: PAPR/OF options supported */
> -#define OV5_LPAR		0x80	/* logical partitioning supported */
> -#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
> +/* Option vector 5: PAPR/OF options supported
> + * Thses bits are also used for the platform_has_feature() call so
      ^^^^^
typo

> + * we encode the vector index in the define and use the OV5_FEAT()
> + * and OV5_INDX() macros to extract the desired information.
> + */
> +#define OV5_FEAT(x)	((x) & 0xff)
> +#define OV5_INDX(x)	((x) >> 8)
> +#define OV5_LPAR		0x0280	/* logical partitioning supported */
> +#define OV5_SPLPAR		0x0240	/* shared-processor LPAR supported */

Wouldn't it be clearer to say

#define OV5_LPAR	(OV5_INDX(0x2) | OV5_FEAT(0x80))

etc?

> @@ -145,6 +141,7 @@
>   * followed by # option vectors - 1, followed by the option vectors.
>   */
>  extern unsigned char ibm_architecture_vec[];
> +bool platform_has_feature(unsigned int);

"extern", please (if nothing else, for consistency).

> +static __initdata struct vec5_fw_feature
> +vec5_fw_features_table[FIRMWARE_MAX_FEATURES] = {

Why make this array FIRMWARE_MAX_FEATURES (63) long?  You could just
restrict the for loop below to ARRAY_SIZE(vec5_fw_features_table).

> +	{FW_FEATURE_TYPE1_AFFINITY,	OV5_TYPE1_AFFINITY},
> +};
> +
> +void __init fw_vec5_feature_init(const char *vec5, unsigned long len)
> +{
> +	unsigned int index, feat;
> +	int i;
> +
> +	pr_debug(" -> fw_vec5_feature_init()\n");
> +
> +	for (i = 0; i < FIRMWARE_MAX_FEATURES; i++) {
> +		if (!vec5_fw_features_table[i].feature)
> +			continue;

And this test could go away.

I realise that you have just copied the existing code, but you should not
do that blindly.  Maybe you could even add an (earlier) patch that fixes
the existing code.
-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH v3 5/12] Update firmware_has_feature() to check architecture bits

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

Hi Nathan,

On Mon, 22 Apr 2013 13:38:47 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>
> +/* Option vector 5: PAPR/OF options supported
> + * Thses bits are also used for the platform_has_feature() call so

You talk about platform_has_feature(), but that does not exist (I assume
it existed in a previous version of the patch set).

> +bool platform_has_feature(unsigned int);

Ditto.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH v3 12/12] Add /proc interface to control topology updates

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Hi Nathan,

On Mon, 22 Apr 2013 13:47:55 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>
>  #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
>  extern int start_topology_update(void);
>  extern int stop_topology_update(void);
> +extern inline int prrn_is_enabled(void);

You really can't do "extern inline" with no body ...

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH v3 12/12] Add /proc interface to control topology updates

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]

Hi Nathan,

On Mon, 22 Apr 2013 13:47:55 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>
> +inline int prrn_is_enabled(void)
> +{
> +	return prrn_enabled;
> +}

We generally leave these "inline"s up to the compiler these days i.e.
remove the "inline".

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH v3 12/12] Add /proc interface to control topology updates

[Michael Ellerman]

On Tue, Apr 23, 2013 at 12:00:26PM +1000, Stephen Rothwell wrote:
> Hi Nathan,
> 
> On Mon, 22 Apr 2013 13:47:55 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
> >
> >  #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
> >  extern int start_topology_update(void);
> >  extern int stop_topology_update(void);
> > +extern inline int prrn_is_enabled(void);
> 
> You really can't do "extern inline" with no body ...

No you can't, and at least with my compiler it causes a build error.

cheers

Re: [PATCH v3 1/12] Create a powerpc update_devicetree interface

[Nathan Fontenot]

On 04/22/2013 07:15 PM, Benjamin Herrenschmidt wrote:
> On Mon, 2013-04-22 at 13:30 -0500, Nathan Fontenot wrote:
> 
>> This patch exposes a method for updating the device tree via
>> ppc_md.update_devicetree that takes a single 32-bit value as a parameter.
>> For pseries platforms this is the existing pseries_devicetree_update routine
>> which is updated to take the new parameter which is a scope value
>> to indicate the the reason for making the rtas calls. This parameter is
>> required by the ibm,update-nodes/ibm,update-properties RTAS calls, and
>> the appropriate value is contained within the RTAS event for PRRN
>> notifications. In pseries_devicetree_update() it was previously
>> hard-coded to 1, the scope value for partition migration.
> 
> I think that's too much abstraction.... (see below)
> 
> Also you add this helper:
> 
>> Index: powerpc/arch/powerpc/kernel/rtas.c
>> ===================================================================
>> --- powerpc.orig/arch/powerpc/kernel/rtas.c	2013-03-08 19:23:06.000000000 -0600
>> +++ powerpc/arch/powerpc/kernel/rtas.c	2013-04-17 13:02:29.000000000 -0500
>> @@ -1085,3 +1085,13 @@
>>  	timebase = 0;
>>  	arch_spin_unlock(&timebase_lock);
>>  }
>> +
>> +int update_devicetree(s32 scope)
>> +{
>> +	int rc = 0;
>> +
>> +	if (ppc_md.update_devicetree)
>> +		rc = ppc_md.update_devicetree(scope);
>> +
>> +	return rc;
>> +}
> 
> But then don't use it afaik (you call directly ppc_md.update_... from
> prrn_work_fn().
> 
> In the end, the caller (PRRN stuff), while in rtasd, is really pseries
> specific and the resulting update_device_tree() as well, so I don't
> think we need the ppc_md. hook in the middle with that "oddball" scope
> parameter which is not defined outside of pseries specific areas.
> 
> In this case, it might be better to make sure the PRRN related stuff in
> rtasd is inside an ifdef CONFIG_PPC_PSERIES and have it call directly
> into pseries_update_devicetree().
> 
> It makes the code somewhat easier to follow and I doubt anybody else
> will ever use that specific hook, at least not in its current form. If
> we need an abstraction later, we can add one then.
> 

ok, good. I was not crazy about using ppc_md to do this and if you're fine
with putting the pseries specific stuff in ifdef CONFIG_PPC_PSERIES I'll
update the code to do that.

Question concerning rtas code. There is quite a bit of pseries specific 
pieces in the general powerpc/kernel directory. Has there been, or should
there be, any effort to move that to the pseries directory?

-Nathan

Re: [PATCH v3 5/12] Update firmware_has_feature() to check architecture bits

[Nathan Fontenot]

On 04/22/2013 08:50 PM, Stephen Rothwell wrote:
> Hi Nathan,
> 
> On Mon, 22 Apr 2013 13:38:47 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>>
>> -/* Option vector 5: PAPR/OF options supported */
>> -#define OV5_LPAR		0x80	/* logical partitioning supported */
>> -#define OV5_SPLPAR		0x40	/* shared-processor LPAR supported */
>> +/* Option vector 5: PAPR/OF options supported
>> + * Thses bits are also used for the platform_has_feature() call so
>       ^^^^^
> typo

will fix.

> 
>> + * we encode the vector index in the define and use the OV5_FEAT()
>> + * and OV5_INDX() macros to extract the desired information.
>> + */
>> +#define OV5_FEAT(x)	((x) & 0xff)
>> +#define OV5_INDX(x)	((x) >> 8)
>> +#define OV5_LPAR		0x0280	/* logical partitioning supported */
>> +#define OV5_SPLPAR		0x0240	/* shared-processor LPAR supported */
> 
> Wouldn't it be clearer to say
> 
> #define OV5_LPAR	(OV5_INDX(0x2) | OV5_FEAT(0x80))

The defines won't work the way you used them, they were designed to take the
combined value, i.e. 0x0280, and parse out the index and the feature.

I do think having macros to create the actual values as your example does is easier
to read. We could do something like...

#define OV5_FEAT(x)	((x) & 0xff)
#define OV5_SETINDX(x)	((x) << 8)
#define OV5_GETINDX(x)	((x) >> 8)

#define OV5_LPAR	(OV5_SETINDX(0x2) | OV5_FEAT(0x80))

Thoughts?

> 
> etc?
> 
>> @@ -145,6 +141,7 @@
>>   * followed by # option vectors - 1, followed by the option vectors.
>>   */
>>  extern unsigned char ibm_architecture_vec[];
>> +bool platform_has_feature(unsigned int);
> 
> "extern", please (if nothing else, for consistency).
> 

That shouldn't really be there, its an artifact from a previous patch. I'll remove it.

>> +static __initdata struct vec5_fw_feature
>> +vec5_fw_features_table[FIRMWARE_MAX_FEATURES] = {
> 
> Why make this array FIRMWARE_MAX_FEATURES (63) long?  You could just
> restrict the for loop below to ARRAY_SIZE(vec5_fw_features_table).
> 
>> +	{FW_FEATURE_TYPE1_AFFINITY,	OV5_TYPE1_AFFINITY},
>> +};
>> +
>> +void __init fw_vec5_feature_init(const char *vec5, unsigned long len)
>> +{
>> +	unsigned int index, feat;
>> +	int i;
>> +
>> +	pr_debug(" -> fw_vec5_feature_init()\n");
>> +
>> +	for (i = 0; i < FIRMWARE_MAX_FEATURES; i++) {
>> +		if (!vec5_fw_features_table[i].feature)
>> +			continue;
> 
> And this test could go away.
> 
> I realise that you have just copied the existing code, but you should not
> do that blindly.  Maybe you could even add an (earlier) patch that fixes
> the existing code.

I think that could be done easily enough.

Thanks for looking,
-Nathan

Re: [PATCH v3 7/12] Use stop machine to update cpu maps

[Nathan Fontenot]

On 04/22/2013 07:24 PM, Benjamin Herrenschmidt wrote:
> On Mon, 2013-04-22 at 13:41 -0500, Nathan Fontenot wrote:
>> From: Jesse Larrew <jlarrew@linux.vnet.ibm.com>
>>
>> Platform events such as partition migration or the new PRRN firmware
>> feature can cause the NUMA characteristics of a CPU to change, and these
>> changes will be reflected in the device tree nodes for the affected
>> CPUs.
>>
>> This patch registers a handler for Open Firmware device tree updates
>> and reconfigures the CPU and node maps whenever the associativity
>> changes. Currently, this is accomplished by marking the affected CPUs in
>> the cpu_associativity_changes_mask and allowing
>> arch_update_cpu_topology() to retrieve the new associativity information
>> using hcall_vphn().
>>
>> Protecting the NUMA cpu maps from concurrent access during an update
>> operation will be addressed in a subsequent patch in this series.
> 
> I see no more mention of stop_machine() ... is the patch subject stale ?
> 

Nope, just me mistakenly putting the wrong subject for this patch. I'll
correct it in the next version.

-Nathan

Re: [PATCH v3 12/12] Add /proc interface to control topology updates

[Nathan Fontenot]

On 04/22/2013 09:49 PM, Michael Ellerman wrote:
> On Tue, Apr 23, 2013 at 12:00:26PM +1000, Stephen Rothwell wrote:
>> Hi Nathan,
>>
>> On Mon, 22 Apr 2013 13:47:55 -0500 Nathan Fontenot <nfont@linux.vnet.ibm.com> wrote:
>>>
>>>  #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR)
>>>  extern int start_topology_update(void);
>>>  extern int stop_topology_update(void);
>>> +extern inline int prrn_is_enabled(void);
>>
>> You really can't do "extern inline" with no body ...
> 
> No you can't, and at least with my compiler it causes a build error.
> 

Easy enough, no more inline for this.

This for looking,
-Nathan

Re: [PATCH v3 1/12] Create a powerpc update_devicetree interface

[Benjamin Herrenschmidt]

On Tue, 2013-04-23 at 13:46 -0500, Nathan Fontenot wrote:
> ok, good. I was not crazy about using ppc_md to do this and if you're fine
> with putting the pseries specific stuff in ifdef CONFIG_PPC_PSERIES I'll
> update the code to do that.
> 
> Question concerning rtas code. There is quite a bit of pseries specific 
> pieces in the general powerpc/kernel directory. Has there been, or should
> there be, any effort to move that to the pseries directory?

It's a good question ... it's shared in part with CHRP, it might make
sense to split it but only if the split can be done cleanly, which I
haven't looked into.

Cheers,
Ben.