[PATCH 0/3] CPU hotplug support for Versatile platforms

[PATCH 0/3] CPU hotplug support for Versatile platforms

[Will Deacon]

This patch series fixes some issues with the RealView CPU hotplug code and then
factors out the common code into plat-versatile. Finally, hotplug support is
added to mach-vexpress using the new platform code.

Taken against 2.6.35 and tested on RealView PB11MPCore and Versatile Express
[ct-ca9x4] boards.

Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>

Will Deacon (3):
  ARM: realview: fix CPU hotplug support for SMP platforms
  ARM: plat-versatile: factor out common hotplug code
  ARM: vexpress: add support for CPU hotplug to ct-ca9x4 tile

 arch/arm/mach-realview/hotplug.c               |   89 +++++++-----------------
 arch/arm/mach-vexpress/Makefile                |    1 +
 arch/arm/mach-vexpress/ct-ca9x4.c              |   43 +++++++++++
 arch/arm/mach-vexpress/hotplug.c               |   58 +++++++++++++++
 arch/arm/plat-versatile/Makefile               |    1 +
 arch/arm/plat-versatile/hotplug.c              |   57 +++++++++++++++
 arch/arm/plat-versatile/include/plat/hotplug.h |    6 ++
 7 files changed, 192 insertions(+), 63 deletions(-)
 create mode 100644 arch/arm/mach-vexpress/hotplug.c
 create mode 100644 arch/arm/plat-versatile/hotplug.c
 create mode 100644 arch/arm/plat-versatile/include/plat/hotplug.h

[PATCH 1/3] ARM: realview: fix CPU hotplug support for SMP platforms

[Will Deacon]

The current CPU hotplug functions for RealView boards suffer from a number
of problems:

- The location of the SMP/AMP bit in the Auxiliary Control Register is
  correct only for 11MPCore.

- The I-cache is not flushed when a core leaves lowpower mode

- The assembly routines for entering/leaving the lowpower state can
  be made more readable by using macros for operations such as dsb().

This patch fixes these problems for the RealView boards and has been
tested successfully on the PB11MPCore board.

Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/mach-realview/hotplug.c |   39 +++++++++++++++++++++++--------------
 1 files changed, 24 insertions(+), 15 deletions(-)

diff --git a/arch/arm/mach-realview/hotplug.c b/arch/arm/mach-realview/hotplug.c
index f95521a..fdb79dd 100644
--- a/arch/arm/mach-realview/hotplug.c
+++ b/arch/arm/mach-realview/hotplug.c
@@ -8,11 +8,15 @@
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  */
+#include <linux/init.h>
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <linux/smp.h>
 #include <linux/completion.h>
 
+#include <mach/board-eb.h>
+#include <mach/board-pbx.h>
+
 #include <asm/cacheflush.h>
 
 extern volatile int pen_release;
@@ -21,42 +25,50 @@ static DECLARE_COMPLETION(cpu_killed);
 
 static inline void cpu_enter_lowpower(void)
 {
-	unsigned int v;
+	unsigned int v, smp_ctrl;
+
+	smp_ctrl = (core_tile_pbxa9mp() || core_tile_a9mp()) ? 0x40 : 0x20;
 
 	flush_cache_all();
+	dsb();
 	asm volatile(
-	"	mcr	p15, 0, %1, c7, c5, 0\n"
-	"	mcr	p15, 0, %1, c7, c10, 4\n"
 	/*
 	 * Turn off coherency
 	 */
 	"	mrc	p15, 0, %0, c1, c0, 1\n"
-	"	bic	%0, %0, #0x20\n"
+	"	bic	%0, %0, %1\n"
 	"	mcr	p15, 0, %0, c1, c0, 1\n"
+	/* DSB */
+	"       mcr     p15, 0, %2, c7, c10, 4\n"
+	/* Disable D-cache */
 	"	mrc	p15, 0, %0, c1, c0, 0\n"
 	"	bic	%0, %0, #0x04\n"
 	"	mcr	p15, 0, %0, c1, c0, 0\n"
 	  : "=&r" (v)
-	  : "r" (0)
-	  : "cc");
+	  : "r" (smp_ctrl), "r" (0)
+	  : "memory");
 }
 
 static inline void cpu_leave_lowpower(void)
 {
-	unsigned int v;
+	unsigned int v, smp_ctrl;
 
+	smp_ctrl = (core_tile_pbxa9mp() || core_tile_a9mp()) ? 0x40 : 0x20;
+
+	flush_cache_all();
+	dsb();
 	asm volatile(	"mrc	p15, 0, %0, c1, c0, 0\n"
 	"	orr	%0, %0, #0x04\n"
 	"	mcr	p15, 0, %0, c1, c0, 0\n"
 	"	mrc	p15, 0, %0, c1, c0, 1\n"
-	"	orr	%0, %0, #0x20\n"
+	"	orr	%0, %0, %1\n"
 	"	mcr	p15, 0, %0, c1, c0, 1\n"
 	  : "=&r" (v)
-	  :
-	  : "cc");
+	  : "r" (smp_ctrl)
+	  : "memory");
 }
 
-static inline void platform_do_lowpower(unsigned int cpu)
+static void __ref platform_do_lowpower(unsigned int cpu)
 {
 	/*
 	 * there is no power-control hardware on this platform, so all
@@ -67,10 +79,7 @@ static inline void platform_do_lowpower(unsigned int cpu)
 		/*
 		 * here's the WFI
 		 */
-		asm(".word	0xe320f003\n"
-		    :
-		    :
-		    : "memory", "cc");
+		asm volatile("wfi" : : : "memory");
 
 		if (pen_release == cpu) {
 			/*
-- 
1.6.3.3

[PATCH 2/3] ARM: plat-versatile: factor out common hotplug code

[Will Deacon]

Common hotplug routines for the Versatile Platforms can be factored
out under plat-versatile leaving only the platform_do_lowpower function
to be implemented by the BSP code.

This patch factors out the hotplug code and changes the RealView hotplug
implementation to use the new framework.

Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/mach-realview/hotplug.c               |   52 +--------------------
 arch/arm/plat-versatile/Makefile               |    1 +
 arch/arm/plat-versatile/hotplug.c              |   57 ++++++++++++++++++++++++
 arch/arm/plat-versatile/include/plat/hotplug.h |    6 +++
 4 files changed, 67 insertions(+), 49 deletions(-)
 create mode 100644 arch/arm/plat-versatile/hotplug.c
 create mode 100644 arch/arm/plat-versatile/include/plat/hotplug.h

diff --git a/arch/arm/mach-realview/hotplug.c b/arch/arm/mach-realview/hotplug.c
index fdb79dd..f80b933 100644
--- a/arch/arm/mach-realview/hotplug.c
+++ b/arch/arm/mach-realview/hotplug.c
@@ -16,13 +16,12 @@
 
 #include <mach/board-eb.h>
 #include <mach/board-pbx.h>
+#include <plat/hotplug.h>
 
 #include <asm/cacheflush.h>
 
 extern volatile int pen_release;
 
-static DECLARE_COMPLETION(cpu_killed);
-
 static inline void cpu_enter_lowpower(void)
 {
 	unsigned int v, smp_ctrl;
@@ -68,8 +67,9 @@ static inline void cpu_leave_lowpower(void)
 	  : "memory");
 }
 
-static void __ref platform_do_lowpower(unsigned int cpu)
+void __ref platform_do_lowpower(unsigned int cpu)
 {
+	cpu_enter_lowpower();
 	/*
 	 * there is no power-control hardware on this platform, so all
 	 * we can do is put the core into WFI; this is safe as the calling
@@ -100,51 +100,5 @@ static void __ref platform_do_lowpower(unsigned int cpu)
 		printk("CPU%u: spurious wakeup call\n", cpu);
 #endif
 	}
-}
-
-int platform_cpu_kill(unsigned int cpu)
-{
-	return wait_for_completion_timeout(&cpu_killed, 5000);
-}
-
-/*
- * platform-specific code to shutdown a CPU
- *
- * Called with IRQs disabled
- */
-void platform_cpu_die(unsigned int cpu)
-{
-#ifdef DEBUG
-	unsigned int this_cpu = hard_smp_processor_id();
-
-	if (cpu != this_cpu) {
-		printk(KERN_CRIT "Eek! platform_cpu_die running on %u, should be %u\n",
-			   this_cpu, cpu);
-		BUG();
-	}
-#endif
-
-	printk(KERN_NOTICE "CPU%u: shutdown\n", cpu);
-	complete(&cpu_killed);
-
-	/*
-	 * we're ready for shutdown now, so do it
-	 */
-	cpu_enter_lowpower();
-	platform_do_lowpower(cpu);
-
-	/*
-	 * bring this CPU back into the world of cache
-	 * coherency, and then restore interrupts
-	 */
 	cpu_leave_lowpower();
 }
-
-int platform_cpu_disable(unsigned int cpu)
-{
-	/*
-	 * we don't allow CPU 0 to be shutdown (it is still too special
-	 * e.g. clock tick interrupts)
-	 */
-	return cpu == 0 ? -EPERM : 0;
-}
diff --git a/arch/arm/plat-versatile/Makefile b/arch/arm/plat-versatile/Makefile
index 9b1a668..e4f043b 100644
--- a/arch/arm/plat-versatile/Makefile
+++ b/arch/arm/plat-versatile/Makefile
@@ -2,3 +2,4 @@ obj-y	:= clock.o
 obj-$(CONFIG_ARM_TIMER_SP804) += timer-sp.o
 obj-$(CONFIG_ARCH_REALVIEW) += sched-clock.o
 obj-$(CONFIG_ARCH_VERSATILE) += sched-clock.o
+obj-$(CONFIG_HOTPLUG_CPU) += hotplug.o
diff --git a/arch/arm/plat-versatile/hotplug.c b/arch/arm/plat-versatile/hotplug.c
new file mode 100644
index 0000000..15ed6bd
--- /dev/null
+++ b/arch/arm/plat-versatile/hotplug.c
@@ -0,0 +1,57 @@
+/*
+ *  linux/arch/arm/plat-versatile/hotplug.c
+ *
+ *  Copyright (C) 2010 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/smp.h>
+#include <linux/completion.h>
+
+#include <plat/hotplug.h>
+
+static DECLARE_COMPLETION(cpu_killed);
+
+int platform_cpu_kill(unsigned int cpu)
+{
+	return wait_for_completion_timeout(&cpu_killed, 5000);
+}
+
+/*
+ * platform-specific code to shutdown a CPU
+ *
+ * Called with IRQs disabled
+ */
+void platform_cpu_die(unsigned int cpu)
+{
+#ifdef DEBUG
+	unsigned int this_cpu = hard_smp_processor_id();
+
+	if (cpu != this_cpu) {
+		printk(KERN_CRIT "Eek! platform_cpu_die running on %u, should be %u\n",
+			   this_cpu, cpu);
+		BUG();
+	}
+#endif
+
+	printk(KERN_NOTICE "CPU%u: shutdown\n", cpu);
+	complete(&cpu_killed);
+
+	/*
+	 * Call the CPU-specific low-power routine.
+	 */
+	platform_do_lowpower(cpu);
+}
+
+int platform_cpu_disable(unsigned int cpu)
+{
+	/*
+	 * we don't allow CPU 0 to be shutdown (it is still too special
+	 * e.g. clock tick interrupts)
+	 */
+	return cpu == 0 ? -EPERM : 0;
+}
diff --git a/arch/arm/plat-versatile/include/plat/hotplug.h b/arch/arm/plat-versatile/include/plat/hotplug.h
new file mode 100644
index 0000000..f36df1c
--- /dev/null
+++ b/arch/arm/plat-versatile/include/plat/hotplug.h
@@ -0,0 +1,6 @@
+#ifndef PLAT_HOTPLUG_H
+#define PLAT_HOTPLUG_H
+
+void platform_do_lowpower(unsigned int cpu);
+
+#endif
-- 
1.6.3.3

[PATCH 3/3] ARM: vexpress: add support for CPU hotplug to ct-ca9x4 tile

[Will Deacon]

The Versatile Express platform can support a quad-core Cortex-A9 tile running
SMP Linux.

This patch adds support for CPU hotplug when running in this configuration.

Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/mach-vexpress/Makefile   |    1 +
 arch/arm/mach-vexpress/ct-ca9x4.c |   43 +++++++++++++++++++++++++++
 arch/arm/mach-vexpress/hotplug.c  |   58 +++++++++++++++++++++++++++++++++++++
 3 files changed, 102 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm/mach-vexpress/hotplug.c

diff --git a/arch/arm/mach-vexpress/Makefile b/arch/arm/mach-vexpress/Makefile
index 1b71b77..15a34f0 100644
--- a/arch/arm/mach-vexpress/Makefile
+++ b/arch/arm/mach-vexpress/Makefile
@@ -6,3 +6,4 @@ obj-y					:= v2m.o
 obj-$(CONFIG_ARCH_VEXPRESS_CA9X4)	+= ct-ca9x4.o
 obj-$(CONFIG_SMP)			+= platsmp.o headsmp.o
 obj-$(CONFIG_LOCAL_TIMERS)		+= localtimer.o
+obj-$(CONFIG_HOTPLUG_CPU)		+= hotplug.o
diff --git a/arch/arm/mach-vexpress/ct-ca9x4.c b/arch/arm/mach-vexpress/ct-ca9x4.c
index 6353459..ff350e6 100644
--- a/arch/arm/mach-vexpress/ct-ca9x4.c
+++ b/arch/arm/mach-vexpress/ct-ca9x4.c
@@ -9,6 +9,7 @@
 #include <linux/amba/bus.h>
 #include <linux/amba/clcd.h>
 
+#include <asm/cacheflush.h>
 #include <asm/clkdev.h>
 #include <asm/pgtable.h>
 #include <asm/hardware/arm_timer.h>
@@ -220,6 +221,48 @@ static struct platform_device pmu_device = {
 	.resource	= pmu_resources,
 };
 
+#ifdef CONFIG_HOTPLUG_CPU
+void cpu_enter_lowpower(void)
+{
+	unsigned int v;
+
+	flush_cache_all();
+	dsb();
+	asm volatile(
+	/*
+	 * Turn off coherency
+	 */
+	"	mrc	p15, 0, %0, c1, c0, 1\n"
+	"	bic	%0, %0, #0x40\n"
+	"	mcr	p15, 0, %0, c1, c0, 1\n"
+	"	dsb\n"
+	/* Disable D-cache */
+	"	mrc	p15, 0, %0, c1, c0, 0\n"
+	"	bic	%0, %0, #0x04\n"
+	"	mcr	p15, 0, %0, c1, c0, 0\n"
+	  : "=&r" (v)
+	  : "r" (0)
+	  : "memory");
+}
+
+void cpu_leave_lowpower(void)
+{
+	unsigned int v;
+
+	flush_cache_all();
+	dsb();
+	asm volatile(	"mrc	p15, 0, %0, c1, c0, 0\n"
+	"	orr	%0, %0, #0x04\n"
+	"	mcr	p15, 0, %0, c1, c0, 0\n"
+	"	mrc	p15, 0, %0, c1, c0, 1\n"
+	"	orr	%0, %0, #0x40\n"
+	"	mcr	p15, 0, %0, c1, c0, 1\n"
+	  : "=&r" (v)
+	  :
+	  : "memory");
+}
+#endif
+
 static void ct_ca9x4_init(void)
 {
 	int i;
diff --git a/arch/arm/mach-vexpress/hotplug.c b/arch/arm/mach-vexpress/hotplug.c
new file mode 100644
index 0000000..9fc2476
--- /dev/null
+++ b/arch/arm/mach-vexpress/hotplug.c
@@ -0,0 +1,58 @@
+/*
+ *  linux/arch/arm/mach-vexpress/hotplug.c
+ *
+ *  Copyright (C) 2010 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/smp.h>
+#include <linux/completion.h>
+
+#include <plat/hotplug.h>
+
+extern volatile int pen_release;
+
+/* CPU-specific functions implemented in the tile support code */
+extern void cpu_enter_lowpower(void);
+extern void cpu_leave_lowpower(void);
+
+void __ref platform_do_lowpower(unsigned int cpu)
+{
+	cpu_enter_lowpower();
+	/*
+	 * there is no power-control hardware on this platform, so all
+	 * we can do is put the core into WFI; this is safe as the calling
+	 * code will have already disabled interrupts
+	 */
+	for (;;) {
+		/*
+		 * here's the WFI
+		 */
+		asm volatile("wfi" : : : "memory");
+
+		if (pen_release == cpu) {
+			/*
+			 * OK, proper wakeup, we're done
+			 */
+			break;
+		}
+
+		/*
+		 * getting here, means that we have come out of WFI without
+		 * having been woken up - this shouldn't happen
+		 *
+		 * The trouble is, letting people know about this is not really
+		 * possible, since we are currently running incoherently, and
+		 * therefore cannot safely call printk() or anything else
+		 */
+#ifdef DEBUG
+		printk("CPU%u: spurious wakeup call\n", cpu);
+#endif
+	}
+	cpu_leave_lowpower();
+}
-- 
1.6.3.3

[PATCH 3/3] ARM: vexpress: add support for CPU hotplug to ct-ca9x4 tile

[Russell King - ARM Linux]

On Tue, Aug 17, 2010 at 04:58:04PM +0100, Will Deacon wrote:
> The Versatile Express platform can support a quad-core Cortex-A9 tile running
> SMP Linux.
> 
> This patch adds support for CPU hotplug when running in this configuration.

This ties the core tile support into the generic versatile express code,
something which the current code structure is careful to avoid.  Please
ensure that we continue to avoid making the generic code rely upon
CA9x4 code.

[PATCH 3/3] ARM: vexpress: add support for CPU hotplug to ct-ca9x4 tile

[Will Deacon]

Hi Russell,

> On Tue, Aug 17, 2010 at 04:58:04PM +0100, Will Deacon wrote:
> > The Versatile Express platform can support a quad-core Cortex-A9 tile running
> > SMP Linux.
> >
> > This patch adds support for CPU hotplug when running in this configuration.
> 
> This ties the core tile support into the generic versatile express code,
> something which the current code structure is careful to avoid.  Please
> ensure that we continue to avoid making the generic code rely upon
> CA9x4 code.

The cpu_{enter,leave}_lowpower functions are implemented in the tile code
so they should be ok. I suppose the problem is that we might be able to do
better than a WFI on some tiles. How about this?:


diff --git a/arch/arm/mach-vexpress/hotplug.c b/arch/arm/mach-vexpress/hotplug.c
new file mode 100644
index 0000000..672e434
--- /dev/null
+++ b/arch/arm/mach-vexpress/hotplug.c
@@ -0,0 +1,47 @@
+/*
+ *  linux/arch/arm/mach-vexpress/hotplug.c
+ *
+ *  Copyright (C) 2010 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/smp.h>
+#include <linux/completion.h>
+
+#include <plat/hotplug.h>
+
+extern volatile int pen_release;
+
+/* CPU-specific functions implemented in the tile support code */
+extern void cpu_enter_lowpower(void);
+extern int cpu_do_lowpower(void);
+extern void cpu_leave_lowpower(void);
+
+void __ref platform_do_lowpower(unsigned int cpu)
+{
+       cpu_enter_lowpower();
+
+       if (cpu_do_lowpower() == -ENODEV) {
+               /*
+                * Tile does not have any low-power hardware so we
+                * put the core into a WFI.
+                */
+               for (;;) {
+                       asm volatile("wfi" : : : "memory");
+
+                       if (pen_release == cpu)
+                               break;
+
+#ifdef DEBUG
+                       printk("CPU%u: spurious wakeup call\n", cpu);
+#endif
+               }
+       }
+
+       cpu_leave_lowpower();
+}


The alternative would be to merge all the cpu_* functions into one, but
then we'd end up replicating the wfi code across all the tiles without
additional power-saving hardware.

Cheers,

Will

[PATCH 3/3] ARM: vexpress: add support for CPU hotplug to ct-ca9x4 tile

[Russell King - ARM Linux]

On Wed, Sep 08, 2010 at 05:46:02PM +0100, Will Deacon wrote:
> Hi Russell,
> 
> > On Tue, Aug 17, 2010 at 04:58:04PM +0100, Will Deacon wrote:
> > > The Versatile Express platform can support a quad-core Cortex-A9 tile running
> > > SMP Linux.
> > >
> > > This patch adds support for CPU hotplug when running in this configuration.
> > 
> > This ties the core tile support into the generic versatile express code,
> > something which the current code structure is careful to avoid.  Please
> > ensure that we continue to avoid making the generic code rely upon
> > CA9x4 code.
> 
> The cpu_{enter,leave}_lowpower functions are implemented in the tile code
> so they should be ok. I suppose the problem is that we might be able to do
> better than a WFI on some tiles. How about this?:

What I was referring to is that generic code _directly_ calls tile code.
That means if you want to build in support for two tiles, you're going
to have symbols clashing as two tiles will define functions called
'cpu_enter_lowpower' etc.