[PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B. Prabhu]

Fast sleep is an idle state, where the core and the L1 and L2
caches are brought down to a threshold voltage. This also means that
the communication between L2 and L3 caches have to be fenced. However
the current P8 chips have a bug wherein this fencing between L2 and
L3 caches get delayed by a cpu cycle. This can delay L3 response to
the other cpus if they request for data during this time. Thus they
would fetch the same data from the memory which could lead to data
corruption if L3 cache is not flushed.
Patch 4 adds support to work around this.

'Deep Winkle' is a deeper idle state where core and private L2 are powered
off. While it offers higher power savings, it is at the cost of losing
hypervisor register state and higher latency.
Patch 5-9 adds support for winkle and uses it for offline cpus.

Patch 1 - Moves parameters required discover idle states to a location 
common to both cpuidle driver and powernv core code
Patch 2 - Populates idle state details from device tree
Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle


Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: devicetree@vger.kernel.org
Cc: linux-pm@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org

Preeti U Murthy (2):
  cpuidle/powernv: Populate cpuidle state details by querying the
    device-tree
  powerpc/powernv/cpuidle: Add workaround to enable fastsleep

Shreyas B. Prabhu (6):
  powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
    fast-sleep
  powerpc/powernv: Add OPAL call to save and restore
  powerpc: Adding macro for accessing Thread Switch Control Register
  powerpc/powernv: Add winkle infrastructure
  powerpc/powernv: Discover and enable winkle
  powerpc/powernv: Enter deepest supported idle state in offline

Srivatsa S. Bhat (1):
  powerpc/powernv: Enable Offline CPUs to enter deep idle states

 arch/powerpc/include/asm/machdep.h             |   4 +
 arch/powerpc/include/asm/opal.h                |  10 ++
 arch/powerpc/include/asm/paca.h                |   3 +
 arch/powerpc/include/asm/ppc-opcode.h          |   2 +
 arch/powerpc/include/asm/processor.h           |   6 +-
 arch/powerpc/include/asm/reg.h                 |   1 +
 arch/powerpc/kernel/asm-offsets.c              |   1 +
 arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
 arch/powerpc/kernel/idle.c                     |  30 ++++
 arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
 arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
 arch/powerpc/platforms/powernv/powernv.h       |   8 +
 arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/smp.c           |  13 +-
 arch/powerpc/platforms/powernv/subcore.c       |  15 ++
 drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
 16 files changed, 439 insertions(+), 33 deletions(-)

-- 
1.9.0

[PATCH 1/9] powerpc/powernv: Enable Offline CPUs to enter deep idle states

[Shreyas B. Prabhu]

From: "Srivatsa S. Bhat" <srivatsa@MIT.EDU>

The offline cpus should enter deep idle states so as to gain maximum
powersavings when the entire core is offline. To do so the offline path
must be made aware of the available deepest idle state. Hence probe the
device tree for the possible idle states in powernv core code and
expose the deepest idle state through flags.

Since the  device tree is probed by the cpuidle driver as well, move
the parameters required to discover the idle states into an appropriate
common place to both the driver and the powernv core code.

Another point is that fastsleep idle state may require workarounds in
the kernel to function properly. This workaround is introduced in the
subsequent patches. However neither the cpuidle driver or the hotplug
path need be bothered about this workaround.

They will be taken care of by the core powernv code.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: linux-pm@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Srivatsa S. Bhat <srivatsa@MIT.EDU>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
[ Changelog modified by preeti@linux.vnet.ibm.com ]
Signed-off-by: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/opal.h          |  4 +++
 arch/powerpc/platforms/powernv/powernv.h |  7 +++++
 arch/powerpc/platforms/powernv/setup.c   | 51 ++++++++++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/smp.c     | 11 ++++++-
 drivers/cpuidle/cpuidle-powernv.c        |  7 ++---
 5 files changed, 75 insertions(+), 5 deletions(-)

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 86055e5..28b8342 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -772,6 +772,10 @@ extern struct kobject *opal_kobj;
 /* /ibm,opal */
 extern struct device_node *opal_node;
 
+/* Flags used for idle state discovery from the device tree */
+#define IDLE_INST_NAP	0x00010000 /* nap instruction can be used */
+#define IDLE_INST_SLEEP	0x00020000 /* sleep instruction can be used */
+
 /* API functions */
 int64_t opal_invalid_call(void);
 int64_t opal_console_write(int64_t term_number, __be64 *length,
diff --git a/arch/powerpc/platforms/powernv/powernv.h b/arch/powerpc/platforms/powernv/powernv.h
index 75501bf..31ece13 100644
--- a/arch/powerpc/platforms/powernv/powernv.h
+++ b/arch/powerpc/platforms/powernv/powernv.h
@@ -23,6 +23,13 @@ static inline int pnv_pci_dma_set_mask(struct pci_dev *pdev, u64 dma_mask)
 }
 #endif
 
+/* Flags to indicate which of the CPU idle states are available for use */
+
+#define IDLE_USE_NAP		(1UL << 0)
+#define IDLE_USE_SLEEP		(1UL << 1)
+
+extern unsigned int pnv_get_supported_cpuidle_states(void);
+
 extern void pnv_lpc_init(void);
 
 bool cpu_core_split_required(void);
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index 5a0e2dc..2dca1d8 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -282,6 +282,57 @@ static void __init pnv_setup_machdep_rtas(void)
 }
 #endif /* CONFIG_PPC_POWERNV_RTAS */
 
+static unsigned int supported_cpuidle_states;
+
+unsigned int pnv_get_supported_cpuidle_states(void)
+{
+	return supported_cpuidle_states;
+}
+
+static int __init pnv_probe_idle_states(void)
+{
+	struct device_node *power_mgt;
+	struct property *prop;
+	int dt_idle_states;
+	u32 *flags;
+	int i;
+
+	supported_cpuidle_states = 0;
+
+	if (cpuidle_disable != IDLE_NO_OVERRIDE)
+		return 0;
+
+	if (!firmware_has_feature(FW_FEATURE_OPALv3))
+		return 0;
+
+	power_mgt = of_find_node_by_path("/ibm,opal/power-mgt");
+	if (!power_mgt) {
+		pr_warn("opal: PowerMgmt Node not found\n");
+		return 0;
+	}
+
+	prop = of_find_property(power_mgt, "ibm,cpu-idle-state-flags", NULL);
+	if (!prop) {
+		pr_warn("DT-PowerMgmt: missing ibm,cpu-idle-state-flags\n");
+		return 0;
+	}
+
+	dt_idle_states = prop->length / sizeof(u32);
+	flags = (u32 *) prop->value;
+
+	for (i = 0; i < dt_idle_states; i++) {
+		if (flags[i] & IDLE_INST_NAP)
+			supported_cpuidle_states |= IDLE_USE_NAP;
+
+		if (flags[i] & IDLE_INST_SLEEP)
+			supported_cpuidle_states |= IDLE_USE_SLEEP;
+	}
+
+	return 0;
+}
+
+subsys_initcall(pnv_probe_idle_states);
+
 static int __init pnv_probe(void)
 {
 	unsigned long root = of_get_flat_dt_root();
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 5fcfcf4..3ad31d2 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -149,6 +149,7 @@ static int pnv_smp_cpu_disable(void)
 static void pnv_smp_cpu_kill_self(void)
 {
 	unsigned int cpu;
+	unsigned long idle_states;
 
 	/* Standard hot unplug procedure */
 	local_irq_disable();
@@ -159,13 +160,21 @@ static void pnv_smp_cpu_kill_self(void)
 	generic_set_cpu_dead(cpu);
 	smp_wmb();
 
+	idle_states = pnv_get_supported_cpuidle_states();
+
 	/* We don't want to take decrementer interrupts while we are offline,
 	 * so clear LPCR:PECE1. We keep PECE2 enabled.
 	 */
 	mtspr(SPRN_LPCR, mfspr(SPRN_LPCR) & ~(u64)LPCR_PECE1);
 	while (!generic_check_cpu_restart(cpu)) {
 		ppc64_runlatch_off();
-		power7_nap(1);
+
+		/* If sleep is supported, go to sleep, instead of nap */
+		if (idle_states & IDLE_USE_SLEEP)
+			power7_sleep();
+		else
+			power7_nap(1);
+
 		ppc64_runlatch_on();
 
 		/* Reenable IRQs briefly to clear the IPI that woke us */
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index a64be57..23d2743 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -16,13 +16,12 @@
 
 #include <asm/machdep.h>
 #include <asm/firmware.h>
+#include <asm/opal.h>
 #include <asm/runlatch.h>
 
 /* Flags and constants used in PowerNV platform */
 
 #define MAX_POWERNV_IDLE_STATES	8
-#define IDLE_USE_INST_NAP	0x00010000 /* Use nap instruction */
-#define IDLE_USE_INST_SLEEP	0x00020000 /* Use sleep instruction */
 
 struct cpuidle_driver powernv_idle_driver = {
 	.name             = "powernv_idle",
@@ -185,7 +184,7 @@ static int powernv_add_idle_states(void)
 	for (i = 0; i < dt_idle_states; i++) {
 
 		flags = be32_to_cpu(idle_state_flags[i]);
-		if (flags & IDLE_USE_INST_NAP) {
+		if (flags & IDLE_INST_NAP) {
 			/* Add NAP state */
 			strcpy(powernv_states[nr_idle_states].name, "Nap");
 			strcpy(powernv_states[nr_idle_states].desc, "Nap");
@@ -196,7 +195,7 @@ static int powernv_add_idle_states(void)
 			nr_idle_states++;
 		}
 
-		if (flags & IDLE_USE_INST_SLEEP) {
+		if (flags & IDLE_INST_SLEEP) {
 			/* Add FASTSLEEP state */
 			strcpy(powernv_states[nr_idle_states].name, "FastSleep");
 			strcpy(powernv_states[nr_idle_states].desc, "FastSleep");
-- 
1.9.0

[PATCH 2/9] cpuidle/powernv: Populate cpuidle state details by querying the device-tree

[Shreyas B. Prabhu]

From: Preeti U Murthy <preeti@linux.vnet.ibm.com>

We hard code the metrics relevant for cpuidle states in the kernel today.
Instead pick them up from the device tree so that they remain relevant
and updated for the system that the kernel is running on.

Cc: linux-pm@vger.kernel.org
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Rob Herring <robh+dt@kernel.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: devicetree@vger.kernel.org
Signed-off-by: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 drivers/cpuidle/cpuidle-powernv.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index 23d2743..3ceff53 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -162,7 +162,8 @@ static int powernv_add_idle_states(void)
 	int nr_idle_states = 1; /* Snooze */
 	int dt_idle_states;
 	const __be32 *idle_state_flags;
-	u32 len_flags, flags;
+	const __be32 *idle_state_latency;
+	u32 len_flags, flags, latency_ns;
 	int i;
 
 	/* Currently we have snooze statically defined */
@@ -178,19 +179,33 @@ static int powernv_add_idle_states(void)
 		pr_warn("DT-PowerMgmt: missing ibm,cpu-idle-state-flags\n");
 		return nr_idle_states;
 	}
+	idle_state_latency = of_get_property(power_mgt,
+			"ibm,cpu-idle-state-latencies-ns", NULL);
+	if (!idle_state_latency) {
+		pr_warn("DT-PowerMgmt: missing ibm,cpu-idle-state-latencies-ns\n");
+		return nr_idle_states;
+	}
 
 	dt_idle_states = len_flags / sizeof(u32);
 
 	for (i = 0; i < dt_idle_states; i++) {
 
 		flags = be32_to_cpu(idle_state_flags[i]);
+
+		/* Cpuidle accepts exit_latency in us and we estimate best case
+		 * target residency to be 10x exit_latency
+		 */
+		latency_ns = be32_to_cpu(idle_state_latency[i]);
+
 		if (flags & IDLE_INST_NAP) {
 			/* Add NAP state */
 			strcpy(powernv_states[nr_idle_states].name, "Nap");
 			strcpy(powernv_states[nr_idle_states].desc, "Nap");
 			powernv_states[nr_idle_states].flags = CPUIDLE_FLAG_TIME_VALID;
-			powernv_states[nr_idle_states].exit_latency = 10;
-			powernv_states[nr_idle_states].target_residency = 100;
+			powernv_states[nr_idle_states].exit_latency =
+					((unsigned int)latency_ns) / 1000;
+			powernv_states[nr_idle_states].target_residency =
+					((unsigned int)latency_ns / 100);
 			powernv_states[nr_idle_states].enter = &nap_loop;
 			nr_idle_states++;
 		}
@@ -201,8 +216,10 @@ static int powernv_add_idle_states(void)
 			strcpy(powernv_states[nr_idle_states].desc, "FastSleep");
 			powernv_states[nr_idle_states].flags =
 				CPUIDLE_FLAG_TIME_VALID | CPUIDLE_FLAG_TIMER_STOP;
-			powernv_states[nr_idle_states].exit_latency = 300;
-			powernv_states[nr_idle_states].target_residency = 1000000;
+			powernv_states[nr_idle_states].exit_latency =
+					((unsigned int)latency_ns) / 1000;
+			powernv_states[nr_idle_states].target_residency =
+					((unsigned int)latency_ns / 100);
 			powernv_states[nr_idle_states].enter = &fastsleep_loop;
 			nr_idle_states++;
 		}
-- 
1.9.0

[PATCH 3/9] powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from fast-sleep

[Shreyas B. Prabhu]

When guests have to be launched, the secondary threads which are offline
are woken up to run the guests. Today these threads wake up from nap
and check if they have to run guests. Now that the offline secondary
threads can go to fastsleep or going ahead a deeper idle state such as winkle,
add this check in the wakeup from any of the deep idle states path as well.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Suggested-by: "Srivatsa S. Bhat" <srivatsa@mit.edu>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
[ Changelog added by <preeti@linux.vnet.ibm.com> ]
Signed-off-by: Preeti U Murthy <preeti@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/exceptions-64s.S | 35 ++++++++++++++++-------------------
 1 file changed, 16 insertions(+), 19 deletions(-)

diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 050f79a..c64f3cc0 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -100,25 +100,8 @@ system_reset_pSeries:
 	SET_SCRATCH0(r13)
 #ifdef CONFIG_PPC_P7_NAP
 BEGIN_FTR_SECTION
-	/* Running native on arch 2.06 or later, check if we are
-	 * waking up from nap. We only handle no state loss and
-	 * supervisor state loss. We do -not- handle hypervisor
-	 * state loss at this time.
-	 */
-	mfspr	r13,SPRN_SRR1
-	rlwinm.	r13,r13,47-31,30,31
-	beq	9f
 
-	/* waking up from powersave (nap) state */
-	cmpwi	cr1,r13,2
-	/* Total loss of HV state is fatal, we could try to use the
-	 * PIR to locate a PACA, then use an emergency stack etc...
-	 * OPAL v3 based powernv platforms have new idle states
-	 * which fall in this catagory.
-	 */
-	bgt	cr1,8f
 	GET_PACA(r13)
-
 #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
 	li	r0,KVM_HWTHREAD_IN_KERNEL
 	stb	r0,HSTATE_HWTHREAD_STATE(r13)
@@ -131,13 +114,27 @@ BEGIN_FTR_SECTION
 1:
 #endif
 
+	/* Running native on arch 2.06 or later, check if we are
+	 * waking up from nap. We only handle no state loss and
+	 * supervisor state loss. We do -not- handle hypervisor
+	 * state loss at this time.
+	 */
+	mfspr	r13,SPRN_SRR1
+	rlwinm.	r13,r13,47-31,30,31
+	beq	9f
+
+	/* waking up from powersave (nap) state */
+	cmpwi	cr1,r13,2
+	GET_PACA(r13)
+
+	bgt	cr1,8f
+
 	beq	cr1,2f
 	b	power7_wakeup_noloss
 2:	b	power7_wakeup_loss
 
 	/* Fast Sleep wakeup on PowerNV */
-8:	GET_PACA(r13)
-	b 	power7_wakeup_tb_loss
+8:	b 	power7_wakeup_tb_loss
 
 9:
 END_FTR_SECTION_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
-- 
1.9.0

[PATCH 4/9] powerpc/powernv/cpuidle: Add workaround to enable fastsleep

[Shreyas B. Prabhu]

From: Preeti U Murthy <preeti@linux.vnet.ibm.com>

Fast sleep is an idle state, where the core and the L1 and L2
caches are brought down to a threshold voltage. This also means that
the communication between L2 and L3 caches have to be fenced. However
the current P8 chips have a bug wherein this fencing between L2 and
L3 caches get delayed by a cpu cycle. This can delay L3 response to
the other cpus if they request for data during this time. Thus they
would fetch the same data from the memory which could lead to data
corruption if L3 cache is not flushed.

The cpu idle states save power at a core level and not at a thread level.
Hence powersavings is based on the shallowest idle state that a thread
of a core is in. The above issue in fastsleep will arise only when
all the threads in a core either enter fastsleep or some of them enter
any deeper idle states, with only a few being in fastsleep. This patch
therefore implements a workaround this bug  by ensuring
that, each time a cpu goes to fastsleep, it checks if it is the last
thread in the core to enter fastsleep. If so, it needs to make an opal
call to get around the above mentioned fastsleep problem in the hardware
before issuing the sleep instruction.

Similarly when a thread in a core comes out of fastsleep, it needs
to verify if its the first thread in the core to come out of fastsleep
and issue the opal call to revert the changes made while entering
fastsleep.

For the same reason mentioned above we need to take care of offline threads
as well since we allow them to enter fastsleep and with support for
deep winkle soon coming in they can enter winkle as well.  We therefore
ensure that even offline threads make the above mentioned opal calls
similarly, so that as long as the threads in a core are in and
idle state >= fastsleep, we have the workaround in place. Whenever a
thread comes out of either of these states, it needs to verify if the
opal call has been made and if so it will revert it. For now this patch
ensures that offline threads enter fastsleep.

We need to be able to synchronize the cpus in a core which are entering
and exiting fastsleep so as to ensure that the last thread in the core
to enter fastsleep and the first to exit fastsleep *only* issue the opal
call. To do so, we need a per-core lock and counter. The counter is
required to keep track of the number of threads in a core which are in
idle state >= fastsleep. To make the implementation of this simple, we
introduce a per-cpu lock and counter and every thread always takes the
primary thread's lock, modifies the primary thread's counter. This
effectively makes them per-core entities.

But the workaround is abstracted in the powernv core code and neither
the hotplug path nor the cpuidle driver need to bother about it. All
they need to know is if fastsleep, with error or no error is present as
an idle state.

Cc: linux-pm@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Signed-off-by: Preeti U Murthy <preeti@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/machdep.h             |   3 +
 arch/powerpc/include/asm/opal.h                |   3 +
 arch/powerpc/include/asm/processor.h           |   4 +-
 arch/powerpc/kernel/idle.c                     |  19 ++++
 arch/powerpc/kernel/idle_power7.S              |   2 +-
 arch/powerpc/platforms/powernv/opal-wrappers.S |   1 +
 arch/powerpc/platforms/powernv/setup.c         | 139 ++++++++++++++++++-------
 drivers/cpuidle/cpuidle-powernv.c              |   8 +-
 8 files changed, 140 insertions(+), 39 deletions(-)

diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h
index b125cea..f37014f 100644
--- a/arch/powerpc/include/asm/machdep.h
+++ b/arch/powerpc/include/asm/machdep.h
@@ -298,6 +298,9 @@ struct machdep_calls {
 #ifdef CONFIG_MEMORY_HOTREMOVE
 	int (*remove_memory)(u64, u64);
 #endif
+	/* Idle handlers */
+	void		(*setup_idle)(void);
+	unsigned long	(*power7_sleep)(void);
 };
 
 extern void e500_idle(void);
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 28b8342..166d572 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -149,6 +149,7 @@ struct opal_sg_list {
 #define OPAL_DUMP_INFO2				94
 #define OPAL_PCI_EEH_FREEZE_SET			97
 #define OPAL_HANDLE_HMI				98
+#define OPAL_CONFIG_IDLE_STATE			99
 #define OPAL_REGISTER_DUMP_REGION		101
 #define OPAL_UNREGISTER_DUMP_REGION		102
 
@@ -775,6 +776,7 @@ extern struct device_node *opal_node;
 /* Flags used for idle state discovery from the device tree */
 #define IDLE_INST_NAP	0x00010000 /* nap instruction can be used */
 #define IDLE_INST_SLEEP	0x00020000 /* sleep instruction can be used */
+#define IDLE_INST_SLEEP_ER1	0x00080000 /* Use sleep with work around*/
 
 /* API functions */
 int64_t opal_invalid_call(void);
@@ -975,6 +977,7 @@ extern int opal_handle_hmi_exception(struct pt_regs *regs);
 
 extern void opal_shutdown(void);
 extern int opal_resync_timebase(void);
+int64_t opal_config_idle_state(uint64_t state, uint64_t enter);
 
 extern void opal_lpc_init(void);
 
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index dda7ac4..41953cd 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -451,8 +451,10 @@ extern unsigned long cpuidle_disable;
 enum idle_boot_override {IDLE_NO_OVERRIDE = 0, IDLE_POWERSAVE_OFF};
 
 extern int powersave_nap;	/* set if nap mode can be used in idle loop */
+extern void arch_setup_idle(void);
 extern void power7_nap(int check_irq);
-extern void power7_sleep(void);
+extern unsigned long power7_sleep(void);
+extern unsigned long __power7_sleep(void);
 extern void flush_instruction_cache(void);
 extern void hard_reset_now(void);
 extern void poweroff_now(void);
diff --git a/arch/powerpc/kernel/idle.c b/arch/powerpc/kernel/idle.c
index d7216c9..1f268e0 100644
--- a/arch/powerpc/kernel/idle.c
+++ b/arch/powerpc/kernel/idle.c
@@ -32,6 +32,9 @@
 #include <asm/machdep.h>
 #include <asm/runlatch.h>
 #include <asm/smp.h>
+#include <asm/cputhreads.h>
+#include <asm/firmware.h>
+#include <asm/opal.h>
 
 
 unsigned long cpuidle_disable = IDLE_NO_OVERRIDE;
@@ -78,6 +81,22 @@ void arch_cpu_idle(void)
 	HMT_medium();
 	ppc64_runlatch_on();
 }
+void arch_setup_idle(void)
+{
+	if (ppc_md.setup_idle)
+		ppc_md.setup_idle();
+}
+
+unsigned long power7_sleep(void)
+{
+	unsigned long ret;
+
+	if (ppc_md.power7_sleep)
+		ret = ppc_md.power7_sleep();
+	else
+		ret = __power7_sleep();
+	return ret;
+}
 
 int powersave_nap;
 
diff --git a/arch/powerpc/kernel/idle_power7.S b/arch/powerpc/kernel/idle_power7.S
index be05841..c3481c9 100644
--- a/arch/powerpc/kernel/idle_power7.S
+++ b/arch/powerpc/kernel/idle_power7.S
@@ -129,7 +129,7 @@ _GLOBAL(power7_nap)
 	b	power7_powersave_common
 	/* No return */
 
-_GLOBAL(power7_sleep)
+_GLOBAL(__power7_sleep)
 	li	r3,1
 	li	r4,1
 	b	power7_powersave_common
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index 2e6ce1b..8d1e724 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -245,5 +245,6 @@ OPAL_CALL(opal_sensor_read,			OPAL_SENSOR_READ);
 OPAL_CALL(opal_get_param,			OPAL_GET_PARAM);
 OPAL_CALL(opal_set_param,			OPAL_SET_PARAM);
 OPAL_CALL(opal_handle_hmi,			OPAL_HANDLE_HMI);
+OPAL_CALL(opal_config_idle_state,		OPAL_CONFIG_IDLE_STATE);
 OPAL_CALL(opal_register_dump_region,		OPAL_REGISTER_DUMP_REGION);
 OPAL_CALL(opal_unregister_dump_region,		OPAL_UNREGISTER_DUMP_REGION);
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index 2dca1d8..9d9a898 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -36,9 +36,20 @@
 #include <asm/opal.h>
 #include <asm/kexec.h>
 #include <asm/smp.h>
+#include <asm/cputhreads.h>
 
 #include "powernv.h"
 
+/* Per-cpu structures to keep track of cpus of a core that
+ * are in idle states >= fastsleep so as to call opal for
+ * sleep setup when the entire core is ready to go to fastsleep.
+ *
+ * We need sometihng similar to a per-core lock. For now we
+ * achieve this by taking the lock of the primary thread in the core.
+ */
+static DEFINE_PER_CPU(spinlock_t, fastsleep_override_lock);
+static DEFINE_PER_CPU(int, fastsleep_cnt);
+
 static void __init pnv_setup_arch(void)
 {
 	set_arch_panic_timeout(10, ARCH_PANIC_TIMEOUT);
@@ -254,35 +265,8 @@ static unsigned long pnv_memory_block_size(void)
 }
 #endif
 
-static void __init pnv_setup_machdep_opal(void)
-{
-	ppc_md.get_boot_time = opal_get_boot_time;
-	ppc_md.get_rtc_time = opal_get_rtc_time;
-	ppc_md.set_rtc_time = opal_set_rtc_time;
-	ppc_md.restart = pnv_restart;
-	ppc_md.power_off = pnv_power_off;
-	ppc_md.halt = pnv_halt;
-	ppc_md.machine_check_exception = opal_machine_check;
-	ppc_md.mce_check_early_recovery = opal_mce_check_early_recovery;
-	ppc_md.hmi_exception_early = opal_hmi_exception_early;
-	ppc_md.handle_hmi_exception = opal_handle_hmi_exception;
-}
-
-#ifdef CONFIG_PPC_POWERNV_RTAS
-static void __init pnv_setup_machdep_rtas(void)
-{
-	if (rtas_token("get-time-of-day") != RTAS_UNKNOWN_SERVICE) {
-		ppc_md.get_boot_time = rtas_get_boot_time;
-		ppc_md.get_rtc_time = rtas_get_rtc_time;
-		ppc_md.set_rtc_time = rtas_set_rtc_time;
-	}
-	ppc_md.restart = rtas_restart;
-	ppc_md.power_off = rtas_power_off;
-	ppc_md.halt = rtas_halt;
-}
-#endif /* CONFIG_PPC_POWERNV_RTAS */
-
 static unsigned int supported_cpuidle_states;
+static int need_fastsleep_workaround;
 
 unsigned int pnv_get_supported_cpuidle_states(void)
 {
@@ -292,12 +276,13 @@ unsigned int pnv_get_supported_cpuidle_states(void)
 static int __init pnv_probe_idle_states(void)
 {
 	struct device_node *power_mgt;
-	struct property *prop;
 	int dt_idle_states;
-	u32 *flags;
+	const __be32 *idle_state_flags;
+	u32 len_flags, flags;
 	int i;
 
 	supported_cpuidle_states = 0;
+	need_fastsleep_workaround = 0;
 
 	if (cpuidle_disable != IDLE_NO_OVERRIDE)
 		return 0;
@@ -311,21 +296,28 @@ static int __init pnv_probe_idle_states(void)
 		return 0;
 	}
 
-	prop = of_find_property(power_mgt, "ibm,cpu-idle-state-flags", NULL);
-	if (!prop) {
+	idle_state_flags = of_get_property(power_mgt,
+			"ibm,cpu-idle-state-flags", &len_flags);
+	if (!idle_state_flags) {
 		pr_warn("DT-PowerMgmt: missing ibm,cpu-idle-state-flags\n");
 		return 0;
 	}
 
-	dt_idle_states = prop->length / sizeof(u32);
-	flags = (u32 *) prop->value;
+	dt_idle_states = len_flags / sizeof(u32);
 
 	for (i = 0; i < dt_idle_states; i++) {
-		if (flags[i] & IDLE_INST_NAP)
+
+		flags = be32_to_cpu(idle_state_flags[i]);
+		if (flags & IDLE_INST_NAP)
 			supported_cpuidle_states |= IDLE_USE_NAP;
 
-		if (flags[i] & IDLE_INST_SLEEP)
+		if (flags & IDLE_INST_SLEEP)
 			supported_cpuidle_states |= IDLE_USE_SLEEP;
+
+		if (flags & IDLE_INST_SLEEP_ER1) {
+			supported_cpuidle_states |= IDLE_USE_SLEEP;
+			need_fastsleep_workaround = 1;
+		}
 	}
 
 	return 0;
@@ -333,6 +325,81 @@ static int __init pnv_probe_idle_states(void)
 
 subsys_initcall(pnv_probe_idle_states);
 
+static void pnv_setup_idle(void)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		spin_lock_init(&per_cpu(fastsleep_override_lock, cpu));
+		per_cpu(fastsleep_cnt, cpu) = threads_per_core;
+	}
+}
+
+static void
+pnv_apply_fastsleep_workaround(bool enter_fastsleep, int primary_thread)
+{
+	if (enter_fastsleep) {
+		spin_lock(&per_cpu(fastsleep_override_lock, primary_thread));
+		if (--(per_cpu(fastsleep_cnt, primary_thread)) == 0)
+			opal_config_idle_state(1, 1);
+		spin_unlock(&per_cpu(fastsleep_override_lock, primary_thread));
+	} else {
+		spin_lock(&per_cpu(fastsleep_override_lock, primary_thread));
+		if ((per_cpu(fastsleep_cnt, primary_thread)) == 0)
+			opal_config_idle_state(1, 0);
+		per_cpu(fastsleep_cnt, primary_thread)++;
+		spin_unlock(&per_cpu(fastsleep_override_lock, primary_thread));
+	}
+}
+
+static unsigned long pnv_power7_sleep(void)
+{
+	int cpu, primary_thread;
+	unsigned long srr1;
+
+	cpu = smp_processor_id();
+	primary_thread = cpu_first_thread_sibling(cpu);
+
+	if (need_fastsleep_workaround) {
+		pnv_apply_fastsleep_workaround(1, primary_thread);
+		srr1 = __power7_sleep();
+		pnv_apply_fastsleep_workaround(0, primary_thread);
+	} else {
+		srr1 = __power7_sleep();
+	}
+	return srr1;
+}
+
+static void __init pnv_setup_machdep_opal(void)
+{
+	ppc_md.get_boot_time = opal_get_boot_time;
+	ppc_md.get_rtc_time = opal_get_rtc_time;
+	ppc_md.set_rtc_time = opal_set_rtc_time;
+	ppc_md.restart = pnv_restart;
+	ppc_md.power_off = pnv_power_off;
+	ppc_md.halt = pnv_halt;
+	ppc_md.machine_check_exception = opal_machine_check;
+	ppc_md.mce_check_early_recovery = opal_mce_check_early_recovery;
+	ppc_md.hmi_exception_early = opal_hmi_exception_early;
+	ppc_md.handle_hmi_exception = opal_handle_hmi_exception;
+	ppc_md.setup_idle = pnv_setup_idle;
+	ppc_md.power7_sleep = pnv_power7_sleep;
+}
+
+#ifdef CONFIG_PPC_POWERNV_RTAS
+static void __init pnv_setup_machdep_rtas(void)
+{
+	if (rtas_token("get-time-of-day") != RTAS_UNKNOWN_SERVICE) {
+		ppc_md.get_boot_time = rtas_get_boot_time;
+		ppc_md.get_rtc_time = rtas_get_rtc_time;
+		ppc_md.set_rtc_time = rtas_set_rtc_time;
+	}
+	ppc_md.restart = rtas_restart;
+	ppc_md.power_off = rtas_power_off;
+	ppc_md.halt = rtas_halt;
+}
+#endif /* CONFIG_PPC_POWERNV_RTAS */
+
 static int __init pnv_probe(void)
 {
 	unsigned long root = of_get_flat_dt_root();
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index 3ceff53..92ad134 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -18,6 +18,7 @@
 #include <asm/firmware.h>
 #include <asm/opal.h>
 #include <asm/runlatch.h>
+#include <asm/processor.h>
 
 /* Flags and constants used in PowerNV platform */
 
@@ -210,7 +211,8 @@ static int powernv_add_idle_states(void)
 			nr_idle_states++;
 		}
 
-		if (flags & IDLE_INST_SLEEP) {
+		if ((flags & IDLE_INST_SLEEP_ER1) ||
+				(flags & IDLE_INST_SLEEP)) {
 			/* Add FASTSLEEP state */
 			strcpy(powernv_states[nr_idle_states].name, "FastSleep");
 			strcpy(powernv_states[nr_idle_states].desc, "FastSleep");
@@ -264,6 +266,10 @@ static int __init powernv_processor_idle_init(void)
 
 	register_cpu_notifier(&setup_hotplug_notifier);
 	printk(KERN_DEBUG "powernv_idle_driver registered\n");
+
+	/* If any idle states require special
+	 * initializations before cpuidle kicks in */
+	arch_setup_idle();
 	return 0;
 }
 
-- 
1.9.0

[PATCH 5/9] powerpc/powernv: Add OPAL call to save and restore

[Shreyas B. Prabhu]

PORE can be programmed to restore hypervisor registers when waking up
from deep cpu idle states like winkle.

Add call to pass SPR address and value to OPAL, which in turn will
program PORE to restore the register state.

Cc: linuxppc-dev@lists.ozlabs.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Suggested-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/opal.h                | 2 ++
 arch/powerpc/platforms/powernv/opal-wrappers.S | 1 +
 2 files changed, 3 insertions(+)

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 166d572..d376020 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -150,6 +150,7 @@ struct opal_sg_list {
 #define OPAL_PCI_EEH_FREEZE_SET			97
 #define OPAL_HANDLE_HMI				98
 #define OPAL_CONFIG_IDLE_STATE			99
+#define OPAL_SLW_SET_REG			100
 #define OPAL_REGISTER_DUMP_REGION		101
 #define OPAL_UNREGISTER_DUMP_REGION		102
 
@@ -978,6 +979,7 @@ extern int opal_handle_hmi_exception(struct pt_regs *regs);
 extern void opal_shutdown(void);
 extern int opal_resync_timebase(void);
 int64_t opal_config_idle_state(uint64_t state, uint64_t enter);
+int64_t opal_slw_set_reg(uint64_t cpu_pir, uint64_t sprn, uint64_t val);
 
 extern void opal_lpc_init(void);
 
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index 8d1e724..12e5d46 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -246,5 +246,6 @@ OPAL_CALL(opal_get_param,			OPAL_GET_PARAM);
 OPAL_CALL(opal_set_param,			OPAL_SET_PARAM);
 OPAL_CALL(opal_handle_hmi,			OPAL_HANDLE_HMI);
 OPAL_CALL(opal_config_idle_state,		OPAL_CONFIG_IDLE_STATE);
+OPAL_CALL(opal_slw_set_reg,			OPAL_SLW_SET_REG);
 OPAL_CALL(opal_register_dump_region,		OPAL_REGISTER_DUMP_REGION);
 OPAL_CALL(opal_unregister_dump_region,		OPAL_UNREGISTER_DUMP_REGION);
-- 
1.9.0

[PATCH 6/9] powerpc: Adding macro for accessing Thread Switch Control Register

[Shreyas B. Prabhu]

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/reg.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index 0c05059..cb65a73 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -371,6 +371,7 @@
 #define SPRN_DBAT7L	0x23F	/* Data BAT 7 Lower Register */
 #define SPRN_DBAT7U	0x23E	/* Data BAT 7 Upper Register */
 #define SPRN_PPR	0x380	/* SMT Thread status Register */
+#define SPRN_TSCR	0x399	/* Thread Switch Control Register */
 
 #define SPRN_DEC	0x016		/* Decrement Register */
 #define SPRN_DER	0x095		/* Debug Enable Regsiter */
-- 
1.9.0

[PATCH 7/9] powerpc/powernv: Add winkle infrastructure

[Shreyas B. Prabhu]

Winkle causes power to be gated off to the entire chiplet. Hence the
hypervisor/firmware state in the entire chiplet is lost.

This patch adds necessary infrastructure to support winkle. Specifically
does following:
- Before entering winkle, save state of registers that need to be
  restored on wake up (SDR1, HFSCR)

- SRR1 bits 46:47 which is used to identify which power saving mode cpu
  woke up from is '11' for both winkle and sleep. Hence introduce a flag
  in PACA to distinguish b/w winkle and sleep.

- Upon waking up, restore all saved registers, recover slb

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Suggested-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/machdep.h     |  1 +
 arch/powerpc/include/asm/paca.h        |  3 ++
 arch/powerpc/include/asm/ppc-opcode.h  |  2 +
 arch/powerpc/include/asm/processor.h   |  2 +
 arch/powerpc/kernel/asm-offsets.c      |  1 +
 arch/powerpc/kernel/exceptions-64s.S   |  4 +-
 arch/powerpc/kernel/idle.c             | 11 +++++
 arch/powerpc/kernel/idle_power7.S      | 81 +++++++++++++++++++++++++++++++++-
 arch/powerpc/platforms/powernv/setup.c | 24 ++++++++++
 9 files changed, 126 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h
index f37014f..0a3ced9 100644
--- a/arch/powerpc/include/asm/machdep.h
+++ b/arch/powerpc/include/asm/machdep.h
@@ -301,6 +301,7 @@ struct machdep_calls {
 	/* Idle handlers */
 	void		(*setup_idle)(void);
 	unsigned long	(*power7_sleep)(void);
+	unsigned long	(*power7_winkle)(void);
 };
 
 extern void e500_idle(void);
diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index a5139ea..3358f09 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -158,6 +158,9 @@ struct paca_struct {
 	 * early exception handler for use by high level C handler
 	 */
 	struct opal_machine_check_event *opal_mc_evt;
+
+	/* Flag to distinguish b/w sleep and winkle */
+	u8 offline_state;
 #endif
 #ifdef CONFIG_PPC_BOOK3S_64
 	/* Exclusive emergency stack pointer for machine check exception. */
diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
index 6f85362..5155be7 100644
--- a/arch/powerpc/include/asm/ppc-opcode.h
+++ b/arch/powerpc/include/asm/ppc-opcode.h
@@ -194,6 +194,7 @@
 
 #define PPC_INST_NAP			0x4c000364
 #define PPC_INST_SLEEP			0x4c0003a4
+#define PPC_INST_WINKLE			0x4c0003e4
 
 /* A2 specific instructions */
 #define PPC_INST_ERATWE			0x7c0001a6
@@ -374,6 +375,7 @@
 
 #define PPC_NAP			stringify_in_c(.long PPC_INST_NAP)
 #define PPC_SLEEP		stringify_in_c(.long PPC_INST_SLEEP)
+#define PPC_WINKLE		stringify_in_c(.long PPC_INST_WINKLE)
 
 /* BHRB instructions */
 #define PPC_CLRBHRB		stringify_in_c(.long PPC_INST_CLRBHRB)
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 41953cd..00e3df9 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -455,6 +455,8 @@ extern void arch_setup_idle(void);
 extern void power7_nap(int check_irq);
 extern unsigned long power7_sleep(void);
 extern unsigned long __power7_sleep(void);
+extern unsigned long power7_winkle(void);
+extern unsigned long __power7_winkle(void);
 extern void flush_instruction_cache(void);
 extern void hard_reset_now(void);
 extern void poweroff_now(void);
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 9d7dede..ea98817 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -731,6 +731,7 @@ int main(void)
 	DEFINE(OPAL_MC_SRR0, offsetof(struct opal_machine_check_event, srr0));
 	DEFINE(OPAL_MC_SRR1, offsetof(struct opal_machine_check_event, srr1));
 	DEFINE(PACA_OPAL_MC_EVT, offsetof(struct paca_struct, opal_mc_evt));
+	DEFINE(PACAOFFLINESTATE, offsetof(struct paca_struct, offline_state));
 #endif
 
 	return 0;
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index c64f3cc0..6c6db2b 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -133,8 +133,8 @@ BEGIN_FTR_SECTION
 	b	power7_wakeup_noloss
 2:	b	power7_wakeup_loss
 
-	/* Fast Sleep wakeup on PowerNV */
-8:	b 	power7_wakeup_tb_loss
+	/* Fast Sleep / Winkle wakeup on PowerNV */
+8:	b 	power7_wakeup_hv_state_loss
 
 9:
 END_FTR_SECTION_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
diff --git a/arch/powerpc/kernel/idle.c b/arch/powerpc/kernel/idle.c
index 1f268e0..ed46217 100644
--- a/arch/powerpc/kernel/idle.c
+++ b/arch/powerpc/kernel/idle.c
@@ -98,6 +98,17 @@ unsigned long power7_sleep(void)
 	return ret;
 }
 
+unsigned long power7_winkle(void)
+{
+	unsigned long ret;
+
+	if (ppc_md.power7_winkle)
+		ret = ppc_md.power7_winkle();
+	else
+		ret = __power7_winkle();
+	return ret;
+}
+
 int powersave_nap;
 
 #ifdef CONFIG_SYSCTL
diff --git a/arch/powerpc/kernel/idle_power7.S b/arch/powerpc/kernel/idle_power7.S
index c3481c9..87b2556 100644
--- a/arch/powerpc/kernel/idle_power7.S
+++ b/arch/powerpc/kernel/idle_power7.S
@@ -18,6 +18,13 @@
 #include <asm/hw_irq.h>
 #include <asm/kvm_book3s_asm.h>
 #include <asm/opal.h>
+#include <asm/mmu-hash64.h>
+
+/*
+ * Use volatile GPRs' space to save essential SPRs before entering winkle
+ */
+#define _SDR1	GPR3
+#define _TSCR	GPR4
 
 #undef DEBUG
 
@@ -39,6 +46,7 @@
  * Pass requested state in r3:
  * 	0 - nap
  * 	1 - sleep
+ *	2 - winkle
  *
  * To check IRQ_HAPPENED in r4
  * 	0 - don't check
@@ -109,9 +117,27 @@ _GLOBAL(power7_enter_nap_mode)
 #endif
 	cmpwi	cr0,r3,1
 	beq	2f
+	cmpwi	cr0,r3,2
+	beq	3f
 	IDLE_STATE_ENTER_SEQ(PPC_NAP)
 	/* No return */
-2:	IDLE_STATE_ENTER_SEQ(PPC_SLEEP)
+2:
+	li	r4,1
+	stb	r4,PACAOFFLINESTATE(r13)
+	IDLE_STATE_ENTER_SEQ(PPC_SLEEP)
+	/* No return */
+
+3:
+	mfspr   r4,SPRN_SDR1
+	std     r4,_SDR1(r1)
+
+	mfspr   r4,SPRN_TSCR
+	std     r4,_TSCR(r1)
+
+	/* Enter winkle */
+        li      r4,0
+        stb     r4,PACAOFFLINESTATE(r13)
+	IDLE_STATE_ENTER_SEQ(PPC_WINKLE)
 	/* No return */
 
 _GLOBAL(power7_idle)
@@ -187,6 +213,59 @@ ALT_FTR_SECTION_END_NESTED_IFSET(CPU_FTR_ARCH_207S, 66);		\
 20:	nop;
 
 
+_GLOBAL(__power7_winkle)
+	li	r3,2
+	li	r4,1
+	b	power7_powersave_common
+	/* No return */
+
+_GLOBAL(power7_wakeup_hv_state_loss)
+	/* Check paca flag to diffentiate b/w fast sleep and winkle */
+	lbz	r4,PACAOFFLINESTATE(13)
+	cmpwi	cr0,r4,0
+	bne	power7_wakeup_tb_loss
+
+	ld      r2,PACATOC(r13);
+	ld      r1,PACAR1(r13)
+
+	bl 	__restore_cpu_power8
+
+	/* Time base re-sync */
+	li	r3,OPAL_RESYNC_TIMEBASE
+	bl	opal_call_realmode;
+
+	/* Restore SLB  from PACA */
+	ld	r8,PACA_SLBSHADOWPTR(r13)
+
+	.rept	SLB_NUM_BOLTED
+	li	r3, SLBSHADOW_SAVEAREA
+	LDX_BE	r5, r8, r3
+	addi	r3, r3, 8
+	LDX_BE	r6, r8, r3
+	andis.	r7,r5,SLB_ESID_V@h
+	beq	1f
+	slbmte	r6,r5
+1:	addi	r8,r8,16
+	.endr
+
+	ld      r4,_SDR1(r1)
+	mtspr   SPRN_SDR1,r4
+
+	ld      r4,_TSCR(r1)
+	mtspr   SPRN_TSCR,r4
+
+	REST_NVGPRS(r1)
+	REST_GPR(2, r1)
+	ld      r3,_CCR(r1)
+	ld      r4,_MSR(r1)
+	ld      r5,_NIP(r1)
+	addi    r1,r1,INT_FRAME_SIZE
+	mtcr    r3
+	mfspr   r3,SPRN_SRR1            /* Return SRR1 */
+	mtspr   SPRN_SRR1,r4
+	mtspr   SPRN_SRR0,r5
+	rfid
+
 _GLOBAL(power7_wakeup_tb_loss)
 	ld	r2,PACATOC(r13);
 	ld	r1,PACAR1(r13)
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index 9d9a898..f45b52d 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -370,6 +370,29 @@ static unsigned long pnv_power7_sleep(void)
 	return srr1;
 }
 
+/*
+ * We need to keep track of offline cpus also for calling
+ * fastsleep workaround appropriately
+ */
+static unsigned long pnv_power7_winkle(void)
+{
+	int cpu, primary_thread;
+	unsigned long srr1;
+
+	cpu = smp_processor_id();
+	primary_thread = cpu_first_thread_sibling(cpu);
+
+	if (need_fastsleep_workaround) {
+		pnv_apply_fastsleep_workaround(1, primary_thread);
+		srr1 = __power7_winkle();
+		pnv_apply_fastsleep_workaround(0, primary_thread);
+	} else {
+			srr1 = __power7_winkle();
+	}
+	return srr1;
+}
+
+
 static void __init pnv_setup_machdep_opal(void)
 {
 	ppc_md.get_boot_time = opal_get_boot_time;
@@ -384,6 +407,7 @@ static void __init pnv_setup_machdep_opal(void)
 	ppc_md.handle_hmi_exception = opal_handle_hmi_exception;
 	ppc_md.setup_idle = pnv_setup_idle;
 	ppc_md.power7_sleep = pnv_power7_sleep;
+	ppc_md.power7_winkle = pnv_power7_winkle;
 }
 
 #ifdef CONFIG_PPC_POWERNV_RTAS
-- 
1.9.0

[PATCH 8/9] powerpc/powernv: Discover and enable winkle

[Shreyas B. Prabhu]

Discover winkle from device tree. If supported make OPAL calls
necessary to save HIDs, HMEER, HSPRG0 and LPCR.
Also make OPAL call when the HID0 value is modified during
split/unsplit of cores.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/opal.h          |  1 +
 arch/powerpc/platforms/powernv/powernv.h |  1 +
 arch/powerpc/platforms/powernv/setup.c   | 75 ++++++++++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/subcore.c | 15 +++++++
 4 files changed, 92 insertions(+)

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index d376020..a77957f 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -778,6 +778,7 @@ extern struct device_node *opal_node;
 #define IDLE_INST_NAP	0x00010000 /* nap instruction can be used */
 #define IDLE_INST_SLEEP	0x00020000 /* sleep instruction can be used */
 #define IDLE_INST_SLEEP_ER1	0x00080000 /* Use sleep with work around*/
+#define IDLE_INST_WINKLE	0x00040000 /* winkle instruction can be used */
 
 /* API functions */
 int64_t opal_invalid_call(void);
diff --git a/arch/powerpc/platforms/powernv/powernv.h b/arch/powerpc/platforms/powernv/powernv.h
index 31ece13..76b37f8 100644
--- a/arch/powerpc/platforms/powernv/powernv.h
+++ b/arch/powerpc/platforms/powernv/powernv.h
@@ -27,6 +27,7 @@ static inline int pnv_pci_dma_set_mask(struct pci_dev *pdev, u64 dma_mask)
 
 #define IDLE_USE_NAP		(1UL << 0)
 #define IDLE_USE_SLEEP		(1UL << 1)
+#define IDLE_USE_WINKLE                (1UL << 3)
 
 extern unsigned int pnv_get_supported_cpuidle_states(void);
 
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index f45b52d..13c5e49 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -273,6 +273,65 @@ unsigned int pnv_get_supported_cpuidle_states(void)
 	return supported_cpuidle_states;
 }
 
+int pnv_save_sprs_for_winkle(void)
+{
+	int cpu;
+	int rc;
+
+	/*
+	* hid0, hid1, hid4, hid5, hmeer and lpcr values are symmetric accross
+	* all cpus at boot. Get these reg values of current cpu and use the
+	* same accross all cpus.
+	*/
+	uint64_t lpcr_val = mfspr(SPRN_LPCR);
+	uint64_t hid0_val = mfspr(SPRN_HID0);
+	uint64_t hid1_val = mfspr(SPRN_HID1);
+	uint64_t hid4_val = mfspr(SPRN_HID4);
+	uint64_t hid5_val = mfspr(SPRN_HID5);
+	uint64_t hmeer_val = mfspr(SPRN_HMEER);
+
+	for_each_possible_cpu(cpu) {
+		uint64_t pir = get_hard_smp_processor_id(cpu);
+		uint64_t local_paca_ptr = (uint64_t)&paca[cpu];
+
+		rc = opal_slw_set_reg(pir, SPRN_HSPRG0, local_paca_ptr);
+		if (rc != 0)
+			return rc;
+
+		rc = opal_slw_set_reg(pir, SPRN_LPCR, lpcr_val);
+		if (rc != 0)
+			return rc;
+
+		/* HIDs are per core registers */
+		if (cpu_thread_in_core(cpu) == 0) {
+
+			rc = opal_slw_set_reg(pir, SPRN_HMEER, hmeer_val);
+			if (rc != 0)
+				return rc;
+
+			rc = opal_slw_set_reg(pir, SPRN_HID0, hid0_val);
+			if (rc != 0)
+				return rc;
+
+			rc = opal_slw_set_reg(pir, SPRN_HID1, hid1_val);
+			if (rc != 0)
+				return rc;
+
+			rc = opal_slw_set_reg(pir, SPRN_HID4, hid4_val);
+			if (rc != 0)
+				return rc;
+
+			rc = opal_slw_set_reg(pir, SPRN_HID5, hid5_val);
+			if (rc != 0)
+				return rc;
+
+		}
+
+	}
+
+	return 0;
+
+}
 static int __init pnv_probe_idle_states(void)
 {
 	struct device_node *power_mgt;
@@ -318,6 +377,22 @@ static int __init pnv_probe_idle_states(void)
 			supported_cpuidle_states |= IDLE_USE_SLEEP;
 			need_fastsleep_workaround = 1;
 		}
+
+		if (flags & IDLE_INST_WINKLE) {
+			/*
+			 * If winkle is supported, save HSPRG0, HIDs and LPCR
+			 * contents via OPAL. Enable winkle only if this
+			 * succeeds.
+			 */
+			int opal_ret_val = pnv_save_sprs_for_winkle();
+
+			if (!opal_ret_val)
+				supported_cpuidle_states |= IDLE_USE_WINKLE;
+			else
+				pr_warn("opal: opal_slw_set_reg failed with rc=%d, disabling winkle\n",
+						opal_ret_val);
+		}
+
 	}
 
 	return 0;
diff --git a/arch/powerpc/platforms/powernv/subcore.c b/arch/powerpc/platforms/powernv/subcore.c
index 894ecb3..47c70666e 100644
--- a/arch/powerpc/platforms/powernv/subcore.c
+++ b/arch/powerpc/platforms/powernv/subcore.c
@@ -24,6 +24,7 @@
 #include <asm/smp.h>
 
 #include "subcore.h"
+#include "powernv.h"
 
 
 /*
@@ -159,6 +160,18 @@ static void wait_for_sync_step(int step)
 	mb();
 }
 
+static void update_hid_in_slw(u64 hid0)
+{
+	u64 idle_states = pnv_get_supported_cpuidle_states();
+
+	if (idle_states & IDLE_USE_WINKLE) {
+		/* OPAL call to patch slw with the new HID0 value */
+		u64 cpu_pir = hard_smp_processor_id();
+
+		opal_slw_set_reg(cpu_pir, SPRN_HID0, hid0);
+	}
+}
+
 static void unsplit_core(void)
 {
 	u64 hid0, mask;
@@ -178,6 +191,7 @@ static void unsplit_core(void)
 	hid0 = mfspr(SPRN_HID0);
 	hid0 &= ~HID0_POWER8_DYNLPARDIS;
 	mtspr(SPRN_HID0, hid0);
+	update_hid_in_slw(hid0);
 
 	while (mfspr(SPRN_HID0) & mask)
 		cpu_relax();
@@ -214,6 +228,7 @@ static void split_core(int new_mode)
 	hid0  = mfspr(SPRN_HID0);
 	hid0 |= HID0_POWER8_DYNLPARDIS | split_parms[i].value;
 	mtspr(SPRN_HID0, hid0);
+	update_hid_in_slw(hid0);
 
 	/* Wait for it to happen */
 	while (!(mfspr(SPRN_HID0) & split_parms[i].mask))
-- 
1.9.0

[PATCH 9/9] powerpc/powernv: Enter deepest supported idle state in offline

[Shreyas B. Prabhu]

Enter winkle during offline if supported, else revert to sleep or nap.

Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/smp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 3ad31d2..e3fc2c9 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -169,8 +169,10 @@ static void pnv_smp_cpu_kill_self(void)
 	while (!generic_check_cpu_restart(cpu)) {
 		ppc64_runlatch_off();
 
-		/* If sleep is supported, go to sleep, instead of nap */
-		if (idle_states & IDLE_USE_SLEEP)
+		/* Go to deepest supported idle state */
+		if (idle_states & IDLE_USE_WINKLE)
+			power7_winkle();
+		else if (idle_states & IDLE_USE_SLEEP)
 			power7_sleep();
 		else
 			power7_nap(1);
-- 
1.9.0

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B Prabhu]

Hi,
Any updates on this patch series?

On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
> Fast sleep is an idle state, where the core and the L1 and L2
> caches are brought down to a threshold voltage. This also means that
> the communication between L2 and L3 caches have to be fenced. However
> the current P8 chips have a bug wherein this fencing between L2 and
> L3 caches get delayed by a cpu cycle. This can delay L3 response to
> the other cpus if they request for data during this time. Thus they
> would fetch the same data from the memory which could lead to data
> corruption if L3 cache is not flushed.
> Patch 4 adds support to work around this.
> 
> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
> off. While it offers higher power savings, it is at the cost of losing
> hypervisor register state and higher latency.
> Patch 5-9 adds support for winkle and uses it for offline cpus.
> 
> Patch 1 - Moves parameters required discover idle states to a location 
> common to both cpuidle driver and powernv core code
> Patch 2 - Populates idle state details from device tree
> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
> 
> 
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> Cc: Rob Herring <robh+dt@kernel.org>
> Cc: Grant Likely <grant.likely@linaro.org>
> Cc: devicetree@vger.kernel.org
> Cc: linux-pm@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> 
> Preeti U Murthy (2):
>   cpuidle/powernv: Populate cpuidle state details by querying the
>     device-tree
>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
> 
> Shreyas B. Prabhu (6):
>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
>     fast-sleep
>   powerpc/powernv: Add OPAL call to save and restore
>   powerpc: Adding macro for accessing Thread Switch Control Register
>   powerpc/powernv: Add winkle infrastructure
>   powerpc/powernv: Discover and enable winkle
>   powerpc/powernv: Enter deepest supported idle state in offline
> 
> Srivatsa S. Bhat (1):
>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
> 
>  arch/powerpc/include/asm/machdep.h             |   4 +
>  arch/powerpc/include/asm/opal.h                |  10 ++
>  arch/powerpc/include/asm/paca.h                |   3 +
>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
>  arch/powerpc/include/asm/processor.h           |   6 +-
>  arch/powerpc/include/asm/reg.h                 |   1 +
>  arch/powerpc/kernel/asm-offsets.c              |   1 +
>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
>  arch/powerpc/kernel/idle.c                     |  30 ++++
>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
>  16 files changed, 439 insertions(+), 33 deletions(-)
> 

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B Prabhu]

Hi,

In this patch series we use winkle for offlined cores. I successfully
tested the working of this with subcore functionality.

Test scenario was as follows:
1. Set SMT mode to 1, Set subores-per-core to 1
2. Offline a core, in this case cpu 32 (sending it to winkle)
3. Set subcores-per-core to 4
4. Online the core
5. Start a guest (Topology 1 core 2 threads) on a subcore, in this case
on cpu 36

This works without any glitch.

Thanks,
Shreyas

On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
> Fast sleep is an idle state, where the core and the L1 and L2
> caches are brought down to a threshold voltage. This also means that
> the communication between L2 and L3 caches have to be fenced. However
> the current P8 chips have a bug wherein this fencing between L2 and
> L3 caches get delayed by a cpu cycle. This can delay L3 response to
> the other cpus if they request for data during this time. Thus they
> would fetch the same data from the memory which could lead to data
> corruption if L3 cache is not flushed.
> Patch 4 adds support to work around this.
> 
> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
> off. While it offers higher power savings, it is at the cost of losing
> hypervisor register state and higher latency.
> Patch 5-9 adds support for winkle and uses it for offline cpus.
> 
> Patch 1 - Moves parameters required discover idle states to a location 
> common to both cpuidle driver and powernv core code
> Patch 2 - Populates idle state details from device tree
> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
> 
> 
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> Cc: Rob Herring <robh+dt@kernel.org>
> Cc: Grant Likely <grant.likely@linaro.org>
> Cc: devicetree@vger.kernel.org
> Cc: linux-pm@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> 
> Preeti U Murthy (2):
>   cpuidle/powernv: Populate cpuidle state details by querying the
>     device-tree
>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
> 
> Shreyas B. Prabhu (6):
>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
>     fast-sleep
>   powerpc/powernv: Add OPAL call to save and restore
>   powerpc: Adding macro for accessing Thread Switch Control Register
>   powerpc/powernv: Add winkle infrastructure
>   powerpc/powernv: Discover and enable winkle
>   powerpc/powernv: Enter deepest supported idle state in offline
> 
> Srivatsa S. Bhat (1):
>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
> 
>  arch/powerpc/include/asm/machdep.h             |   4 +
>  arch/powerpc/include/asm/opal.h                |  10 ++
>  arch/powerpc/include/asm/paca.h                |   3 +
>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
>  arch/powerpc/include/asm/processor.h           |   6 +-
>  arch/powerpc/include/asm/reg.h                 |   1 +
>  arch/powerpc/kernel/asm-offsets.c              |   1 +
>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
>  arch/powerpc/kernel/idle.c                     |  30 ++++
>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
>  16 files changed, 439 insertions(+), 33 deletions(-)
> 

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B Prabhu]

Hi,
Any updates on this patch series?

On Thursday 18 September 2014 08:41 AM, Shreyas B Prabhu wrote:
> Hi,
> 
> In this patch series we use winkle for offlined cores. I successfully
> tested the working of this with subcore functionality.
> 
> Test scenario was as follows:
> 1. Set SMT mode to 1, Set subores-per-core to 1
> 2. Offline a core, in this case cpu 32 (sending it to winkle)
> 3. Set subcores-per-core to 4
> 4. Online the core
> 5. Start a guest (Topology 1 core 2 threads) on a subcore, in this case
> on cpu 36
> 
> This works without any glitch.
> 
> Thanks,
> Shreyas
> 
> On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
>> Fast sleep is an idle state, where the core and the L1 and L2
>> caches are brought down to a threshold voltage. This also means that
>> the communication between L2 and L3 caches have to be fenced. However
>> the current P8 chips have a bug wherein this fencing between L2 and
>> L3 caches get delayed by a cpu cycle. This can delay L3 response to
>> the other cpus if they request for data during this time. Thus they
>> would fetch the same data from the memory which could lead to data
>> corruption if L3 cache is not flushed.
>> Patch 4 adds support to work around this.
>>
>> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
>> off. While it offers higher power savings, it is at the cost of losing
>> hypervisor register state and higher latency.
>> Patch 5-9 adds support for winkle and uses it for offline cpus.
>>
>> Patch 1 - Moves parameters required discover idle states to a location 
>> common to both cpuidle driver and powernv core code
>> Patch 2 - Populates idle state details from device tree
>> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
>>
>>
>> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>> Cc: Paul Mackerras <paulus@samba.org>
>> Cc: Michael Ellerman <mpe@ellerman.id.au>
>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
>> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
>> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
>> Cc: Rob Herring <robh+dt@kernel.org>
>> Cc: Grant Likely <grant.likely@linaro.org>
>> Cc: devicetree@vger.kernel.org
>> Cc: linux-pm@vger.kernel.org
>> Cc: linuxppc-dev@lists.ozlabs.org
>>
>> Preeti U Murthy (2):
>>   cpuidle/powernv: Populate cpuidle state details by querying the
>>     device-tree
>>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
>>
>> Shreyas B. Prabhu (6):
>>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
>>     fast-sleep
>>   powerpc/powernv: Add OPAL call to save and restore
>>   powerpc: Adding macro for accessing Thread Switch Control Register
>>   powerpc/powernv: Add winkle infrastructure
>>   powerpc/powernv: Discover and enable winkle
>>   powerpc/powernv: Enter deepest supported idle state in offline
>>
>> Srivatsa S. Bhat (1):
>>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
>>
>>  arch/powerpc/include/asm/machdep.h             |   4 +
>>  arch/powerpc/include/asm/opal.h                |  10 ++
>>  arch/powerpc/include/asm/paca.h                |   3 +
>>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
>>  arch/powerpc/include/asm/processor.h           |   6 +-
>>  arch/powerpc/include/asm/reg.h                 |   1 +
>>  arch/powerpc/kernel/asm-offsets.c              |   1 +
>>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
>>  arch/powerpc/kernel/idle.c                     |  30 ++++
>>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
>>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
>>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
>>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
>>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
>>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
>>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
>>  16 files changed, 439 insertions(+), 33 deletions(-)
>>

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Rafael J. Wysocki]

On Monday, September 29, 2014 03:53:06 PM Shreyas B Prabhu wrote:
> Hi,
> Any updates on this patch series?

I have a couple of patches from there in my tree it seems.  Please have a look
at linux-pm.git/linux-next and please let me know if that's the case.


> On Thursday 18 September 2014 08:41 AM, Shreyas B Prabhu wrote:
> > Hi,
> > 
> > In this patch series we use winkle for offlined cores. I successfully
> > tested the working of this with subcore functionality.
> > 
> > Test scenario was as follows:
> > 1. Set SMT mode to 1, Set subores-per-core to 1
> > 2. Offline a core, in this case cpu 32 (sending it to winkle)
> > 3. Set subcores-per-core to 4
> > 4. Online the core
> > 5. Start a guest (Topology 1 core 2 threads) on a subcore, in this case
> > on cpu 36
> > 
> > This works without any glitch.
> > 
> > Thanks,
> > Shreyas
> > 
> > On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
> >> Fast sleep is an idle state, where the core and the L1 and L2
> >> caches are brought down to a threshold voltage. This also means that
> >> the communication between L2 and L3 caches have to be fenced. However
> >> the current P8 chips have a bug wherein this fencing between L2 and
> >> L3 caches get delayed by a cpu cycle. This can delay L3 response to
> >> the other cpus if they request for data during this time. Thus they
> >> would fetch the same data from the memory which could lead to data
> >> corruption if L3 cache is not flushed.
> >> Patch 4 adds support to work around this.
> >>
> >> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
> >> off. While it offers higher power savings, it is at the cost of losing
> >> hypervisor register state and higher latency.
> >> Patch 5-9 adds support for winkle and uses it for offline cpus.
> >>
> >> Patch 1 - Moves parameters required discover idle states to a location 
> >> common to both cpuidle driver and powernv core code
> >> Patch 2 - Populates idle state details from device tree
> >> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
> >>
> >>
> >> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> >> Cc: Paul Mackerras <paulus@samba.org>
> >> Cc: Michael Ellerman <mpe@ellerman.id.au>
> >> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
> >> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
> >> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> >> Cc: Rob Herring <robh+dt@kernel.org>
> >> Cc: Grant Likely <grant.likely@linaro.org>
> >> Cc: devicetree@vger.kernel.org
> >> Cc: linux-pm@vger.kernel.org
> >> Cc: linuxppc-dev@lists.ozlabs.org
> >>
> >> Preeti U Murthy (2):
> >>   cpuidle/powernv: Populate cpuidle state details by querying the
> >>     device-tree
> >>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
> >>
> >> Shreyas B. Prabhu (6):
> >>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
> >>     fast-sleep
> >>   powerpc/powernv: Add OPAL call to save and restore
> >>   powerpc: Adding macro for accessing Thread Switch Control Register
> >>   powerpc/powernv: Add winkle infrastructure
> >>   powerpc/powernv: Discover and enable winkle
> >>   powerpc/powernv: Enter deepest supported idle state in offline
> >>
> >> Srivatsa S. Bhat (1):
> >>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
> >>
> >>  arch/powerpc/include/asm/machdep.h             |   4 +
> >>  arch/powerpc/include/asm/opal.h                |  10 ++
> >>  arch/powerpc/include/asm/paca.h                |   3 +
> >>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
> >>  arch/powerpc/include/asm/processor.h           |   6 +-
> >>  arch/powerpc/include/asm/reg.h                 |   1 +
> >>  arch/powerpc/kernel/asm-offsets.c              |   1 +
> >>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
> >>  arch/powerpc/kernel/idle.c                     |  30 ++++
> >>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
> >>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
> >>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
> >>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
> >>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
> >>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
> >>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
> >>  16 files changed, 439 insertions(+), 33 deletions(-)
> >>
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B Prabhu]

Hi Rafael,

On Tuesday 30 September 2014 04:58 AM, Rafael J. Wysocki wrote:
> On Monday, September 29, 2014 03:53:06 PM Shreyas B Prabhu wrote:
>> Hi,
>> Any updates on this patch series?
> 
> I have a couple of patches from there in my tree it seems.  Please have a look
> at linux-pm.git/linux-next and please let me know if that's the case.
> 

I checked linux-pm.git/linux-net (Last commit 067c17382165). None of the patches
in this series are present in the tree. 


> 
>> On Thursday 18 September 2014 08:41 AM, Shreyas B Prabhu wrote:
>>> Hi,
>>>
>>> In this patch series we use winkle for offlined cores. I successfully
>>> tested the working of this with subcore functionality.
>>>
>>> Test scenario was as follows:
>>> 1. Set SMT mode to 1, Set subores-per-core to 1
>>> 2. Offline a core, in this case cpu 32 (sending it to winkle)
>>> 3. Set subcores-per-core to 4
>>> 4. Online the core
>>> 5. Start a guest (Topology 1 core 2 threads) on a subcore, in this case
>>> on cpu 36
>>>
>>> This works without any glitch.
>>>
>>> Thanks,
>>> Shreyas
>>>
>>> On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
>>>> Fast sleep is an idle state, where the core and the L1 and L2
>>>> caches are brought down to a threshold voltage. This also means that
>>>> the communication between L2 and L3 caches have to be fenced. However
>>>> the current P8 chips have a bug wherein this fencing between L2 and
>>>> L3 caches get delayed by a cpu cycle. This can delay L3 response to
>>>> the other cpus if they request for data during this time. Thus they
>>>> would fetch the same data from the memory which could lead to data
>>>> corruption if L3 cache is not flushed.
>>>> Patch 4 adds support to work around this.
>>>>
>>>> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
>>>> off. While it offers higher power savings, it is at the cost of losing
>>>> hypervisor register state and higher latency.
>>>> Patch 5-9 adds support for winkle and uses it for offline cpus.
>>>>
>>>> Patch 1 - Moves parameters required discover idle states to a location 
>>>> common to both cpuidle driver and powernv core code
>>>> Patch 2 - Populates idle state details from device tree
>>>> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
>>>>
>>>>
>>>> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>>>> Cc: Paul Mackerras <paulus@samba.org>
>>>> Cc: Michael Ellerman <mpe@ellerman.id.au>
>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
>>>> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
>>>> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
>>>> Cc: Rob Herring <robh+dt@kernel.org>
>>>> Cc: Grant Likely <grant.likely@linaro.org>
>>>> Cc: devicetree@vger.kernel.org
>>>> Cc: linux-pm@vger.kernel.org
>>>> Cc: linuxppc-dev@lists.ozlabs.org
>>>>
>>>> Preeti U Murthy (2):
>>>>   cpuidle/powernv: Populate cpuidle state details by querying the
>>>>     device-tree
>>>>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
>>>>
>>>> Shreyas B. Prabhu (6):
>>>>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
>>>>     fast-sleep
>>>>   powerpc/powernv: Add OPAL call to save and restore
>>>>   powerpc: Adding macro for accessing Thread Switch Control Register
>>>>   powerpc/powernv: Add winkle infrastructure
>>>>   powerpc/powernv: Discover and enable winkle
>>>>   powerpc/powernv: Enter deepest supported idle state in offline
>>>>
>>>> Srivatsa S. Bhat (1):
>>>>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
>>>>
>>>>  arch/powerpc/include/asm/machdep.h             |   4 +
>>>>  arch/powerpc/include/asm/opal.h                |  10 ++
>>>>  arch/powerpc/include/asm/paca.h                |   3 +
>>>>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
>>>>  arch/powerpc/include/asm/processor.h           |   6 +-
>>>>  arch/powerpc/include/asm/reg.h                 |   1 +
>>>>  arch/powerpc/kernel/asm-offsets.c              |   1 +
>>>>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
>>>>  arch/powerpc/kernel/idle.c                     |  30 ++++
>>>>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
>>>>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
>>>>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
>>>>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
>>>>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
>>>>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
>>>>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
>>>>  16 files changed, 439 insertions(+), 33 deletions(-)
>>>>
>>
> 

Re: [PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Rafael J. Wysocki]

On Tuesday, September 30, 2014 01:42:05 PM Shreyas B Prabhu wrote:
> Hi Rafael,
> 
> On Tuesday 30 September 2014 04:58 AM, Rafael J. Wysocki wrote:
> > On Monday, September 29, 2014 03:53:06 PM Shreyas B Prabhu wrote:
> >> Hi,
> >> Any updates on this patch series?
> > 
> > I have a couple of patches from there in my tree it seems.  Please have a look
> > at linux-pm.git/linux-next and please let me know if that's the case.
> > 
> 
> I checked linux-pm.git/linux-net (Last commit 067c17382165). None of the patches
> in this series are present in the tree. 

OK, thanks for checking.

Would you mind resending it with any ACKs or Reviewed-by tags received so far?


> >> On Thursday 18 September 2014 08:41 AM, Shreyas B Prabhu wrote:
> >>> Hi,
> >>>
> >>> In this patch series we use winkle for offlined cores. I successfully
> >>> tested the working of this with subcore functionality.
> >>>
> >>> Test scenario was as follows:
> >>> 1. Set SMT mode to 1, Set subores-per-core to 1
> >>> 2. Offline a core, in this case cpu 32 (sending it to winkle)
> >>> 3. Set subcores-per-core to 4
> >>> 4. Online the core
> >>> 5. Start a guest (Topology 1 core 2 threads) on a subcore, in this case
> >>> on cpu 36
> >>>
> >>> This works without any glitch.
> >>>
> >>> Thanks,
> >>> Shreyas
> >>>
> >>> On Monday 25 August 2014 11:31 PM, Shreyas B. Prabhu wrote:
> >>>> Fast sleep is an idle state, where the core and the L1 and L2
> >>>> caches are brought down to a threshold voltage. This also means that
> >>>> the communication between L2 and L3 caches have to be fenced. However
> >>>> the current P8 chips have a bug wherein this fencing between L2 and
> >>>> L3 caches get delayed by a cpu cycle. This can delay L3 response to
> >>>> the other cpus if they request for data during this time. Thus they
> >>>> would fetch the same data from the memory which could lead to data
> >>>> corruption if L3 cache is not flushed.
> >>>> Patch 4 adds support to work around this.
> >>>>
> >>>> 'Deep Winkle' is a deeper idle state where core and private L2 are powered
> >>>> off. While it offers higher power savings, it is at the cost of losing
> >>>> hypervisor register state and higher latency.
> >>>> Patch 5-9 adds support for winkle and uses it for offline cpus.
> >>>>
> >>>> Patch 1 - Moves parameters required discover idle states to a location 
> >>>> common to both cpuidle driver and powernv core code
> >>>> Patch 2 - Populates idle state details from device tree
> >>>> Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle
> >>>>
> >>>>
> >>>> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> >>>> Cc: Paul Mackerras <paulus@samba.org>
> >>>> Cc: Michael Ellerman <mpe@ellerman.id.au>
> >>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >>>> Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
> >>>> Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
> >>>> Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
> >>>> Cc: Rob Herring <robh+dt@kernel.org>
> >>>> Cc: Grant Likely <grant.likely@linaro.org>
> >>>> Cc: devicetree@vger.kernel.org
> >>>> Cc: linux-pm@vger.kernel.org
> >>>> Cc: linuxppc-dev@lists.ozlabs.org
> >>>>
> >>>> Preeti U Murthy (2):
> >>>>   cpuidle/powernv: Populate cpuidle state details by querying the
> >>>>     device-tree
> >>>>   powerpc/powernv/cpuidle: Add workaround to enable fastsleep
> >>>>
> >>>> Shreyas B. Prabhu (6):
> >>>>   powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
> >>>>     fast-sleep
> >>>>   powerpc/powernv: Add OPAL call to save and restore
> >>>>   powerpc: Adding macro for accessing Thread Switch Control Register
> >>>>   powerpc/powernv: Add winkle infrastructure
> >>>>   powerpc/powernv: Discover and enable winkle
> >>>>   powerpc/powernv: Enter deepest supported idle state in offline
> >>>>
> >>>> Srivatsa S. Bhat (1):
> >>>>   powerpc/powernv: Enable Offline CPUs to enter deep idle states
> >>>>
> >>>>  arch/powerpc/include/asm/machdep.h             |   4 +
> >>>>  arch/powerpc/include/asm/opal.h                |  10 ++
> >>>>  arch/powerpc/include/asm/paca.h                |   3 +
> >>>>  arch/powerpc/include/asm/ppc-opcode.h          |   2 +
> >>>>  arch/powerpc/include/asm/processor.h           |   6 +-
> >>>>  arch/powerpc/include/asm/reg.h                 |   1 +
> >>>>  arch/powerpc/kernel/asm-offsets.c              |   1 +
> >>>>  arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
> >>>>  arch/powerpc/kernel/idle.c                     |  30 ++++
> >>>>  arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
> >>>>  arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
> >>>>  arch/powerpc/platforms/powernv/powernv.h       |   8 +
> >>>>  arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
> >>>>  arch/powerpc/platforms/powernv/smp.c           |  13 +-
> >>>>  arch/powerpc/platforms/powernv/subcore.c       |  15 ++
> >>>>  drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
> >>>>  16 files changed, 439 insertions(+), 33 deletions(-)
> >>>>
> >>
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

[PATCH 0/9] powerpc/powernv: Support for fastsleep and winkle

[Shreyas B. Prabhu]

Fast sleep is an idle state, where the core and the L1 and L2
caches are brought down to a threshold voltage. This also means that
the communication between L2 and L3 caches have to be fenced. However
the current P8 chips have a bug wherein this fencing between L2 and
L3 caches get delayed by a cpu cycle. This can delay L3 response to
the other cpus if they request for data during this time. Thus they
would fetch the same data from the memory which could lead to data
corruption if L3 cache is not flushed.
Patch 4 adds support to work around this.

'Deep Winkle' is a deeper idle state where core and private L2 are powered
off. While it offers higher power savings, it is at the cost of losing
hypervisor register state and higher latency.
Patch 5-9 adds support for winkle and uses it for offline cpus.

Patch 1 - Moves parameters required discover idle states to a location 
common to both cpuidle driver and powernv core code
Patch 2 - Populates idle state details from device tree
Patch 3 - Enables cpus to run guest after waking up from fastsleep/winkle


Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Srivatsa S. Bhat <srivatsa@MIT.EDU>
Cc: Preeti U. Murthy <preeti@linux.vnet.ibm.com>
Cc: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Cc: Rob Herring <robh+dt@kernel.org>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: devicetree@vger.kernel.org
Cc: linux-pm@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org

Preeti U Murthy (2):
  cpuidle/powernv: Populate cpuidle state details by querying the
    device-tree
  powerpc/powernv/cpuidle: Add workaround to enable fastsleep

Shreyas B. Prabhu (6):
  powerpc/kvm/book3s_hv: Enable CPUs to run guest after waking up from
    fast-sleep
  powerpc/powernv: Add OPAL call to save and restore
  powerpc: Adding macro for accessing Thread Switch Control Register
  powerpc/powernv: Add winkle infrastructure
  powerpc/powernv: Discover and enable winkle
  powerpc/powernv: Enter deepest supported idle state in offline

Srivatsa S. Bhat (1):
  powerpc/powernv: Enable Offline CPUs to enter deep idle states

 arch/powerpc/include/asm/machdep.h             |   4 +
 arch/powerpc/include/asm/opal.h                |  10 ++
 arch/powerpc/include/asm/paca.h                |   3 +
 arch/powerpc/include/asm/ppc-opcode.h          |   2 +
 arch/powerpc/include/asm/processor.h           |   6 +-
 arch/powerpc/include/asm/reg.h                 |   1 +
 arch/powerpc/kernel/asm-offsets.c              |   1 +
 arch/powerpc/kernel/exceptions-64s.S           |  37 ++---
 arch/powerpc/kernel/idle.c                     |  30 ++++
 arch/powerpc/kernel/idle_power7.S              |  83 +++++++++-
 arch/powerpc/platforms/powernv/opal-wrappers.S |   2 +
 arch/powerpc/platforms/powernv/powernv.h       |   8 +
 arch/powerpc/platforms/powernv/setup.c         | 217 +++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/smp.c           |  13 +-
 arch/powerpc/platforms/powernv/subcore.c       |  15 ++
 drivers/cpuidle/cpuidle-powernv.c              |  40 ++++-
 16 files changed, 439 insertions(+), 33 deletions(-)

-- 
1.9.0