[PATCH 0/3] cpu: idle state framework for offline CPUs.

[PATCH 0/3] cpu: idle state framework for offline CPUs.

[Gautham R Shenoy]

Hi,

**** RFC not for inclusion ****

When we perform a CPU-Offline operation today, we do not put the CPU
into the most energy efficient state. On x86, it loops in hlt as opposed to
going to one of the low-power C-states. On pSeries, we call rtas_stop_self()
and hand over the vCPU back to the resource pool, thereby deallocating
the vCPU.

Thus, when applications or platforms desire to put a particular CPU
to an extended low-power state for a short while, currently they have to
piggy-back on scheduler heuristics such as sched_mc_powersavings or play with
exclusive Cpusets. The former does a good job based on the workload, but fails
to provide any guarentee that the CPU won't be used for the next <> seconds,
while the latter might conflict with the existing CPUsets configurations.

There were efforts to alleviate these problems and various proposals have been
put forth. They include putting the CPU to the deepest possible idle-state
when offlined [1], removing the desired CPU from the topmost-cpuset [2],
a driver which forces a high-priority idle thread to run on the desired CPU
thereby putting it to idle [3].

In this patch-series, we propose to extend the CPU-Hotplug infrastructure
and allow the system administrator to choose the desired state the CPU should
go to when it is offlined. We think this approach addresses the concerns about
determinism as well as transparency, since CPU-Hotplug already provides
notification mechanism which the userspace can listen to for any change
in the configuration and correspondingly readjust any previously set
cpu-affinities. Also, approaches such as [1] can make use of this
extended infrastructure instead of putting the CPU to an arbitrary C-state
when it is offlined, thereby providing the system administrator a rope to hang
himself with should he feel the need to do so.

This patch-series tries to achieve this by implementing an architecture
independent framework that exposes sysfs tunables to allow the
system-adminstrator to choose the offline-state of a CPU.

	/sys/devices/system/cpu/cpu<number>/available_offline_states
and
	/sys/devices/system/cpu/cpu<number>/preferred_offline_states

For the purpose of proof-of-concept, we've implemented the backend for
pSeries. For pSeries, we define two available_offline_states. They are:

	deallocate: This is default behaviour which on an offline, deallocates
	the vCPU by invoking rtas_stop_self() and hands it back to
	the resource pool.

	deactivate: This calls H_CEDE, which will request the hypervisor to
	idle the vCPU in the lowest power mode and give it back as soon as
	we need it.


Any feedback on the patchset will be immensely valuable.

References:
-----------
[1] Pallipadi, Venkatesh: x86: Make offline cpus to go to deepest idle state
using mwait (URL: http://lkml.org/lkml/2009/5/22/431)

[2] Li, Shaohua: cpuset: add new API to change cpuset top group's cpus
(URL: http://lkml.org/lkml/2009/5/19/54)

[3] Li, Shaohua: new ACPI processor driver to force CPUs idle
(URL: http://www.spinics.net/lists/linux-acpi/msg22863.html)


Changelog:
---

Gautham R Shenoy (3):
      pSeries: cpu: Cede CPU during a deactivate-offline
      cpu: Implement cpu-offline-state callbacks for pSeries.
      cpu: Offline state Framework.


 arch/powerpc/platforms/pseries/hotplug-cpu.c    |  160 ++++++++++++++++++++++-
 arch/powerpc/platforms/pseries/offline_driver.h |   17 ++
 arch/powerpc/platforms/pseries/plpar_wrappers.h |    6 +
 arch/powerpc/platforms/pseries/smp.c            |   18 ++-
 drivers/base/cpu.c                              |  111 ++++++++++++++++
 include/linux/cpu.h                             |   15 ++
 6 files changed, 319 insertions(+), 8 deletions(-)
 create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h

-- 
Thanks and Regards
gautham.

[PATCH 1/3] cpu: Offline state Framework.

[Gautham R Shenoy]

Provide an interface by which the system administrator can decide what state
should the CPU go to when it is offlined.

To query the available offline states, on needs to perform a read on:
/sys/devices/system/cpu/cpu<number>/available_offline_states

To query or set the preferred offline state for a particular CPU, one needs to
use the sysfs interface

/sys/devices/system/cpu/cpu<number>/preferred_offline_state

This patch implements the architecture independent bits of the
cpu-offline-state framework.

The architecture specific bits are expected to register the actual code which
implements the callbacks when the above mentioned sysfs interfaces are read or
written into. Thus the values provided by reading available_offline_states are
expected to vary with the architecture.

Signed-off-by: Gautham R Shenoy <ego@in.ibm.com>
---
 drivers/base/cpu.c  |  111 +++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/cpu.h |   15 +++++++
 2 files changed, 126 insertions(+), 0 deletions(-)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index e62a4cc..1a63de0 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -56,26 +56,137 @@ static ssize_t __ref store_online(struct sys_device *dev, struct sysdev_attribut
 }
 static SYSDEV_ATTR(online, 0644, show_online, store_online);
 
+static struct cpu_offline_driver *cpu_offline_driver;
+static SYSDEV_ATTR(available_offline_states, 0444, NULL, NULL);
+static SYSDEV_ATTR(preferred_offline_state, 0644, NULL, NULL);
+
+/* Should be called with cpu_add_remove_lock held */
+void cpu_offline_driver_add_cpu(struct sys_device *cpu_sys_dev)
+{
+	if (!cpu_offline_driver)
+		return;
+
+	sysdev_create_file(cpu_sys_dev, &attr_available_offline_states);
+	sysdev_create_file(cpu_sys_dev, &attr_preferred_offline_state);
+}
+
+/* Should be called with cpu_add_remove_lock held */
+void cpu_offline_driver_remove_cpu(struct sys_device *cpu_sys_dev)
+{
+	if (!cpu_offline_driver)
+		return;
+
+	sysdev_remove_file(cpu_sys_dev, &attr_available_offline_states);
+	sysdev_remove_file(cpu_sys_dev, &attr_preferred_offline_state);
+
+}
+
 static void __cpuinit register_cpu_control(struct cpu *cpu)
 {
 	sysdev_create_file(&cpu->sysdev, &attr_online);
+	cpu_offline_driver_add_cpu(&cpu->sysdev);
 }
+
 void unregister_cpu(struct cpu *cpu)
 {
 	int logical_cpu = cpu->sysdev.id;
 
 	unregister_cpu_under_node(logical_cpu, cpu_to_node(logical_cpu));
 
+	cpu_offline_driver_remove_cpu(&cpu->sysdev);
 	sysdev_remove_file(&cpu->sysdev, &attr_online);
 
 	sysdev_unregister(&cpu->sysdev);
 	per_cpu(cpu_sys_devices, logical_cpu) = NULL;
 	return;
 }
+
+static int __cpuinit
+cpu_driver_callback(struct notifier_block *nfb, unsigned long action,
+								void *hcpu)
+{
+	struct sys_device *cpu_sysdev = per_cpu(cpu_sys_devices,
+						(unsigned long)(hcpu));
+
+	switch (action) {
+	case CPU_DEAD:
+	case CPU_DEAD_FROZEN:
+		cpu_offline_driver_remove_cpu(cpu_sysdev);
+		break;
+
+	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
+		cpu_offline_driver_add_cpu(cpu_sysdev);
+		break;
+	default:
+		break;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block __cpuinitdata cpu_driver_notifier = {
+	.notifier_call = cpu_driver_callback,
+	.priority = 0
+};
+
+int register_cpu_offline_driver(struct cpu_offline_driver *arch_cpu_driver)
+{
+	int ret = 0;
+	cpu_maps_update_begin();
+
+	if (cpu_offline_driver != NULL) {
+		ret = -EEXIST;
+		goto out_unlock;
+	}
+
+	if (!(arch_cpu_driver->show_available_states &&
+	      arch_cpu_driver->show_preferred_state &&
+	      arch_cpu_driver->store_preferred_state)) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	attr_available_offline_states.show =
+		arch_cpu_driver->show_available_states;
+	attr_preferred_offline_state.show =
+		arch_cpu_driver->show_preferred_state;
+	attr_preferred_offline_state.store =
+		arch_cpu_driver->store_preferred_state;
+
+	cpu_offline_driver = arch_cpu_driver;
+
+out_unlock:
+	cpu_maps_update_done();
+	if (!ret)
+		register_cpu_notifier(&cpu_driver_notifier);
+	return ret;
+}
+
+void unregister_cpu_offline_driver(struct cpu_offline_driver *arch_cpu_driver)
+{
+	cpu_maps_update_begin();
+
+	if (!cpu_offline_driver) {
+		WARN_ON(1);
+		cpu_maps_update_done();
+		return;
+	}
+
+	cpu_offline_driver = NULL;
+	attr_available_offline_states.show = NULL;
+	attr_preferred_offline_state.show = NULL;
+	attr_preferred_offline_state.store = NULL;
+
+	cpu_maps_update_done();
+	unregister_cpu_notifier(&cpu_driver_notifier);
+}
+
 #else /* ... !CONFIG_HOTPLUG_CPU */
 static inline void register_cpu_control(struct cpu *cpu)
 {
 }
+
 #endif /* CONFIG_HOTPLUG_CPU */
 
 #ifdef CONFIG_KEXEC
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 4d668e0..e2150be 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -51,6 +51,21 @@ struct notifier_block;
 #ifdef CONFIG_HOTPLUG_CPU
 extern int register_cpu_notifier(struct notifier_block *nb);
 extern void unregister_cpu_notifier(struct notifier_block *nb);
+
+struct cpu_offline_driver {
+	ssize_t (*show_available_states)(struct sys_device *dev,
+			struct sysdev_attribute *attr, char *buf);
+	ssize_t (*show_preferred_state)(struct sys_device *dev,
+			struct sysdev_attribute *attr, char *buf);
+
+	ssize_t (*store_preferred_state)(struct sys_device *dev,
+			struct sysdev_attribute *attr,
+			const char *buf, size_t count);
+};
+
+extern int register_cpu_offline_driver(struct cpu_offline_driver *driver);
+extern void unregister_cpu_offline_driver(struct cpu_offline_driver *driver);
+
 #else
 
 #ifndef MODULE

[PATCH 2/3] cpu: Implement cpu-offline-state callbacks for pSeries.

[Gautham R Shenoy]

This patch implements the callbacks to handle the reads/writes into the sysfs
interfaces

/sys/devices/system/cpu/cpu<number>/available_offline_states
and
/sys/devices/system/cpu/cpu<number>/preferred_offline_state

Currently, the patch defines two states which the processor can go to when it
is offlined. They are

- deallocate: The current behaviour when the cpu is offlined.
  The CPU would call make an rtas_stop_self() call and hand over the
  CPU back to the resource pool, thereby effectively deallocating
  that vCPU from the LPAR.

- deactivate: This is expected to cede the processor to the hypervisor, so
  that on processors which support appropriate low-power states, they can
  be exploited. This can be considered as an extended tickless idle state.

The patch only implements the callbacks which will display the available
states, and record the preferred states. The code bits to call
rtas_stop_self() or H_CEDE, depending on the preferred_offline_state is
implemented in the next patch.

Signed-off-by: Gautham R Shenoy <ego@in.ibm.com>
---
 arch/powerpc/platforms/pseries/hotplug-cpu.c    |   90 +++++++++++++++++++++++
 arch/powerpc/platforms/pseries/offline_driver.h |   16 ++++
 2 files changed, 106 insertions(+), 0 deletions(-)
 create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h

diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index a20ead8..f15de99 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -30,6 +30,95 @@
 #include <asm/pSeries_reconfig.h>
 #include "xics.h"
 #include "plpar_wrappers.h"
+#include "offline_driver.h"
+
+struct cpu_offline_state {
+	enum cpu_state_vals state_val;
+	const char *state_name;
+} pSeries_cpu_offline_states[] = {
+	{CPU_DEACTIVATE, "deactivate"},
+	{CPU_DEALLOCATE, "deallocate"},
+};
+
+DEFINE_PER_CPU(enum cpu_state_vals, preferred_offline_state) = CPU_DEALLOCATE;
+
+ssize_t pSeries_show_available_states(struct sys_device *dev,
+			struct sysdev_attribute *attr, char *buf)
+{
+	int state;
+	ssize_t ret = 0;
+
+	for (state = CPU_DEACTIVATE; state < CPU_MAX_OFFLINE_STATES; state++) {
+		if (state == CPU_STATE_ONLINE)
+			continue;
+
+		if (ret >= (ssize_t) ((PAGE_SIZE / sizeof(char))
+					- (CPU_STATES_LEN + 2)))
+			goto out;
+		ret += scnprintf(&buf[ret], CPU_STATES_LEN, "%s ",
+				pSeries_cpu_offline_states[state].state_name);
+	}
+
+out:
+	ret += sprintf(&buf[ret], "\n");
+	return ret;
+}
+
+ssize_t pSeries_show_preferred_state(struct sys_device *dev,
+			struct sysdev_attribute *attr, char *buf)
+{
+	struct cpu *cpu = container_of(dev, struct cpu, sysdev);
+	int state = per_cpu(preferred_offline_state, cpu->sysdev.id);
+
+	return scnprintf(buf, CPU_STATES_LEN, "%s\n",
+			pSeries_cpu_offline_states[state].state_name);
+}
+
+ssize_t pSeries_store_preferred_state(struct sys_device *dev,
+			struct sysdev_attribute *attr,
+			const char *buf, size_t count)
+{
+	struct cpu *cpu = container_of(dev, struct cpu, sysdev);
+	unsigned int ret = -EINVAL;
+	char state_name[CPU_STATES_LEN];
+	int i;
+	cpu_maps_update_begin();
+	ret = sscanf(buf, "%15s", state_name);
+
+	if (ret != 1) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	for (i = CPU_DEACTIVATE; i < CPU_MAX_OFFLINE_STATES; i++)
+		if (!strnicmp(state_name,
+				pSeries_cpu_offline_states[i].state_name,
+				CPU_STATES_LEN))
+			break;
+
+	if (i == CPU_MAX_OFFLINE_STATES) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	per_cpu(preferred_offline_state, cpu->sysdev.id) =
+				pSeries_cpu_offline_states[i].state_val;
+	ret = 0;
+
+out_unlock:
+	cpu_maps_update_done();
+
+	if (ret)
+		return ret;
+	else
+		return count;
+}
+
+struct cpu_offline_driver pSeries_offline_driver = {
+	.show_available_states = pSeries_show_available_states,
+	.show_preferred_state = pSeries_show_preferred_state,
+	.store_preferred_state = pSeries_store_preferred_state,
+};
 
 /* This version can't take the spinlock, because it never returns */
 static struct rtas_args rtas_stop_self_args = {
@@ -281,6 +370,7 @@ static int __init pseries_cpu_hotplug_init(void)
 	ppc_md.cpu_die = pseries_mach_cpu_die;
 	smp_ops->cpu_disable = pseries_cpu_disable;
 	smp_ops->cpu_die = pseries_cpu_die;
+	register_cpu_offline_driver(&pSeries_offline_driver);
 
 	/* Processors can be added/removed only on LPAR */
 	if (firmware_has_feature(FW_FEATURE_LPAR))
diff --git a/arch/powerpc/platforms/pseries/offline_driver.h b/arch/powerpc/platforms/pseries/offline_driver.h
new file mode 100644
index 0000000..bdae76a
--- /dev/null
+++ b/arch/powerpc/platforms/pseries/offline_driver.h
@@ -0,0 +1,16 @@
+#ifndef _OFFLINE_DRIVER_H_
+#define _OFFLINE_DRIVER_H_
+
+#define CPU_STATES_LEN	16
+
+/* Cpu offline states go here */
+enum cpu_state_vals {
+	CPU_DEACTIVATE,
+	CPU_DEALLOCATE,
+	CPU_STATE_ONLINE,
+	CPU_MAX_OFFLINE_STATES
+};
+
+DECLARE_PER_CPU(enum cpu_state_vals, preferred_offline_state);
+
+#endif

[PATCH 3/3] pSeries: cpu: Cede CPU during a deactivate-offline

[Gautham R Shenoy]

Implements the pSeries specific code bits to put the CPU into
rtas_stop_self() state or H_CEDE state depending on the
preferred_offline_state value for that CPU.

Signed-off-by: Gautham R Shenoy <ego@in.ibm.com>
---
 arch/powerpc/platforms/pseries/hotplug-cpu.c    |   70 +++++++++++++++++++++--
 arch/powerpc/platforms/pseries/offline_driver.h |    3 +
 arch/powerpc/platforms/pseries/plpar_wrappers.h |    6 ++
 arch/powerpc/platforms/pseries/smp.c            |   18 ++++++
 4 files changed, 88 insertions(+), 9 deletions(-)

diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index f15de99..5b47d6c 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -41,6 +41,8 @@ struct cpu_offline_state {
 };
 
 DEFINE_PER_CPU(enum cpu_state_vals, preferred_offline_state) = CPU_DEALLOCATE;
+DEFINE_PER_CPU(enum cpu_state_vals, cpu_current_state);
+DEFINE_PER_CPU(int, cpu_offline_ack);
 
 ssize_t pSeries_show_available_states(struct sys_device *dev,
 			struct sysdev_attribute *attr, char *buf)
@@ -148,8 +150,47 @@ static void pseries_mach_cpu_die(void)
 	local_irq_disable();
 	idle_task_exit();
 	xics_teardown_cpu();
-	unregister_slb_shadow(hard_smp_processor_id(), __pa(get_slb_shadow()));
-	rtas_stop_self();
+	if (__get_cpu_var(preferred_offline_state) == CPU_DEACTIVATE) {
+
+		__get_cpu_var(cpu_offline_ack) = 1;
+		get_lppaca()->idle = 1;
+		if (!get_lppaca()->shared_proc)
+			get_lppaca()->donate_dedicated_cpu = 1;
+
+		printk(KERN_INFO "cpu %u (hwid %u) ceding\n",
+			smp_processor_id(), hard_smp_processor_id());
+
+		while (__get_cpu_var(cpu_current_state)	== CPU_DEACTIVATE) {
+			cede_processor();
+			printk(KERN_INFO "cpu %u (hwid %u) Returned from cede.\
+				Decrementer value: %x. Timebase value:%llx \n",
+			       smp_processor_id(), hard_smp_processor_id(),
+			       get_dec(), get_tb());
+		}
+
+		printk(KERN_INFO "cpu %u (hwid %u) Received online PROD \n",
+		       smp_processor_id(), hard_smp_processor_id());
+
+		if (!get_lppaca()->shared_proc)
+			get_lppaca()->donate_dedicated_cpu = 0;
+		get_lppaca()->idle = 0;
+
+		unregister_slb_shadow(hard_smp_processor_id(),
+						__pa(get_slb_shadow()));
+
+		/**
+		 * NOTE: Calling start_secondary() here, is not a very nice
+		 * way of beginning a new context.
+		 *
+		 * We need to reset the stack-pointer.
+		 * Find a cleaner way to do this.
+		 */
+		start_secondary(NULL);
+	} else {
+		unregister_slb_shadow(hard_smp_processor_id(),
+						__pa(get_slb_shadow()));
+		rtas_stop_self();
+	}
 	/* Should never get here... */
 	BUG();
 	for(;;);
@@ -192,6 +233,10 @@ static int pseries_cpu_disable(void)
 
 	/* FIXME: abstract this to not be platform specific later on */
 	xics_migrate_irqs_away();
+	__get_cpu_var(cpu_current_state) =
+		__get_cpu_var(preferred_offline_state);
+
+	__get_cpu_var(cpu_offline_ack) = 0;
 	return 0;
 }
 
@@ -201,11 +246,22 @@ static void pseries_cpu_die(unsigned int cpu)
 	int cpu_status;
 	unsigned int pcpu = get_hard_smp_processor_id(cpu);
 
-	for (tries = 0; tries < 25; tries++) {
-		cpu_status = query_cpu_stopped(pcpu);
-		if (cpu_status == 0 || cpu_status == -1)
-			break;
-		cpu_relax();
+	if (per_cpu(preferred_offline_state, cpu) == CPU_DEACTIVATE) {
+		/* Wait for some Ack */
+		for (tries = 0; tries < 10000; tries++) {
+			cpu_status = !per_cpu(cpu_offline_ack, cpu);
+			if (!cpu_status)
+				break;
+			cpu_relax();
+		}
+
+	} else {
+		for (tries = 0; tries < 25; tries++) {
+			cpu_status = query_cpu_stopped(pcpu);
+			if (cpu_status == 0 || cpu_status == -1)
+				break;
+			cpu_relax();
+		}
 	}
 	if (cpu_status != 0) {
 		printk("Querying DEAD? cpu %i (%i) shows %i\n",
diff --git a/arch/powerpc/platforms/pseries/offline_driver.h b/arch/powerpc/platforms/pseries/offline_driver.h
index bdae76a..571e085 100644
--- a/arch/powerpc/platforms/pseries/offline_driver.h
+++ b/arch/powerpc/platforms/pseries/offline_driver.h
@@ -12,5 +12,6 @@ enum cpu_state_vals {
 };
 
 DECLARE_PER_CPU(enum cpu_state_vals, preferred_offline_state);
-
+DECLARE_PER_CPU(enum cpu_state_vals, cpu_current_state);
+extern int start_secondary(void *unused);
 #endif
diff --git a/arch/powerpc/platforms/pseries/plpar_wrappers.h b/arch/powerpc/platforms/pseries/plpar_wrappers.h
index a24a6b2..ab7f5ab 100644
--- a/arch/powerpc/platforms/pseries/plpar_wrappers.h
+++ b/arch/powerpc/platforms/pseries/plpar_wrappers.h
@@ -14,6 +14,12 @@ static inline long cede_processor(void)
 	return plpar_hcall_norets(H_CEDE);
 }
 
+static inline long prod_processor(unsigned long cpu)
+{
+	unsigned long hcpuid = get_hard_smp_processor_id(cpu);
+	return plpar_hcall_norets(H_PROD, hcpuid);
+}
+
 static inline long vpa_call(unsigned long flags, unsigned long cpu,
 		unsigned long vpa)
 {
diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
index 1f8f6cf..15d96a2 100644
--- a/arch/powerpc/platforms/pseries/smp.c
+++ b/arch/powerpc/platforms/pseries/smp.c
@@ -48,6 +48,7 @@
 #include "plpar_wrappers.h"
 #include "pseries.h"
 #include "xics.h"
+#include "offline_driver.h"
 
 
 /*
@@ -86,7 +87,10 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)
 	/* Fixup atomic count: it exited inside IRQ handler. */
 	task_thread_info(paca[lcpu].__current)->preempt_count	= 0;
 
-	/* 
+	if (per_cpu(preferred_offline_state, lcpu) == CPU_DEACTIVATE)
+		goto out;
+
+	/*
 	 * If the RTAS start-cpu token does not exist then presume the
 	 * cpu is already spinning.
 	 */
@@ -100,6 +104,7 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)
 		return 0;
 	}
 
+out:
 	return 1;
 }
 
@@ -119,6 +124,7 @@ static void __devinit smp_xics_setup_cpu(int cpu)
 
 static void __devinit smp_pSeries_kick_cpu(int nr)
 {
+	long rc;
 	BUG_ON(nr < 0 || nr >= NR_CPUS);
 
 	if (!smp_startup_cpu(nr))
@@ -130,6 +136,16 @@ static void __devinit smp_pSeries_kick_cpu(int nr)
 	 * the processor will continue on to secondary_start
 	 */
 	paca[nr].cpu_start = 1;
+
+	per_cpu(cpu_current_state, nr) = CPU_STATE_ONLINE;
+	if (per_cpu(preferred_offline_state, nr) == CPU_DEACTIVATE) {
+
+		printk(KERN_INFO "Prodding processor %d to go online\n", nr);
+		rc = prod_processor(nr);
+		if (rc != H_SUCCESS)
+			panic("Prod to wake up processor %d returned \
+				with error code: %ld. Dying\n", nr, rc);
+	}
 }
 
 static int smp_pSeries_cpu_bootable(unsigned int nr)

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Shaohua Li]

Hi,

On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> and allow the system administrator to choose the desired state the CPU should
> go to when it is offlined. We think this approach addresses the concerns about
> determinism as well as transparency, since CPU-Hotplug already provides
> notification mechanism which the userspace can listen to for any change
> in the configuration and correspondingly readjust any previously set
> cpu-affinities.
Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
even userspace can get a notification.

> Also, approaches such as [1] can make use of this
> extended infrastructure instead of putting the CPU to an arbitrary C-state
> when it is offlined, thereby providing the system administrator a rope to hang
> himself with should he feel the need to do so.
I didn't see the reason why administrator needs to know which state offline cpu
should stay. Don't know about powerpc side, but in x86 side, it appears deepest
C-state is already preferred.

Thanks,
Shaohua

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Shaohua Li]

Hi,

On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> and allow the system administrator to choose the desired state the CPU should
> go to when it is offlined. We think this approach addresses the concerns about
> determinism as well as transparency, since CPU-Hotplug already provides
> notification mechanism which the userspace can listen to for any change
> in the configuration and correspondingly readjust any previously set
> cpu-affinities.
Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
even userspace can get a notification.

> Also, approaches such as [1] can make use of this
> extended infrastructure instead of putting the CPU to an arbitrary C-state
> when it is offlined, thereby providing the system administrator a rope to hang
> himself with should he feel the need to do so.
I didn't see the reason why administrator needs to know which state offline cpu
should stay. Don't know about powerpc side, but in x86 side, it appears deepest
C-state is already preferred.

Thanks,
Shaohua

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Vaidyanathan Srinivasan]

* Shaohua Li <shaohua.li@intel.com> [2009-08-06 09:58:55]:

> Hi,
> 
> On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > and allow the system administrator to choose the desired state the CPU should
> > go to when it is offlined. We think this approach addresses the concerns about
> > determinism as well as transparency, since CPU-Hotplug already provides
> > notification mechanism which the userspace can listen to for any change
> > in the configuration and correspondingly readjust any previously set
> > cpu-affinities.
> Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> even userspace can get a notification.

Hi Shaohua,

Notification to userspace and an opportunity to react to the situation
is certainly better to manage as compared to under-the-cover reduction
in capacity.

We are already doing something to this effect in virtualized guests
where VCPU entitlement and assignment can be changed by the hypervisor
based on resource constraints.  This framework with notification and
determinism will help manage cpu capacity better.

I agree with Peter that scheduler approach is better, but it is not
deterministic.  This is simpler and cleaner alternative to keep the
complexity in userspace and provide only framework to control/notify
kernel.
 
> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.

Yes that is what we would expect, but the deepest sleep state may be
restricted by BIOS or other system level parameters.  This was the
main objection to Venki's deepest sleep state for offline cpus patch.
There could be higher level policy that restricts the deepest level
C-states for various reasons.  This framework is to provide an
opportunity to adhere to the policy with userspace inputs.  This
framework also helps to choose different states in a virtualized
system.

--Vaidy

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Vaidyanathan Srinivasan]

* Shaohua Li <shaohua.li@intel.com> [2009-08-06 09:58:55]:

> Hi,
> 
> On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > and allow the system administrator to choose the desired state the CPU should
> > go to when it is offlined. We think this approach addresses the concerns about
> > determinism as well as transparency, since CPU-Hotplug already provides
> > notification mechanism which the userspace can listen to for any change
> > in the configuration and correspondingly readjust any previously set
> > cpu-affinities.
> Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> even userspace can get a notification.

Hi Shaohua,

Notification to userspace and an opportunity to react to the situation
is certainly better to manage as compared to under-the-cover reduction
in capacity.

We are already doing something to this effect in virtualized guests
where VCPU entitlement and assignment can be changed by the hypervisor
based on resource constraints.  This framework with notification and
determinism will help manage cpu capacity better.

I agree with Peter that scheduler approach is better, but it is not
deterministic.  This is simpler and cleaner alternative to keep the
complexity in userspace and provide only framework to control/notify
kernel.
 
> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.

Yes that is what we would expect, but the deepest sleep state may be
restricted by BIOS or other system level parameters.  This was the
main objection to Venki's deepest sleep state for offline cpus patch.
There could be higher level policy that restricts the deepest level
C-states for various reasons.  This framework is to provide an
opportunity to adhere to the policy with userspace inputs.  This
framework also helps to choose different states in a virtualized
system.

--Vaidy

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Gautham R Shenoy]

Hi Shaohua,

On Thu, Aug 06, 2009 at 09:58:55AM +0800, Shaohua Li wrote:
> Hi,
> 
> On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > and allow the system administrator to choose the desired state the CPU should
> > go to when it is offlined. We think this approach addresses the concerns about
> > determinism as well as transparency, since CPU-Hotplug already provides
> > notification mechanism which the userspace can listen to for any change
> > in the configuration and correspondingly readjust any previously set
> > cpu-affinities.
> Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> even userspace can get a notification.

I think Peter's problem was more to do with the kernel offlining the CPUs
behind the scenes, right ?

We don't do that in this patch series. The option to offline the CPUs is
very much with the admin. The patch-series only provides the interface
that helps the admin choose the state the CPU must reside in when it
goes offline.

> 
> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.

We can still provide a sane default value based on what states are
available and what the BIOS limits us to. Thus we can still use the
idle-state-offline patch that Venki posted sometime ago, right ?

> 
> Thanks,
> Shaohua

-- 
Thanks and Regards
gautham

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Gautham R Shenoy]

Hi Shaohua,

On Thu, Aug 06, 2009 at 09:58:55AM +0800, Shaohua Li wrote:
> Hi,
> 
> On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > and allow the system administrator to choose the desired state the CPU should
> > go to when it is offlined. We think this approach addresses the concerns about
> > determinism as well as transparency, since CPU-Hotplug already provides
> > notification mechanism which the userspace can listen to for any change
> > in the configuration and correspondingly readjust any previously set
> > cpu-affinities.
> Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> even userspace can get a notification.

I think Peter's problem was more to do with the kernel offlining the CPUs
behind the scenes, right ?

We don't do that in this patch series. The option to offline the CPUs is
very much with the admin. The patch-series only provides the interface
that helps the admin choose the state the CPU must reside in when it
goes offline.

> 
> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.

We can still provide a sane default value based on what states are
available and what the BIOS limits us to. Thus we can still use the
idle-state-offline patch that Venki posted sometime ago, right ?

> 
> Thanks,
> Shaohua

-- 
Thanks and Regards
gautham

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Thu, 2009-08-06 at 10:03 +0530, Vaidyanathan Srinivasan wrote:
> This was the
> main objection to Venki's deepest sleep state for offline cpus patch.

Well, my main objection was that is was a single raw function pointer
without any management layer around it.

We have the exact same mess with the idle routine - and that has bitten
us in the past.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Thu, 2009-08-06 at 10:03 +0530, Vaidyanathan Srinivasan wrote:
> This was the
> main objection to Venki's deepest sleep state for offline cpus patch.

Well, my main objection was that is was a single raw function pointer
without any management layer around it.

We have the exact same mess with the idle routine - and that has bitten
us in the past.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Thu, 2009-08-06 at 17:03 +0200, Peter Zijlstra wrote:
> On Thu, 2009-08-06 at 10:03 +0530, Vaidyanathan Srinivasan wrote:
> > This was the
> > main objection to Venki's deepest sleep state for offline cpus patch.
> 
> Well, my main objection was that is was a single raw function pointer
> without any management layer around it.
> 
> We have the exact same mess with the idle routine - and that has bitten
> us in the past.

That said, I still think it makes sense to share this with the idle
routine, we can make some callbacks unsuitable for hot-unplug just fine.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Thu, 2009-08-06 at 17:03 +0200, Peter Zijlstra wrote:
> On Thu, 2009-08-06 at 10:03 +0530, Vaidyanathan Srinivasan wrote:
> > This was the
> > main objection to Venki's deepest sleep state for offline cpus patch.
> 
> Well, my main objection was that is was a single raw function pointer
> without any management layer around it.
> 
> We have the exact same mess with the idle routine - and that has bitten
> us in the past.

That said, I still think it makes sense to share this with the idle
routine, we can make some callbacks unsuitable for hot-unplug just fine.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Shaohua Li]

On Thu, Aug 06, 2009 at 09:48:44PM +0800, Gautham R Shenoy wrote:
> Hi Shaohua,
> 
> On Thu, Aug 06, 2009 at 09:58:55AM +0800, Shaohua Li wrote:
> > Hi,
> > 
> > On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > > and allow the system administrator to choose the desired state the CPU should
> > > go to when it is offlined. We think this approach addresses the concerns about
> > > determinism as well as transparency, since CPU-Hotplug already provides
> > > notification mechanism which the userspace can listen to for any change
> > > in the configuration and correspondingly readjust any previously set
> > > cpu-affinities.
> > Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> > even userspace can get a notification.
> 
> I think Peter's problem was more to do with the kernel offlining the CPUs
> behind the scenes, right ?
> 
> We don't do that in this patch series. The option to offline the CPUs is
> very much with the admin. The patch-series only provides the interface
> that helps the admin choose the state the CPU must reside in when it
> goes offline.
but the goal is to use cpu offline to save power, right? So we still have
Peter's problem.
 
> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> 
> We can still provide a sane default value based on what states are
> available and what the BIOS limits us to. Thus we can still use the
> idle-state-offline patch that Venki posted sometime ago, right ?
My original concern about Venki's patch is the C-state limition, but Venki
thought if CPU has the limition, CPU should disable specific C-state, so this
isn't a problem. I had no objection about the infrastructure itself, but just
wonder why we need it.

Thanks,
Shaohua

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Shaohua Li]

On Thu, Aug 06, 2009 at 09:48:44PM +0800, Gautham R Shenoy wrote:
> Hi Shaohua,
> 
> On Thu, Aug 06, 2009 at 09:58:55AM +0800, Shaohua Li wrote:
> > Hi,
> > 
> > On Wed, Aug 05, 2009 at 10:25:53PM +0800, Gautham R Shenoy wrote:
> > > In this patch-series, we propose to extend the CPU-Hotplug infrastructure
> > > and allow the system administrator to choose the desired state the CPU should
> > > go to when it is offlined. We think this approach addresses the concerns about
> > > determinism as well as transparency, since CPU-Hotplug already provides
> > > notification mechanism which the userspace can listen to for any change
> > > in the configuration and correspondingly readjust any previously set
> > > cpu-affinities.
> > Peter dislikes any approach (including cpuhotplug) which breaks userspace policy,
> > even userspace can get a notification.
> 
> I think Peter's problem was more to do with the kernel offlining the CPUs
> behind the scenes, right ?
> 
> We don't do that in this patch series. The option to offline the CPUs is
> very much with the admin. The patch-series only provides the interface
> that helps the admin choose the state the CPU must reside in when it
> goes offline.
but the goal is to use cpu offline to save power, right? So we still have
Peter's problem.
 
> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> 
> We can still provide a sane default value based on what states are
> available and what the BIOS limits us to. Thus we can still use the
> idle-state-offline patch that Venki posted sometime ago, right ?
My original concern about Venki's patch is the C-state limition, but Venki
thought if CPU has the limition, CPU should disable specific C-state, so this
isn't a problem. I had no objection about the infrastructure itself, but just
wonder why we need it.

Thanks,
Shaohua

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.
> 

Agreed, deepest c-state is always best, there's no need to make it configurable.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> > Also, approaches such as [1] can make use of this
> > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > when it is offlined, thereby providing the system administrator a rope to hang
> > himself with should he feel the need to do so.
> I didn't see the reason why administrator needs to know which state offline cpu
> should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> C-state is already preferred.
> 

Agreed, deepest c-state is always best, there's no need to make it configurable.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> 
> Yes that is what we would expect, but the deepest sleep state may be
> restricted by BIOS or other system level parameters.  This was the
> main objection to Venki's deepest sleep state for offline cpus
> patch.

'May be restricted'? Kernel already needs to know about that
restriction, just do the right thing...

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> 
> Yes that is what we would expect, but the deepest sleep state may be
> restricted by BIOS or other system level parameters.  This was the
> main objection to Venki's deepest sleep state for offline cpus
> patch.

'May be restricted'? Kernel already needs to know about that
restriction, just do the right thing...

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Rafael J. Wysocki]

On Sunday 09 August 2009, Pavel Machek wrote:
> Hi!
> 
> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> > 
> 
> Agreed, deepest c-state is always best, there's no need to make it configurable.

Unless it doesn't work.

Rafael

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Rafael J. Wysocki]

On Sunday 09 August 2009, Pavel Machek wrote:
> Hi!
> 
> > > Also, approaches such as [1] can make use of this
> > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > when it is offlined, thereby providing the system administrator a rope to hang
> > > himself with should he feel the need to do so.
> > I didn't see the reason why administrator needs to know which state offline cpu
> > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > C-state is already preferred.
> > 
> 
> Agreed, deepest c-state is always best, there's no need to make it configurable.

Unless it doesn't work.

Rafael

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Vaidyanathan Srinivasan]

* Rafael J. Wysocki <rjw@sisk.pl> [2009-08-09 15:22:02]:

> On Sunday 09 August 2009, Pavel Machek wrote:
> > Hi!
> > 
> > > > Also, approaches such as [1] can make use of this
> > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > himself with should he feel the need to do so.
> > > I didn't see the reason why administrator needs to know which state offline cpu
> > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > C-state is already preferred.
> > > 
> > 
> > Agreed, deepest c-state is always best, there's no need to make it configurable.
> 
> Unless it doesn't work.

Yes, this is one of the reason.  Kernel will know about the deepest
sleep state and any restrictions and should set that as default in the
preferred_offline state. End-users and sys-admins need not change it,
default will be the deepest sleep state supported and allowed by the
system which may be different than the one supported by the processor.

This framework could allow the default to be set easily by other
userspace tools.  These restrictions apply to cpuidle governor as
well, hence at some level both these subsystems should be in sync.

As described in this patch series, the meaning of offline cpu is
different in a virtualized system and this framework provides
flexibility of choice there.  Platforms today do have a choice on what
state an offline cpu should reside, this framework is one of the
possible methods to exploit that choice and not restrict it due to
lack of OS flexibility.

--Vaidy

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Vaidyanathan Srinivasan]

* Rafael J. Wysocki <rjw@sisk.pl> [2009-08-09 15:22:02]:

> On Sunday 09 August 2009, Pavel Machek wrote:
> > Hi!
> > 
> > > > Also, approaches such as [1] can make use of this
> > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > himself with should he feel the need to do so.
> > > I didn't see the reason why administrator needs to know which state offline cpu
> > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > C-state is already preferred.
> > > 
> > 
> > Agreed, deepest c-state is always best, there's no need to make it configurable.
> 
> Unless it doesn't work.

Yes, this is one of the reason.  Kernel will know about the deepest
sleep state and any restrictions and should set that as default in the
preferred_offline state. End-users and sys-admins need not change it,
default will be the deepest sleep state supported and allowed by the
system which may be different than the one supported by the processor.

This framework could allow the default to be set easily by other
userspace tools.  These restrictions apply to cpuidle governor as
well, hence at some level both these subsystems should be in sync.

As described in this patch series, the meaning of offline cpu is
different in a virtualized system and this framework provides
flexibility of choice there.  Platforms today do have a choice on what
state an offline cpu should reside, this framework is one of the
possible methods to exploit that choice and not restrict it due to
lack of OS flexibility.

--Vaidy

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

On Sun 2009-08-09 15:22:02, Rafael J. Wysocki wrote:
> On Sunday 09 August 2009, Pavel Machek wrote:
> > Hi!
> > 
> > > > Also, approaches such as [1] can make use of this
> > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > himself with should he feel the need to do so.
> > > I didn't see the reason why administrator needs to know which state offline cpu
> > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > C-state is already preferred.
> > > 
> > 
> > Agreed, deepest c-state is always best, there's no need to make it configurable.
> 
> Unless it doesn't work.

If it does not work, machine will not boot. We already have
max_cstate= kernel command line option to work around that...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

On Sun 2009-08-09 15:22:02, Rafael J. Wysocki wrote:
> On Sunday 09 August 2009, Pavel Machek wrote:
> > Hi!
> > 
> > > > Also, approaches such as [1] can make use of this
> > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > himself with should he feel the need to do so.
> > > I didn't see the reason why administrator needs to know which state offline cpu
> > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > C-state is already preferred.
> > > 
> > 
> > Agreed, deepest c-state is always best, there's no need to make it configurable.
> 
> Unless it doesn't work.

If it does not work, machine will not boot. We already have
max_cstate= kernel command line option to work around that...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pallipadi, Venkatesh]

On Mon, 2009-08-10 at 01:19 -0700, Pavel Machek wrote:
> On Sun 2009-08-09 15:22:02, Rafael J. Wysocki wrote:
> > On Sunday 09 August 2009, Pavel Machek wrote:
> > > Hi!
> > > 
> > > > > Also, approaches such as [1] can make use of this
> > > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > > himself with should he feel the need to do so.
> > > > I didn't see the reason why administrator needs to know which state offline cpu
> > > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > > C-state is already preferred.
> > > > 
> > > 
> > > Agreed, deepest c-state is always best, there's no need to make it configurable.
> > 
> > Unless it doesn't work.
> 
> If it does not work, machine will not boot. We already have
> max_cstate= kernel command line option to work around that...
> 

On x86, my earlier patch was selecting deepest C-state based on CPU
capability. There, "unless it doesn't work" will not hold good. If a CPU
C-state is reported in cpuid it will work. If there is any errata we
will workaround it as we do with anything else in the kernel.

My concern about having this interface for offline CPU is
- How are we going to populate this possible states. On x86, there are
2-3 CPU mwait cstates with each having few sub C-states. And there can
be some BIOS supplied C-states which are IO port invocations which will
map to some CPU mwait state,sub state pair mentioned above and we won't
know anything about this mapping. So, we want to give all these options
in sysfs?
- Having all these states and having no information on what basis one
should change this or on what reasons one should change this will result
in some userspace superpowersaver daemon using this in a wrong way and
some distro shipping it.

Also, I don't think using just the ACPI/BIOS supplied states in _CST is
right thing to do for offline. _CST is meant for C-state and BIOS may
not include some C-state in _CST if the system manufacturer thinks that
the latency is too high for the state to be used as a C-state. That
limitation applies for C-state as the cpu is expected to come out of
C-state often and execute code handle interrupts etc. But, that
restriction does not apply for offline online which is not as frequent
as C-state entry and it already has big latency with startup IPI, and a
whole baggage of CPU setup code. So, using BIOS CST info for CPU offline
state doesn't seem right.

May be having (to pick a number) 3 possible offline states for all
platforms with one for halt equivalent and one for deepest possible that
CPU can handle and one for deepest possible that platform likes for
C-states may make sense. Will keeps things simpler in terms of usage
expectations and possibly reduce the misuse oppurtubity?

Thanks,
Venki

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pallipadi, Venkatesh]

On Mon, 2009-08-10 at 01:19 -0700, Pavel Machek wrote:
> On Sun 2009-08-09 15:22:02, Rafael J. Wysocki wrote:
> > On Sunday 09 August 2009, Pavel Machek wrote:
> > > Hi!
> > > 
> > > > > Also, approaches such as [1] can make use of this
> > > > > extended infrastructure instead of putting the CPU to an arbitrary C-state
> > > > > when it is offlined, thereby providing the system administrator a rope to hang
> > > > > himself with should he feel the need to do so.
> > > > I didn't see the reason why administrator needs to know which state offline cpu
> > > > should stay. Don't know about powerpc side, but in x86 side, it appears deepest
> > > > C-state is already preferred.
> > > > 
> > > 
> > > Agreed, deepest c-state is always best, there's no need to make it configurable.
> > 
> > Unless it doesn't work.
> 
> If it does not work, machine will not boot. We already have
> max_cstate= kernel command line option to work around that...
> 

On x86, my earlier patch was selecting deepest C-state based on CPU
capability. There, "unless it doesn't work" will not hold good. If a CPU
C-state is reported in cpuid it will work. If there is any errata we
will workaround it as we do with anything else in the kernel.

My concern about having this interface for offline CPU is
- How are we going to populate this possible states. On x86, there are
2-3 CPU mwait cstates with each having few sub C-states. And there can
be some BIOS supplied C-states which are IO port invocations which will
map to some CPU mwait state,sub state pair mentioned above and we won't
know anything about this mapping. So, we want to give all these options
in sysfs?
- Having all these states and having no information on what basis one
should change this or on what reasons one should change this will result
in some userspace superpowersaver daemon using this in a wrong way and
some distro shipping it.

Also, I don't think using just the ACPI/BIOS supplied states in _CST is
right thing to do for offline. _CST is meant for C-state and BIOS may
not include some C-state in _CST if the system manufacturer thinks that
the latency is too high for the state to be used as a C-state. That
limitation applies for C-state as the cpu is expected to come out of
C-state often and execute code handle interrupts etc. But, that
restriction does not apply for offline online which is not as frequent
as C-state entry and it already has big latency with startup IPI, and a
whole baggage of CPU setup code. So, using BIOS CST info for CPU offline
state doesn't seem right.

May be having (to pick a number) 3 possible offline states for all
platforms with one for halt equivalent and one for deepest possible that
CPU can handle and one for deepest possible that platform likes for
C-states may make sense. Will keeps things simpler in terms of usage
expectations and possibly reduce the misuse oppurtubity?

Thanks,
Venki

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 10, 2009 at 05:22:17PM -0700, Pallipadi, Venkatesh wrote:
> Also, I don't think using just the ACPI/BIOS supplied states in _CST is
> right thing to do for offline. _CST is meant for C-state and BIOS may
> not include some C-state in _CST if the system manufacturer thinks that
> the latency is too high for the state to be used as a C-state. That
> limitation applies for C-state as the cpu is expected to come out of
> C-state often and execute code handle interrupts etc. But, that
> restriction does not apply for offline online which is not as frequent
> as C-state entry and it already has big latency with startup IPI, and a
> whole baggage of CPU setup code. So, using BIOS CST info for CPU offline
> state doesn't seem right.
> 
> May be having (to pick a number) 3 possible offline states for all
> platforms with one for halt equivalent and one for deepest possible that
> CPU can handle and one for deepest possible that platform likes for
> C-states may make sense. Will keeps things simpler in terms of usage
> expectations and possibly reduce the misuse oppurtubity?

Yes, I think we should let specific archs advertise a small set
of possible offline states and let the cpu state be set to one of
those only keeping the platform implementation robust.

Here is variant of the original proposal from Gautham -

/sys/devices/system/cpu/cpu<number>/available_states

For example, available state for an Intel platform could be exported as
"online dealloc C1 C6"

online = fully up
dealloc = offline and de-allocated (as in virtualized environment)
C1 = C1 or C1E halt
C6 = C6 sleep

/sys/devices/system/cpu/cpu<number>/state

Writing any of the available states to this file triggers transition to that 
state barring some transitions that are disallowed to keep things simple
(e.g. dealloc cpus support only transition to online).

/sys/devices/system/cpu/cpu<number>/online

Backward compatibility - online = 0 changes state to C6 or dealloc depending
on the platform. online = 1 changes state to online.

Would this make sense ?

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 10, 2009 at 05:22:17PM -0700, Pallipadi, Venkatesh wrote:
> Also, I don't think using just the ACPI/BIOS supplied states in _CST is
> right thing to do for offline. _CST is meant for C-state and BIOS may
> not include some C-state in _CST if the system manufacturer thinks that
> the latency is too high for the state to be used as a C-state. That
> limitation applies for C-state as the cpu is expected to come out of
> C-state often and execute code handle interrupts etc. But, that
> restriction does not apply for offline online which is not as frequent
> as C-state entry and it already has big latency with startup IPI, and a
> whole baggage of CPU setup code. So, using BIOS CST info for CPU offline
> state doesn't seem right.
> 
> May be having (to pick a number) 3 possible offline states for all
> platforms with one for halt equivalent and one for deepest possible that
> CPU can handle and one for deepest possible that platform likes for
> C-states may make sense. Will keeps things simpler in terms of usage
> expectations and possibly reduce the misuse oppurtubity?

Yes, I think we should let specific archs advertise a small set
of possible offline states and let the cpu state be set to one of
those only keeping the platform implementation robust.

Here is variant of the original proposal from Gautham -

/sys/devices/system/cpu/cpu<number>/available_states

For example, available state for an Intel platform could be exported as
"online dealloc C1 C6"

online = fully up
dealloc = offline and de-allocated (as in virtualized environment)
C1 = C1 or C1E halt
C6 = C6 sleep

/sys/devices/system/cpu/cpu<number>/state

Writing any of the available states to this file triggers transition to that 
state barring some transitions that are disallowed to keep things simple
(e.g. dealloc cpus support only transition to online).

/sys/devices/system/cpu/cpu<number>/online

Backward compatibility - online = 0 changes state to C6 or dealloc depending
on the platform. online = 1 changes state to online.

Would this make sense ?

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> May be having (to pick a number) 3 possible offline states for all
> platforms with one for halt equivalent and one for deepest possible that
> CPU can handle and one for deepest possible that platform likes for
> C-states may make sense. Will keeps things simpler in terms of usage
> expectations and possibly reduce the misuse oppurtubity?

Maybe just going to the deepest offline state automatically is the
easiest option?

cpu hotplug/unplug should be rare-enough operation that the latencies
do not really matter, right?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

Hi!

> May be having (to pick a number) 3 possible offline states for all
> platforms with one for halt equivalent and one for deepest possible that
> CPU can handle and one for deepest possible that platform likes for
> C-states may make sense. Will keeps things simpler in terms of usage
> expectations and possibly reduce the misuse oppurtubity?

Maybe just going to the deepest offline state automatically is the
easiest option?

cpu hotplug/unplug should be rare-enough operation that the latencies
do not really matter, right?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Wed, 2009-08-12 at 13:58 +0200, Pavel Machek wrote:
> Hi!
> 
> > May be having (to pick a number) 3 possible offline states for all
> > platforms with one for halt equivalent and one for deepest possible that
> > CPU can handle and one for deepest possible that platform likes for
> > C-states may make sense. Will keeps things simpler in terms of usage
> > expectations and possibly reduce the misuse oppurtubity?
> 
> Maybe just going to the deepest offline state automatically is the
> easiest option?
> 
> cpu hotplug/unplug should be rare-enough operation that the latencies
> do not really matter, right?

Ha, it uses kstopmachine, anybody caring about hotplug latencies is
insane.

And yeah, I'm not quite sure what this user-interface is good for
either. Having an in-kernel management layer where you can register
various idle routines makes sense. But exposing it to userspace,.. not
so much.

The idle thread can select an idle routine under constraints of the QoS
latency constraints, the unplug loop however should indeed select the
one that is available to dead cpus (not all idle routines are available
from what people tell me), and yields the best power savings.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Wed, 2009-08-12 at 13:58 +0200, Pavel Machek wrote:
> Hi!
> 
> > May be having (to pick a number) 3 possible offline states for all
> > platforms with one for halt equivalent and one for deepest possible that
> > CPU can handle and one for deepest possible that platform likes for
> > C-states may make sense. Will keeps things simpler in terms of usage
> > expectations and possibly reduce the misuse oppurtubity?
> 
> Maybe just going to the deepest offline state automatically is the
> easiest option?
> 
> cpu hotplug/unplug should be rare-enough operation that the latencies
> do not really matter, right?

Ha, it uses kstopmachine, anybody caring about hotplug latencies is
insane.

And yeah, I'm not quite sure what this user-interface is good for
either. Having an in-kernel management layer where you can register
various idle routines makes sense. But exposing it to userspace,.. not
so much.

The idle thread can select an idle routine under constraints of the QoS
latency constraints, the unplug loop however should indeed select the
one that is available to dead cpus (not all idle routines are available
from what people tell me), and yields the best power savings.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Wed, Aug 12, 2009 at 01:58:06PM +0200, Pavel Machek wrote:
> Hi!
> 
> > May be having (to pick a number) 3 possible offline states for all
> > platforms with one for halt equivalent and one for deepest possible that
> > CPU can handle and one for deepest possible that platform likes for
> > C-states may make sense. Will keeps things simpler in terms of usage
> > expectations and possibly reduce the misuse oppurtubity?
> 
> Maybe just going to the deepest offline state automatically is the
> easiest option?

In a native system, I think we should the platform-specific code
export what makes sense. That may be just the lowest possible
state only. Or may be more than one.

In a virtualized system, we would want to do at least the following -

1. An offline configuration state where the hypervisor can
take the cpu back and allocate it to another VM.

2. A low-power state where the guest indicates it doesn't need the
CPU (and can be put in low power state) but doesn't want to give up 
its allocated cpu share. IOW, no visible configuration changes.

So, in any case we would probably want more than one states.

> cpu hotplug/unplug should be rare-enough operation that the latencies
> do not really matter, right?

As of now, from the platform perspective, I don't think low-power
state latencies matter in this code path. The only thing that might
have any relevance is electrical power-off technology and whether
there may be any h/w specific issues restricting its use. I don't know
that there will be any, but it may not be a good idea to prevent
platforms from requiring the use of multiple "offline" states.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Wed, Aug 12, 2009 at 01:58:06PM +0200, Pavel Machek wrote:
> Hi!
> 
> > May be having (to pick a number) 3 possible offline states for all
> > platforms with one for halt equivalent and one for deepest possible that
> > CPU can handle and one for deepest possible that platform likes for
> > C-states may make sense. Will keeps things simpler in terms of usage
> > expectations and possibly reduce the misuse oppurtubity?
> 
> Maybe just going to the deepest offline state automatically is the
> easiest option?

In a native system, I think we should the platform-specific code
export what makes sense. That may be just the lowest possible
state only. Or may be more than one.

In a virtualized system, we would want to do at least the following -

1. An offline configuration state where the hypervisor can
take the cpu back and allocate it to another VM.

2. A low-power state where the guest indicates it doesn't need the
CPU (and can be put in low power state) but doesn't want to give up 
its allocated cpu share. IOW, no visible configuration changes.

So, in any case we would probably want more than one states.

> cpu hotplug/unplug should be rare-enough operation that the latencies
> do not really matter, right?

As of now, from the platform perspective, I don't think low-power
state latencies matter in this code path. The only thing that might
have any relevance is electrical power-off technology and whether
there may be any h/w specific issues restricting its use. I don't know
that there will be any, but it may not be a good idea to prevent
platforms from requiring the use of multiple "offline" states.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Len Brown]

On Thu, 13 Aug 2009, Dipankar Sarma wrote:

> On Wed, Aug 12, 2009 at 01:58:06PM +0200, Pavel Machek wrote:
> > Hi!
> > 
> > > May be having (to pick a number) 3 possible offline states for all
> > > platforms with one for halt equivalent and one for deepest possible that
> > > CPU can handle and one for deepest possible that platform likes for
> > > C-states may make sense. Will keeps things simpler in terms of usage
> > > expectations and possibly reduce the misuse oppurtubity?
> > 
> > Maybe just going to the deepest offline state automatically is the
> > easiest option?

I agree with Pavel and PeterZ on this.

> In a native system, I think we should the platform-specific code
> export what makes sense. That may be just the lowest possible
> state only. Or may be more than one.

For x86, it is 1 state.

> In a virtualized system, we would want to do at least the following -

Are you talking about Linux as a para-virtualized guest here?

> 1. An offline configuration state where the hypervisor can
> take the cpu back and allocate it to another VM.

The hypervisor gets control no matter what idle state
the guest enters, yes?

> 2. A low-power state where the guest indicates it doesn't need the
> CPU (and can be put in low power state) but doesn't want to give up 
> its allocated cpu share. IOW, no visible configuration changes.
> 
> So, in any case we would probably want more than one states.

How are #1 and #2 different when the hypervisor
gets control in all idle states?  I assert that
they are the same, and thus 1 state will suffice.

> > cpu hotplug/unplug should be rare-enough operation that the latencies
> > do not really matter, right?
> 
> As of now, from the platform perspective, I don't think low-power
> state latencies matter in this code path.

Yay, something we all agree on.

> The only thing that might
> have any relevance is electrical power-off technology and whether
> there may be any h/w specific issues restricting its use.
> I don't know that there will be any, but it may not be a good idea to prevent
> platforms from requiring the use of multiple "offline" states.

Can you give an example of such restrictions? (I can't)

I do not advocate integrating features into the kernel
if they can not be tested and it is uncear that they are needed.
Keep it simple.  Add complexity only when it is clear that you must.

thanks,
Len Brown, Intel Open Source Technology Center.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Len Brown]

On Thu, 13 Aug 2009, Dipankar Sarma wrote:

> On Wed, Aug 12, 2009 at 01:58:06PM +0200, Pavel Machek wrote:
> > Hi!
> > 
> > > May be having (to pick a number) 3 possible offline states for all
> > > platforms with one for halt equivalent and one for deepest possible that
> > > CPU can handle and one for deepest possible that platform likes for
> > > C-states may make sense. Will keeps things simpler in terms of usage
> > > expectations and possibly reduce the misuse oppurtubity?
> > 
> > Maybe just going to the deepest offline state automatically is the
> > easiest option?

I agree with Pavel and PeterZ on this.

> In a native system, I think we should the platform-specific code
> export what makes sense. That may be just the lowest possible
> state only. Or may be more than one.

For x86, it is 1 state.

> In a virtualized system, we would want to do at least the following -

Are you talking about Linux as a para-virtualized guest here?

> 1. An offline configuration state where the hypervisor can
> take the cpu back and allocate it to another VM.

The hypervisor gets control no matter what idle state
the guest enters, yes?

> 2. A low-power state where the guest indicates it doesn't need the
> CPU (and can be put in low power state) but doesn't want to give up 
> its allocated cpu share. IOW, no visible configuration changes.
> 
> So, in any case we would probably want more than one states.

How are #1 and #2 different when the hypervisor
gets control in all idle states?  I assert that
they are the same, and thus 1 state will suffice.

> > cpu hotplug/unplug should be rare-enough operation that the latencies
> > do not really matter, right?
> 
> As of now, from the platform perspective, I don't think low-power
> state latencies matter in this code path.

Yay, something we all agree on.

> The only thing that might
> have any relevance is electrical power-off technology and whether
> there may be any h/w specific issues restricting its use.
> I don't know that there will be any, but it may not be a good idea to prevent
> platforms from requiring the use of multiple "offline" states.

Can you give an example of such restrictions? (I can't)

I do not advocate integrating features into the kernel
if they can not be tested and it is uncear that they are needed.
Keep it simple.  Add complexity only when it is clear that you must.

thanks,
Len Brown, Intel Open Source Technology Center.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Wed, Aug 12, 2009 at 08:45:18PM -0400, Len Brown wrote:
> On Thu, 13 Aug 2009, Dipankar Sarma wrote:
> > In a native system, I think we should the platform-specific code
> > export what makes sense. That may be just the lowest possible
> > state only. Or may be more than one.
> 
> For x86, it is 1 state.

Native x86, yes. For virtualized systems, that may not be the
case depending on how the hypervisor behaves.

> > In a virtualized system, we would want to do at least the following -
> 
> Are you talking about Linux as a para-virtualized guest here?
> 
> > 1. An offline configuration state where the hypervisor can
> > take the cpu back and allocate it to another VM.
> 
> The hypervisor gets control no matter what idle state
> the guest enters, yes?

The hypervisor may get control, but what they do may depend
on that the guest OS wished/hinted for - config change or just
shutdown unused cpu if possible for a while.

> > 2. A low-power state where the guest indicates it doesn't need the
> > CPU (and can be put in low power state) but doesn't want to give up 
> > its allocated cpu share. IOW, no visible configuration changes.
> > 
> > So, in any case we would probably want more than one states.
> 
> How are #1 and #2 different when the hypervisor
> gets control in all idle states?  I assert that
> they are the same, and thus 1 state will suffice.

It depends on the hypervisor implementation. On pseries (powerpc)
hypervisor, for example, they are different. By offlining a vcpu
(and in turn shutting a cpu), you will actually create a configuration
change in the VM that is visible to other systems management tools
which may not be what the system administrator wanted. Ideally,
we would like to distinguish between these two states.

Hope that suffices as an example.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Wed, Aug 12, 2009 at 08:45:18PM -0400, Len Brown wrote:
> On Thu, 13 Aug 2009, Dipankar Sarma wrote:
> > In a native system, I think we should the platform-specific code
> > export what makes sense. That may be just the lowest possible
> > state only. Or may be more than one.
> 
> For x86, it is 1 state.

Native x86, yes. For virtualized systems, that may not be the
case depending on how the hypervisor behaves.

> > In a virtualized system, we would want to do at least the following -
> 
> Are you talking about Linux as a para-virtualized guest here?
> 
> > 1. An offline configuration state where the hypervisor can
> > take the cpu back and allocate it to another VM.
> 
> The hypervisor gets control no matter what idle state
> the guest enters, yes?

The hypervisor may get control, but what they do may depend
on that the guest OS wished/hinted for - config change or just
shutdown unused cpu if possible for a while.

> > 2. A low-power state where the guest indicates it doesn't need the
> > CPU (and can be put in low power state) but doesn't want to give up 
> > its allocated cpu share. IOW, no visible configuration changes.
> > 
> > So, in any case we would probably want more than one states.
> 
> How are #1 and #2 different when the hypervisor
> gets control in all idle states?  I assert that
> they are the same, and thus 1 state will suffice.

It depends on the hypervisor implementation. On pseries (powerpc)
hypervisor, for example, they are different. By offlining a vcpu
(and in turn shutting a cpu), you will actually create a configuration
change in the VM that is visible to other systems management tools
which may not be what the system administrator wanted. Ideally,
we would like to distinguish between these two states.

Hope that suffices as an example.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

> > > 2. A low-power state where the guest indicates it doesn't need the
> > > CPU (and can be put in low power state) but doesn't want to give up 
> > > its allocated cpu share. IOW, no visible configuration changes.
> > > 
> > > So, in any case we would probably want more than one states.
> > 
> > How are #1 and #2 different when the hypervisor
> > gets control in all idle states?  I assert that
> > they are the same, and thus 1 state will suffice.
> 
> It depends on the hypervisor implementation. On pseries (powerpc)
> hypervisor, for example, they are different. By offlining a vcpu
> (and in turn shutting a cpu), you will actually create a configuration
> change in the VM that is visible to other systems management tools
> which may not be what the system administrator wanted. Ideally,
> we would like to distinguish between these two states.
> 
> Hope that suffices as an example.

So... you have something like "physically pulling out hotplug cpu" on
powerpc.

But maybe it is useful to take already offline cpus (from linux side),
and make that visible to hypervisor, too.

So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
would be more useful for hypervisor case?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Pavel Machek]

> > > 2. A low-power state where the guest indicates it doesn't need the
> > > CPU (and can be put in low power state) but doesn't want to give up 
> > > its allocated cpu share. IOW, no visible configuration changes.
> > > 
> > > So, in any case we would probably want more than one states.
> > 
> > How are #1 and #2 different when the hypervisor
> > gets control in all idle states?  I assert that
> > they are the same, and thus 1 state will suffice.
> 
> It depends on the hypervisor implementation. On pseries (powerpc)
> hypervisor, for example, they are different. By offlining a vcpu
> (and in turn shutting a cpu), you will actually create a configuration
> change in the VM that is visible to other systems management tools
> which may not be what the system administrator wanted. Ideally,
> we would like to distinguish between these two states.
> 
> Hope that suffices as an example.

So... you have something like "physically pulling out hotplug cpu" on
powerpc.

But maybe it is useful to take already offline cpus (from linux side),
and make that visible to hypervisor, too.

So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
would be more useful for hypervisor case?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > 
> > It depends on the hypervisor implementation. On pseries (powerpc)
> > hypervisor, for example, they are different. By offlining a vcpu
> > (and in turn shutting a cpu), you will actually create a configuration
> > change in the VM that is visible to other systems management tools
> > which may not be what the system administrator wanted. Ideally,
> > we would like to distinguish between these two states.
> > 
> > Hope that suffices as an example.
> 
> So... you have something like "physically pulling out hotplug cpu" on
> powerpc.

If any system can do physical unplug, then it should do "offline"
with configuration changes reflected in the hypervisor and
other system configuration software.

> But maybe it is useful to take already offline cpus (from linux side),
> and make that visible to hypervisor, too.
> 
> So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> would be more useful for hypervisor case?

On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
de-allocates it from the VM hands over the control to hypervisor. The
hypervisors may do whatever it wants including allocating it to
another VM. Once gone, the original VM may not get it back depending
on the situation.

The point I am making is that we may not always want to *release*
the CPU to hypervisor and induce a configuration change. That needs
to be reflected by extending the existing user interface - hence
the proposal for - /sys/devices/system/cpu/cpu<#>/state and
/sys/devices/system/cpu/cpu<#>/available_states. It allows
ceding to hypervisor without de-allocating. It is a minor
extension of the existing interface keeping backwards compatibility
and platforms can allow what make sense.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > 
> > It depends on the hypervisor implementation. On pseries (powerpc)
> > hypervisor, for example, they are different. By offlining a vcpu
> > (and in turn shutting a cpu), you will actually create a configuration
> > change in the VM that is visible to other systems management tools
> > which may not be what the system administrator wanted. Ideally,
> > we would like to distinguish between these two states.
> > 
> > Hope that suffices as an example.
> 
> So... you have something like "physically pulling out hotplug cpu" on
> powerpc.

If any system can do physical unplug, then it should do "offline"
with configuration changes reflected in the hypervisor and
other system configuration software.

> But maybe it is useful to take already offline cpus (from linux side),
> and make that visible to hypervisor, too.
> 
> So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> would be more useful for hypervisor case?

On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
de-allocates it from the VM hands over the control to hypervisor. The
hypervisors may do whatever it wants including allocating it to
another VM. Once gone, the original VM may not get it back depending
on the situation.

The point I am making is that we may not always want to *release*
the CPU to hypervisor and induce a configuration change. That needs
to be reflected by extending the existing user interface - hence
the proposal for - /sys/devices/system/cpu/cpu<#>/state and
/sys/devices/system/cpu/cpu<#>/available_states. It allows
ceding to hypervisor without de-allocating. It is a minor
extension of the existing interface keeping backwards compatibility
and platforms can allow what make sense.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Balbir Singh]

* Dipankar Sarma <dipankar@in.ibm.com> [2009-08-16 23:56:29]:

> On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > > 
> > > It depends on the hypervisor implementation. On pseries (powerpc)
> > > hypervisor, for example, they are different. By offlining a vcpu
> > > (and in turn shutting a cpu), you will actually create a configuration
> > > change in the VM that is visible to other systems management tools
> > > which may not be what the system administrator wanted. Ideally,
> > > we would like to distinguish between these two states.
> > > 
> > > Hope that suffices as an example.
> > 
> > So... you have something like "physically pulling out hotplug cpu" on
> > powerpc.
> 
> If any system can do physical unplug, then it should do "offline"
> with configuration changes reflected in the hypervisor and
> other system configuration software.
> 
> > But maybe it is useful to take already offline cpus (from linux side),
> > and make that visible to hypervisor, too.
> > 
> > So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> > would be more useful for hypervisor case?
> 
> On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
> de-allocates it from the VM hands over the control to hypervisor. The
> hypervisors may do whatever it wants including allocating it to
> another VM. Once gone, the original VM may not get it back depending
> on the situation.
> 
> The point I am making is that we may not always want to *release*
> the CPU to hypervisor and induce a configuration change. That needs
> to be reflected by extending the existing user interface - hence
> the proposal for - /sys/devices/system/cpu/cpu<#>/state and
> /sys/devices/system/cpu/cpu<#>/available_states. It allows
> ceding to hypervisor without de-allocating. It is a minor
> extension of the existing interface keeping backwards compatibility
> and platforms can allow what make sense.
>


Agreed, I've tried to come with a little ASCII art to depict your
scenairos graphically


        +--------+ don't need (offline)
        |  OS    +----------->+------------+
        +--+-----+            | hypervisor +-----> Reuse CPU
           |                  |            |       for something
           |                  |            |       else
           |                  |            |   (visible to users)
           |                  |            |    as resource changed
           |                  +----------- +
           V (needed, but can cede)
       +------------+
       | hypervisor | Don't reuse CPU
       |            |  (CPU ceded)
       |            | give back to OS
       +------------+ when needed.
                        (Not visible to
                        users as so resource
                        binding changed)



-- 
	Balbir

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Balbir Singh]

* Dipankar Sarma <dipankar@in.ibm.com> [2009-08-16 23:56:29]:

> On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > > 
> > > It depends on the hypervisor implementation. On pseries (powerpc)
> > > hypervisor, for example, they are different. By offlining a vcpu
> > > (and in turn shutting a cpu), you will actually create a configuration
> > > change in the VM that is visible to other systems management tools
> > > which may not be what the system administrator wanted. Ideally,
> > > we would like to distinguish between these two states.
> > > 
> > > Hope that suffices as an example.
> > 
> > So... you have something like "physically pulling out hotplug cpu" on
> > powerpc.
> 
> If any system can do physical unplug, then it should do "offline"
> with configuration changes reflected in the hypervisor and
> other system configuration software.
> 
> > But maybe it is useful to take already offline cpus (from linux side),
> > and make that visible to hypervisor, too.
> > 
> > So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> > would be more useful for hypervisor case?
> 
> On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
> de-allocates it from the VM hands over the control to hypervisor. The
> hypervisors may do whatever it wants including allocating it to
> another VM. Once gone, the original VM may not get it back depending
> on the situation.
> 
> The point I am making is that we may not always want to *release*
> the CPU to hypervisor and induce a configuration change. That needs
> to be reflected by extending the existing user interface - hence
> the proposal for - /sys/devices/system/cpu/cpu<#>/state and
> /sys/devices/system/cpu/cpu<#>/available_states. It allows
> ceding to hypervisor without de-allocating. It is a minor
> extension of the existing interface keeping backwards compatibility
> and platforms can allow what make sense.
>


Agreed, I've tried to come with a little ASCII art to depict your
scenairos graphically


        +--------+ don't need (offline)
        |  OS    +----------->+------------+
        +--+-----+            | hypervisor +-----> Reuse CPU
           |                  |            |       for something
           |                  |            |       else
           |                  |            |   (visible to users)
           |                  |            |    as resource changed
           |                  +----------- +
           V (needed, but can cede)
       +------------+
       | hypervisor | Don't reuse CPU
       |            |  (CPU ceded)
       |            | give back to OS
       +------------+ when needed.
                        (Not visible to
                        users as so resource
                        binding changed)



-- 
	Balbir

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> * Dipankar Sarma <dipankar@in.ibm.com> [2009-08-16 23:56:29]:
> 
> > On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > > > 
> > > > It depends on the hypervisor implementation. On pseries (powerpc)
> > > > hypervisor, for example, they are different. By offlining a vcpu
> > > > (and in turn shutting a cpu), you will actually create a configuration
> > > > change in the VM that is visible to other systems management tools
> > > > which may not be what the system administrator wanted. Ideally,
> > > > we would like to distinguish between these two states.
> > > > 
> > > > Hope that suffices as an example.
> > > 
> > > So... you have something like "physically pulling out hotplug cpu" on
> > > powerpc.
> > 
> > If any system can do physical unplug, then it should do "offline"
> > with configuration changes reflected in the hypervisor and
> > other system configuration software.
> > 
> > > But maybe it is useful to take already offline cpus (from linux side),
> > > and make that visible to hypervisor, too.
> > > 
> > > So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> > > would be more useful for hypervisor case?
> > 
> > On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
> > de-allocates it from the VM hands over the control to hypervisor. The
> > hypervisors may do whatever it wants including allocating it to
> > another VM. Once gone, the original VM may not get it back depending
> > on the situation.
> > 
> > The point I am making is that we may not always want to *release*
> > the CPU to hypervisor and induce a configuration change. That needs
> > to be reflected by extending the existing user interface - hence
> > the proposal for - /sys/devices/system/cpu/cpu<#>/state and
> > /sys/devices/system/cpu/cpu<#>/available_states. It allows
> > ceding to hypervisor without de-allocating. It is a minor
> > extension of the existing interface keeping backwards compatibility
> > and platforms can allow what make sense.
> >
> 
> 
> Agreed, I've tried to come with a little ASCII art to depict your
> scenairos graphically
> 
> 
>         +--------+ don't need (offline)
>         |  OS    +----------->+------------+
>         +--+-----+            | hypervisor +-----> Reuse CPU
>            |                  |            |       for something
>            |                  |            |       else
>            |                  |            |   (visible to users)
>            |                  |            |    as resource changed
>            |                  +----------- +
>            V (needed, but can cede)
>        +------------+
>        | hypervisor | Don't reuse CPU
>        |            |  (CPU ceded)
>        |            | give back to OS
>        +------------+ when needed.
>                         (Not visible to
>                         users as so resource
>                         binding changed)

I still don't get it... _why_ should this be exposed in the guest
kernel? Why not let the hypervisor manage a guest's offline cpus in a
way it sees fit?

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> * Dipankar Sarma <dipankar@in.ibm.com> [2009-08-16 23:56:29]:
> 
> > On Fri, Aug 14, 2009 at 01:30:21PM +0200, Pavel Machek wrote:
> > > > 
> > > > It depends on the hypervisor implementation. On pseries (powerpc)
> > > > hypervisor, for example, they are different. By offlining a vcpu
> > > > (and in turn shutting a cpu), you will actually create a configuration
> > > > change in the VM that is visible to other systems management tools
> > > > which may not be what the system administrator wanted. Ideally,
> > > > we would like to distinguish between these two states.
> > > > 
> > > > Hope that suffices as an example.
> > > 
> > > So... you have something like "physically pulling out hotplug cpu" on
> > > powerpc.
> > 
> > If any system can do physical unplug, then it should do "offline"
> > with configuration changes reflected in the hypervisor and
> > other system configuration software.
> > 
> > > But maybe it is useful to take already offline cpus (from linux side),
> > > and make that visible to hypervisor, too.
> > > 
> > > So maybe something like "echo 1 > /sys/devices/system/cpu/cpu1/unplug"
> > > would be more useful for hypervisor case?
> > 
> > On pseries, we do an RTAS call ("stop-cpu") which effectively permantently
> > de-allocates it from the VM hands over the control to hypervisor. The
> > hypervisors may do whatever it wants including allocating it to
> > another VM. Once gone, the original VM may not get it back depending
> > on the situation.
> > 
> > The point I am making is that we may not always want to *release*
> > the CPU to hypervisor and induce a configuration change. That needs
> > to be reflected by extending the existing user interface - hence
> > the proposal for - /sys/devices/system/cpu/cpu<#>/state and
> > /sys/devices/system/cpu/cpu<#>/available_states. It allows
> > ceding to hypervisor without de-allocating. It is a minor
> > extension of the existing interface keeping backwards compatibility
> > and platforms can allow what make sense.
> >
> 
> 
> Agreed, I've tried to come with a little ASCII art to depict your
> scenairos graphically
> 
> 
>         +--------+ don't need (offline)
>         |  OS    +----------->+------------+
>         +--+-----+            | hypervisor +-----> Reuse CPU
>            |                  |            |       for something
>            |                  |            |       else
>            |                  |            |   (visible to users)
>            |                  |            |    as resource changed
>            |                  +----------- +
>            V (needed, but can cede)
>        +------------+
>        | hypervisor | Don't reuse CPU
>        |            |  (CPU ceded)
>        |            | give back to OS
>        +------------+ when needed.
>                         (Not visible to
>                         users as so resource
>                         binding changed)

I still don't get it... _why_ should this be exposed in the guest
kernel? Why not let the hypervisor manage a guest's offline cpus in a
way it sees fit?

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Sun, Aug 16, 2009 at 11:53:22PM +0200, Peter Zijlstra wrote:
> On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> > Agreed, I've tried to come with a little ASCII art to depict your
> > scenairos graphically
> > 
> > 
> >         +--------+ don't need (offline)
> >         |  OS    +----------->+------------+
> >         +--+-----+            | hypervisor +-----> Reuse CPU
> >            |                  |            |       for something
> >            |                  |            |       else
> >            |                  |            |   (visible to users)
> >            |                  |            |    as resource changed
> >            |                  +----------- +
> >            V (needed, but can cede)
> >        +------------+
> >        | hypervisor | Don't reuse CPU
> >        |            |  (CPU ceded)
> >        |            | give back to OS
> >        +------------+ when needed.
> >                         (Not visible to
> >                         users as so resource
> >                         binding changed)
> 
> I still don't get it... _why_ should this be exposed in the guest
> kernel? Why not let the hypervisor manage a guest's offline cpus in a
> way it sees fit?

For most parts, we do. The guest kernel doesn't manage the offline
CPU state. That is typically done by the hypervisor. However, offline
operation as defined now always result in a VM resize in some hypervisor
systems (like pseries) - it would be convenient to have a non-resize
offline operation which lets the guest cede the cpu to hypervisor
with the hint that the VM shouldn't be resized and the guest needs the guarantee
to get the cpu back any time. The hypervisor can do whatever it wants
with the ceded CPU including putting it in a low power state, but
not change the physical cpu shares of the VM. The pseries hypervisor,
for example, clearly distinguishes between the two - "rtas-stop-self" call
to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
is that we allow this with an extension to existing interfaces because it 
makes sense to allow sort of "hibernation" of the cpus without changing any
configuration of the VMs.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Sun, Aug 16, 2009 at 11:53:22PM +0200, Peter Zijlstra wrote:
> On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> > Agreed, I've tried to come with a little ASCII art to depict your
> > scenairos graphically
> > 
> > 
> >         +--------+ don't need (offline)
> >         |  OS    +----------->+------------+
> >         +--+-----+            | hypervisor +-----> Reuse CPU
> >            |                  |            |       for something
> >            |                  |            |       else
> >            |                  |            |   (visible to users)
> >            |                  |            |    as resource changed
> >            |                  +----------- +
> >            V (needed, but can cede)
> >        +------------+
> >        | hypervisor | Don't reuse CPU
> >        |            |  (CPU ceded)
> >        |            | give back to OS
> >        +------------+ when needed.
> >                         (Not visible to
> >                         users as so resource
> >                         binding changed)
> 
> I still don't get it... _why_ should this be exposed in the guest
> kernel? Why not let the hypervisor manage a guest's offline cpus in a
> way it sees fit?

For most parts, we do. The guest kernel doesn't manage the offline
CPU state. That is typically done by the hypervisor. However, offline
operation as defined now always result in a VM resize in some hypervisor
systems (like pseries) - it would be convenient to have a non-resize
offline operation which lets the guest cede the cpu to hypervisor
with the hint that the VM shouldn't be resized and the guest needs the guarantee
to get the cpu back any time. The hypervisor can do whatever it wants
with the ceded CPU including putting it in a low power state, but
not change the physical cpu shares of the VM. The pseries hypervisor,
for example, clearly distinguishes between the two - "rtas-stop-self" call
to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
is that we allow this with an extension to existing interfaces because it 
makes sense to allow sort of "hibernation" of the cpus without changing any
configuration of the VMs.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> On Sun, Aug 16, 2009 at 11:53:22PM +0200, Peter Zijlstra wrote:
> > On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> > > Agreed, I've tried to come with a little ASCII art to depict your
> > > scenairos graphically
> > > 
> > > 
> > >         +--------+ don't need (offline)
> > >         |  OS    +----------->+------------+
> > >         +--+-----+            | hypervisor +-----> Reuse CPU
> > >            |                  |            |       for something
> > >            |                  |            |       else
> > >            |                  |            |   (visible to users)
> > >            |                  |            |    as resource changed
> > >            |                  +----------- +
> > >            V (needed, but can cede)
> > >        +------------+
> > >        | hypervisor | Don't reuse CPU
> > >        |            |  (CPU ceded)
> > >        |            | give back to OS
> > >        +------------+ when needed.
> > >                         (Not visible to
> > >                         users as so resource
> > >                         binding changed)
> > 
> > I still don't get it... _why_ should this be exposed in the guest
> > kernel? Why not let the hypervisor manage a guest's offline cpus in a
> > way it sees fit?
> 
> For most parts, we do. The guest kernel doesn't manage the offline
> CPU state. That is typically done by the hypervisor. However, offline
> operation as defined now always result in a VM resize in some hypervisor
> systems (like pseries) - it would be convenient to have a non-resize
> offline operation which lets the guest cede the cpu to hypervisor
> with the hint that the VM shouldn't be resized and the guest needs the guarantee
> to get the cpu back any time. The hypervisor can do whatever it wants
> with the ceded CPU including putting it in a low power state, but
> not change the physical cpu shares of the VM. The pseries hypervisor,
> for example, clearly distinguishes between the two - "rtas-stop-self" call
> to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> is that we allow this with an extension to existing interfaces because it 
> makes sense to allow sort of "hibernation" of the cpus without changing any
> configuration of the VMs.

>From my POV the thing you call cede is the only sane thing to do for a
guest. Let the hypervisor management interface deal with resizing guests
if and when that's needed.

Thing is, you don't want a guest to be able to influence the amount of
cpu shares attributed to it. You want that in explicit control of
whomever manages the hypervisor.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Peter Zijlstra]

On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> On Sun, Aug 16, 2009 at 11:53:22PM +0200, Peter Zijlstra wrote:
> > On Mon, 2009-08-17 at 01:14 +0530, Balbir Singh wrote:
> > > Agreed, I've tried to come with a little ASCII art to depict your
> > > scenairos graphically
> > > 
> > > 
> > >         +--------+ don't need (offline)
> > >         |  OS    +----------->+------------+
> > >         +--+-----+            | hypervisor +-----> Reuse CPU
> > >            |                  |            |       for something
> > >            |                  |            |       else
> > >            |                  |            |   (visible to users)
> > >            |                  |            |    as resource changed
> > >            |                  +----------- +
> > >            V (needed, but can cede)
> > >        +------------+
> > >        | hypervisor | Don't reuse CPU
> > >        |            |  (CPU ceded)
> > >        |            | give back to OS
> > >        +------------+ when needed.
> > >                         (Not visible to
> > >                         users as so resource
> > >                         binding changed)
> > 
> > I still don't get it... _why_ should this be exposed in the guest
> > kernel? Why not let the hypervisor manage a guest's offline cpus in a
> > way it sees fit?
> 
> For most parts, we do. The guest kernel doesn't manage the offline
> CPU state. That is typically done by the hypervisor. However, offline
> operation as defined now always result in a VM resize in some hypervisor
> systems (like pseries) - it would be convenient to have a non-resize
> offline operation which lets the guest cede the cpu to hypervisor
> with the hint that the VM shouldn't be resized and the guest needs the guarantee
> to get the cpu back any time. The hypervisor can do whatever it wants
> with the ceded CPU including putting it in a low power state, but
> not change the physical cpu shares of the VM. The pseries hypervisor,
> for example, clearly distinguishes between the two - "rtas-stop-self" call
> to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> is that we allow this with an extension to existing interfaces because it 
> makes sense to allow sort of "hibernation" of the cpus without changing any
> configuration of the VMs.

>From my POV the thing you call cede is the only sane thing to do for a
guest. Let the hypervisor management interface deal with resizing guests
if and when that's needed.

Thing is, you don't want a guest to be able to influence the amount of
cpu shares attributed to it. You want that in explicit control of
whomever manages the hypervisor.

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 17, 2009 at 09:15:57AM +0200, Peter Zijlstra wrote:
> On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> > For most parts, we do. The guest kernel doesn't manage the offline
> > CPU state. That is typically done by the hypervisor. However, offline
> > operation as defined now always result in a VM resize in some hypervisor
> > systems (like pseries) - it would be convenient to have a non-resize
> > offline operation which lets the guest cede the cpu to hypervisor
> > with the hint that the VM shouldn't be resized and the guest needs the guarantee
> > to get the cpu back any time. The hypervisor can do whatever it wants
> > with the ceded CPU including putting it in a low power state, but
> > not change the physical cpu shares of the VM. The pseries hypervisor,
> > for example, clearly distinguishes between the two - "rtas-stop-self" call
> > to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> > is that we allow this with an extension to existing interfaces because it 
> > makes sense to allow sort of "hibernation" of the cpus without changing any
> > configuration of the VMs.
> 
> >From my POV the thing you call cede is the only sane thing to do for a
> guest. Let the hypervisor management interface deal with resizing guests
> if and when that's needed.

That is more or less how it currently works - atleast for pseries hypervisor. 
The current "offline" operation with "rtas-stop-self" call I mentioned
earlier is initiated by the hypervisor management interfaces/tool in
pseries system. This wakes up a guest system tool that echoes "1"
to the offline file resulting in the configuration change.
The OS involvement is necessary to evacuate tasks/interrupts
from the released CPU. We don't really want to initiate this from guests.

> Thing is, you don't want a guest to be able to influence the amount of
> cpu shares attributed to it. You want that in explicit control of
> whomever manages the hypervisor.

Agreed. But given a fixed cpu share by the hypervisor management tools,
we would like to be able to cede cpus to hypervisor leaving the hypervisor
configuration intact. This, we don't have at the moment and want to just
extend the current interface for this.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 17, 2009 at 09:15:57AM +0200, Peter Zijlstra wrote:
> On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> > For most parts, we do. The guest kernel doesn't manage the offline
> > CPU state. That is typically done by the hypervisor. However, offline
> > operation as defined now always result in a VM resize in some hypervisor
> > systems (like pseries) - it would be convenient to have a non-resize
> > offline operation which lets the guest cede the cpu to hypervisor
> > with the hint that the VM shouldn't be resized and the guest needs the guarantee
> > to get the cpu back any time. The hypervisor can do whatever it wants
> > with the ceded CPU including putting it in a low power state, but
> > not change the physical cpu shares of the VM. The pseries hypervisor,
> > for example, clearly distinguishes between the two - "rtas-stop-self" call
> > to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> > is that we allow this with an extension to existing interfaces because it 
> > makes sense to allow sort of "hibernation" of the cpus without changing any
> > configuration of the VMs.
> 
> >From my POV the thing you call cede is the only sane thing to do for a
> guest. Let the hypervisor management interface deal with resizing guests
> if and when that's needed.

That is more or less how it currently works - atleast for pseries hypervisor. 
The current "offline" operation with "rtas-stop-self" call I mentioned
earlier is initiated by the hypervisor management interfaces/tool in
pseries system. This wakes up a guest system tool that echoes "1"
to the offline file resulting in the configuration change.
The OS involvement is necessary to evacuate tasks/interrupts
from the released CPU. We don't really want to initiate this from guests.

> Thing is, you don't want a guest to be able to influence the amount of
> cpu shares attributed to it. You want that in explicit control of
> whomever manages the hypervisor.

Agreed. But given a fixed cpu share by the hypervisor management tools,
we would like to be able to cede cpus to hypervisor leaving the hypervisor
configuration intact. This, we don't have at the moment and want to just
extend the current interface for this.

Thanks
Dipankar

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 17, 2009 at 01:28:15PM +0530, Dipankar Sarma wrote:
> On Mon, Aug 17, 2009 at 09:15:57AM +0200, Peter Zijlstra wrote:
> > On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> > > For most parts, we do. The guest kernel doesn't manage the offline
> > > CPU state. That is typically done by the hypervisor. However, offline
> > > operation as defined now always result in a VM resize in some hypervisor
> > > systems (like pseries) - it would be convenient to have a non-resize
> > > offline operation which lets the guest cede the cpu to hypervisor
> > > with the hint that the VM shouldn't be resized and the guest needs the guarantee
> > > to get the cpu back any time. The hypervisor can do whatever it wants
> > > with the ceded CPU including putting it in a low power state, but
> > > not change the physical cpu shares of the VM. The pseries hypervisor,
> > > for example, clearly distinguishes between the two - "rtas-stop-self" call
> > > to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> > > is that we allow this with an extension to existing interfaces because it 
> > > makes sense to allow sort of "hibernation" of the cpus without changing any
> > > configuration of the VMs.
> > 
> > >From my POV the thing you call cede is the only sane thing to do for a
> > guest. Let the hypervisor management interface deal with resizing guests
> > if and when that's needed.
> 
> That is more or less how it currently works - atleast for pseries hypervisor. 
> The current "offline" operation with "rtas-stop-self" call I mentioned
> earlier is initiated by the hypervisor management interfaces/tool in
> pseries system. This wakes up a guest system tool that echoes "1"
> to the offline file resulting in the configuration change.

Should have said - echoes "0" to the online file. 

You don't necessarily need this in the guest Linux as long as there is
a way for hypervisor tools to internally move Linux tasks/interrupts
from a vcpu - async event handled by the kernel, for example.
But I think it is too late for that - the interface has long been
exported.


> The OS involvement is necessary to evacuate tasks/interrupts
> from the released CPU. We don't really want to initiate this from guests.
> 
> > Thing is, you don't want a guest to be able to influence the amount of
> > cpu shares attributed to it. You want that in explicit control of
> > whomever manages the hypervisor.
> 
> Agreed. But given a fixed cpu share by the hypervisor management tools,
> we would like to be able to cede cpus to hypervisor leaving the hypervisor
> configuration intact. This, we don't have at the moment and want to just
> extend the current interface for this.
> 
> Thanks
> Dipankar
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [PATCH 0/3] cpu: idle state framework for offline CPUs.

[Dipankar Sarma]

On Mon, Aug 17, 2009 at 01:28:15PM +0530, Dipankar Sarma wrote:
> On Mon, Aug 17, 2009 at 09:15:57AM +0200, Peter Zijlstra wrote:
> > On Mon, 2009-08-17 at 11:54 +0530, Dipankar Sarma wrote:
> > > For most parts, we do. The guest kernel doesn't manage the offline
> > > CPU state. That is typically done by the hypervisor. However, offline
> > > operation as defined now always result in a VM resize in some hypervisor
> > > systems (like pseries) - it would be convenient to have a non-resize
> > > offline operation which lets the guest cede the cpu to hypervisor
> > > with the hint that the VM shouldn't be resized and the guest needs the guarantee
> > > to get the cpu back any time. The hypervisor can do whatever it wants
> > > with the ceded CPU including putting it in a low power state, but
> > > not change the physical cpu shares of the VM. The pseries hypervisor,
> > > for example, clearly distinguishes between the two - "rtas-stop-self" call
> > > to resize VM vs. H_CEDE hypercall with a hint. What I am suggesting
> > > is that we allow this with an extension to existing interfaces because it 
> > > makes sense to allow sort of "hibernation" of the cpus without changing any
> > > configuration of the VMs.
> > 
> > >From my POV the thing you call cede is the only sane thing to do for a
> > guest. Let the hypervisor management interface deal with resizing guests
> > if and when that's needed.
> 
> That is more or less how it currently works - atleast for pseries hypervisor. 
> The current "offline" operation with "rtas-stop-self" call I mentioned
> earlier is initiated by the hypervisor management interfaces/tool in
> pseries system. This wakes up a guest system tool that echoes "1"
> to the offline file resulting in the configuration change.

Should have said - echoes "0" to the online file. 

You don't necessarily need this in the guest Linux as long as there is
a way for hypervisor tools to internally move Linux tasks/interrupts
from a vcpu - async event handled by the kernel, for example.
But I think it is too late for that - the interface has long been
exported.


> The OS involvement is necessary to evacuate tasks/interrupts
> from the released CPU. We don't really want to initiate this from guests.
> 
> > Thing is, you don't want a guest to be able to influence the amount of
> > cpu shares attributed to it. You want that in explicit control of
> > whomever manages the hypervisor.
> 
> Agreed. But given a fixed cpu share by the hypervisor management tools,
> we would like to be able to cede cpus to hypervisor leaving the hypervisor
> configuration intact. This, we don't have at the moment and want to just
> extend the current interface for this.
> 
> Thanks
> Dipankar
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>