[PATCH v4 0/7] x86: BSP or CPU0 online/offline

[PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

BSP or CPU0 has been the last obstacle to CPU hotplug on x86. This patch set
implements BSP online and offline and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any
thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is
required to be offline or hot replaced to keep the system run.

v4: Add __read_mostly for bsp_hotpluggable variable. Add my email address in
cpu-hotplug.txt document. A wording change in comment.

v3: Register a pm notifier to check if CPU0 is online before hibernate/suspend.
Small wording changes in document and print info.

v2: Add locking changes between cpu hotplug and hibernate/suspend. Change PIC
irq bound to CPU0 detection.

Fenghua Yu (7):
  x86/topology.c: Support functions for BSP online/offline
  x86/common.c: Init BSP data during BSP online
  x86/mtrr/main.c: Ask the first online CPU to save mtrr
  x86/smpboot.c: Don't offline BSP if any irq can not be migrated out
    of it
  Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
    online/offline feature
  x86/i387.c: Thread xstate is initialized only on BSP once
  x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline

 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++
 Documentation/kernel-parameters.txt |   13 ++++++++++
 arch/x86/include/asm/processor.h    |    1 +
 arch/x86/kernel/cpu/common.c        |   13 ++++++++--
 arch/x86/kernel/cpu/mtrr/main.c     |    9 +++++-
 arch/x86/kernel/i387.c              |    9 ++++++-
 arch/x86/kernel/smpboot.c           |   43 ++++++++++++++++++++++++++++-----
 arch/x86/kernel/topology.c          |   24 +++++++++++++-----
 arch/x86/power/cpu.c                |   44 +++++++++++++++++++++++++++++++++++
 9 files changed, 155 insertions(+), 20 deletions(-)

[PATCH v4 1/7] x86/topology.c: Support functions for BSP online/offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel
parameter bsp_hotplug can enable BSP hotplug feature.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/topology.c |   24 +++++++++++++++++-------
 1 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
index 76ee977..4493868 100644
--- a/arch/x86/kernel/topology.c
+++ b/arch/x86/kernel/topology.c
@@ -35,18 +35,28 @@
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+static int __read_mostly bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 1;
+}
+
+__setup("bsp_hotplug", enable_bsp_hotplug);
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-	 * CPU0 cannot be offlined due to several
-	 * restrictions and assumptions in kernel. This basically
-	 * doesn't add a control file, one cannot attempt to offline
-	 * BSP.
+	 * Resume from suspend/hibernate depends on BSP. PIC interrupts depend
+	 * on BSP.
 	 *
-	 * Also certain PCI quirks require not to enable hotplug control
-	 * for all CPU's.
+	 * If the BSP depencies are under control, one can tell kernel to
+	 * enable BSP hotplug. This basically adds a control file and
+	 * one can attempt to offline BSP.
 	 */
-	if (num)
+	if (num || bsp_hotpluggable)
 		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
 
 	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
-- 
1.6.0.3

[PATCH v4 2/7] x86/common.c: Init BSP data during BSP online

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

During BSP online, enable x2apic, set_numa_node, add numa mask, and enable
sep cpu for BSP.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/processor.h |    1 +
 arch/x86/kernel/cpu/common.c     |   13 ++++++++++---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index b650435..4057161 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -162,6 +162,7 @@ extern struct pt_regs *idle_regs(struct pt_regs *);
 
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
+extern void identify_boot_cpu_online(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
 extern void print_cpu_info(struct cpuinfo_x86 *);
 extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c);
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index aa003b1..faa3bda 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -916,6 +916,14 @@ void __init identify_boot_cpu(void)
 #endif
 }
 
+void __cpuinit identify_boot_cpu_online(void)
+{
+	numa_add_cpu(smp_processor_id());
+#ifdef CONFIG_X86_32
+	enable_sep_cpu();
+#endif
+}
+
 void __cpuinit identify_secondary_cpu(struct cpuinfo_x86 *c)
 {
 	BUG_ON(c == &boot_cpu_data);
@@ -1163,7 +1171,7 @@ void __cpuinit cpu_init(void)
 	oist = &per_cpu(orig_ist, cpu);
 
 #ifdef CONFIG_NUMA
-	if (cpu != 0 && percpu_read(numa_node) == 0 &&
+	if (percpu_read(numa_node) == 0 &&
 	    early_cpu_to_node(cpu) != NUMA_NO_NODE)
 		set_numa_node(early_cpu_to_node(cpu));
 #endif
@@ -1195,8 +1203,7 @@ void __cpuinit cpu_init(void)
 	barrier();
 
 	x86_configure_nx();
-	if (cpu != 0)
-		enable_x2apic();
+	enable_x2apic();
 
 	/*
 	 * set up and load the per-CPU TSS
-- 
1.6.0.3

[PATCH v4 3/7] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Ask the first online CPU to save mtrr instead of asking BSP. BSP could be
offline when mtrr_save_state() is called.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/cpu/mtrr/main.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
index 6b96110..e4c1a41 100644
--- a/arch/x86/kernel/cpu/mtrr/main.c
+++ b/arch/x86/kernel/cpu/mtrr/main.c
@@ -695,11 +695,16 @@ void mtrr_ap_init(void)
 }
 
 /**
- * Save current fixed-range MTRR state of the BSP
+ * Save current fixed-range MTRR state of the first cpu in cpu_online_mask.
  */
 void mtrr_save_state(void)
 {
-	smp_call_function_single(0, mtrr_save_fixed_ranges, NULL, 1);
+	int first_cpu;
+
+	get_online_cpus();
+	first_cpu = cpumask_first(cpu_online_mask);
+	smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
+	put_online_cpus();
 }
 
 void set_mtrr_aps_delayed_init(void)
-- 
1.6.0.3

[PATCH v4 4/7] x86/smpboot.c: Don't offline BSP if any irq can not be migrated out of it

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Don't offline BSP if any irq can not be migrated out of it.

Call identify_boot_cpu_online() for BSP in smp_callin() and continue to online
BSP in native_cpu_up().

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/smpboot.c |   43 ++++++++++++++++++++++++++++++++++++-------
 1 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 9f548cb..83838c5 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -136,8 +136,8 @@ EXPORT_PER_CPU_SYMBOL(cpu_info);
 atomic_t init_deasserted;
 
 /*
- * Report back to the Boot Processor.
- * Running on AP.
+ * Report back to the Boot Processor during boot time or to the caller processor
+ * during CPU online.
  */
 static void __cpuinit smp_callin(void)
 {
@@ -224,6 +224,13 @@ static void __cpuinit smp_callin(void)
 	smp_store_cpu_info(cpuid);
 
 	/*
+	 * This function won't run on the BSP during boot time. It run
+	 * on BSP only when BSP is offlined and onlined again.
+	 */
+	if (cpuid == 0)
+		identify_boot_cpu_online();
+
+	/*
 	 * This must be done before setting cpu_online_mask
 	 * or calling notify_cpu_starting.
 	 */
@@ -839,7 +846,7 @@ int __cpuinit native_cpu_up(unsigned int cpu)
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || apicid == boot_cpu_physical_apicid ||
+	if (apicid == BAD_APICID ||
 	    !physid_isset(apicid, phys_cpu_present_map)) {
 		printk(KERN_ERR "%s: bad cpu %d\n", __func__, cpu);
 		return -EINVAL;
@@ -1283,12 +1290,34 @@ int native_cpu_disable(void)
 	 * Perhaps use cpufreq to drop frequency, but that could go
 	 * into generic code.
 	 *
-	 * We won't take down the boot processor on i386 due to some
+	 * We won't take down the boot processor on i386 if some
 	 * interrupts only being able to be serviced by the BSP.
-	 * Especially so if we're not using an IOAPIC	-zwane
+	 * Especially so if we're not using an IOAPIC
+	 * -original comment from zwane, changed by fyu
 	 */
-	if (cpu == 0)
-		return -EBUSY;
+	if (cpu == 0) {
+		int irq;
+		struct irq_desc *desc;
+		struct irq_data *data;
+		struct irq_chip *chip;
+
+		for_each_irq_desc(irq, desc) {
+			raw_spin_lock(&desc->lock);
+			if (!irq_has_action(irq)) {
+				raw_spin_unlock(&desc->lock);
+				continue;
+			}
+
+			data = irq_desc_get_irq_data(desc);
+			chip = irq_data_get_irq_chip(data);
+			if (!chip->irq_set_affinity) {
+				pr_debug("irq%d can't move out of BSP\n", irq);
+				raw_spin_unlock(&desc->lock);
+				return -EBUSY;
+			}
+			raw_spin_unlock(&desc->lock);
+		}
+	}
 
 	clear_local_APIC();
 
-- 
1.6.0.3

[PATCH v4 5/7] Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Add x86 CPU0 online/offline feature in the two documentations.

By default BSP is not hotpluggable. Kernel parameter bsp_hotplug enables BSP
online/offline feature.

The documentations point out two BSP dependencies. First, resume from hibernate
or suspend always starts from BSP. So hibernate and suspend are prevented if BSP
is offline. Another dependency is PIC interrupts always go to BSP.

It's said that some machines may depend on CPU0 to poweroff/reboot. But I
haven't seen such dependency on a few tested machines.

Please let me know if you see any CPU0 dependencies on your machine.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++++++
 Documentation/kernel-parameters.txt |   13 +++++++++++++
 2 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
index a20bfd4..0ca600d 100644
--- a/Documentation/cpu-hotplug.txt
+++ b/Documentation/cpu-hotplug.txt
@@ -207,6 +207,25 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If given kernel option bsp_hotplug, CPU0 is removable.
+
+But some features depend on BSP. Known dependencies are:
+1. Resume from hibernate/suspend depends on BSP. Hibernate/suspend will fail if
+BSP is offline and you need to online BSP before hibernate/suspend can continue.
+2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on BSP on some machines although I haven't
+seen any poweroff/reboot failure so far after BSP is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of BSP.
+
+If the dependencies are under your control, you can turn on bsp_hotplug.
+
+--Fenghua Yu <fenghua.yu@intel.com>
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index a0c5c5f..55582db 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1842,6 +1842,19 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
 
+	bsp_hotplug	[X86] BSP (aka CPU0) is hotpluggable.
+			Some features depend on BSP. Known dependencies are:
+			1. Resume from suspend/hibernate depends on BSP.
+			Suspend/hibernate will fail if BSP is offline and you
+			need to online BSP before suspend/hibernate.
+			2. PIC interrupts also depend on BSP. BSP can't be
+			removed if a PIC interrupt is detected.
+			It's said poweroff/reboot may depend on BSP on some
+			machines although I haven't seen such issues so far
+			after BSP is offline on a few tested machines.
+			If the dependencies are under your control, you can
+			turn on bsp_hotplug.
+
 	nptcg=		[IA-64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
-- 
1.6.0.3

[PATCH v4 6/7] x86/i387.c: Thread xstate is initialized only on BSP once

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

init_thread_xstate() is only called on BSP once to avoid overriding xstate_size.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/i387.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
index 739d859..72652ff 100644
--- a/arch/x86/kernel/i387.c
+++ b/arch/x86/kernel/i387.c
@@ -93,6 +93,7 @@ void __cpuinit fpu_init(void)
 {
 	unsigned long cr0;
 	unsigned long cr4_mask = 0;
+	static int once = 1;
 
 	if (cpu_has_fxsr)
 		cr4_mask |= X86_CR4_OSFXSR;
@@ -107,8 +108,14 @@ void __cpuinit fpu_init(void)
 		cr0 |= X86_CR0_EM;
 	write_cr0(cr0);
 
-	if (!smp_processor_id())
+	/*
+	 * init_thread_xstate is only called on BSP once to avoid overriding
+	 * xstate_size.
+	 */
+	if (!smp_processor_id() && once) {
+		once = 0;
 		init_thread_xstate();
+	}
 
 	mxcsr_feature_mask_init();
 	/* clean state in init */
-- 
1.6.0.3

[PATCH v4 7/7] x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Because x86 BIOS requires CPU0 to resume from sleep, suspend or hibernate
can't be executed if CPU0 is detected offline.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/power/cpu.c |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index f10c0af..a4ec084 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -236,3 +236,47 @@ void restore_processor_state(void)
 #ifdef CONFIG_X86_32
 EXPORT_SYMBOL(restore_processor_state);
 #endif
+
+/*
+ * When bsp_check() is called in hibernate and suspend, cpu hotplug
+ * is disabled already. So it's unnessary to handle race condition between
+ * cpumask query and cpu hotplug.
+ */
+static int bsp_check(void)
+{
+	if (cpumask_first(cpu_online_mask) != 0) {
+		printk(KERN_WARNING "CPU0 is offline.\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int bsp_pm_callback(struct notifier_block *nb, unsigned long action,
+			   void *ptr)
+{
+	int ret = 0;
+
+	switch (action) {
+	case PM_SUSPEND_PREPARE:
+	case PM_HIBERNATION_PREPARE:
+		ret = bsp_check();
+		break;
+	default:
+		break;
+	}
+	return notifier_from_errno(ret);
+}
+
+static int __init bsp_pm_check_init(void)
+{
+	/*
+	 * Set this bsp_pm_callback as lower priority than
+	 * cpu_hotplug_pm_callback. So cpu_hotplug_pm_callback will be called
+	 * earlier to disable cpu hotplug before bsp online check.
+	 */
+	pm_notifier(bsp_pm_callback, -INT_MAX);
+	return 0;
+}
+
+core_initcall(bsp_pm_check_init);
-- 
1.6.0.3

Re: [PATCH v4 6/7] x86/i387.c: Thread xstate is initialized only on BSP once

[Brian Gerst]

On Sat, Nov 12, 2011 at 12:26 AM, Fenghua Yu <fenghua.yu@intel.com> wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
>
> init_thread_xstate() is only called on BSP once to avoid overriding xstate_size.
>
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
> ---
>  arch/x86/kernel/i387.c |    9 ++++++++-
>  1 files changed, 8 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
> index 739d859..72652ff 100644
> --- a/arch/x86/kernel/i387.c
> +++ b/arch/x86/kernel/i387.c
> @@ -93,6 +93,7 @@ void __cpuinit fpu_init(void)
>  {
>        unsigned long cr0;
>        unsigned long cr4_mask = 0;
> +       static int once = 1;
>
>        if (cpu_has_fxsr)
>                cr4_mask |= X86_CR4_OSFXSR;
> @@ -107,8 +108,14 @@ void __cpuinit fpu_init(void)
>                cr0 |= X86_CR0_EM;
>        write_cr0(cr0);
>
> -       if (!smp_processor_id())
> +       /*
> +        * init_thread_xstate is only called on BSP once to avoid overriding
> +        * xstate_size.
> +        */
> +       if (!smp_processor_id() && once) {
> +               once = 0;
>                init_thread_xstate();
> +       }

Just check if xstate_size is 0 instead of adding a new flag.

--
Brian Gerst

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Fenghua Yu <fenghua.yu@intel.com> wrote:

> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> BSP or CPU0 has been the last obstacle to CPU hotplug on x86. 
> This patch set implements BSP online and offline and removes 
> this obstacle to CPU hotplug.
> 
> RAS needs the feature. If socket0 needs to be hotplugged for 
> any reason (any thread on socket0 is bad, shared cache issue, 
> uncore issue, etc), CPU0 is required to be offline or hot 
> replaced to keep the system run.
> 
> v4: Add __read_mostly for bsp_hotpluggable variable. Add my 
> email address in cpu-hotplug.txt document. A wording change in 
> comment.
> 
> v3: Register a pm notifier to check if CPU0 is online before 
> hibernate/suspend. Small wording changes in document and print 
> info.
> 
> v2: Add locking changes between cpu hotplug and 
> hibernate/suspend. Change PIC irq bound to CPU0 detection.
> 
> Fenghua Yu (7):
>   x86/topology.c: Support functions for BSP online/offline
>   x86/common.c: Init BSP data during BSP online
>   x86/mtrr/main.c: Ask the first online CPU to save mtrr
>   x86/smpboot.c: Don't offline BSP if any irq can not be migrated out
>     of it
>   Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
>     online/offline feature
>   x86/i387.c: Thread xstate is initialized only on BSP once
>   x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline
> 
>  Documentation/cpu-hotplug.txt       |   19 +++++++++++++++
>  Documentation/kernel-parameters.txt |   13 ++++++++++
>  arch/x86/include/asm/processor.h    |    1 +
>  arch/x86/kernel/cpu/common.c        |   13 ++++++++--
>  arch/x86/kernel/cpu/mtrr/main.c     |    9 +++++-
>  arch/x86/kernel/i387.c              |    9 ++++++-
>  arch/x86/kernel/smpboot.c           |   43 ++++++++++++++++++++++++++++-----
>  arch/x86/kernel/topology.c          |   24 +++++++++++++-----
>  arch/x86/power/cpu.c                |   44 +++++++++++++++++++++++++++++++++++
>  9 files changed, 155 insertions(+), 20 deletions(-)

This is an interesting but totally scary feature to me! :-)

One aspect that makes it essentially undebuggable is that right 
now it needs a boot parameter to even activate. Few people will 
test it this way.

So at minimum i'd suggest adding a new Kconfig switch ak'a 
CONFIG_BOOTPARAM_HOTPLUG_BOOT_CPU (disabled by default) which 
alpha-testers, randconfig setups and distributions with suicidal 
tendencies could enable by default.

Also, could you please enumerate all limitations that could 
possibly happen? The documentation has this list right now:

+1. Resume from hibernate/suspend depends on BSP. Hibernate/suspend will fail if
+BSP is offline and you need to online BSP before hibernate/suspend can continue.

This needs to be fixed on some other fashion than warning people 
in documentation that it would break.

Firstly, at minimum a suspend/hibernate attempt should fail in 
some deterministic fashion.

Secondly, and more importantly, is there *any* hardware in 
existence that has a BIOS that can suspend/resume successfully 
with BSP offlined? If such hardware exists then we need to 
support it properly - initially perhaps by whitelisting such 
systems.

Then if demand for this picks up some more intelligent method of 
cooperating with the firmware could be added: the firmware could 
actually signal to us whether it supports suspend/resume from 
other than the boot CPU.

+2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on BSP on some machines although I haven't
+seen any poweroff/reboot failure so far after BSP is offline on a few tested
+machines.

We need a debug feature for this: CONFIG_DEBUG_BOOT_CPU_OFF=y or 
such (disabled by default): this feature would offline the boot 
CPU as soon as possible, and boot up userspace with the boot CPU 
offlined.

So these are the things we need to even begin considering such a 
patch-set for mainline.

Thanks,

	Ingo

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> Secondly, and more importantly, is there *any* hardware in 
> existence that has a BIOS that can suspend/resume successfully 
> with BSP offlined? If such hardware exists then we need to 
> support it properly - initially perhaps by whitelisting such 
> systems.

I suspect the answer to that is 'no' - as resume is really just 
a fresh bootup of the physical CPU and BIOSen just start on the 
boot CPU, no questions asked.

So the right approach there would be to detect the case where we 
boot up back from S2RAM resume on an offlined CPU (the BSP is 
really just one of the possibilities - in theory a S2RAM resume 
could boot back up on any of the APs as well) - the resume code 
should move off that CPU ASAP and keep that CPU offlined.

But the hibernation angle should be considered. Hibernation 
already has to deal with the case where someone physically 
unplugs a CPU and then resumes from the disk image, right? How 
does the hibernation code handle that case currently?

Thanks,

	Ingo

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 02:28 PM, Ingo Molnar wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
>> Secondly, and more importantly, is there *any* hardware in 
>> existence that has a BIOS that can suspend/resume successfully 
>> with BSP offlined? If such hardware exists then we need to 
>> support it properly - initially perhaps by whitelisting such 
>> systems.
> 
> I suspect the answer to that is 'no' - as resume is really just 
> a fresh bootup of the physical CPU and BIOSen just start on the 
> boot CPU, no questions asked.
> 
> So the right approach there would be to detect the case where we 
> boot up back from S2RAM resume on an offlined CPU (the BSP is 
> really just one of the possibilities - in theory a S2RAM resume 
> could boot back up on any of the APs as well) - the resume code 
> should move off that CPU ASAP and keep that CPU offlined.
> 
> But the hibernation angle should be considered. Hibernation 
> already has to deal with the case where someone physically 
> unplugs a CPU and then resumes from the disk image, right? How 
> does the hibernation code handle that case currently?
> 


Oh, wait a minute. Are we talking about physical CPU Hotplug in between
hibernation and restore? AFAIK, currently we don't handle that at all. 
One of the things I recall in this context is that, while developing
the x86 microcode update optimization patch (now, mainline commit Id
7098944), it was pointed out that physical CPU hotplug on x86, and that
too *in-between* hibernation and restore, is very far-fetched and not
handled at present.

I had posted another patch to go along with the above optimization patch
so that we don't break stuff when physical cpu hotplug comes into picture.
Link: http://thread.gmane.org/gmane.linux.kernel/1205405/focus=1205784
May be it will come of use now, as it is, or in a better form.

Regards,
Srivatsa S. Bhat
IBM Linux Technology Center

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> wrote:

> > But the hibernation angle should be considered. Hibernation 
> > already has to deal with the case where someone physically 
> > unplugs a CPU and then resumes from the disk image, right? 
> > How does the hibernation code handle that case currently?
> 
> Oh, wait a minute. Are we talking about physical CPU Hotplug 
> in between hibernation and restore? AFAIK, currently we don't 
> handle that at all. [...]

Well, not hotplug, but plain old-fashioned: 'hibernate the box, 
take out a CPU physically and thaw' kind of hardware change.

How will the hibernation code handle this case?

Thanks,

	Ingo

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 04:05 PM, Ingo Molnar wrote:

> 
> * Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> wrote:
> 
>>> But the hibernation angle should be considered. Hibernation 
>>> already has to deal with the case where someone physically 
>>> unplugs a CPU and then resumes from the disk image, right? 
>>> How does the hibernation code handle that case currently?
>>
>> Oh, wait a minute. Are we talking about physical CPU Hotplug 
>> in between hibernation and restore? AFAIK, currently we don't 
>> handle that at all. [...]
> 
> Well, not hotplug, but plain old-fashioned: 'hibernate the box, 
> take out a CPU physically and thaw' kind of hardware change.
>


Well, actually that's what I meant, taking this same idea a bit
further: hibernate the box, take out a CPU physically, insert a
slightly different CPU (which needs a different microcode image)
and thaw.

In this case, the other patch that I mentioned in my previous mail
would be required (or an equivalent), because the optimization
patch which is now in mainline, would apply the same old microcode
image on this new CPU too, blindly.

 
> How will the hibernation code handle this case?


I am not familiar with how the rest of the hibernation code handles
the case you described. I just happen to know about the microcode
case needing some more work for this to work reliably.

Regards,
Srivatsa S. Bhat
IBM Linux Technology Center

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 04:17 PM, Srivatsa S. Bhat wrote:

> On 12/06/2011 04:05 PM, Ingo Molnar wrote:
> 
>>
>> * Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> wrote:
>>
>>>> But the hibernation angle should be considered. Hibernation 
>>>> already has to deal with the case where someone physically 
>>>> unplugs a CPU and then resumes from the disk image, right? 
>>>> How does the hibernation code handle that case currently?
>>>
>>> Oh, wait a minute. Are we talking about physical CPU Hotplug 
>>> in between hibernation and restore? AFAIK, currently we don't 
>>> handle that at all. [...]
>>
>> Well, not hotplug, but plain old-fashioned: 'hibernate the box, 
>> take out a CPU physically and thaw' kind of hardware change.
>>
> 
> 
> Well, actually that's what I meant, taking this same idea a bit
> further: hibernate the box, take out a CPU physically, insert a
> slightly different CPU (which needs a different microcode image)
> and thaw.
> 
> In this case, the other patch that I mentioned in my previous mail
> would be required (or an equivalent), because the optimization
> patch which is now in mainline, would apply the same old microcode
> image on this new CPU too, blindly.
> 


By the way, this problem is not tied to CPU0 alone, it exists for any
CPU! (as long as we are talking about plugging in/out CPUs physically).

Regards,
Srivatsa S. Bhat

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Borislav Petkov]

On Tue, Dec 06, 2011 at 04:17:58PM +0530, Srivatsa S. Bhat wrote:
> In this case, the other patch that I mentioned in my previous mail
> would be required (or an equivalent), because the optimization
> patch which is now in mainline, would apply the same old microcode
> image on this new CPU too, blindly.

Not blindly, the microcode is still verified.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Borislav Petkov]

On Tue, Dec 06, 2011 at 04:55:02PM +0530, Srivatsa S. Bhat wrote:
> By the way, this problem is not tied to CPU0 alone, it exists for any
> CPU! (as long as we are talking about plugging in/out CPUs physically).

Just a reminder: before you guys go and wander off into the woods of
hypothetical with this, please make sure this use case is relevant
enough for the trouble. The only real reason given so far AFAICT was RAS
and to be able to offline BSP in order to prolong system life before
maintenance.

When you take it down for maintenance eventually, you don't need to
suspend but simply poweroff.

Or...

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Borislav Petkov <bp@amd64.org> wrote:

> On Tue, Dec 06, 2011 at 04:55:02PM +0530, Srivatsa S. Bhat wrote:
>
> > By the way, this problem is not tied to CPU0 alone, it 
> > exists for any CPU! (as long as we are talking about 
> > plugging in/out CPUs physically).
> 
> Just a reminder: before you guys go and wander off into the 
> woods of hypothetical with this, please make sure this use 
> case is relevant enough for the trouble. The only real reason 
> given so far AFAICT was RAS and to be able to offline BSP in 
> order to prolong system life before maintenance.
> 
> When you take it down for maintenance eventually, you don't 
> need to suspend but simply poweroff.

I think it's definitely a marginal and speculative feature - but 
the patches don't look overly complicated, so i'm not 
*completely* against removing various boot-CPU assumptions 
(although i'm predisposed against it) - if it is correct and if 
there's someone interested in doing proper patches.

All in one, the quality threshold for inclusion is very high but 
not an infinite number.

The specific point i tried to make about s2ram is to make sure 
it does not break during normal usage: for example someone 
offlines the boot CPU, but the box then gets suspended - that 
should not hang or crash.

Thanks,

	Ingo

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 06:30 PM, Borislav Petkov wrote:

> On Tue, Dec 06, 2011 at 04:17:58PM +0530, Srivatsa S. Bhat wrote:
>> In this case, the other patch that I mentioned in my previous mail
>> would be required (or an equivalent), because the optimization
>> patch which is now in mainline, would apply the same old microcode
>> image on this new CPU too, blindly.
> 
> Not blindly, the microcode is still verified.
> 


I saw your other mail about the validity of this whole scenario, and I
kind of agree to your point.

My thoughts below might not be so relevant/significant considering that,
but anyways, just to understand what you said above:
I didn't quite find where the microcode is verified if the kernel happens
to have the microcode image already.

>From what I saw, microcode image is not freed nor invalidated when a
CPU goes down (which was introduced by the optimization patch). And hence,
when the CPU comes back up, the call sequence would look something like:

case CPU_ONLINE:
case CPU_ONLINE_FROZEN:
    * microcode_update_cpu(cpu);
        * microcode_resume_cpu(cpu) /* Because uci->valid was 1 */
            * eventually calls apply_microcode_amd(cpu) for AMD or
                               apply_microcode(cpu) for Intel

And both these functions simply write the microcode image to the CPU
and then check whether the _write_ was successful, (not whether the required
microcode version was applied).

That was why, to take care of this, the other patch (below) was written, IMHO.
http://thread.gmane.org/gmane.linux.kernel/1205405/
 
Regards,
Srivatsa S. Bhat

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 02:12 PM, Ingo Molnar wrote:

> 
> * Fenghua Yu <fenghua.yu@intel.com> wrote:
> 
>> From: Fenghua Yu <fenghua.yu@intel.com>
>>
>> BSP or CPU0 has been the last obstacle to CPU hotplug on x86. 
>> This patch set implements BSP online and offline and removes 
>> this obstacle to CPU hotplug.

[snip]
> 

> Also, could you please enumerate all limitations that could 
> possibly happen? The documentation has this list right now:
> 
> +1. Resume from hibernate/suspend depends on BSP. Hibernate/suspend will fail if
> +BSP is offline and you need to online BSP before hibernate/suspend can continue.
> 
> This needs to be fixed on some other fashion than warning people 
> in documentation that it would break.
> 


Actually, this patchset does more than just warn people. It has
checks to see if the CPU0 is offline, and if so, it fails the
suspend/hibernate attempt. See patch 7/7
(x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline)

> Firstly, at minimum a suspend/hibernate attempt should fail in 
> some deterministic fashion.


It does, as mentioned above. In fact, this patchset does it
proactively: whether the hardware/firmware supports suspend/resume
with BSP offlined or not, it just prevents anybody from doing
suspend/hibernate when the boot CPU is offline.

I am not saying that this is the *right* way to do it; I am just
pointing out that this patchset _does_ handle it.

> 
> Secondly, and more importantly, is there *any* hardware in 
> existence that has a BIOS that can suspend/resume successfully 
> with BSP offlined? If such hardware exists then we need to 
> support it properly - initially perhaps by whitelisting such 
> systems.
> 
> Then if demand for this picks up some more intelligent method of 
> cooperating with the firmware could be added: the firmware could 
> actually signal to us whether it supports suspend/resume from 
> other than the boot CPU.
> 

 
Regards,
Srivatsa S. Bhat

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Borislav Petkov]

On Tue, Dec 06, 2011 at 07:34:37PM +0530, Srivatsa S. Bhat wrote:
> > Not blindly, the microcode is still verified.
> > 
> I saw your other mail about the validity of this whole scenario, and I
> kind of agree to your point.
> 
> My thoughts below might not be so relevant/significant considering that,
> but anyways, just to understand what you said above:
> I didn't quite find where the microcode is verified if the kernel happens
> to have the microcode image already.

Sorry, I might not've been clear enough: it is still verified by the
_hardware_, regardless of whether the ucode loader checks it or not.
Remember, hw guys don't trust sw guys and the reverse holds true too.

:-)

HTH.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Srivatsa S. Bhat]

On 12/06/2011 07:45 PM, Borislav Petkov wrote:

> On Tue, Dec 06, 2011 at 07:34:37PM +0530, Srivatsa S. Bhat wrote:
>>> Not blindly, the microcode is still verified.
>>>
>> I saw your other mail about the validity of this whole scenario, and I
>> kind of agree to your point.
>>
>> My thoughts below might not be so relevant/significant considering that,
>> but anyways, just to understand what you said above:
>> I didn't quite find where the microcode is verified if the kernel happens
>> to have the microcode image already.
> 
> Sorry, I might not've been clear enough: it is still verified by the
> _hardware_, regardless of whether the ucode loader checks it or not.
> Remember, hw guys don't trust sw guys and the reverse holds true too.
> 
> :-)
> 


Oh.. Ok :-) And thank you for the explanation!

Regards,
Srivatsa S. Bhat

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Van De Ven, Arjan]

> But the hibernation angle should be considered. Hibernation already has to deal
> with the case where someone physically unplugs a CPU and then resumes from
> the disk image, right? How does the hibernation code handle that case
> currently?

it'll fail the resume.

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Yu, Fenghua]

> 
> The specific point i tried to make about s2ram is to make sure
> it does not break during normal usage: for example someone
> offlines the boot CPU, but the box then gets suspended - that
> should not hang or crash.
> 
This has been considered in the patch #7/7. There shouldn't be a problem here (in theory).

Suspend/hibernate calls callback bsp_pm_callback() and must pass this check before it can continue.

When CPU0 is offline, bsp_pm_callback() returns -ENODEV and suspend/hibernate fails. When CPU0 is online, bsp_pm_callback() returns 0 and suspend/hibernate can continue.

So when CPU0 is offline, system cannot suspend/hibernate. Suspend/hibernate can succeed only when CPU0 is online.

Plus the patch #7/7 considers race condition by pm_notifier_call_chain()'s priority in pm call chain. So there should be race issue here.

Thanks.

-Fenghua

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Yu, Fenghua]

> From: Borislav Petkov [mailto:bp@amd64.org]
> On Tue, Dec 06, 2011 at 04:55:02PM +0530, Srivatsa S. Bhat wrote:
> > By the way, this problem is not tied to CPU0 alone, it exists for any
> > CPU! (as long as we are talking about plugging in/out CPUs
> physically).
> 
> Just a reminder: before you guys go and wander off into the woods of
> hypothetical with this, please make sure this use case is relevant
> enough for the trouble. The only real reason given so far AFAICT was
> RAS
> and to be able to offline BSP in order to prolong system life before
> maintenance.
> 
> When you take it down for maintenance eventually, you don't need to
> suspend but simply poweroff.

Agree with you. To maintain a system with a bad CPU, either you hot plug or hot replace the CPU, or you power off then replace the CPU. Replacing the CPU between suspend and resume doesn't seem a normal RAS behavior.

If you choose hotplug or hot replace the CPU, then this patchset is useful.

CPU0 offline/online is a must-have if you want to replace a socket which has the BSP in it for any errors in uncore or a sibling CPU. Not mention that CPU0 itself is bad and you want to hot unplug it.

Thanks.

-Fenghua

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Yu, Fenghua <fenghua.yu@intel.com> wrote:

> > When you take it down for maintenance eventually, you don't 
> > need to suspend but simply poweroff.
> 
> Agree with you. To maintain a system with a bad CPU, either 
> you hot plug or hot replace the CPU, or you power off then 
> replace the CPU. Replacing the CPU between suspend and resume 
> doesn't seem a normal RAS behavior.

More importantly, you generally *cannot* realistically continue 
with a bad CPU anyway - the system will crash or will show signs 
of corruptions and you *want* a full powerdown and a clean 
reboot.

The usecases for real CPU hotplug look pretty limited to me:

 - Special hardware environments that are deeply redundant and 
   can warn about 'soft' failures well before hard failures
   which gives a realistic window of time for a maintenance
   hot-swap. [Such hardware actually exists, i even worked with
   an x86 one eons ago.]

 - Swapping slower CPUs for a faster CPUs, without any downtime.
   Given that mixed steppings and mixed frequencies are
   generally pretty unpredictable even with no hotswap in the
   picture, i can see hw designers (and qa test matrix
   engineers) cringe at the idea.

Thanks,

	Ingo

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Luck, Tony]

> More importantly, you generally *cannot* realistically continue 
> with a bad CPU anyway - the system will crash or will show signs 
> of corruptions and you *want* a full powerdown and a clean 
> reboot.

See the "Enhanced cache error reporting" section in the Intel
Software Developers manual (section 15.4 in volume 3B of the
latest edition).  Intel provides what is probably a very early
notification in many cases that a processors cache is experiencing
problems. At the time of the notification the system is still
functioning correctly.  The SDM suggests that when the "yellow"
status is signaled you should schedule service "within a few weeks".

24x7 systems with a lot of sockets & cores, and highly paranoid
administrators, might want to take action to stop using the cores
that share the cache with problems sooner than they can schedule
downtime.

>- Special hardware environments that are deeply redundant and 
>   can warn about 'soft' failures well before hard failures
>   which gives a realistic window of time for a maintenance
>   hot-swap. [Such hardware actually exists, i even worked with
>   an x86 one eons ago.]

So not so special any more - every Xeon since Core Duo has the
cache error reporting capability.

-Tony

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Luck, Tony <tony.luck@intel.com> wrote:

> > More importantly, you generally *cannot* realistically 
> > continue with a bad CPU anyway - the system will crash or 
> > will show signs of corruptions and you *want* a full 
> > powerdown and a clean reboot.
> 
> See the "Enhanced cache error reporting" section in the Intel 
> Software Developers manual (section 15.4 in volume 3B of the 
> latest edition).  Intel provides what is probably a very early 
> notification in many cases that a processors cache is 
> experiencing problems. At the time of the notification the 
> system is still functioning correctly.  The SDM suggests that 
> when the "yellow" status is signaled you should schedule 
> service "within a few weeks".

The question is, how realistically does this report true CPU 
troubles, statistically? The on-die cache might have the highest 
transistor count, but it's not under nearly the same thermal 
stress as functional units.

If 90% of all hard CPU failures can be predicted that way then 
it's probably useful. If it's only 20%, then not so much.

Also, it's still all theoretical until there's systems out there 
where the CPU socket is physically hotpluggable. If there's such 
plans in the works then sure, theory becomes reality and then 
it's all useful - and then we can do these patches (and more).

Thanks,

	Ingo

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Luck, Tony]

> The question is, how realistically does this report true CPU 
> troubles, statistically? The on-die cache might have the highest 
> transistor count, but it's not under nearly the same thermal 
> stress as functional units.
>
> If 90% of all hard CPU failures can be predicted that way then 
> it's probably useful. If it's only 20%, then not so much.

Intel doesn't release error rates - so I can't help with data here.

> Also, it's still all theoretical until there's systems out there 
> where the CPU socket is physically hotpluggable. If there's such 
> plans in the works then sure, theory becomes reality and then 
> it's all useful - and then we can do these patches (and more).

No - physical removal of the cpu is not a requirement for this
to be useful.  If you have a system that is reporting cache
problems via the "yellow" status in the MCi_STATUS msr, then
there is benefit in simply soft off-lining the cores that share
that cache - assuming that leaves you some online cores! A single
socket system with L3 cache troubles is not helped - but problems
in L1/L2 cache, or on multi-socket systems can be avoided (and
are already being avoided for the cases where CPU0 is not involved).

Physical removal of the cpu is a problem for Linux since Nehalem
(when memory controller moved on-die). Take away the cpu, and you
lose access to the memory connected to that socket - and we don't
have general solutions for memory removal.

-Tony

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Luck, Tony <tony.luck@intel.com> wrote:

> > The question is, how realistically does this report true CPU 
> > troubles, statistically? The on-die cache might have the 
> > highest transistor count, but it's not under nearly the same 
> > thermal stress as functional units.
> >
> > If 90% of all hard CPU failures can be predicted that way 
> > then it's probably useful. If it's only 20%, then not so 
> > much.
> 
> Intel doesn't release error rates - so I can't help with data 
> here.

Well, precise data won't be needed - but we need *something* 
indicative to justify the feature - faith alone won't be enough.

Is there any third party research on this? I remember that 
Google released hard drive failure stats a few years ago, maybe 
there's some approximate data about CPU "soft" failure rates. 
Even anecdotal data and speculation/estimation would be a start 
- it could be contradicted later on by more precise data, once 
people start using the "generic CPU hot-unplug" feature. (which 
this feature should really be named, instead of the 'BSP unplug' 
name.)

> > Also, it's still all theoretical until there's systems out 
> > there where the CPU socket is physically hotpluggable. If 
> > there's such plans in the works then sure, theory becomes 
> > reality and then it's all useful - and then we can do these 
> > patches (and more).
> 
> No - physical removal of the cpu is not a requirement for this 
> to be useful. [...]

Indeed, you are right, i stand corrected there.

Okay, i'm convinced, i guess we can do this.

> [...]
>
> Physical removal of the cpu is a problem for Linux since 
> Nehalem (when memory controller moved on-die). Take away the 
> cpu, and you lose access to the memory connected to that 
> socket - and we don't have general solutions for memory 
> removal.

It's possible technically but not the easiest of features - also 
i suspect Linus would object to the naive breaking of the 
semi-linear kernel mapping we do today ;-)

But if someone implements that in a sane way, using at least 2MB 
granular mappings [or maybe ORDER_MAX granular mappings], which 
keeps 2MB TLBs, and uses a quick hash table for __pa() and 
__va(), i would definitely take a look at how ugly it ends up 
being. Our hibernation code already gives us a generic way to 
quiescence all DMA activity on the system, so most of the 
building blocks are in place.

Thanks,

	Ingo

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Yu, Fenghua]

> 
> * Fenghua Yu <fenghua.yu@intel.com> wrote:
> Then if demand for this picks up some more intelligent method of
> cooperating with the firmware could be added: the firmware could
> actually signal to us whether it supports suspend/resume from
> other than the boot CPU.

We started to think how to handle the resume issue in firmware,
e.g. the way you talked, or change boot CPU to another online CPU
to execute resume procedure.

A firmware solution is a long run. Currently we don't suspend/hibernate
when BSP is offline. If a firmware solution is available, we can
change the sanity check to allow suspend/hibernate if CPU0 is offline
for new firmware. But even when the solution is available in new
firmware, we still need to do the same sanity check on legacy firmware.

Thanks.

-Fenghua

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Yu, Fenghua <fenghua.yu@intel.com> wrote:

> > 
> > * Fenghua Yu <fenghua.yu@intel.com> wrote:
> > Then if demand for this picks up some more intelligent method of
> > cooperating with the firmware could be added: the firmware could
> > actually signal to us whether it supports suspend/resume from
> > other than the boot CPU.
> 
> We started to think how to handle the resume issue in 
> firmware, e.g. the way you talked, or change boot CPU to 
> another online CPU to execute resume procedure.
> 
> A firmware solution is a long run. Currently we don't 
> suspend/hibernate when BSP is offline. If a firmware solution 
> is available, we can change the sanity check to allow 
> suspend/hibernate if CPU0 is offline for new firmware. But 
> even when the solution is available in new firmware, we still 
> need to do the same sanity check on legacy firmware.

Right.

You convinced me, if there's no objections i guess we could try 
this feature carefully, if it's named in some more descriptive 
manner such as:

   x86: Arbitrary CPU hot(un)plug support

Nobody outside x86 devs knows 'BSP' means and why they should 
care.

The minor usability/testability/self-test improvements i 
suggested need to be added as well.

Thanks,

	Ingo

RE: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Yu, Fenghua]

> We need a debug feature for this: CONFIG_DEBUG_BOOT_CPU_OFF=y or
> such (disabled by default): this feature would offline the boot
> CPU as soon as possible, and boot up userspace with the boot CPU
> offlined.

I would think the earliest time when CPU0 can be offline is right after do_initcalls() because do_initcalls() installs some cpu hotplug notifier.

But this will change generic code (e.g. add arch hook in init/main.c or add a new initcall level which is after level 7s).

Another way is to offline CPU0 (or any CPU) in /etc/rc.local. By this way, we don't change kernel. And this is safe, but a bit later than in kernel.

Any comment?

Thanks.

-Fenghua

Re: [PATCH v4 0/7] x86: BSP or CPU0 online/offline

[Ingo Molnar]

* Yu, Fenghua <fenghua.yu@intel.com> wrote:

> > We need a debug feature for this: 
> > CONFIG_DEBUG_BOOT_CPU_OFF=y or such (disabled by default): 
> > this feature would offline the boot CPU as soon as possible, 
> > and boot up userspace with the boot CPU offlined.
> 
> I would think the earliest time when CPU0 can be offline is 
> right after do_initcalls() because do_initcalls() installs 
> some cpu hotplug notifier.
> 
> But this will change generic code (e.g. add arch hook in 
> init/main.c or add a new initcall level which is after level 
> 7s).
> 
> Another way is to offline CPU0 (or any CPU) in /etc/rc.local. 
> By this way, we don't change kernel. And this is safe, but a 
> bit later than in kernel.
> 
> Any comment?

I think we should try to do this in the kernel, to keep the 
debug feature independent of user-space code. This also helps
us clean init code from boot CPU assumptions.

Not sure we need a new initlevel - why not just offline the CPU? 
I suspect the cpu hotplug notifier installation that is done in 
do_initcalls() should be moved earlier so that the boot CPU can 
be offlined as early as a secondary CPU is up and running.

Thanks,

	Ingo