[PATCH v5 0/12] x86: Arbitrary CPU hot(un)plug support

[PATCH v5 0/12] x86: Arbitrary CPU hot(un)plug support

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

CPU0 or BSP (Bootstrap Processor) has been the last processor that can not be
hot removed on x86. This patch set implements CPU0 or BSP online and offline
and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any
thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is
required to be offline or hot replaced to keep the system run. For example,
starting with Core Duo, if you have a system that is reporting cache problems
via the "yellow" status in the MCi_STATUS msr, then there is benefit in simply
soft off-lining the cores that share that cache - assuming that leaves you at
least one online core. A single socket system with L3 cache troubles is not
helped - but problems in L1/L2 cache, or on multi-socket systems can be avoided.
They are already being avoided for the cases where CPU0 is not involved.
This patchset can help L1/L2 cache problem in CPU0 or L3 cache problem on the
socket with CPU0 in a multi-socket system.

v5: Add CONFIG_BOOTPARAM_HOTPLUG_CPU0 and CONFIG_DEBUG_HOTPLUG_CPU0. Simplify
duplicate xstate_size init check. Wakeup CPU0 via nmi instead INITs. Add
mcheck_cpu_init when CPU0 online. Change variable bsp_hotpluggable to
cpu0_hotpluggable with __initdata qualifier.

v4: Add __read_mostly for internal bsp_hotpluggable variable. Add my email
address in cpu-hotplug.txt document. A wording change in comment.

v3: Register a pm notifier to check if CPU0 is online before hibernate/suspend.
Small wording changes in document and print info.

v2: Add locking changes between cpu hotplug and hibernate/suspend. Change PIC
irq bound to CPU0 detection.

Fenghua Yu (12):
  Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
    online/offline feature
  x86/Kconfig: Add config switch for CPU0 hotplug
  x86/topology.c: Support functions for CPU0 online/offline
  x86/smpboot.c: Don't offline CPU0 if any irq can not be migrated out
    of it and remove CPU0 check in smp_callin()
  x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline
  x86/head_64.S: Define start_cpu0
  x86/head_32.S: Define start_cpu0
  x86/smpboot.c: Wake up CPU0 via NMI instead of INITs
  x86/common.c: Init CPU0 data during CPU0 online
  x86/mtrr/main.c: Ask the first online CPU to save mtrr
  x86/i387.c: Thread xstate is initialized only on CPU0 once
  x86/topology.c: debug CPU0 hotplug

 Documentation/cpu-hotplug.txt       |   24 +++++++
 Documentation/kernel-parameters.txt |   14 ++++
 arch/x86/Kconfig                    |   44 ++++++++++++
 arch/x86/include/asm/cpu.h          |    1 +
 arch/x86/include/asm/processor.h    |    1 +
 arch/x86/kernel/cpu/common.c        |   17 ++++-
 arch/x86/kernel/cpu/mtrr/main.c     |    9 ++-
 arch/x86/kernel/head_32.S           |   17 ++++-
 arch/x86/kernel/head_64.S           |   15 ++++
 arch/x86/kernel/i387.c              |    6 ++-
 arch/x86/kernel/smpboot.c           |  124 +++++++++++++++++++++++++++++++----
 arch/x86/kernel/topology.c          |   66 +++++++++++++++++--
 arch/x86/power/cpu.c                |   44 ++++++++++++
 13 files changed, 352 insertions(+), 30 deletions(-)

[PATCH v5 01/12] Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

If CONFIG_BOOTPARAM_HOTPLUG_CPU0 is turned on, CPU0 is hotpluggable. Otherwise,
by default CPU0 is not hotpluggable and kernel parameter cpu0_hotplug enables
CPU0 online/offline feature.

The documentations point out two known CPU0 dependencies. First, resume from
hibernate or suspend always starts from CPU0. So hibernate and suspend are
prevented if CPU0 is offline. Another dependency is PIC interrupts always go
to CPU0.

It's said that some machines may depend on CPU0 to poweroff/reboot. But I
haven't seen such dependency on a few tested machines.

Please let me know if you see any CPU0 dependencies on your machine.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 Documentation/cpu-hotplug.txt       |   24 ++++++++++++++++++++++++
 Documentation/kernel-parameters.txt |   14 ++++++++++++++
 2 files changed, 38 insertions(+), 0 deletions(-)

diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
index a20bfd4..e6f8f0e 100644
--- a/Documentation/cpu-hotplug.txt
+++ b/Documentation/cpu-hotplug.txt
@@ -207,6 +207,30 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If kernel is compiled with CONFIG_BOOTPARAM_HOTPLUG_CPU0=y, CPU0 is
+removable by default. Otherwise, CPU0 is also removable by kernel option
+cpu0_hotplug.
+
+But some features depend on CPU0. Two known dependencies are:
+
+1. Resume from hibernate/suspend depends on CPU0. Hibernate/suspend will fail if
+CPU0 is offline and you need to online CPU0 before hibernate/suspend can
+continue.
+2. PIC interrupts also depend on CPU0. CPU0 can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on CPU0 on some machines although I haven't
+seen any poweroff/reboot failure so far after CPU0 is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of CPU0.
+
+If the dependencies are under your control, you can turn on CPU0 hotplug feature
+either by CONFIG_BOOTPARAM_HOTPLUG_CPU0 or by kernel parameter cpu0_hotplug.
+
+--Fenghua Yu <fenghua.yu@intel.com>
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index b2d3efe..a5f0499 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1846,6 +1846,20 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
 
+	cpu0_hotplug	[X86] Turn on CPU0 hotplug feature when
+			CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off.
+			Some features depend on CPU0. Known dependencies are:
+			1. Resume from suspend/hibernate depends on CPU0.
+			Suspend/hibernate will fail if CPU0 is offline and you
+			need to online CPU0 before suspend/hibernate.
+			2. PIC interrupts also depend on CPU0. CPU0 can't be
+			removed if a PIC interrupt is detected.
+			It's said poweroff/reboot may depend on CPU0 on some
+			machines although I haven't seen such issues so far
+			after CPU0 is offline on a few tested machines.
+			If the dependencies are under your control, you can
+			turn on cpu0_hotplug.
+
 	nptcg=		[IA-64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
-- 
1.6.0.3

[PATCH v5 02/12] x86/Kconfig: Add config switch for CPU0 hotplug

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

New config switch CONFIG_BOOTPARAM_HOTPLUG_CPU0 sets default state of whether
the CPU0 hotplug is on or off.

If the switch is off, CPU0 is not hotpluggable by default. But the CPU0 hotplug
feature can still be turned on by kernel parameter cpu0_hotplug at boot.

If the switch is on, CPU0 is always hotpluggable. The default value of
the switch is off.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/Kconfig |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index d6f1e91..c2cf195 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1719,6 +1719,35 @@ config HOTPLUG_CPU
 	    automatically on SMP systems. )
 	  Say N if you want to disable CPU hotplug.
 
+config BOOTPARAM_HOTPLUG_CPU0
+	bool "Set default setting of cpu0_hotpluggable"
+	default n
+	depends on HOTPLUG_CPU && EXPERIMENTAL
+	---help---
+	  Set whether default state of cpu0_hotpluggable is on or off.
+
+	  Say Y here to enable CPU0 hotplug by default. If this switch
+	  is turned on, there is no need to give cpu0_hotplug kernel
+	  parameter and the CPU0 hotplug feature is enabled by default.
+
+	  Please note: there are two known CPU0 dependencies if you want
+	  to enable the CPU0 hotplug feature either by this switch or by
+	  cpu0_hotplug kernel parameter.
+
+	  First, resume from hibernate or suspend always starts from CPU0.
+	  So hibernate and suspend are prevented if CPU0 is offline.
+
+	  Second dependency is PIC interrupts always go to CPU0. CPU0 can not
+	  offline if any interrupt can not migrate out of CPU0. There may
+	  be other CPU0 dependencies.
+
+	  Please make sure the dependencies are under your control before
+	  you enable this feature.
+
+	  Say N if you don't want to enable CPU0 hotplug feature by default.
+	  You still can enable the CPU0 hotplug feature at boot by kernel
+	  parameter cpu0_hotplug.
+
 config COMPAT_VDSO
 	def_bool y
 	prompt "Compat VDSO support"
-- 
1.6.0.3

[PATCH v5 03/12] x86/topology.c: Support functions for CPU0 online/offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

If CONFIG_BOOTPARAM_HOTPLUG_CPU is turned on, CPU0 hotplug feature is enabled
by default.

If CONFIG_BOOTPARAM_HOTPLUG_CPU is not turned on, CPU0 hotplug feature is not
enabled by default. The kernel parameter cpu0_hotplug can enable CPU0 hotplug
feature at boot.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/topology.c |   27 ++++++++++++++++++++-------
 1 files changed, 20 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
index 76ee977..8cec15c 100644
--- a/arch/x86/kernel/topology.c
+++ b/arch/x86/kernel/topology.c
@@ -35,18 +35,31 @@
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+#ifdef CONFIG_BOOTPARAM_HOTPLUG_CPU0
+static int __initdata cpu0_hotpluggable = 1;
+#else
+static int __initdata cpu0_hotpluggable;
+static int __init enable_cpu0_hotplug(char *str)
+{
+	cpu0_hotpluggable = 1;
+	return 1;
+}
+
+__setup("cpu0_hotplug", enable_cpu0_hotplug);
+#endif
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-	 * CPU0 cannot be offlined due to several
-	 * restrictions and assumptions in kernel. This basically
-	 * doesn't add a control file, one cannot attempt to offline
-	 * BSP.
+	 * Two known BSP/CPU0 dependencies: Resume from suspend/hibernate
+	 * depends on BSP. PIC interrupts depend on BSP.
 	 *
-	 * Also certain PCI quirks require not to enable hotplug control
-	 * for all CPU's.
+	 * If the BSP depencies are under control, one can tell kernel to
+	 * enable BSP hotplug. This basically adds a control file and
+	 * one can attempt to offline BSP.
 	 */
-	if (num)
+	if (num || cpu0_hotpluggable)
 		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
 
 	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
-- 
1.6.0.3

Re: [PATCH v5 03/12] x86/topology.c: Support functions for CPU0 online/offline

[Ben Hutchings]

On Wed, 2012-01-11 at 09:04 -0800, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> If CONFIG_BOOTPARAM_HOTPLUG_CPU is turned on, CPU0 hotplug feature is enabled
> by default.
> 
> If CONFIG_BOOTPARAM_HOTPLUG_CPU is not turned on, CPU0 hotplug feature is not
> enabled by default. The kernel parameter cpu0_hotplug can enable CPU0 hotplug
> feature at boot.
[...]
>  int __ref arch_register_cpu(int num)
>  {
>  	/*
> -	 * CPU0 cannot be offlined due to several
> -	 * restrictions and assumptions in kernel. This basically
> -	 * doesn't add a control file, one cannot attempt to offline
> -	 * BSP.
> +	 * Two known BSP/CPU0 dependencies: Resume from suspend/hibernate
> +	 * depends on BSP. PIC interrupts depend on BSP.
>  	 *
> -	 * Also certain PCI quirks require not to enable hotplug control
> -	 * for all CPU's.
> +	 * If the BSP depencies are under control, one can tell kernel to
> +	 * enable BSP hotplug. This basically adds a control file and
> +	 * one can attempt to offline BSP.
>  	 */
> -	if (num)
> +	if (num || cpu0_hotpluggable)
>  		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
>  
>  	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);

This change belongs at the end of the series.  It should not be possible
to enable CPU0 hotplug until after the hotplug logic can do it
correctly, and this might break bisection.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

RE: [PATCH v5 03/12] x86/topology.c: Support functions for CPU0 online/offline

[Yu, Fenghua]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2250 bytes --]


> On Wed, 2012-01-11 at 09:04 -0800, Fenghua Yu wrote:
> > From: Fenghua Yu <fenghua.yu@intel.com>
> >
> > If CONFIG_BOOTPARAM_HOTPLUG_CPU is turned on, CPU0 hotplug feature is
> enabled
> > by default.
> >
> > If CONFIG_BOOTPARAM_HOTPLUG_CPU is not turned on, CPU0 hotplug
> feature is not
> > enabled by default. The kernel parameter cpu0_hotplug can enable CPU0
> hotplug
> > feature at boot.
> [...]
> >  int __ref arch_register_cpu(int num)
> >  {
> >  	/*
> > -	 * CPU0 cannot be offlined due to several
> > -	 * restrictions and assumptions in kernel. This basically
> > -	 * doesn't add a control file, one cannot attempt to offline
> > -	 * BSP.
> > +	 * Two known BSP/CPU0 dependencies: Resume from suspend/hibernate
> > +	 * depends on BSP. PIC interrupts depend on BSP.
> >  	 *
> > -	 * Also certain PCI quirks require not to enable hotplug control
> > -	 * for all CPU's.
> > +	 * If the BSP depencies are under control, one can tell kernel to
> > +	 * enable BSP hotplug. This basically adds a control file and
> > +	 * one can attempt to offline BSP.
> >  	 */
> > -	if (num)
> > +	if (num || cpu0_hotpluggable)
> >  		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
> >
> >  	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
> 
> This change belongs at the end of the series.  It should not be
> possible
> to enable CPU0 hotplug until after the hotplug logic can do it
> correctly, and this might break bisection.

Quote from https://www.linux.com/how-to-participate-in-the-linux-community
"It can be tempting to add a whole new infrastructure with a series of patches, but to leave that infrastructure unused until the final patch in the series enables the whole thing. This temptation should be avoided if possible; if that series adds regressions, bisection will finger the last patch as the one which caused the problem, even though the real bug is elsewhere. Whenever possible, a patch which adds new code should make that code active immediately."

So this patch currently is in the right place in the patch set unless I miss something.

Thanks.

-Fenghua
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

RE: [PATCH v5 03/12] x86/topology.c: Support functions for CPU0 online/offline

[Ben Hutchings]

On Tue, 2012-01-24 at 22:31 +0000, Yu, Fenghua wrote:
> > On Wed, 2012-01-11 at 09:04 -0800, Fenghua Yu wrote:
> > > From: Fenghua Yu <fenghua.yu@intel.com>
> > >
> > > If CONFIG_BOOTPARAM_HOTPLUG_CPU is turned on, CPU0 hotplug feature is
> > enabled
> > > by default.
> > >
> > > If CONFIG_BOOTPARAM_HOTPLUG_CPU is not turned on, CPU0 hotplug
> > feature is not
> > > enabled by default. The kernel parameter cpu0_hotplug can enable CPU0
> > hotplug
> > > feature at boot.
> > [...]
> > >  int __ref arch_register_cpu(int num)
> > >  {
> > >  	/*
> > > -	 * CPU0 cannot be offlined due to several
> > > -	 * restrictions and assumptions in kernel. This basically
> > > -	 * doesn't add a control file, one cannot attempt to offline
> > > -	 * BSP.
> > > +	 * Two known BSP/CPU0 dependencies: Resume from suspend/hibernate
> > > +	 * depends on BSP. PIC interrupts depend on BSP.
> > >  	 *
> > > -	 * Also certain PCI quirks require not to enable hotplug control
> > > -	 * for all CPU's.
> > > +	 * If the BSP depencies are under control, one can tell kernel to
> > > +	 * enable BSP hotplug. This basically adds a control file and
> > > +	 * one can attempt to offline BSP.
> > >  	 */
> > > -	if (num)
> > > +	if (num || cpu0_hotpluggable)
> > >  		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
> > >
> > >  	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
> > 
> > This change belongs at the end of the series.  It should not be
> > possible
> > to enable CPU0 hotplug until after the hotplug logic can do it
> > correctly, and this might break bisection.
> 
> Quote from https://www.linux.com/how-to-participate-in-the-linux-community
> "It can be tempting to add a whole new infrastructure with a series of
> patches, but to leave that infrastructure unused until the final patch
> in the series enables the whole thing. This temptation should be
> avoided if possible; if that series adds regressions, bisection will
> finger the last patch as the one which caused the problem, even though
> the real bug is elsewhere. Whenever possible, a patch which adds new
> code should make that code active immediately."
> 
> So this patch currently is in the right place in the patch set unless
> I miss something.

You're giving undue weight to that guidance.  It is far more important
that you do not enable features that don't work!

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

RE: [PATCH v5 03/12] x86/topology.c: Support functions for CPU0 online/offline

[Yu, Fenghua]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1227 bytes --]

> > Quote from https://www.linux.com/how-to-participate-in-the-linux-
> community
> > "It can be tempting to add a whole new infrastructure with a series
> of
> > patches, but to leave that infrastructure unused until the final
> patch
> > in the series enables the whole thing. This temptation should be
> > avoided if possible; if that series adds regressions, bisection will
> > finger the last patch as the one which caused the problem, even
> though
> > the real bug is elsewhere. Whenever possible, a patch which adds new
> > code should make that code active immediately."
> >
> > So this patch currently is in the right place in the patch set unless
> > I miss something.
> 
> You're giving undue weight to that guidance.  It is far more important
> that you do not enable features that don't work!

Either way has reasonable arguments. This should be a generic question. But for this specific patch set, the above quote does make more sense to me as far as git bisect is concerned. Maybe someone else in the list can provide more insight?

Thanks.

-Fenghua
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

[PATCH v5 04/12] x86/smpboot.c: Don't offline CPU0 if any irq can not be migrated out of it and remove CPU0 check in smp_callin()

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Don't offline CPU0 if any irq can not be migrated out of it.

Call identify_boot_cpu_online() for CPU0 in smp_callin() and continue to online
CPU0 in native_cpu_up().

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/smpboot.c |   42 +++++++++++++++++++++++++++++++++++-------
 1 files changed, 35 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 66d250c..a3c4be3 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -136,8 +136,8 @@ EXPORT_PER_CPU_SYMBOL(cpu_info);
 atomic_t init_deasserted;
 
 /*
- * Report back to the Boot Processor.
- * Running on AP.
+ * Report back to the Boot Processor during boot time or to the caller processor
+ * during CPU online.
  */
 static void __cpuinit smp_callin(void)
 {
@@ -230,6 +230,13 @@ static void __cpuinit smp_callin(void)
 	pr_debug("Stack at about %p\n", &cpuid);
 
 	/*
+	 * This function won't run on the BSP during boot time. It run
+	 * on BSP only when BSP is offlined and onlined again.
+	 */
+	if (cpuid == 0)
+		identify_boot_cpu_online();
+
+	/*
 	 * This must be done before setting cpu_online_mask
 	 * or calling notify_cpu_starting.
 	 */
@@ -845,7 +852,7 @@ int __cpuinit native_cpu_up(unsigned int cpu)
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || apicid == boot_cpu_physical_apicid ||
+	if (apicid == BAD_APICID ||
 	    !physid_isset(apicid, phys_cpu_present_map) ||
 	    (!x2apic_mode && apicid >= 255)) {
 		printk(KERN_ERR "%s: bad cpu %d\n", __func__, cpu);
@@ -1291,12 +1298,33 @@ int native_cpu_disable(void)
 	 * Perhaps use cpufreq to drop frequency, but that could go
 	 * into generic code.
 	 *
-	 * We won't take down the boot processor on i386 due to some
+	 * We won't take down the boot processor on x86 if some
 	 * interrupts only being able to be serviced by the BSP.
-	 * Especially so if we're not using an IOAPIC	-zwane
+	 * Especially so if we're not using an IOAPIC
 	 */
-	if (cpu == 0)
-		return -EBUSY;
+	if (cpu == 0) {
+		int irq;
+		struct irq_desc *desc;
+		struct irq_data *data;
+		struct irq_chip *chip;
+
+		for_each_irq_desc(irq, desc) {
+			raw_spin_lock(&desc->lock);
+			if (!irq_has_action(irq)) {
+				raw_spin_unlock(&desc->lock);
+				continue;
+			}
+
+			data = irq_desc_get_irq_data(desc);
+			chip = irq_data_get_irq_chip(data);
+			if (!chip->irq_set_affinity) {
+				pr_debug("irq%d can't move out of BSP\n", irq);
+				raw_spin_unlock(&desc->lock);
+				return -EBUSY;
+			}
+			raw_spin_unlock(&desc->lock);
+		}
+	}
 
 	clear_local_APIC();
 
-- 
1.6.0.3

[PATCH v5 05/12] x86/power/cpu.c: Don't hibernate/suspend if CPU0 is offline

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Because x86 BIOS requires CPU0 to resume from sleep, suspend or hibernate
can't be executed if CPU0 is detected offline. To make suspend or hibernate and
further resume succeed, CPU0 must be online.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/power/cpu.c |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index f10c0af..a4ec084 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -236,3 +236,47 @@ void restore_processor_state(void)
 #ifdef CONFIG_X86_32
 EXPORT_SYMBOL(restore_processor_state);
 #endif
+
+/*
+ * When bsp_check() is called in hibernate and suspend, cpu hotplug
+ * is disabled already. So it's unnessary to handle race condition between
+ * cpumask query and cpu hotplug.
+ */
+static int bsp_check(void)
+{
+	if (cpumask_first(cpu_online_mask) != 0) {
+		printk(KERN_WARNING "CPU0 is offline.\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
+static int bsp_pm_callback(struct notifier_block *nb, unsigned long action,
+			   void *ptr)
+{
+	int ret = 0;
+
+	switch (action) {
+	case PM_SUSPEND_PREPARE:
+	case PM_HIBERNATION_PREPARE:
+		ret = bsp_check();
+		break;
+	default:
+		break;
+	}
+	return notifier_from_errno(ret);
+}
+
+static int __init bsp_pm_check_init(void)
+{
+	/*
+	 * Set this bsp_pm_callback as lower priority than
+	 * cpu_hotplug_pm_callback. So cpu_hotplug_pm_callback will be called
+	 * earlier to disable cpu hotplug before bsp online check.
+	 */
+	pm_notifier(bsp_pm_callback, -INT_MAX);
+	return 0;
+}
+
+core_initcall(bsp_pm_check_init);
-- 
1.6.0.3

[PATCH v5 06/12] x86/head_64.S: Define start_cpu0

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

start_cpu0() is defined in head_64.S for 64-bit. The function sets up stack and
jumps to start_secondary() for CPU0 wake up.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/head_64.S |   15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 40f4eb3..e1677b9 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -253,6 +253,21 @@ ENTRY(secondary_startup_64)
 	pushq	%rax		# target address in negative space
 	lretq
 
+#ifdef CONFIG_HOTPLUG_CPU
+/*
+ * Boot CPU0 entry point. It's called from play_dead(). Everything has been set
+ * up already except stack. We just set up stack here. Then call
+ * start_secondary().
+ */
+ENTRY(start_cpu0)
+	movq stack_start(%rip),%rsp
+	movq	initial_code(%rip),%rax
+	pushq	$0		# fake return address to stop unwinder
+	pushq	$__KERNEL_CS	# set correct cs
+	pushq	%rax		# target address in negative space
+	lretq
+#endif
+
 	/* SMP bootup changes these two */
 	__REFDATA
 	.align	8
-- 
1.6.0.3

[PATCH v5 07/12] x86/head_32.S: Define start_cpu0

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

start_cpu0() is defined in head_32.S for 32-bit. The function sets up stack and
jumps to start_secondary() for CPU0 wake up.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/head_32.S |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
index ce0be7c..5a38a1a 100644
--- a/arch/x86/kernel/head_32.S
+++ b/arch/x86/kernel/head_32.S
@@ -265,6 +265,20 @@ num_subarch_entries = (. - subarch_entries) / 4
 	jmp default_entry
 #endif /* CONFIG_PARAVIRT */
 
+__CPUINIT
+
+#ifdef CONFIG_HOTPLUG_CPU
+/*
+ * Boot CPU0 entry point. It's called from play_dead(). Everything has been set
+ * up already except stack. We just set up stack here. Then call
+ * start_secondary().
+ */
+ENTRY(start_cpu0)
+	movl stack_start, %ecx
+	movl %ecx, %esp
+	jmp  *(initial_code)
+#endif
+
 /*
  * Non-boot CPU entry point; entered from trampoline.S
  * We can't lgdt here, because lgdt itself uses a data segment, but
@@ -273,9 +287,6 @@ num_subarch_entries = (. - subarch_entries) / 4
  * If cpu hotplug is not supported then this code can go in init section
  * which will be freed later
  */
-
-__CPUINIT
-
 #ifdef CONFIG_SMP
 ENTRY(startup_32_smp)
 	cld
-- 
1.6.0.3

[PATCH v5 08/12] x86/smpboot.c: Wake up CPU0 via NMI instead of INITs

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Instead of waiting for STARTUP after INITs, BSP will execute
the BIOS boot-strap code which is not a desired behavior for
waking up BSP. To avoid the boot-strap code, wake up CPU0 by
NMI instead.

This works to wake up soft offlined CPU0 only. If CPU0 is hard offlined (i.e.
physically hot removed and then hot added), NMI won't wake it up. We'll change
this code in the future to wake up hard offlined CPU0 if real platform and
request are available.

AP is still waken up as before by INIT, INIT, STARTUP sequence.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/cpu.h |    1 +
 arch/x86/kernel/smpboot.c  |   82 ++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 76 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h
index 4564c8e..a119572 100644
--- a/arch/x86/include/asm/cpu.h
+++ b/arch/x86/include/asm/cpu.h
@@ -28,6 +28,7 @@ struct x86_cpu {
 #ifdef CONFIG_HOTPLUG_CPU
 extern int arch_register_cpu(int num);
 extern void arch_unregister_cpu(int);
+extern void __cpuinit start_cpu0(void);
 #endif
 
 DECLARE_PER_CPU(int, cpu_state);
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index a3c4be3..0d93599 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -149,15 +149,17 @@ static void __cpuinit smp_callin(void)
 	 * we may get here before an INIT-deassert IPI reaches
 	 * our local APIC.  We have to wait for the IPI or we'll
 	 * lock up on an APIC access.
+	 *
+	 * Since CPU0 is not wakened up by INIT, it doesn't wait for the IPI.
 	 */
-	if (apic->wait_for_init_deassert)
+	cpuid = smp_processor_id();
+	if (apic->wait_for_init_deassert && cpuid != 0)
 		apic->wait_for_init_deassert(&init_deasserted);
 
 	/*
 	 * (This works even if the APIC is not enabled.)
 	 */
 	phys_id = read_apic_id();
-	cpuid = smp_processor_id();
 	if (cpumask_test_cpu(cpuid, cpu_callin_mask)) {
 		panic("%s: phys CPU#%d, CPU#%d already present??\n", __func__,
 					phys_id, cpuid);
@@ -251,6 +253,8 @@ static void __cpuinit smp_callin(void)
 	cpumask_set_cpu(cpuid, cpu_callin_mask);
 }
 
+static int cpu0_logical_apicid;
+static int enable_start_cpu0;
 /*
  * Activate a secondary processor.
  */
@@ -265,6 +269,8 @@ notrace static void __cpuinit start_secondary(void *unused)
 	preempt_disable();
 	smp_callin();
 
+	enable_start_cpu0 = 0;
+
 #ifdef CONFIG_X86_32
 	/* switch away from the initial page table */
 	load_cr3(swapper_pg_dir);
@@ -493,7 +499,7 @@ void __inquire_remote_apic(int apicid)
  * won't ... remember to clear down the APIC, etc later.
  */
 int __cpuinit
-wakeup_secondary_cpu_via_nmi(int logical_apicid, unsigned long start_eip)
+wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip)
 {
 	unsigned long send_status, accept_status = 0;
 	int maxlvt;
@@ -501,7 +507,7 @@ wakeup_secondary_cpu_via_nmi(int logical_apicid, unsigned long start_eip)
 	/* Target chip */
 	/* Boot on the stack */
 	/* Kick the second */
-	apic_icr_write(APIC_DM_NMI | apic->dest_logical, logical_apicid);
+	apic_icr_write(APIC_DM_NMI | apic->dest_logical, apicid);
 
 	pr_debug("Waiting for send to finish...\n");
 	send_status = safe_apic_wait_icr_idle();
@@ -677,6 +683,17 @@ static void __cpuinit announce_cpu(int cpu, int apicid)
 			node, cpu, apicid);
 }
 
+static int wakeup_cpu0_nmi(unsigned int cmd, struct pt_regs *regs)
+{
+	int cpu;
+
+	cpu = smp_processor_id();
+	if (cpu == 0 && !cpu_online(cpu) && enable_start_cpu0)
+		return NMI_HANDLED;
+
+	return NMI_DONE;
+}
+
 /*
  * NOTE - on most systems this is a PHYSICAL apic ID, but on multiquad
  * (ie clustered apic addressing mode), this is a LOGICAL apic ID.
@@ -771,8 +788,47 @@ do_rest:
 	 */
 	if (apic->wakeup_secondary_cpu)
 		boot_error = apic->wakeup_secondary_cpu(apicid, start_ip);
-	else
-		boot_error = wakeup_secondary_cpu_via_init(apicid, start_ip);
+	else {
+		if (cpu)
+			/*
+			 * Wake up AP by INIT, INIT, STARTUP sequence.
+			 */
+			boot_error = wakeup_secondary_cpu_via_init(apicid,
+								   start_ip);
+		else {
+			/*
+			 * Instead of waiting for STARTUP after INITs, BSP will
+			 * execute the BIOS boot-strap code which is not a
+			 * desired behavior for waking up BSP. To avoid the
+			 * boot-strap code, wake up CPU0 by NMI instead.
+			 *
+			 * This works to wake up soft offlined CPU0 only. If
+			 * CPU0 is hard offlined (i.e. physically hot removed
+			 * and then hot added), NMI won't wake it up. We'll
+			 * change this code in the future to wake up hard
+			 * offlined CPU0 if real platform and request are
+			 * available.
+			 */
+			int id;
+
+			enable_start_cpu0 = 1;
+			/*
+			 * Register a NMI handler to help wake up CPU0.
+			 */
+			boot_error = register_nmi_handler(NMI_LOCAL,
+					 wakeup_cpu0_nmi, 0, "wake_cpu0");
+
+			if (!boot_error) {
+				if (apic->dest_logical == APIC_DEST_LOGICAL)
+					id = cpu0_logical_apicid;
+				else
+					id = apicid;
+				boot_error = wakeup_secondary_cpu_via_nmi(id,
+								start_ip);
+				unregister_nmi_handler(NMI_LOCAL, "wake_cpu0");
+			}
+		}
+	}
 
 	if (!boot_error) {
 		/*
@@ -1088,6 +1144,8 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus)
 	 */
 	setup_local_APIC();
 
+	cpu0_logical_apicid = GET_APIC_LOGICAL_ID(apic_read(APIC_LDR));
+
 	/*
 	 * Enable IO APIC before setting up error vector
 	 */
@@ -1431,6 +1489,11 @@ static inline void mwait_play_dead(void)
 		__monitor(mwait_ptr, 0, 0);
 		mb();
 		__mwait(eax, 0);
+		/*
+		 * If NMI wakes up CPU0, start offlined CPU0.
+		 */
+		if (smp_processor_id() == 0 && enable_start_cpu0)
+			start_cpu0();
 	}
 }
 
@@ -1441,10 +1504,15 @@ static inline void hlt_play_dead(void)
 
 	while (1) {
 		native_halt();
+		/*
+		 * If NMI wakes up CPU0, start offlined CPU0.
+		 */
+		if (smp_processor_id() == 0 && enable_start_cpu0)
+			start_cpu0();
 	}
 }
 
-void native_play_dead(void)
+void __ref native_play_dead(void)
 {
 	play_dead_common();
 	tboot_shutdown(TB_SHUTDOWN_WFS);
-- 
1.6.0.3

Re: [PATCH v5 08/12] x86/smpboot.c: Wake up CPU0 via NMI instead of INITs

[Brian Gerst]

On Wed, Jan 11, 2012 at 12:04 PM, Fenghua Yu <fenghua.yu@intel.com> wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
>
> Instead of waiting for STARTUP after INITs, BSP will execute
> the BIOS boot-strap code which is not a desired behavior for
> waking up BSP. To avoid the boot-strap code, wake up CPU0 by
> NMI instead.
>
> This works to wake up soft offlined CPU0 only. If CPU0 is hard offlined (i.e.
> physically hot removed and then hot added), NMI won't wake it up. We'll change
> this code in the future to wake up hard offlined CPU0 if real platform and
> request are available.
>
> AP is still waken up as before by INIT, INIT, STARTUP sequence.

Would clearing IA32_APIC_BASE_MSR.BSP avoid calling the BIOS code when
the cpu is brought back online?  Setting that flag in another cpu that
is still online might also allow suspend/resume to work.

--
Brian Gerst

[PATCH v5 09/12] x86/common.c: Init CPU0 data during CPU0 online

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

During CPU0 online, enable x2apic, re-init mcheck, set_numa_node, add numa mask,
and enable sep cpu for CPU0.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/processor.h |    1 +
 arch/x86/kernel/cpu/common.c     |   17 ++++++++++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index aa9088c..861d0e9 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -160,6 +160,7 @@ extern struct pt_regs *idle_regs(struct pt_regs *);
 
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
+extern void identify_boot_cpu_online(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
 extern void print_cpu_info(struct cpuinfo_x86 *);
 extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c);
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index d43cad7..300e3e2 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -911,6 +911,18 @@ void __init identify_boot_cpu(void)
 #endif
 }
 
+void __cpuinit identify_boot_cpu_online(void)
+{
+	mcheck_cpu_init(&boot_cpu_data);
+
+#ifdef CONFIG_NUMA
+	numa_add_cpu(smp_processor_id());
+#endif
+#ifdef CONFIG_X86_32
+	enable_sep_cpu();
+#endif
+}
+
 void __cpuinit identify_secondary_cpu(struct cpuinfo_x86 *c)
 {
 	BUG_ON(c == &boot_cpu_data);
@@ -1189,7 +1201,7 @@ void __cpuinit cpu_init(void)
 	oist = &per_cpu(orig_ist, cpu);
 
 #ifdef CONFIG_NUMA
-	if (cpu != 0 && percpu_read(numa_node) == 0 &&
+	if (percpu_read(numa_node) == 0 &&
 	    early_cpu_to_node(cpu) != NUMA_NO_NODE)
 		set_numa_node(early_cpu_to_node(cpu));
 #endif
@@ -1221,8 +1233,7 @@ void __cpuinit cpu_init(void)
 	barrier();
 
 	x86_configure_nx();
-	if (cpu != 0)
-		enable_x2apic();
+	enable_x2apic();
 
 	/*
 	 * set up and load the per-CPU TSS
-- 
1.6.0.3

[PATCH v5 10/12] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Ask the first online CPU to save mtrr instead of asking BSP. BSP could be
offline when mtrr_save_state() is called.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/cpu/mtrr/main.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
index 6b96110..e4c1a41 100644
--- a/arch/x86/kernel/cpu/mtrr/main.c
+++ b/arch/x86/kernel/cpu/mtrr/main.c
@@ -695,11 +695,16 @@ void mtrr_ap_init(void)
 }
 
 /**
- * Save current fixed-range MTRR state of the BSP
+ * Save current fixed-range MTRR state of the first cpu in cpu_online_mask.
  */
 void mtrr_save_state(void)
 {
-	smp_call_function_single(0, mtrr_save_fixed_ranges, NULL, 1);
+	int first_cpu;
+
+	get_online_cpus();
+	first_cpu = cpumask_first(cpu_online_mask);
+	smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
+	put_online_cpus();
 }
 
 void set_mtrr_aps_delayed_init(void)
-- 
1.6.0.3

Re: [PATCH v5 10/12] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Brian Gerst]

On Wed, Jan 11, 2012 at 12:04 PM, Fenghua Yu <fenghua.yu@intel.com> wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
>
> Ask the first online CPU to save mtrr instead of asking BSP. BSP could be
> offline when mtrr_save_state() is called.

If you can use any non-boot cpu to save the MTRRs why not just use the
current cpu?  They should all be in sync anyways.

--
Brian Gerst

Re: [PATCH v5 10/12] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[H. Peter Anvin]

On 01/12/2012 04:33 AM, Brian Gerst wrote:
> On Wed, Jan 11, 2012 at 12:04 PM, Fenghua Yu <fenghua.yu@intel.com> wrote:
>> From: Fenghua Yu <fenghua.yu@intel.com>
>>
>> Ask the first online CPU to save mtrr instead of asking BSP. BSP could be
>> offline when mtrr_save_state() is called.
> 
> If you can use any non-boot cpu to save the MTRRs why not just use the
> current cpu?  They should all be in sync anyways.
> 

A much bigger question: why do we ever bother saving the MTRR state per
se?  We examine the MTRR state -- we have to -- during boot, and it
should never diverge from the state set by the OS from that point on --
we'll need to set it back to that.  So we should just keep track of what
the correct MTRR state is at all times.

	-hpa


-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

RE: [PATCH v5 10/12] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Yu, Fenghua]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1508 bytes --]

> From: H. Peter Anvin [mailto:hpa@zytor.com]
> Sent: Sunday, January 15, 2012 4:08 PM
> On 01/12/2012 04:33 AM, Brian Gerst wrote:
> > On Wed, Jan 11, 2012 at 12:04 PM, Fenghua Yu <fenghua.yu@intel.com>
> wrote:
> >> From: Fenghua Yu <fenghua.yu@intel.com>
> >>
> >> Ask the first online CPU to save mtrr instead of asking BSP. BSP
> could be
> >> offline when mtrr_save_state() is called.
> >
> > If you can use any non-boot cpu to save the MTRRs why not just use
> the
> > current cpu?  They should all be in sync anyways.
> >
> 
> A much bigger question: why do we ever bother saving the MTRR state per
> se?  We examine the MTRR state -- we have to -- during boot, and it
> should never diverge from the state set by the OS from that point on --
> we'll need to set it back to that.  So we should just keep track of
> what
> the correct MTRR state is at all times.

The fixed MTRR state is saved when each CPU is up:
        /*
         * Save current MTRR state in case it was changed since early boot
         * (e.g. by the ACPI SMI) to initialize new CPUs with MTRRs in sync:
         */
        mtrr_save_state();
It's saved because MTRR state could be changed since early boot according to the above comment. I haven't found other info source except the comment. If that's true, we may need to save the state, right?

Thanks.

-Fenghua

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

RE: [PATCH v5 10/12] x86/mtrr/main.c: Ask the first online CPU to save mtrr

[Yu, Fenghua]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 800 bytes --]

On 01/12/2012 04:33 AM, Brian Gerst wrote:
> On Wed, Jan 11, 2012 at 12:04 PM, Fenghua Yu <fenghua.yu@intel.com> wrote:
>> From: Fenghua Yu <fenghua.yu@intel.com>
>>
>> Ask the first online CPU to save mtrr instead of asking BSP. BSP
could be
>> offline when mtrr_save_state() is called.
>
> If you can use any non-boot cpu to save the MTRRs why not just use the
> current cpu?  They should all be in sync anyways.

The current cpu can not be used to save the MTRRs because its MTRRs are not initialized yet at this point. Later on, set_mtrr() will be called to initialize MTRRs on the current CPU (and all booted CPUs).

Thanks.

-Fenghua
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

[PATCH v5 11/12] x86/i387.c: Thread xstate is initialized only on CPU0 once

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

init_thread_xstate() is only called once to avoid overriding xstate_size during
boot time or during CPU hotplug.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/i387.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
index 739d859..cca99fa 100644
--- a/arch/x86/kernel/i387.c
+++ b/arch/x86/kernel/i387.c
@@ -107,7 +107,11 @@ void __cpuinit fpu_init(void)
 		cr0 |= X86_CR0_EM;
 	write_cr0(cr0);
 
-	if (!smp_processor_id())
+	/*
+	 * init_thread_xstate is only called once to avoid overriding
+	 * xstate_size during boot time or during CPU hotplug.
+	 */
+	if (xstate_size == 0)
 		init_thread_xstate();
 
 	mxcsr_feature_mask_init();
-- 
1.6.0.3

[PATCH v5 12/12] x86/topology.c: debug CPU0 hotplug

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

CONFIG_DEBUG_HOTPLUG_CPU0 is for debuging the CPU0 hotplug feature. The switch
offlines CPU0 as soon as possible and boots userspace up with CPU0 offlined.
User can online CPU0 back after boot time. The default value of the switch is
off.

To debug CPU0 hotplug, you need to enable CPU0 offline/online feature by either
turning on CONFIG_BOOTPARAM_HOTPLUG_CPU0 during compilation or giving
cpu0_hotplug kernel parameter at boot.

It's safe and early place to take down CPU0 after all hotplug notifiers
are installed and SMP is booted.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/Kconfig           |   15 +++++++++++++++
 arch/x86/kernel/topology.c |   39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 0 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c2cf195..36b6a2c 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1748,6 +1748,21 @@ config BOOTPARAM_HOTPLUG_CPU0
 	  You still can enable the CPU0 hotplug feature at boot by kernel
 	  parameter cpu0_hotplug.
 
+config DEBUG_HOTPLUG_CPU0
+	def_bool n
+	prompt "Debug CPU0 hotplug"
+	depends on HOTPLUG_CPU && EXPERIMENTAL
+	---help---
+	  Enabling this option offlines CPU0 (if CPU0 can be offlined) as
+	  soon as possible and boots up userspace with CPU0 offlined. User
+	  can online CPU0 back after boot time.
+
+	  To debug CPU0 hotplug, you need to enable CPU0 offline/online
+	  feature by either turning on CONFIG_BOOTPARAM_HOTPLUG_CPU0 during
+	  compilation or giving cpu0_hotplug kernel parameter at boot.
+
+	  If unuser, say N.
+
 config COMPAT_VDSO
 	def_bool y
 	prompt "Compat VDSO support"
diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
index 8cec15c..3f697dd 100644
--- a/arch/x86/kernel/topology.c
+++ b/arch/x86/kernel/topology.c
@@ -49,6 +49,45 @@ static int __init enable_cpu0_hotplug(char *str)
 __setup("cpu0_hotplug", enable_cpu0_hotplug);
 #endif
 
+#ifdef CONFIG_DEBUG_HOTPLUG_CPU0
+/*
+ * This function offlines CPU0 as early as possible and allows userspace to
+ * boot up without CPU0. CPU0 can be onlined back by user after boot.
+ *
+ * This is only called for debugging CPU0 offline feature.
+ */
+static void __init _debug_hotplug_cpu(int num)
+{
+	int ret;
+	struct sys_device *dev = get_cpu_sysdev(num);
+
+	/*
+	 * Take hotpluggable online CPU0 down.
+	 */
+	if (num || !per_cpu(cpu_devices, num).cpu.hotpluggable || !dev ||
+	    !cpu_online(num)) {
+		pr_debug("CPU0 can't be offlined.\n");
+		return;
+	}
+
+	cpu_hotplug_driver_lock();
+
+	ret = cpu_down(num);
+	if (!ret)
+		kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
+	else
+		pr_debug("Can't offline CPU%d.\n", num);
+
+	cpu_hotplug_driver_unlock();
+}
+static int __init debug_hotplug_cpu(void)
+{
+	_debug_hotplug_cpu(0);
+	return 0;
+}
+device_initcall_sync(debug_hotplug_cpu);
+#endif /* CONFIG_DEBUG_HOTPLUG_CPU0 */
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-- 
1.6.0.3

Re: [PATCH v5 12/12] x86/topology.c: debug CPU0 hotplug

[Jiang Liu]

On 01/12/2012 01:04 AM, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> CONFIG_DEBUG_HOTPLUG_CPU0 is for debuging the CPU0 hotplug feature. The switch
> offlines CPU0 as soon as possible and boots userspace up with CPU0 offlined.
> User can online CPU0 back after boot time. The default value of the switch is
> off.
> 
> To debug CPU0 hotplug, you need to enable CPU0 offline/online feature by either
> turning on CONFIG_BOOTPARAM_HOTPLUG_CPU0 during compilation or giving
> cpu0_hotplug kernel parameter at boot.
> 
> It's safe and early place to take down CPU0 after all hotplug notifiers
> are installed and SMP is booted.
> 
> Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
> ---
>  arch/x86/Kconfig           |   15 +++++++++++++++
>  arch/x86/kernel/topology.c |   39 +++++++++++++++++++++++++++++++++++++++
>  2 files changed, 54 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index c2cf195..36b6a2c 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1748,6 +1748,21 @@ config BOOTPARAM_HOTPLUG_CPU0
>  	  You still can enable the CPU0 hotplug feature at boot by kernel
>  	  parameter cpu0_hotplug.
>  
> +config DEBUG_HOTPLUG_CPU0
> +	def_bool n
> +	prompt "Debug CPU0 hotplug"
> +	depends on HOTPLUG_CPU && EXPERIMENTAL
> +	---help---
> +	  Enabling this option offlines CPU0 (if CPU0 can be offlined) as
> +	  soon as possible and boots up userspace with CPU0 offlined. User
> +	  can online CPU0 back after boot time.
> +
> +	  To debug CPU0 hotplug, you need to enable CPU0 offline/online
> +	  feature by either turning on CONFIG_BOOTPARAM_HOTPLUG_CPU0 during
> +	  compilation or giving cpu0_hotplug kernel parameter at boot.
> +
> +	  If unuser, say N.
> +
>  config COMPAT_VDSO
>  	def_bool y
>  	prompt "Compat VDSO support"
> diff --git a/arch/x86/kernel/topology.c b/arch/x86/kernel/topology.c
> index 8cec15c..3f697dd 100644
> --- a/arch/x86/kernel/topology.c
> +++ b/arch/x86/kernel/topology.c
> @@ -49,6 +49,45 @@ static int __init enable_cpu0_hotplug(char *str)
>  __setup("cpu0_hotplug", enable_cpu0_hotplug);
>  #endif
>  
> +#ifdef CONFIG_DEBUG_HOTPLUG_CPU0
> +/*
> + * This function offlines CPU0 as early as possible and allows userspace to
> + * boot up without CPU0. CPU0 can be onlined back by user after boot.
> + *
> + * This is only called for debugging CPU0 offline feature.
> + */
> +static void __init _debug_hotplug_cpu(int num)
> +{
> +	int ret;
> +	struct sys_device *dev = get_cpu_sysdev(num);
> +
> +	/*
> +	 * Take hotpluggable online CPU0 down.
> +	 */
> +	if (num || !per_cpu(cpu_devices, num).cpu.hotpluggable || !dev ||
> +	    !cpu_online(num)) {
> +		pr_debug("CPU0 can't be offlined.\n");
> +		return;
> +	}
The usage of parameter num seems a little inconsistent. 
Suggest either to remove the num parameter or to get rid of the assumption
that num is always zero.

> +
> +	cpu_hotplug_driver_lock();
> +
> +	ret = cpu_down(num);
> +	if (!ret)
> +		kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
> +	else
> +		pr_debug("Can't offline CPU%d.\n", num);
> +
> +	cpu_hotplug_driver_unlock();
> +}
> +static int __init debug_hotplug_cpu(void)
> +{
> +	_debug_hotplug_cpu(0);
> +	return 0;
> +}
> +device_initcall_sync(debug_hotplug_cpu);
> +#endif /* CONFIG_DEBUG_HOTPLUG_CPU0 */
> +
>  int __ref arch_register_cpu(int num)
>  {
>  	/*