[PATCH RFC v3 0/3] support hotplug CPUs before reboot

[PATCH RFC v3 0/3] support hotplug CPUs before reboot

[Hsin-Yi Wang]

This series adds a config REBOOT_HOTPLUG_CPU which would hotplug CPUs
before reboot. Architecture code (smp_send_stop) currently would loop
through online secondary CPUs and some may call ipi functions to them.
With this config enabled, ideally we don't need smp_send_stop. But we
keep the code for those don't enable this config and as a backup if
some CPU fails to go offline before reboot.

Also enable this config for arm64 and x86 defconfig as an example.

Change from v2:
* Add another config instead of configed by CONFIG_HOTPLUG_CPU

Hsin-Yi Wang (3):
  reboot: support hotplug CPUs before reboot
  arm64: defconfig: enable REBOOT_HOTPLUG_CPU
  x86: defconfig: enable REBOOT_HOTPLUG_CPU

 arch/Kconfig                      |  6 ++++++
 arch/arm64/configs/defconfig      |  1 +
 arch/x86/configs/i386_defconfig   |  1 +
 arch/x86/configs/x86_64_defconfig |  1 +
 include/linux/cpu.h               |  3 +++
 kernel/cpu.c                      | 19 +++++++++++++++++++
 kernel/reboot.c                   |  8 ++++++++
 7 files changed, 39 insertions(+)

-- 
2.25.0.rc1.283.g88dfdc4193-goog

[PATCH RFC v3 1/3] reboot: support hotplug CPUs before reboot

[Hsin-Yi Wang]

Currently system reboots uses architecture specific codes (smp_send_stop)
to offline non reboot CPUs. Most architecture's implementation is looping
through all non reboot online CPUs and call ipi function to each of them. Some
architecture like arm64, arm, and x86... would set offline masks to cpu without
really offline them. This causes some race condition and kernel warning comes
out sometimes when system reboots.

This patch adds a config REBOOT_HOTPLUG_CPU, which would hotplug cpus in
migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for
checking online cpus would be an empty loop. If architecture don't enable this
config, or some cpus somehow fails to offline, it would fallback to ipi
function.

Signed-off-by: Hsin-Yi Wang <hsinyi@chromium.org>
---
 arch/Kconfig        |  6 ++++++
 include/linux/cpu.h |  3 +++
 kernel/cpu.c        | 19 +++++++++++++++++++
 kernel/reboot.c     |  8 ++++++++
 4 files changed, 36 insertions(+)

diff --git a/arch/Kconfig b/arch/Kconfig
index 48b5e103bdb0..a043b9be1499 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -52,6 +52,12 @@ config OPROFILE_EVENT_MULTIPLEX
 
 	  If unsure, say N.
 
+config REBOOT_HOTPLUG_CPU
+	bool "Support for hotplug CPUs before reboot"
+	depends on HOTPLUG_CPU
+	help
+	  Say Y to do a full hotplug on secondary CPUs before reboot.
+
 config HAVE_OPROFILE
 	bool
 
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 1ca2baf817ed..3bf5ab289954 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -118,6 +118,9 @@ extern void cpu_hotplug_disable(void);
 extern void cpu_hotplug_enable(void);
 void clear_tasks_mm_cpumask(int cpu);
 int cpu_down(unsigned int cpu);
+#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
+extern void offline_secondary_cpus(int primary);
+#endif
 
 #else /* CONFIG_HOTPLUG_CPU */
 
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 9c706af713fb..52afc47dd56a 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1057,6 +1057,25 @@ int cpu_down(unsigned int cpu)
 }
 EXPORT_SYMBOL(cpu_down);
 
+#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
+void offline_secondary_cpus(int primary)
+{
+	int i, err;
+
+	cpu_maps_update_begin();
+
+	for_each_online_cpu(i) {
+		if (i == primary)
+			continue;
+		err = _cpu_down(i, 0, CPUHP_OFFLINE);
+		if (err)
+			pr_warn("Failed to offline cpu %d\n", i);
+	}
+	cpu_hotplug_disabled++;
+
+	cpu_maps_update_done();
+}
+#endif
 #else
 #define takedown_cpu		NULL
 #endif /*CONFIG_HOTPLUG_CPU*/
diff --git a/kernel/reboot.c b/kernel/reboot.c
index c4d472b7f1b4..fda84794ce46 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -7,6 +7,7 @@
 
 #define pr_fmt(fmt)	"reboot: " fmt
 
+#include <linux/cpu.h>
 #include <linux/ctype.h>
 #include <linux/export.h>
 #include <linux/kexec.h>
@@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void)
 	/* The boot cpu is always logical cpu 0 */
 	int cpu = reboot_cpu;
 
+#if !IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
 	cpu_hotplug_disable();
+#endif
 
 	/* Make certain the cpu I'm about to reboot on is online */
 	if (!cpu_online(cpu))
@@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void)
 
 	/* Make certain I only run on the appropriate processor */
 	set_cpus_allowed_ptr(current, cpumask_of(cpu));
+
+	/* Hotplug other cpus if possible */
+#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
+	offline_secondary_cpus(cpu);
+#endif
 }
 
 /**
-- 
2.25.0.rc1.283.g88dfdc4193-goog

[PATCH RFC v3 2/3] arm64: defconfig: enable REBOOT_HOTPLUG_CPU

[Hsin-Yi Wang]

Enable REBOOT_HOTPLUG_CPU for arm64 to hotplug CPUs before reboot if
CONFIG_HOTPLUG_CPU is set.

Signed-off-by: Hsin-Yi Wang <hsinyi@chromium.org>
---
 arch/arm64/configs/defconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 8409aa80e30a..9815fe0ebdb4 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -687,6 +687,7 @@ CONFIG_QCOM_BAM_DMA=y
 CONFIG_QCOM_HIDMA_MGMT=y
 CONFIG_QCOM_HIDMA=y
 CONFIG_RCAR_DMAC=y
+CONFIG_REBOOT_HOTPLUG_CPU=y
 CONFIG_RENESAS_USB_DMAC=m
 CONFIG_VFIO=y
 CONFIG_VFIO_PCI=y
-- 
2.25.0.rc1.283.g88dfdc4193-goog

[PATCH RFC v3 3/3] x86: defconfig: enable REBOOT_HOTPLUG_CPU

[Hsin-Yi Wang]

Enable REBOOT_HOTPLUG_CPU for x86 to hotplug CPUs before reboot if
CONFIG_HOTPLUG_CPU is set.

Signed-off-by: Hsin-Yi Wang <hsinyi@chromium.org>
---
 arch/x86/configs/i386_defconfig   | 1 +
 arch/x86/configs/x86_64_defconfig | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/x86/configs/i386_defconfig b/arch/x86/configs/i386_defconfig
index 59ce9ed58430..2f3e90969cb2 100644
--- a/arch/x86/configs/i386_defconfig
+++ b/arch/x86/configs/i386_defconfig
@@ -46,6 +46,7 @@ CONFIG_KEXEC=y
 CONFIG_CRASH_DUMP=y
 CONFIG_RANDOMIZE_BASE=y
 CONFIG_RANDOMIZE_MEMORY=y
+CONFIG_REBOOT_HOTPLUG_CPU=y
 # CONFIG_COMPAT_VDSO is not set
 CONFIG_HIBERNATION=y
 CONFIG_PM_DEBUG=y
diff --git a/arch/x86/configs/x86_64_defconfig b/arch/x86/configs/x86_64_defconfig
index 0b9654c7a05c..196d1329ee68 100644
--- a/arch/x86/configs/x86_64_defconfig
+++ b/arch/x86/configs/x86_64_defconfig
@@ -43,6 +43,7 @@ CONFIG_KEXEC=y
 CONFIG_CRASH_DUMP=y
 CONFIG_RANDOMIZE_BASE=y
 CONFIG_RANDOMIZE_MEMORY=y
+CONFIG_REBOOT_HOTPLUG_CPU=y
 # CONFIG_COMPAT_VDSO is not set
 CONFIG_HIBERNATION=y
 CONFIG_PM_DEBUG=y
-- 
2.25.0.rc1.283.g88dfdc4193-goog

Re: [PATCH RFC v3 1/3] reboot: support hotplug CPUs before reboot

[Vitaly Kuznetsov]

Hsin-Yi Wang <hsinyi@chromium.org> writes:

> Currently system reboots uses architecture specific codes (smp_send_stop)
> to offline non reboot CPUs. Most architecture's implementation is looping
> through all non reboot online CPUs and call ipi function to each of them. Some
> architecture like arm64, arm, and x86... would set offline masks to cpu without
> really offline them. This causes some race condition and kernel warning comes
> out sometimes when system reboots.
>
> This patch adds a config REBOOT_HOTPLUG_CPU, which would hotplug cpus in
> migrate_to_reboot_cpu(). If non reboot cpus are all offlined here, the loop for
> checking online cpus would be an empty loop. If architecture don't enable this
> config, or some cpus somehow fails to offline, it would fallback to ipi
> function.
>
> Signed-off-by: Hsin-Yi Wang <hsinyi@chromium.org>
> ---
>  arch/Kconfig        |  6 ++++++
>  include/linux/cpu.h |  3 +++
>  kernel/cpu.c        | 19 +++++++++++++++++++
>  kernel/reboot.c     |  8 ++++++++
>  4 files changed, 36 insertions(+)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 48b5e103bdb0..a043b9be1499 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -52,6 +52,12 @@ config OPROFILE_EVENT_MULTIPLEX
>  
>  	  If unsure, say N.
>  
> +config REBOOT_HOTPLUG_CPU
> +	bool "Support for hotplug CPUs before reboot"
> +	depends on HOTPLUG_CPU
> +	help
> +	  Say Y to do a full hotplug on secondary CPUs before reboot.

I'm not sure this should be a configurable option, e.g. in case this is
a good approach in general, why not just use CONFIG_HOTPLUG_CPU in the
code? 

> +
>  config HAVE_OPROFILE
>  	bool
>  
> diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> index 1ca2baf817ed..3bf5ab289954 100644
> --- a/include/linux/cpu.h
> +++ b/include/linux/cpu.h
> @@ -118,6 +118,9 @@ extern void cpu_hotplug_disable(void);
>  extern void cpu_hotplug_enable(void);
>  void clear_tasks_mm_cpumask(int cpu);
>  int cpu_down(unsigned int cpu);
> +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> +extern void offline_secondary_cpus(int primary);
> +#endif
>  
>  #else /* CONFIG_HOTPLUG_CPU */
>  
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 9c706af713fb..52afc47dd56a 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -1057,6 +1057,25 @@ int cpu_down(unsigned int cpu)
>  }
>  EXPORT_SYMBOL(cpu_down);
>  
> +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> +void offline_secondary_cpus(int primary)
> +{
> +	int i, err;
> +
> +	cpu_maps_update_begin();
> +
> +	for_each_online_cpu(i) {
> +		if (i == primary)
> +			continue;
> +		err = _cpu_down(i, 0, CPUHP_OFFLINE);
> +		if (err)
> +			pr_warn("Failed to offline cpu %d\n", i);
> +	}
> +	cpu_hotplug_disabled++;
> +
> +	cpu_maps_update_done();
> +}
> +#endif

This looks like a simplified version of freeze_secondary_cpus(), can
they be merged?


>  #else
>  #define takedown_cpu		NULL
>  #endif /*CONFIG_HOTPLUG_CPU*/
> diff --git a/kernel/reboot.c b/kernel/reboot.c
> index c4d472b7f1b4..fda84794ce46 100644
> --- a/kernel/reboot.c
> +++ b/kernel/reboot.c
> @@ -7,6 +7,7 @@
>  
>  #define pr_fmt(fmt)	"reboot: " fmt
>  
> +#include <linux/cpu.h>
>  #include <linux/ctype.h>
>  #include <linux/export.h>
>  #include <linux/kexec.h>
> @@ -220,7 +221,9 @@ void migrate_to_reboot_cpu(void)
>  	/* The boot cpu is always logical cpu 0 */
>  	int cpu = reboot_cpu;
>  
> +#if !IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
>  	cpu_hotplug_disable();
> +#endif
>  
>  	/* Make certain the cpu I'm about to reboot on is online */
>  	if (!cpu_online(cpu))
> @@ -231,6 +234,11 @@ void migrate_to_reboot_cpu(void)
>  
>  	/* Make certain I only run on the appropriate processor */
>  	set_cpus_allowed_ptr(current, cpumask_of(cpu));
> +
> +	/* Hotplug other cpus if possible */
> +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> +	offline_secondary_cpus(cpu);
> +#endif
>  }
>  
>  /**

-- 
Vitaly

Re: [PATCH RFC v3 1/3] reboot: support hotplug CPUs before reboot

[Hsin-Yi Wang]

On Mon, Jan 13, 2020 at 8:46 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

Thanks for your comments.

> > +config REBOOT_HOTPLUG_CPU
> > +     bool "Support for hotplug CPUs before reboot"
> > +     depends on HOTPLUG_CPU
> > +     help
> > +       Say Y to do a full hotplug on secondary CPUs before reboot.
>
> I'm not sure this should be a configurable option, e.g. in case this is
> a good approach in general, why not just use CONFIG_HOTPLUG_CPU in the
> code?
>
In v2 it uses CONFIG_HOTPLUG_CPU, but I think adding another config is
more flexible. Maybe there are some architecture that supports
HOTPLUG_CPU but doesn't want to do full cpu hotplug before reboot.
(Eg. doing cpu hotplug would make reboot process slower.)
> > +
> >  config HAVE_OPROFILE
> >       bool
> >
> > diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> > index 1ca2baf817ed..3bf5ab289954 100644
> > --- a/include/linux/cpu.h
> > +++ b/include/linux/cpu.h
> > @@ -118,6 +118,9 @@ extern void cpu_hotplug_disable(void);
> >  extern void cpu_hotplug_enable(void);
> >  void clear_tasks_mm_cpumask(int cpu);
> >  int cpu_down(unsigned int cpu);
> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> > +extern void offline_secondary_cpus(int primary);
> > +#endif
> >
> >  #else /* CONFIG_HOTPLUG_CPU */
> >
> > diff --git a/kernel/cpu.c b/kernel/cpu.c
> > index 9c706af713fb..52afc47dd56a 100644
> > --- a/kernel/cpu.c
> > +++ b/kernel/cpu.c
> > @@ -1057,6 +1057,25 @@ int cpu_down(unsigned int cpu)
> >  }
> >  EXPORT_SYMBOL(cpu_down);
> >
> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> > +void offline_secondary_cpus(int primary)
> > +{
> > +     int i, err;
> > +
> > +     cpu_maps_update_begin();
> > +
> > +     for_each_online_cpu(i) {
> > +             if (i == primary)
> > +                     continue;
> > +             err = _cpu_down(i, 0, CPUHP_OFFLINE);
> > +             if (err)
> > +                     pr_warn("Failed to offline cpu %d\n", i);
> > +     }
> > +     cpu_hotplug_disabled++;
> > +
> > +     cpu_maps_update_done();
> > +}
> > +#endif
>
> This looks like a simplified version of freeze_secondary_cpus(), can
> they be merged?
>
Comparing to freeze_secondary_cpus(),  I think it's not necessary to
check pm_wakeup_pending() before _cpu_down() here. Thus it doesn't
need to depend on CONFIG_PM_SLEEP_SMP.
Also I think it could continue to offline other CPUs even one fails,
while freeze_secondary_cpus() would stop once it fails on offlining
one CPU.
Based on these differences, I didn't use freeze_secondary_cpus() here.
As for merging the common part, it might need additional flags to
handle the difference, which might lower the readability.
>

Re: [PATCH RFC v3 1/3] reboot: support hotplug CPUs before reboot

[Vitaly Kuznetsov]

Hsin-Yi Wang <hsinyi@chromium.org> writes:

> On Mon, Jan 13, 2020 at 8:46 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Thanks for your comments.
>
>> > +config REBOOT_HOTPLUG_CPU
>> > +     bool "Support for hotplug CPUs before reboot"
>> > +     depends on HOTPLUG_CPU
>> > +     help
>> > +       Say Y to do a full hotplug on secondary CPUs before reboot.
>>
>> I'm not sure this should be a configurable option, e.g. in case this is
>> a good approach in general, why not just use CONFIG_HOTPLUG_CPU in the
>> code?
>>
> In v2 it uses CONFIG_HOTPLUG_CPU, but I think adding another config is
> more flexible. Maybe there are some architecture that supports
> HOTPLUG_CPU but doesn't want to do full cpu hotplug before reboot.
> (Eg. doing cpu hotplug would make reboot process slower.)

In that case this should be an architectural decision, not a selectable
option. If you want to enable it for certain arches only (and not the
other way around), that would look like

config ARCH_HAS_HOTUNPLUG_CPUS_ON_REBOOT
	bool

...

config X86
        def_bool y
        ...
        select ARCH_HAS_HOTUNPLUG_CPUS_ON_REBOOT

because as a user, I really have no idea if I want to 'unplug secondary
CPUs on reboot' or not.

>> > +
>> >  config HAVE_OPROFILE
>> >       bool
>> >
>> > diff --git a/include/linux/cpu.h b/include/linux/cpu.h
>> > index 1ca2baf817ed..3bf5ab289954 100644
>> > --- a/include/linux/cpu.h
>> > +++ b/include/linux/cpu.h
>> > @@ -118,6 +118,9 @@ extern void cpu_hotplug_disable(void);
>> >  extern void cpu_hotplug_enable(void);
>> >  void clear_tasks_mm_cpumask(int cpu);
>> >  int cpu_down(unsigned int cpu);
>> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
>> > +extern void offline_secondary_cpus(int primary);
>> > +#endif
>> >
>> >  #else /* CONFIG_HOTPLUG_CPU */
>> >
>> > diff --git a/kernel/cpu.c b/kernel/cpu.c
>> > index 9c706af713fb..52afc47dd56a 100644
>> > --- a/kernel/cpu.c
>> > +++ b/kernel/cpu.c
>> > @@ -1057,6 +1057,25 @@ int cpu_down(unsigned int cpu)
>> >  }
>> >  EXPORT_SYMBOL(cpu_down);
>> >
>> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
>> > +void offline_secondary_cpus(int primary)
>> > +{
>> > +     int i, err;
>> > +
>> > +     cpu_maps_update_begin();
>> > +
>> > +     for_each_online_cpu(i) {
>> > +             if (i == primary)
>> > +                     continue;
>> > +             err = _cpu_down(i, 0, CPUHP_OFFLINE);
>> > +             if (err)
>> > +                     pr_warn("Failed to offline cpu %d\n", i);
>> > +     }
>> > +     cpu_hotplug_disabled++;
>> > +
>> > +     cpu_maps_update_done();
>> > +}
>> > +#endif
>>
>> This looks like a simplified version of freeze_secondary_cpus(), can
>> they be merged?
>>
> Comparing to freeze_secondary_cpus(),  I think it's not necessary to
> check pm_wakeup_pending() before _cpu_down() here. Thus it doesn't
> need to depend on CONFIG_PM_SLEEP_SMP.
> Also I think it could continue to offline other CPUs even one fails,
> while freeze_secondary_cpus() would stop once it fails on offlining
> one CPU.
> Based on these differences, I didn't use freeze_secondary_cpus() here.
> As for merging the common part, it might need additional flags to
> handle the difference, which might lower the readability.

I have to admit I'm not convinced (but maintainers may disagree of
course): #ifdefs are there to avoid compiling code which we don't need,
in case a second user emerges we can drop them or #ifdef just some parts
of it, it's not set in stone. Also, in case the only difference is that
you don't want to stop if some CPU fails to offline, a single bool flag
(e.g. 'force') would solve the problem, I don't see a significant
readability change.

-- 
Vitaly

Re: [PATCH RFC v3 1/3] reboot: support hotplug CPUs before reboot

[Hsin-Yi Wang]

On Mon, Jan 13, 2020 at 11:57 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Hsin-Yi Wang <hsinyi@chromium.org> writes:
>
> > On Mon, Jan 13, 2020 at 8:46 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
> >
> > Thanks for your comments.
> >
> >> > +config REBOOT_HOTPLUG_CPU
> >> > +     bool "Support for hotplug CPUs before reboot"
> >> > +     depends on HOTPLUG_CPU
> >> > +     help
> >> > +       Say Y to do a full hotplug on secondary CPUs before reboot.
> >>
> >> I'm not sure this should be a configurable option, e.g. in case this is
> >> a good approach in general, why not just use CONFIG_HOTPLUG_CPU in the
> >> code?
> >>
> > In v2 it uses CONFIG_HOTPLUG_CPU, but I think adding another config is
> > more flexible. Maybe there are some architecture that supports
> > HOTPLUG_CPU but doesn't want to do full cpu hotplug before reboot.
> > (Eg. doing cpu hotplug would make reboot process slower.)
>
> In that case this should be an architectural decision, not a selectable
> option. If you want to enable it for certain arches only (and not the
> other way around), that would look like
>
> config ARCH_HAS_HOTUNPLUG_CPUS_ON_REBOOT
>         bool
>
> ...
>
> config X86
>         def_bool y
>         ...
>         select ARCH_HAS_HOTUNPLUG_CPUS_ON_REBOOT
>
> because as a user, I really have no idea if I want to 'unplug secondary
> CPUs on reboot' or not.
>
Thanks for the example. I would use this way in next version.
> >> > +
> >> >  config HAVE_OPROFILE
> >> >       bool
> >> >
> >> > diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> >> > index 1ca2baf817ed..3bf5ab289954 100644
> >> > --- a/include/linux/cpu.h
> >> > +++ b/include/linux/cpu.h
> >> > @@ -118,6 +118,9 @@ extern void cpu_hotplug_disable(void);
> >> >  extern void cpu_hotplug_enable(void);
> >> >  void clear_tasks_mm_cpumask(int cpu);
> >> >  int cpu_down(unsigned int cpu);
> >> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> >> > +extern void offline_secondary_cpus(int primary);
> >> > +#endif
> >> >
> >> >  #else /* CONFIG_HOTPLUG_CPU */
> >> >
> >> > diff --git a/kernel/cpu.c b/kernel/cpu.c
> >> > index 9c706af713fb..52afc47dd56a 100644
> >> > --- a/kernel/cpu.c
> >> > +++ b/kernel/cpu.c
> >> > @@ -1057,6 +1057,25 @@ int cpu_down(unsigned int cpu)
> >> >  }
> >> >  EXPORT_SYMBOL(cpu_down);
> >> >
> >> > +#if IS_ENABLED(CONFIG_REBOOT_HOTPLUG_CPU)
> >> > +void offline_secondary_cpus(int primary)
> >> > +{
> >> > +     int i, err;
> >> > +
> >> > +     cpu_maps_update_begin();
> >> > +
> >> > +     for_each_online_cpu(i) {
> >> > +             if (i == primary)
> >> > +                     continue;
> >> > +             err = _cpu_down(i, 0, CPUHP_OFFLINE);
> >> > +             if (err)
> >> > +                     pr_warn("Failed to offline cpu %d\n", i);
> >> > +     }
> >> > +     cpu_hotplug_disabled++;
> >> > +
> >> > +     cpu_maps_update_done();
> >> > +}
> >> > +#endif
> >>
> >> This looks like a simplified version of freeze_secondary_cpus(), can
> >> they be merged?
> >>
> > Comparing to freeze_secondary_cpus(),  I think it's not necessary to
> > check pm_wakeup_pending() before _cpu_down() here. Thus it doesn't
> > need to depend on CONFIG_PM_SLEEP_SMP.
> > Also I think it could continue to offline other CPUs even one fails,
> > while freeze_secondary_cpus() would stop once it fails on offlining
> > one CPU.
> > Based on these differences, I didn't use freeze_secondary_cpus() here.
> > As for merging the common part, it might need additional flags to
> > handle the difference, which might lower the readability.
>
> I have to admit I'm not convinced (but maintainers may disagree of
> course): #ifdefs are there to avoid compiling code which we don't need,
> in case a second user emerges we can drop them or #ifdef just some parts
> of it, it's not set in stone. Also, in case the only difference is that
> you don't want to stop if some CPU fails to offline, a single bool flag
> (e.g. 'force') would solve the problem, I don't see a significant
> readability change.
>
Okay, I will merge them with an additional flag for whether it should
check pm_wakeup_pending() and break on error.
> --
> Vitaly
>