[PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[Qais Yousef]

This issue was fixed already by:

commit d66b16f5df4b ("arm64: Don't use disable_nonboot_cpus()")
commit dddf3578e0d4 ("ARM: Don't use disable_nonboot_cpus()")

The only caller of disable_nonboot_cpus() is x86, which is in a proper
suspend/resume path and due to the reverted patch lost its ability to
early abort due to a pending wakeup.

The fix that is being reverted is arguably a better one to backport to
stable trees. But it highlights the confusion about using
disable_nonboot_cpus() API.

This is a preparation to remove disable_nonboot_cpus() in favor of
freeze_secondary_cpus().

This reverts commit e98eac6ff1b45e4e73f2e6031b37c256ccb5d36b.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>
CC: "Rafael J. Wysocki" <rjw@rjwysocki.net>
CC: Len Brown <len.brown@intel.com>
CC: Pavel Machek <pavel@ucw.cz>
CC: Ingo Molnar <mingo@redhat.com>
CC: Borislav Petkov <bp@alien8.de>
CC: "H. Peter Anvin" <hpa@zytor.com>
CC: x86@kernel.org
CC: Todd E Brandt <todd.e.brandt@linux.intel.com>
CC: linux-pm@vger.kernel.org
CC: linux-kernel@vger.kernel.org
---
 include/linux/cpu.h | 12 +++---------
 kernel/cpu.c        |  4 ++--
 2 files changed, 5 insertions(+), 11 deletions(-)

diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index beaed2dc269e..9ead281157d3 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -144,18 +144,12 @@ static inline void get_online_cpus(void) { cpus_read_lock(); }
 static inline void put_online_cpus(void) { cpus_read_unlock(); }
 
 #ifdef CONFIG_PM_SLEEP_SMP
-int __freeze_secondary_cpus(int primary, bool suspend);
-static inline int freeze_secondary_cpus(int primary)
-{
-	return __freeze_secondary_cpus(primary, true);
-}
-
+extern int freeze_secondary_cpus(int primary);
 static inline int disable_nonboot_cpus(void)
 {
-	return __freeze_secondary_cpus(0, false);
+	return freeze_secondary_cpus(0);
 }
-
-void enable_nonboot_cpus(void);
+extern void enable_nonboot_cpus(void);
 
 static inline int suspend_disable_secondary_cpus(void)
 {
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 12ae636e9cb6..30848496cbc7 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1327,7 +1327,7 @@ void bringup_nonboot_cpus(unsigned int setup_max_cpus)
 #ifdef CONFIG_PM_SLEEP_SMP
 static cpumask_var_t frozen_cpus;
 
-int __freeze_secondary_cpus(int primary, bool suspend)
+int freeze_secondary_cpus(int primary)
 {
 	int cpu, error = 0;
 
@@ -1352,7 +1352,7 @@ int __freeze_secondary_cpus(int primary, bool suspend)
 		if (cpu == primary)
 			continue;
 
-		if (suspend && pm_wakeup_pending()) {
+		if (pm_wakeup_pending()) {
 			pr_info("Wakeup pending. Abort CPU freeze\n");
 			error = -EBUSY;
 			break;
-- 
2.17.1

[PATCH 2/3] cpu/hotplug: Remove disable_nonboot_cpus()

[Qais Yousef]

The single user could have called freeze_secondary_cpus() directly.

Since this function was a source of confusion, remove it as it's
just a pointless wrapper.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>
CC: "Rafael J. Wysocki" <rjw@rjwysocki.net>
CC: Len Brown <len.brown@intel.com>
CC: Pavel Machek <pavel@ucw.cz>
CC: Ingo Molnar <mingo@redhat.com>
CC: Borislav Petkov <bp@alien8.de>
CC: "H. Peter Anvin" <hpa@zytor.com>
CC: x86@kernel.org
CC: Todd E Brandt <todd.e.brandt@linux.intel.com>
CC: linux-pm@vger.kernel.org
CC: linux-kernel@vger.kernel.org
---
 Documentation/power/suspend-and-cpuhotplug.rst | 4 ++--
 arch/x86/power/cpu.c                           | 2 +-
 include/linux/cpu.h                            | 5 -----
 kernel/cpu.c                                   | 2 +-
 4 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/Documentation/power/suspend-and-cpuhotplug.rst b/Documentation/power/suspend-and-cpuhotplug.rst
index 572d968c5375..79f7610a6f6f 100644
--- a/Documentation/power/suspend-and-cpuhotplug.rst
+++ b/Documentation/power/suspend-and-cpuhotplug.rst
@@ -48,7 +48,7 @@ More details follow::
                                         |
                                         |
                                         v
-                              disable_nonboot_cpus()
+                              freeze_secondary_cpus()
                                    /* start */
                                         |
                                         v
@@ -83,7 +83,7 @@ More details follow::
                             Release cpu_add_remove_lock
                                         |
                                         v
-                       /* disable_nonboot_cpus() complete */
+                       /* freeze_secondary_cpus() complete */
                                         |
                                         v
                                    Do suspend
diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index 915bb1639763..e0a9ad8829b0 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -307,7 +307,7 @@ int hibernate_resume_nonboot_cpu_disable(void)
 	if (ret)
 		return ret;
 	smp_ops.play_dead = resume_play_dead;
-	ret = disable_nonboot_cpus();
+	ret = freeze_secondary_cpus(0);
 	smp_ops.play_dead = play_dead;
 	return ret;
 }
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 9ead281157d3..a00941c7828b 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -145,10 +145,6 @@ static inline void put_online_cpus(void) { cpus_read_unlock(); }
 
 #ifdef CONFIG_PM_SLEEP_SMP
 extern int freeze_secondary_cpus(int primary);
-static inline int disable_nonboot_cpus(void)
-{
-	return freeze_secondary_cpus(0);
-}
 extern void enable_nonboot_cpus(void);
 
 static inline int suspend_disable_secondary_cpus(void)
@@ -166,7 +162,6 @@ static inline void suspend_enable_secondary_cpus(void)
 }
 
 #else /* !CONFIG_PM_SLEEP_SMP */
-static inline int disable_nonboot_cpus(void) { return 0; }
 static inline void enable_nonboot_cpus(void) {}
 static inline int suspend_disable_secondary_cpus(void) { return 0; }
 static inline void suspend_enable_secondary_cpus(void) { }
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 30848496cbc7..d6d2c829561e 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1376,7 +1376,7 @@ int freeze_secondary_cpus(int primary)
 
 	/*
 	 * Make sure the CPUs won't be enabled by someone else. We need to do
-	 * this even in case of failure as all disable_nonboot_cpus() users are
+	 * this even in case of failure as all freeze_secondary_cpus() users are
 	 * supposed to do enable_nonboot_cpus() on the failure path.
 	 */
 	cpu_hotplug_disabled++;
-- 
2.17.1

[PATCH 3/3] cpu/hotplug: Rename enable_nonboot_cpus()

[Qais Yousef]

Now that disable_nonboot_cpus was removed in favour of
freeze_secondary_cpus(), rename enable_nonboot_cpus() to
thaw_secondary_cpus() to preserve the naming symmetry.

Done automatically via:

	git grep -l enable_nonboot_cpus | xargs sed -i 's/enable_nonboot_cpus/thaw_secondary_cpus/g'

Signed-off-by: Qais Yousef <qais.yousef@arm.com>
CC: "Rafael J. Wysocki" <rjw@rjwysocki.net>
CC: Len Brown <len.brown@intel.com>
CC: Pavel Machek <pavel@ucw.cz>
CC: Ingo Molnar <mingo@redhat.com>
CC: Borislav Petkov <bp@alien8.de>
CC: "H. Peter Anvin" <hpa@zytor.com>
CC: x86@kernel.org
CC: Todd E Brandt <todd.e.brandt@linux.intel.com>
CC: linux-pm@vger.kernel.org
CC: linux-kernel@vger.kernel.org
---
 Documentation/power/suspend-and-cpuhotplug.rst       |  2 +-
 arch/x86/kernel/smpboot.c                            |  4 ++--
 include/linux/cpu.h                                  |  6 +++---
 include/linux/smp.h                                  |  4 ++--
 kernel/cpu.c                                         | 12 ++++++------
 .../pm-graph/config/custom-timeline-functions.cfg    |  2 +-
 tools/power/pm-graph/sleepgraph.py                   |  2 +-
 7 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/Documentation/power/suspend-and-cpuhotplug.rst b/Documentation/power/suspend-and-cpuhotplug.rst
index 79f7610a6f6f..ebedb6c75db9 100644
--- a/Documentation/power/suspend-and-cpuhotplug.rst
+++ b/Documentation/power/suspend-and-cpuhotplug.rst
@@ -93,7 +93,7 @@ More details follow::
 Resuming back is likewise, with the counterparts being (in the order of
 execution during resume):
 
-* enable_nonboot_cpus() which involves::
+* thaw_secondary_cpus() which involves::
 
    |  Acquire cpu_add_remove_lock
    |  Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 69881b2d446c..9c31685af55a 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1372,12 +1372,12 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus)
 	speculative_store_bypass_ht_init();
 }
 
-void arch_enable_nonboot_cpus_begin(void)
+void arch_thaw_secondary_cpus_begin(void)
 {
 	set_mtrr_aps_delayed_init();
 }
 
-void arch_enable_nonboot_cpus_end(void)
+void arch_thaw_secondary_cpus_end(void)
 {
 	mtrr_aps_init();
 }
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index a00941c7828b..52692587f7fe 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -145,7 +145,7 @@ static inline void put_online_cpus(void) { cpus_read_unlock(); }
 
 #ifdef CONFIG_PM_SLEEP_SMP
 extern int freeze_secondary_cpus(int primary);
-extern void enable_nonboot_cpus(void);
+extern void thaw_secondary_cpus(void);
 
 static inline int suspend_disable_secondary_cpus(void)
 {
@@ -158,11 +158,11 @@ static inline int suspend_disable_secondary_cpus(void)
 }
 static inline void suspend_enable_secondary_cpus(void)
 {
-	return enable_nonboot_cpus();
+	return thaw_secondary_cpus();
 }
 
 #else /* !CONFIG_PM_SLEEP_SMP */
-static inline void enable_nonboot_cpus(void) {}
+static inline void thaw_secondary_cpus(void) {}
 static inline int suspend_disable_secondary_cpus(void) { return 0; }
 static inline void suspend_enable_secondary_cpus(void) { }
 #endif /* !CONFIG_PM_SLEEP_SMP */
diff --git a/include/linux/smp.h b/include/linux/smp.h
index cbc9162689d0..04019872c7bc 100644
--- a/include/linux/smp.h
+++ b/include/linux/smp.h
@@ -227,8 +227,8 @@ static inline int get_boot_cpu_id(void)
  */
 extern void arch_disable_smp_support(void);
 
-extern void arch_enable_nonboot_cpus_begin(void);
-extern void arch_enable_nonboot_cpus_end(void);
+extern void arch_thaw_secondary_cpus_begin(void);
+extern void arch_thaw_secondary_cpus_end(void);
 
 void smp_setup_processor_id(void);
 
diff --git a/kernel/cpu.c b/kernel/cpu.c
index d6d2c829561e..8d57c3cfe862 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1377,7 +1377,7 @@ int freeze_secondary_cpus(int primary)
 	/*
 	 * Make sure the CPUs won't be enabled by someone else. We need to do
 	 * this even in case of failure as all freeze_secondary_cpus() users are
-	 * supposed to do enable_nonboot_cpus() on the failure path.
+	 * supposed to do thaw_secondary_cpus() on the failure path.
 	 */
 	cpu_hotplug_disabled++;
 
@@ -1385,15 +1385,15 @@ int freeze_secondary_cpus(int primary)
 	return error;
 }
 
-void __weak arch_enable_nonboot_cpus_begin(void)
+void __weak arch_thaw_secondary_cpus_begin(void)
 {
 }
 
-void __weak arch_enable_nonboot_cpus_end(void)
+void __weak arch_thaw_secondary_cpus_end(void)
 {
 }
 
-void enable_nonboot_cpus(void)
+void thaw_secondary_cpus(void)
 {
 	int cpu, error;
 
@@ -1405,7 +1405,7 @@ void enable_nonboot_cpus(void)
 
 	pr_info("Enabling non-boot CPUs ...\n");
 
-	arch_enable_nonboot_cpus_begin();
+	arch_thaw_secondary_cpus_begin();
 
 	for_each_cpu(cpu, frozen_cpus) {
 		trace_suspend_resume(TPS("CPU_ON"), cpu, true);
@@ -1418,7 +1418,7 @@ void enable_nonboot_cpus(void)
 		pr_warn("Error taking CPU%d up: %d\n", cpu, error);
 	}
 
-	arch_enable_nonboot_cpus_end();
+	arch_thaw_secondary_cpus_end();
 
 	cpumask_clear(frozen_cpus);
 out:
diff --git a/tools/power/pm-graph/config/custom-timeline-functions.cfg b/tools/power/pm-graph/config/custom-timeline-functions.cfg
index 4f80ad7d7275..962e5768681c 100644
--- a/tools/power/pm-graph/config/custom-timeline-functions.cfg
+++ b/tools/power/pm-graph/config/custom-timeline-functions.cfg
@@ -125,7 +125,7 @@ acpi_suspend_begin:
 suspend_console:
 acpi_pm_prepare:
 syscore_suspend:
-arch_enable_nonboot_cpus_end:
+arch_thaw_secondary_cpus_end:
 syscore_resume:
 acpi_pm_finish:
 resume_console:
diff --git a/tools/power/pm-graph/sleepgraph.py b/tools/power/pm-graph/sleepgraph.py
index f7d1c1f62f86..530a9f681410 100755
--- a/tools/power/pm-graph/sleepgraph.py
+++ b/tools/power/pm-graph/sleepgraph.py
@@ -192,7 +192,7 @@ class SystemValues:
 		'suspend_console': {},
 		'acpi_pm_prepare': {},
 		'syscore_suspend': {},
-		'arch_enable_nonboot_cpus_end': {},
+		'arch_thaw_secondary_cpus_end': {},
 		'syscore_resume': {},
 		'acpi_pm_finish': {},
 		'resume_console': {},
-- 
2.17.1

Re: [PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[Rafael J. Wysocki]

On Thursday, April 9, 2020 1:27:40 PM CEST Qais Yousef wrote:
> This issue was fixed already by:
> 
> commit d66b16f5df4b ("arm64: Don't use disable_nonboot_cpus()")
> commit dddf3578e0d4 ("ARM: Don't use disable_nonboot_cpus()")
> 
> The only caller of disable_nonboot_cpus() is x86, which is in a proper
> suspend/resume path and due to the reverted patch lost its ability to
> early abort due to a pending wakeup.
> 
> The fix that is being reverted is arguably a better one to backport to
> stable trees. But it highlights the confusion about using
> disable_nonboot_cpus() API.
> 
> This is a preparation to remove disable_nonboot_cpus() in favor of
> freeze_secondary_cpus().
> 
> This reverts commit e98eac6ff1b45e4e73f2e6031b37c256ccb5d36b.
> 
> Signed-off-by: Qais Yousef <qais.yousef@arm.com>
> CC: "Rafael J. Wysocki" <rjw@rjwysocki.net>
> CC: Len Brown <len.brown@intel.com>
> CC: Pavel Machek <pavel@ucw.cz>
> CC: Ingo Molnar <mingo@redhat.com>
> CC: Borislav Petkov <bp@alien8.de>
> CC: "H. Peter Anvin" <hpa@zytor.com>
> CC: x86@kernel.org
> CC: Todd E Brandt <todd.e.brandt@linux.intel.com>
> CC: linux-pm@vger.kernel.org
> CC: linux-kernel@vger.kernel.org
> ---
>  include/linux/cpu.h | 12 +++---------
>  kernel/cpu.c        |  4 ++--
>  2 files changed, 5 insertions(+), 11 deletions(-)
> 
> diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> index beaed2dc269e..9ead281157d3 100644
> --- a/include/linux/cpu.h
> +++ b/include/linux/cpu.h
> @@ -144,18 +144,12 @@ static inline void get_online_cpus(void) { cpus_read_lock(); }
>  static inline void put_online_cpus(void) { cpus_read_unlock(); }
>  
>  #ifdef CONFIG_PM_SLEEP_SMP
> -int __freeze_secondary_cpus(int primary, bool suspend);
> -static inline int freeze_secondary_cpus(int primary)
> -{
> -	return __freeze_secondary_cpus(primary, true);
> -}
> -
> +extern int freeze_secondary_cpus(int primary);
>  static inline int disable_nonboot_cpus(void)
>  {
> -	return __freeze_secondary_cpus(0, false);
> +	return freeze_secondary_cpus(0);
>  }
> -
> -void enable_nonboot_cpus(void);
> +extern void enable_nonboot_cpus(void);
>  
>  static inline int suspend_disable_secondary_cpus(void)
>  {
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 12ae636e9cb6..30848496cbc7 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -1327,7 +1327,7 @@ void bringup_nonboot_cpus(unsigned int setup_max_cpus)
>  #ifdef CONFIG_PM_SLEEP_SMP
>  static cpumask_var_t frozen_cpus;
>  
> -int __freeze_secondary_cpus(int primary, bool suspend)
> +int freeze_secondary_cpus(int primary)
>  {
>  	int cpu, error = 0;
>  
> @@ -1352,7 +1352,7 @@ int __freeze_secondary_cpus(int primary, bool suspend)
>  		if (cpu == primary)
>  			continue;
>  
> -		if (suspend && pm_wakeup_pending()) {
> +		if (pm_wakeup_pending()) {
>  			pr_info("Wakeup pending. Abort CPU freeze\n");
>  			error = -EBUSY;
>  			break;
> 

I would do this the other way around:

1. Make x86 call freeze_secondary_cpus() directly, rename
   enable_nonboot_cpus() and drop disable_nonboot_cpus().
2. Get rid of __freeze_secondary_cpus().

Re: [PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[Qais Yousef]

On 04/26/20 17:24, Rafael J. Wysocki wrote:
> I would do this the other way around:
> 
> 1. Make x86 call freeze_secondary_cpus() directly, rename
>    enable_nonboot_cpus() and drop disable_nonboot_cpus().

All of this in a single patch?

> 2. Get rid of __freeze_secondary_cpus().

I guess you're implying to drop the revert too and manually unroll it instead.

Could do.

Thanks

--
Qais Yousef

Re: [PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[Rafael J. Wysocki]

On Mon, Apr 27, 2020 at 12:29 PM Qais Yousef <qais.yousef@arm.com> wrote:
>
> On 04/26/20 17:24, Rafael J. Wysocki wrote:
> > I would do this the other way around:
> >
> > 1. Make x86 call freeze_secondary_cpus() directly, rename
> >    enable_nonboot_cpus() and drop disable_nonboot_cpus().
>
> All of this in a single patch?

Well, why not?

Calling freeze_secondary_cpus() directly causes disable_nonboot_cpus()
to be unused (and so it can be dropped in the same patch) and it also
introduces a name mismatch between freeze_ and enable_, which IMO
needs to be addressed right away (also in the same patch).

> > 2. Get rid of __freeze_secondary_cpus().
>
> I guess you're implying to drop the revert too and manually unroll it instead.

IMO the revert is just an extra step with no real value, so why do it?

> Could do.

Thanks!

Re: [PATCH 1/3] Revert "cpu/hotplug: Ignore pm_wakeup_pending() for disable_nonboot_cpus()"

[Qais Yousef]

On 04/29/20 12:40, Rafael J. Wysocki wrote:
> On Mon, Apr 27, 2020 at 12:29 PM Qais Yousef <qais.yousef@arm.com> wrote:
> >
> > On 04/26/20 17:24, Rafael J. Wysocki wrote:
> > > I would do this the other way around:
> > >
> > > 1. Make x86 call freeze_secondary_cpus() directly, rename
> > >    enable_nonboot_cpus() and drop disable_nonboot_cpus().
> >
> > All of this in a single patch?
> 
> Well, why not?

I don't mind, was just clarifying. Usually it's requested to split patches :)

> 
> Calling freeze_secondary_cpus() directly causes disable_nonboot_cpus()
> to be unused (and so it can be dropped in the same patch) and it also
> introduces a name mismatch between freeze_ and enable_, which IMO
> needs to be addressed right away (also in the same patch).
> 
> > > 2. Get rid of __freeze_secondary_cpus().
> >
> > I guess you're implying to drop the revert too and manually unroll it instead.
> 
> IMO the revert is just an extra step with no real value, so why do it?

Works for me. Will send v2 ASAP.

Thanks

--
Qais Yousef