[PATCH v5 0/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[PATCH v5 0/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[James Morse]

Hi all,

These patches allow arm64 to hibernate on any CPU saving the cpu-id in the
arch header, then switch to it during resume using
hibernate_resume_nonboot_cpu_disable().

I hoped to avoid patch 1 by duplicating the logic in the arch code, but
Lorenzo pointed out using cpu_down() indicates that tasks are not frozen,
whereas in reality they are. [0]
Patch 1 lets us specify which cpu disable_nonboot_cpus() should leave
standing.

(All three patches need to be merged together, this series doesn't conflict
 with [1].)

Comments welcome,

This series is based on v4.8-rc2 and can be retrieved from:
git://linux-arm.org/linux-jm.git hibernate/cpuN/v5

Changes since v4:
 * Added freeze_secondary_cpus().
 * Wired up hibernate_resume_nonboot_cpu_disable(), removing macros and
   kconfig symbols from previous approaches.
 * Added check for sleep_cpu being uninitialised when we come to save the
   arch header.

Changes since v3:
 * Split series, at the PM/Hibernate patch, merged arm64 patches to remove
   dependencies.
 * Changed Kconfig symbol name.
 * Fixed logic error '<= 0' when testing for an unrecognised CPU.

Changes since v2:
 * Split wrong-CPU logic into an earlier patch that just returns an error.
 * Changed core code patch to use macros instead of a weak function.
   CONFIG_ARCH_HIBERNATION_HEADER now implies an asm/suspend.h header.
 * Wording in error messages 'hibernate' not 'suspend'.

Changes since v1:
 * Fixed 'Brining' typo.



[v1] http://www.spinics.net/lists/arm-kernel/msg507805.html
[v2] https://www.spinics.net/lists/arm-kernel/msg511654.html
[v3] https://www.spinics.net/lists/arm-kernel/msg514644.html
[v4] https://www.spinics.net/lists/arm-kernel/msg515978.html

[0] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-July/441305.html
[1] https://www.spinics.net/lists/arm-kernel/msg523933.html
[2] hotplug cpu0, kexec, hibernate, resume
-------------------------%<-------------------------
root@ubuntu:~# echo disk > /sys/power/state
[   76.768682] PM: Syncing filesystems ... done.
[   76.774872] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   76.783253] Double checking all user space processes after OOM killer disable
... (elapsed 0.000 seconds)
[   76.794214] PM: Preallocating image memory... done (allocated 84746 pages)
[   80.454878] PM: Allocated 338984 kbytes in 3.65 seconds (92.87 MB/s)
[   80.461269] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   80.491476] PM: freeze of devices complete after 21.037 msecs
[   80.498053] PM: late freeze of devices complete after 0.789 msecs
[   80.505563] PM: noirq freeze of devices complete after 1.387 msecs
[   80.511777] Disabling non-boot CPUs ...
[   80.540044] CPU1: shutdown
[   80.542755] psci: CPU1 killed.
[   80.611931] CPU2: shutdown
[   80.614646] psci: CPU2 killed.
[   80.687816] IRQ11 no longer affine to CPU3
[   80.687831] IRQ20 no longer affine to CPU3
[   80.687872] CPU3: shutdown
[   80.698806] psci: CPU3 killed.
[   80.725600] PM: Creating hibernation image:
[   80.725600] PM: Need to copy 84054 pages
[   80.725600] PM: Hibernation image created (84054 pages copied)
[   80.725609] Enabling non-boot CPUs ...
[   80.758320] Detected VIPT I-cache on CPU1
[   80.758381] CPU1: Booted secondary processor [410fd033]
[   80.758946] CPU1 is up
[   80.810889] Detected VIPT I-cache on CPU2
[   80.810932] CPU2: Booted secondary processor [410fd033]
[   80.811598] CPU2 is up
[   80.863596] Detected VIPT I-cache on CPU3
[   80.863640] CPU3: Booted secondary processor [410fd033]
[   80.864256] CPU3 is up
[   80.876712] PM: noirq thaw of devices complete after 0.802 msecs
[   80.883475] PM: early thaw of devices complete after 0.699 msecs
[   80.947177] PM: thaw of devices complete after 57.665 msecs
[   80.953222] hibernate: Hibernating on CPU 0 [mpidr:0x101]
[   80.961099] PM: Using 3 thread(s) for compression.
[   80.961099] PM: Compressing and saving image data (84219 pages)...
[   80.972154] PM: Image saving progress:   0%
[   81.270381] PM: Image saving progress:  10%
[   81.498242] PM: Image saving progress:  20%
[   81.789014] PM: Image saving progress:  30%
[   84.844984] PM: Image saving progress:  40%
[   87.657013] PM: Image saving progress:  50%
[   88.933987] PM: Image saving progress:  60%
[   90.597472] PM: Image saving progress:  70%
[   92.362916] PM: Image saving progress:  80%
[   93.622648] PM: Image saving progress:  90%
[   95.468855] PM: Image saving progress: 100%
[   99.073785] PM: Image saving done.
[   99.077240] PM: Wrote 336876 kbytes in 18.10 seconds (18.61 MB/s)
[   99.092099] PM: S|
[   99.370393] kvm: exiting hardware virtualization
[   99.535936] reboot: Restarting system

[ ... ]

[    6.074235] Freezing user space processes ... (elapsed 0.000 seconds) done.
[    6.081540] Double checking all user space processes after OOM killer disable
... (elapsed 0.000 seconds)
[    6.119415] random: fast init done
[    6.133638] PM: Using 1 thread(s) for decompression.
[    6.133638] PM: Loading and decompressing image data (84219 pages)...
[    6.145079] hibernate: Hibernated on CPU 1 [mpidr:0x101]
[    6.150412] hibernate: Hibernated on a CPU that is offline! Bringing CPU up.
[    6.198971] Detected VIPT I-cache on CPU1
[    6.199036] CPU1: Booted secondary processor [410fd033]
[    7.177507] PM: Image loading progress:   0%
[    7.816126] PM: Image loading progress:  10%
[    8.176005] PM: Image loading progress:  20%
[    8.568329] PM: Image loading progress:  30%
[    8.991993] PM: Image loading progress:  40%
[    9.385313] PM: Image loading progress:  50%
[    9.708479] PM: Image loading progress:  60%
[   10.082700] PM: Image loading progress:  70%
[   10.478748] PM: Image loading progress:  80%
[   10.876730] PM: Image loading progress:  90%
[   11.275249] PM: Image loading progress: 100%
[   11.280021] PM: Image loading done.
[   11.283534] PM: Read 336876 kbytes in 5.13 seconds (65.66 MB/s)
[   11.316174] PM: quiesce of devices complete after 19.385 msecs
[   11.322864] PM: late quiesce of devices complete after 0.825 msecs
[   11.349479] PM: noirq quiesce of devices complete after 20.401 msecs
[   11.355864] Disabling non-boot CPUs ...
[   11.372531] IRQ7 no longer affine to CPU0
[   11.372537] IRQ9 no longer affine to CPU0
[   11.372544] IRQ11 no longer affine to CPU0
[   11.372559] IRQ20 no longer affine to CPU0
[   11.372595] CPU0: shutdown
[   11.391601] psci: CPU0 killed.
[   80.725613] Enabling non-boot CPUs ...
[   80.758314] Detected VIPT I-cache on CPU1
[   80.758375] CPU1: Booted secondary processor [410fd033]
[   80.758939] CPU1 is up
[   80.810877] Detected VIPT I-cache on CPU2
[   80.810917] CPU2: Booted secondary processor [410fd033]
[   80.811581] CPU2 is up
[   80.859588] Detected VIPT I-cache on CPU3
[   80.859628] CPU3: Booted secondary processor [410fd033]
[   80.860246] CPU3 is up
[   80.873337] PM: noirq restore of devices complete after 1.434 msecs
[   80.880381] PM: early restore of devices complete after 0.701 msecs
[   81.231471] PM: restore of devices complete after 71.157 msecs
[   81.238336] Restarting tasks ... done.
root@ubuntu:~#
-------------------------%<-------------------------

James Morse (3):
  cpu/hotplug: Allow suspend/resume CPU to be specified
  arm64: hibernate: Resume when hibernate image created on non-boot CPU
  Revert "arm64: hibernate: Refuse to hibernate if the boot cpu is
    offline"

 arch/arm64/include/asm/suspend.h |  3 ++
 arch/arm64/kernel/hibernate.c    | 67 +++++++++++++++++++++++++++-------------
 include/linux/cpu.h              |  6 +++-
 kernel/cpu.c                     |  9 +++---
 4 files changed, 59 insertions(+), 26 deletions(-)

-- 
2.8.0.rc3

[PATCH v5 1/3] cpu/hotplug: Allow suspend/resume CPU to be specified

[James Morse]

disable_nonboot_cpus() assumes that the lowest numbered online CPU is
the boot CPU, and that this is the correct CPU to run any power
management code on.

On x86 this is always correct, as CPU0 cannot (easily) by taken offline.

On arm64 CPU0 can be taken offline. For hibernate/resume this means we
may hibernate on a CPU other than CPU0. If the system is rebooted with
kexec 'CPU0' will be assigned to a different physical CPU. This
complicates hibernate/resume as now we can't trust the CPU numbers.
Arch code can find the correct physical CPU, and ensure it is online
before resume from hibernate begins, but also needs to influence
disable_nonboot_cpus()s choice of CPU.

Rename disable_nonboot_cpus() as freeze_secondary_cpus() and add an
argument indicating which CPU should be left standing. Follow the logic
in migrate_to_reboot_cpu() to use the lowest numbered online CPU if the
requested CPU is not online.
Add disable_nonboot_cpus() as an inline function that has the existing
behaviour.

Signed-off-by: James Morse <james.morse@arm.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
---
An alternative is to provide two functions calling a common function,
but this would mean spilling the cpu_maps_update_begin() into these two.

 include/linux/cpu.h | 6 +++++-
 kernel/cpu.c        | 9 +++++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 797d9c8e9a1b..ad4f1f33a74e 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -228,7 +228,11 @@ static inline void cpu_hotplug_done(void) {}
 #endif		/* CONFIG_HOTPLUG_CPU */
 
 #ifdef CONFIG_PM_SLEEP_SMP
-extern int disable_nonboot_cpus(void);
+extern int freeze_secondary_cpus(int primary);
+static inline int disable_nonboot_cpus(void)
+{
+	return freeze_secondary_cpus(0);
+}
 extern void enable_nonboot_cpus(void);
 #else /* !CONFIG_PM_SLEEP_SMP */
 static inline int disable_nonboot_cpus(void) { return 0; }
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 341bf80f80bd..ebbf027dd4a1 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1024,12 +1024,13 @@ EXPORT_SYMBOL_GPL(cpu_up);
 #ifdef CONFIG_PM_SLEEP_SMP
 static cpumask_var_t frozen_cpus;
 
-int disable_nonboot_cpus(void)
+int freeze_secondary_cpus(int primary)
 {
-	int cpu, first_cpu, error = 0;
+	int cpu, error = 0;
 
 	cpu_maps_update_begin();
-	first_cpu = cpumask_first(cpu_online_mask);
+	if (!cpu_online(primary))
+		primary = cpumask_first(cpu_online_mask);
 	/*
 	 * We take down all of the non-boot CPUs in one shot to avoid races
 	 * with the userspace trying to use the CPU hotplug at the same time
@@ -1038,7 +1039,7 @@ int disable_nonboot_cpus(void)
 
 	pr_info("Disabling non-boot CPUs ...\n");
 	for_each_online_cpu(cpu) {
-		if (cpu == first_cpu)
+		if (cpu == primary)
 			continue;
 		trace_suspend_resume(TPS("CPU_OFF"), cpu, true);
 		error = _cpu_down(cpu, 1, CPUHP_OFFLINE);
-- 
2.8.0.rc3

[PATCH v5 2/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[James Morse]

disable_nonboot_cpus() assumes that the lowest numbered online CPU is
the boot CPU, and that this is the correct CPU to run any power
management code on.

On arm64 CPU0 can be taken offline. For hibernate/resume this means we
may hibernate on a CPU other than CPU0. If the system is rebooted with
kexec 'CPU0' will be assigned to a different CPU. This complicates
hibernate/resume as now we can't trust the CPU numbers.

We currently forbid hibernate if CPU0 has been hotplugged out to avoid
this situation without kexec.

Save the MPIDR of the CPU we hibernated on in the hibernate arch-header,
use hibernate_resume_nonboot_cpu_disable() to direct which CPU we should
resume on based on the MPIDR of the CPU we hibernated on. This allows us to
hibernate/resume on any CPU, even if the logical numbers have been
shuffled by kexec.

Signed-off-by: James Morse <james.morse@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
---
This patch should be merged via the same tree as patch 1.

 arch/arm64/include/asm/suspend.h |  3 +++
 arch/arm64/kernel/hibernate.c    | 51 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/arch/arm64/include/asm/suspend.h b/arch/arm64/include/asm/suspend.h
index 024d623f662e..b8a313fd7a09 100644
--- a/arch/arm64/include/asm/suspend.h
+++ b/arch/arm64/include/asm/suspend.h
@@ -47,4 +47,7 @@ int swsusp_arch_resume(void);
 int arch_hibernation_header_save(void *addr, unsigned int max_size);
 int arch_hibernation_header_restore(void *addr);
 
+/* Used to resume on the CPU we hibernated on */
+int hibernate_resume_nonboot_cpu_disable(void);
+
 #endif
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 65d81f965e74..9fcf2fcda876 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -15,6 +15,7 @@
  * License terms: GNU General Public License (GPL) version 2
  */
 #define pr_fmt(x) "hibernate: " x
+#include <linux/cpu.h>
 #include <linux/kvm_host.h>
 #include <linux/mm.h>
 #include <linux/notifier.h>
@@ -26,6 +27,7 @@
 
 #include <asm/barrier.h>
 #include <asm/cacheflush.h>
+#include <asm/cputype.h>
 #include <asm/irqflags.h>
 #include <asm/memory.h>
 #include <asm/mmu_context.h>
@@ -34,6 +36,7 @@
 #include <asm/pgtable-hwdef.h>
 #include <asm/sections.h>
 #include <asm/smp.h>
+#include <asm/smp_plat.h>
 #include <asm/suspend.h>
 #include <asm/sysreg.h>
 #include <asm/virt.h>
@@ -67,6 +70,12 @@ extern char hibernate_el2_vectors[];
 extern char __hyp_stub_vectors[];
 
 /*
+ * The logical cpu number we should resume on, initialised to a non-cpu
+ * number.
+ */
+static int sleep_cpu = -EINVAL;
+
+/*
  * Values that may not change over hibernate/resume. We put the build number
  * and date in here so that we guarantee not to resume with a different
  * kernel.
@@ -88,6 +97,8 @@ static struct arch_hibernate_hdr {
 	 * re-configure el2.
 	 */
 	phys_addr_t	__hyp_stub_vectors;
+
+	u64		sleep_cpu_mpidr;
 } resume_hdr;
 
 static inline void arch_hdr_invariants(struct arch_hibernate_hdr_invariants *i)
@@ -130,12 +141,22 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
 	else
 		hdr->__hyp_stub_vectors = 0;
 
+	/* Save the mpidr of the cpu we called cpu_suspend() on... */
+	if (sleep_cpu < 0) {
+		pr_err("Failing to hibernate on an unkown CPU.\n");
+		return -ENODEV;
+	}
+	hdr->sleep_cpu_mpidr = cpu_logical_map(sleep_cpu);
+	pr_info("Hibernating on CPU %d [mpidr:0x%llx]\n", sleep_cpu,
+		hdr->sleep_cpu_mpidr);
+
 	return 0;
 }
 EXPORT_SYMBOL(arch_hibernation_header_save);
 
 int arch_hibernation_header_restore(void *addr)
 {
+	int ret;
 	struct arch_hibernate_hdr_invariants invariants;
 	struct arch_hibernate_hdr *hdr = addr;
 
@@ -145,6 +166,24 @@ int arch_hibernation_header_restore(void *addr)
 		return -EINVAL;
 	}
 
+	sleep_cpu = get_logical_index(hdr->sleep_cpu_mpidr);
+	pr_info("Hibernated on CPU %d [mpidr:0x%llx]\n", sleep_cpu,
+		hdr->sleep_cpu_mpidr);
+	if (sleep_cpu < 0) {
+		pr_crit("Hibernated on a CPU not known to this kernel!\n");
+		sleep_cpu = -EINVAL;
+		return -EINVAL;
+	}
+	if (!cpu_online(sleep_cpu)) {
+		pr_info("Hibernated on a CPU that is offline! Bringing CPU up.\n");
+		ret = cpu_up(sleep_cpu);
+		if (ret) {
+			pr_err("Failed to bring hibernate-CPU up!\n");
+			sleep_cpu = -EINVAL;
+			return ret;
+		}
+	}
+
 	resume_hdr = *hdr;
 
 	return 0;
@@ -256,6 +295,7 @@ int swsusp_arch_suspend(void)
 	local_dbg_save(flags);
 
 	if (__cpu_suspend_enter(&state)) {
+		sleep_cpu = smp_processor_id();
 		ret = swsusp_save();
 	} else {
 		/* Clean kernel to PoC for secondary core startup */
@@ -267,6 +307,7 @@ int swsusp_arch_suspend(void)
 		 */
 		in_suspend = 0;
 
+		sleep_cpu = -EINVAL;
 		__cpu_suspend_exit();
 	}
 
@@ -507,3 +548,13 @@ static int __init check_boot_cpu_online_init(void)
 	return 0;
 }
 core_initcall(check_boot_cpu_online_init);
+
+int hibernate_resume_nonboot_cpu_disable(void)
+{
+	if (sleep_cpu < 0) {
+		pr_err("Failing to resume from hibernate on an unkown CPU.\n");
+		return -ENODEV;
+	}
+
+	return freeze_secondary_cpus(sleep_cpu);
+}
-- 
2.8.0.rc3

[PATCH v5 3/3] Revert "arm64: hibernate: Refuse to hibernate if the boot cpu is offline"

[James Morse]

Now that we use the MPIDR to resume on the same CPU that we hibernated on,
we no longer need to refuse to hibernate if the boot cpu is offline. (Which
we can't possibly know if kexec causes logical CPUs to be renumbered).

This reverts commit 1fe492ce6482b77807b25d29690a48c46456beee.

Signed-off-by: James Morse <james.morse@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
---
This patch should be merged via the same tree as patch 1.

 arch/arm64/kernel/hibernate.c | 26 --------------------------
 1 file changed, 26 deletions(-)

diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9fcf2fcda876..2dcd7d29f568 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -18,7 +18,6 @@
 #include <linux/cpu.h>
 #include <linux/kvm_host.h>
 #include <linux/mm.h>
-#include <linux/notifier.h>
 #include <linux/pm.h>
 #include <linux/sched.h>
 #include <linux/suspend.h>
@@ -524,31 +523,6 @@ out:
 	return rc;
 }
 
-static int check_boot_cpu_online_pm_callback(struct notifier_block *nb,
-					     unsigned long action, void *ptr)
-{
-	if (action == PM_HIBERNATION_PREPARE &&
-	     cpumask_first(cpu_online_mask) != 0) {
-		pr_warn("CPU0 is offline.\n");
-		return notifier_from_errno(-ENODEV);
-	}
-
-	return NOTIFY_OK;
-}
-
-static int __init check_boot_cpu_online_init(void)
-{
-	/*
-	 * Set this pm_notifier callback with a lower priority than
-	 * cpu_hotplug_pm_callback, so that cpu_hotplug_pm_callback will be
-	 * called earlier to disable cpu hotplug before the cpu online check.
-	 */
-	pm_notifier(check_boot_cpu_online_pm_callback, -INT_MAX);
-
-	return 0;
-}
-core_initcall(check_boot_cpu_online_init);
-
 int hibernate_resume_nonboot_cpu_disable(void)
 {
 	if (sleep_cpu < 0) {
-- 
2.8.0.rc3

Re: [PATCH v5 1/3] cpu/hotplug: Allow suspend/resume CPU to be specified

[Will Deacon]

On Wed, Aug 17, 2016 at 01:50:25PM +0100, James Morse wrote:
> disable_nonboot_cpus() assumes that the lowest numbered online CPU is
> the boot CPU, and that this is the correct CPU to run any power
> management code on.
> 
> On x86 this is always correct, as CPU0 cannot (easily) by taken offline.
> 
> On arm64 CPU0 can be taken offline. For hibernate/resume this means we
> may hibernate on a CPU other than CPU0. If the system is rebooted with
> kexec 'CPU0' will be assigned to a different physical CPU. This
> complicates hibernate/resume as now we can't trust the CPU numbers.
> Arch code can find the correct physical CPU, and ensure it is online
> before resume from hibernate begins, but also needs to influence
> disable_nonboot_cpus()s choice of CPU.
> 
> Rename disable_nonboot_cpus() as freeze_secondary_cpus() and add an
> argument indicating which CPU should be left standing. Follow the logic
> in migrate_to_reboot_cpu() to use the lowest numbered online CPU if the
> requested CPU is not online.
> Add disable_nonboot_cpus() as an inline function that has the existing
> behaviour.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> ---
> An alternative is to provide two functions calling a common function,
> but this would mean spilling the cpu_maps_update_begin() into these two.
> 
>  include/linux/cpu.h | 6 +++++-
>  kernel/cpu.c        | 9 +++++----
>  2 files changed, 10 insertions(+), 5 deletions(-)

Thomas, does this look ok to you? If so, would you prefer to merge this
series via -tip, or have us take this one via the arm64 tree?

Thanks,

Will

> diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> index 797d9c8e9a1b..ad4f1f33a74e 100644
> --- a/include/linux/cpu.h
> +++ b/include/linux/cpu.h
> @@ -228,7 +228,11 @@ static inline void cpu_hotplug_done(void) {}
>  #endif		/* CONFIG_HOTPLUG_CPU */
>  
>  #ifdef CONFIG_PM_SLEEP_SMP
> -extern int disable_nonboot_cpus(void);
> +extern int freeze_secondary_cpus(int primary);
> +static inline int disable_nonboot_cpus(void)
> +{
> +	return freeze_secondary_cpus(0);
> +}
>  extern void enable_nonboot_cpus(void);
>  #else /* !CONFIG_PM_SLEEP_SMP */
>  static inline int disable_nonboot_cpus(void) { return 0; }
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 341bf80f80bd..ebbf027dd4a1 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -1024,12 +1024,13 @@ EXPORT_SYMBOL_GPL(cpu_up);
>  #ifdef CONFIG_PM_SLEEP_SMP
>  static cpumask_var_t frozen_cpus;
>  
> -int disable_nonboot_cpus(void)
> +int freeze_secondary_cpus(int primary)
>  {
> -	int cpu, first_cpu, error = 0;
> +	int cpu, error = 0;
>  
>  	cpu_maps_update_begin();
> -	first_cpu = cpumask_first(cpu_online_mask);
> +	if (!cpu_online(primary))
> +		primary = cpumask_first(cpu_online_mask);
>  	/*
>  	 * We take down all of the non-boot CPUs in one shot to avoid races
>  	 * with the userspace trying to use the CPU hotplug at the same time
> @@ -1038,7 +1039,7 @@ int disable_nonboot_cpus(void)
>  
>  	pr_info("Disabling non-boot CPUs ...\n");
>  	for_each_online_cpu(cpu) {
> -		if (cpu == first_cpu)
> +		if (cpu == primary)
>  			continue;
>  		trace_suspend_resume(TPS("CPU_OFF"), cpu, true);
>  		error = _cpu_down(cpu, 1, CPUHP_OFFLINE);
> -- 
> 2.8.0.rc3
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 1/3] cpu/hotplug: Allow suspend/resume CPU to be specified

[Thomas Gleixner]

On Fri, 26 Aug 2016, Will Deacon wrote:
> On Wed, Aug 17, 2016 at 01:50:25PM +0100, James Morse wrote:
> > disable_nonboot_cpus() assumes that the lowest numbered online CPU is
> > the boot CPU, and that this is the correct CPU to run any power
> > management code on.
> > 
> > On x86 this is always correct, as CPU0 cannot (easily) by taken offline.
> > 
> > On arm64 CPU0 can be taken offline. For hibernate/resume this means we
> > may hibernate on a CPU other than CPU0. If the system is rebooted with
> > kexec 'CPU0' will be assigned to a different physical CPU. This
> > complicates hibernate/resume as now we can't trust the CPU numbers.
> > Arch code can find the correct physical CPU, and ensure it is online
> > before resume from hibernate begins, but also needs to influence
> > disable_nonboot_cpus()s choice of CPU.
> > 
> > Rename disable_nonboot_cpus() as freeze_secondary_cpus() and add an
> > argument indicating which CPU should be left standing. Follow the logic
> > in migrate_to_reboot_cpu() to use the lowest numbered online CPU if the
> > requested CPU is not online.
> > Add disable_nonboot_cpus() as an inline function that has the existing
> > behaviour.
> > 
> > Signed-off-by: James Morse <james.morse@arm.com>
> > Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> > ---
> > An alternative is to provide two functions calling a common function,
> > but this would mean spilling the cpu_maps_update_begin() into these two.
> > 
> >  include/linux/cpu.h | 6 +++++-
> >  kernel/cpu.c        | 9 +++++----
> >  2 files changed, 10 insertions(+), 5 deletions(-)
> 
> Thomas, does this look ok to you? If so, would you prefer to merge this
> series via -tip, or have us take this one via the arm64 tree?

You can take it via ARM64. It's not conflicting with the stuff I have in the
pipeline.

Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Re: [PATCH v5 1/3] cpu/hotplug: Allow suspend/resume CPU to be specified

[Will Deacon]

On Fri, Aug 26, 2016 at 12:09:38PM +0200, Thomas Gleixner wrote:
> On Fri, 26 Aug 2016, Will Deacon wrote:
> > On Wed, Aug 17, 2016 at 01:50:25PM +0100, James Morse wrote:
> > > disable_nonboot_cpus() assumes that the lowest numbered online CPU is
> > > the boot CPU, and that this is the correct CPU to run any power
> > > management code on.
> > > 
> > > On x86 this is always correct, as CPU0 cannot (easily) by taken offline.
> > > 
> > > On arm64 CPU0 can be taken offline. For hibernate/resume this means we
> > > may hibernate on a CPU other than CPU0. If the system is rebooted with
> > > kexec 'CPU0' will be assigned to a different physical CPU. This
> > > complicates hibernate/resume as now we can't trust the CPU numbers.
> > > Arch code can find the correct physical CPU, and ensure it is online
> > > before resume from hibernate begins, but also needs to influence
> > > disable_nonboot_cpus()s choice of CPU.
> > > 
> > > Rename disable_nonboot_cpus() as freeze_secondary_cpus() and add an
> > > argument indicating which CPU should be left standing. Follow the logic
> > > in migrate_to_reboot_cpu() to use the lowest numbered online CPU if the
> > > requested CPU is not online.
> > > Add disable_nonboot_cpus() as an inline function that has the existing
> > > behaviour.
> > > 
> > > Signed-off-by: James Morse <james.morse@arm.com>
> > > Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> > > ---
> > > An alternative is to provide two functions calling a common function,
> > > but this would mean spilling the cpu_maps_update_begin() into these two.
> > > 
> > >  include/linux/cpu.h | 6 +++++-
> > >  kernel/cpu.c        | 9 +++++----
> > >  2 files changed, 10 insertions(+), 5 deletions(-)
> > 
> > Thomas, does this look ok to you? If so, would you prefer to merge this
> > series via -tip, or have us take this one via the arm64 tree?
> 
> You can take it via ARM64. It's not conflicting with the stuff I have in the
> pipeline.
> 
> Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Brill, thanks Thomas.

Will

Re: [PATCH v5 0/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[Rafael J. Wysocki]

On Wednesday, August 17, 2016 01:50:24 PM James Morse wrote:
> Hi all,
> 
> These patches allow arm64 to hibernate on any CPU saving the cpu-id in the
> arch header, then switch to it during resume using
> hibernate_resume_nonboot_cpu_disable().
> 
> I hoped to avoid patch 1 by duplicating the logic in the arch code, but
> Lorenzo pointed out using cpu_down() indicates that tasks are not frozen,
> whereas in reality they are. [0]
> Patch 1 lets us specify which cpu disable_nonboot_cpus() should leave
> standing.
> 
> (All three patches need to be merged together, this series doesn't conflict
>  with [1].)
> 
> Comments welcome,
> 
> This series is based on v4.8-rc2 and can be retrieved from:
> git://linux-arm.org/linux-jm.git hibernate/cpuN/v5
> 
> Changes since v4:
>  * Added freeze_secondary_cpus().
>  * Wired up hibernate_resume_nonboot_cpu_disable(), removing macros and
>    kconfig symbols from previous approaches.
>  * Added check for sleep_cpu being uninitialised when we come to save the
>    arch header.
> 
> Changes since v3:
>  * Split series, at the PM/Hibernate patch, merged arm64 patches to remove
>    dependencies.
>  * Changed Kconfig symbol name.
>  * Fixed logic error '<= 0' when testing for an unrecognised CPU.
> 
> Changes since v2:
>  * Split wrong-CPU logic into an earlier patch that just returns an error.
>  * Changed core code patch to use macros instead of a weak function.
>    CONFIG_ARCH_HIBERNATION_HEADER now implies an asm/suspend.h header.
>  * Wording in error messages 'hibernate' not 'suspend'.
> 
> Changes since v1:
>  * Fixed 'Brining' typo.
> 
> 
> 
> [v1] http://www.spinics.net/lists/arm-kernel/msg507805.html
> [v2] https://www.spinics.net/lists/arm-kernel/msg511654.html
> [v3] https://www.spinics.net/lists/arm-kernel/msg514644.html
> [v4] https://www.spinics.net/lists/arm-kernel/msg515978.html
> 
> [0] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-July/441305.html
> [1] https://www.spinics.net/lists/arm-kernel/msg523933.html
> [2] hotplug cpu0, kexec, hibernate, resume
> -------------------------%<-------------------------
> root@ubuntu:~# echo disk > /sys/power/state
> [   76.768682] PM: Syncing filesystems ... done.
> [   76.774872] Freezing user space processes ... (elapsed 0.001 seconds) done.
> [   76.783253] Double checking all user space processes after OOM killer disable
> ... (elapsed 0.000 seconds)
> [   76.794214] PM: Preallocating image memory... done (allocated 84746 pages)
> [   80.454878] PM: Allocated 338984 kbytes in 3.65 seconds (92.87 MB/s)
> [   80.461269] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
> [   80.491476] PM: freeze of devices complete after 21.037 msecs
> [   80.498053] PM: late freeze of devices complete after 0.789 msecs
> [   80.505563] PM: noirq freeze of devices complete after 1.387 msecs
> [   80.511777] Disabling non-boot CPUs ...
> [   80.540044] CPU1: shutdown
> [   80.542755] psci: CPU1 killed.
> [   80.611931] CPU2: shutdown
> [   80.614646] psci: CPU2 killed.
> [   80.687816] IRQ11 no longer affine to CPU3
> [   80.687831] IRQ20 no longer affine to CPU3
> [   80.687872] CPU3: shutdown
> [   80.698806] psci: CPU3 killed.
> [   80.725600] PM: Creating hibernation image:
> [   80.725600] PM: Need to copy 84054 pages
> [   80.725600] PM: Hibernation image created (84054 pages copied)
> [   80.725609] Enabling non-boot CPUs ...
> [   80.758320] Detected VIPT I-cache on CPU1
> [   80.758381] CPU1: Booted secondary processor [410fd033]
> [   80.758946] CPU1 is up
> [   80.810889] Detected VIPT I-cache on CPU2
> [   80.810932] CPU2: Booted secondary processor [410fd033]
> [   80.811598] CPU2 is up
> [   80.863596] Detected VIPT I-cache on CPU3
> [   80.863640] CPU3: Booted secondary processor [410fd033]
> [   80.864256] CPU3 is up
> [   80.876712] PM: noirq thaw of devices complete after 0.802 msecs
> [   80.883475] PM: early thaw of devices complete after 0.699 msecs
> [   80.947177] PM: thaw of devices complete after 57.665 msecs
> [   80.953222] hibernate: Hibernating on CPU 0 [mpidr:0x101]
> [   80.961099] PM: Using 3 thread(s) for compression.
> [   80.961099] PM: Compressing and saving image data (84219 pages)...
> [   80.972154] PM: Image saving progress:   0%
> [   81.270381] PM: Image saving progress:  10%
> [   81.498242] PM: Image saving progress:  20%
> [   81.789014] PM: Image saving progress:  30%
> [   84.844984] PM: Image saving progress:  40%
> [   87.657013] PM: Image saving progress:  50%
> [   88.933987] PM: Image saving progress:  60%
> [   90.597472] PM: Image saving progress:  70%
> [   92.362916] PM: Image saving progress:  80%
> [   93.622648] PM: Image saving progress:  90%
> [   95.468855] PM: Image saving progress: 100%
> [   99.073785] PM: Image saving done.
> [   99.077240] PM: Wrote 336876 kbytes in 18.10 seconds (18.61 MB/s)
> [   99.092099] PM: S|
> [   99.370393] kvm: exiting hardware virtualization
> [   99.535936] reboot: Restarting system
> 
> [ ... ]
> 
> [    6.074235] Freezing user space processes ... (elapsed 0.000 seconds) done.
> [    6.081540] Double checking all user space processes after OOM killer disable
> ... (elapsed 0.000 seconds)
> [    6.119415] random: fast init done
> [    6.133638] PM: Using 1 thread(s) for decompression.
> [    6.133638] PM: Loading and decompressing image data (84219 pages)...
> [    6.145079] hibernate: Hibernated on CPU 1 [mpidr:0x101]
> [    6.150412] hibernate: Hibernated on a CPU that is offline! Bringing CPU up.
> [    6.198971] Detected VIPT I-cache on CPU1
> [    6.199036] CPU1: Booted secondary processor [410fd033]
> [    7.177507] PM: Image loading progress:   0%
> [    7.816126] PM: Image loading progress:  10%
> [    8.176005] PM: Image loading progress:  20%
> [    8.568329] PM: Image loading progress:  30%
> [    8.991993] PM: Image loading progress:  40%
> [    9.385313] PM: Image loading progress:  50%
> [    9.708479] PM: Image loading progress:  60%
> [   10.082700] PM: Image loading progress:  70%
> [   10.478748] PM: Image loading progress:  80%
> [   10.876730] PM: Image loading progress:  90%
> [   11.275249] PM: Image loading progress: 100%
> [   11.280021] PM: Image loading done.
> [   11.283534] PM: Read 336876 kbytes in 5.13 seconds (65.66 MB/s)
> [   11.316174] PM: quiesce of devices complete after 19.385 msecs
> [   11.322864] PM: late quiesce of devices complete after 0.825 msecs
> [   11.349479] PM: noirq quiesce of devices complete after 20.401 msecs
> [   11.355864] Disabling non-boot CPUs ...
> [   11.372531] IRQ7 no longer affine to CPU0
> [   11.372537] IRQ9 no longer affine to CPU0
> [   11.372544] IRQ11 no longer affine to CPU0
> [   11.372559] IRQ20 no longer affine to CPU0
> [   11.372595] CPU0: shutdown
> [   11.391601] psci: CPU0 killed.
> [   80.725613] Enabling non-boot CPUs ...
> [   80.758314] Detected VIPT I-cache on CPU1
> [   80.758375] CPU1: Booted secondary processor [410fd033]
> [   80.758939] CPU1 is up
> [   80.810877] Detected VIPT I-cache on CPU2
> [   80.810917] CPU2: Booted secondary processor [410fd033]
> [   80.811581] CPU2 is up
> [   80.859588] Detected VIPT I-cache on CPU3
> [   80.859628] CPU3: Booted secondary processor [410fd033]
> [   80.860246] CPU3 is up
> [   80.873337] PM: noirq restore of devices complete after 1.434 msecs
> [   80.880381] PM: early restore of devices complete after 0.701 msecs
> [   81.231471] PM: restore of devices complete after 71.157 msecs
> [   81.238336] Restarting tasks ... done.
> root@ubuntu:~#
> -------------------------%<-------------------------
> 
> James Morse (3):
>   cpu/hotplug: Allow suspend/resume CPU to be specified
>   arm64: hibernate: Resume when hibernate image created on non-boot CPU
>   Revert "arm64: hibernate: Refuse to hibernate if the boot cpu is
>     offline"
> 
>  arch/arm64/include/asm/suspend.h |  3 ++
>  arch/arm64/kernel/hibernate.c    | 67 +++++++++++++++++++++++++++-------------
>  include/linux/cpu.h              |  6 +++-
>  kernel/cpu.c                     |  9 +++---
>  4 files changed, 59 insertions(+), 26 deletions(-)

What's the status of this?

Thanks,
Rafael

Re: [PATCH v5 0/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[James Morse]

Hi Rafael,

On 14/09/16 02:07, Rafael J. Wysocki wrote:
> What's the status of this?

Will has queued it in his for-next/core branch.


Thanks,

James

Re: [PATCH v5 0/3] arm64: hibernate: Resume when hibernate image created on non-boot CPU

[Rafael J. Wysocki]

On Wednesday, September 14, 2016 09:09:26 AM James Morse wrote:
> Hi Rafael,
> 
> On 14/09/16 02:07, Rafael J. Wysocki wrote:
> > What's the status of this?
> 
> Will has queued it in his for-next/core branch.

OK, thanks!