[V4 PATCH 0/2] kexec: crash_kexec_post_notifiers boot option related fixes

[V4 PATCH 0/2] kexec: crash_kexec_post_notifiers boot option related fixes

[Hidehiro Kawai]

Daniel Walker reported problems which happens when
crash_kexec_post_notifiers kernel option is enabled
(https://lkml.org/lkml/2015/6/24/44).

In that case, smp_send_stop() is called before entering kdump routines
which assume other CPUs are still online.  This causes some issues
depending on architectures.  For example, for x86, kdump routines fail
to save other CPUs' registers and disable virtualization extensions.
For MIPS OCTEON, it fails to stop the watchdog timer.

To fix this problem, call a new kdump friendly function,
crash_smp_send_stop(), instead of the smp_send_stop() when
crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
weak function, and it just call smp_send_stop().  Architecture
codes should override it so that kdump can work appropriately.
This patch set supports only x86 and MIPS.

NOTE:
- Right solution would be to place crash_smp_send_stop() before
  __crash_kexec() invocation in all cases and remove smp_send_stop(),
  but we can't do that until all architectures implement own
  crash_smp_send_stop()
- crash_smp_send_stop()-like work is still needed by
  machine_crash_shutdown() because crash_kexec() can be called without
  entering panic()

Changes in V4:
- Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
- Rename panic_smp_send_stop to crash_smp_send_stop
- Don't change the behavior for Xen's PV kernel
- Support MIPS

Changes in V3: https://lkml.org/lkml/2016/7/5/221
- Revise comments, description, and symbol names (the logic doesn't
  change)
- Make crash_kexec_post_notifiers boot option modifiable after boot

Changes in V2: https://lkml.org/lkml/2015/7/23/864
- Replace smp_send_stop() call with crash_kexec version which
  saves cpu states and does cleanups instead of changing execution
  flow
- Drop a fix for Problem 1
- Drop other patches because they aren't needed anymore

V1: https://lkml.org/lkml/2015/7/10/316

---

Hidehiro Kawai (2):
      x86/panic: Replace smp_send_stop() with kdump friendly version in panic path
      mips/panic: Replace smp_send_stop() with kdump friendly version in panic path


 arch/mips/cavium-octeon/setup.c  |   14 +++++++++++
 arch/mips/include/asm/kexec.h    |    1 +
 arch/mips/kernel/crash.c         |   18 ++++++++++++++-
 arch/mips/kernel/machine_kexec.c |    1 +
 arch/x86/include/asm/kexec.h     |    1 +
 arch/x86/include/asm/smp.h       |    1 +
 arch/x86/kernel/crash.c          |   22 +++++++++++++++---
 arch/x86/kernel/smp.c            |    5 ++++
 kernel/panic.c                   |   47 ++++++++++++++++++++++++++++++++------
 9 files changed, 99 insertions(+), 11 deletions(-)


-- 
Hidehiro Kawai
Hitachi, Ltd. Research & Development Group

[V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Hidehiro Kawai]

Daniel Walker reported problems which happens when
crash_kexec_post_notifiers kernel option is enabled
(https://lkml.org/lkml/2015/6/24/44).

In that case, smp_send_stop() is called before entering kdump routines
which assume other CPUs are still online.  As the result, for x86,
kdump routines fail to save other CPUs' registers  and disable
virtualization extensions.

To fix this problem, call a new kdump friendly function,
crash_smp_send_stop(), instead of the smp_send_stop() when
crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
weak function, and it just call smp_send_stop().  Architecture
codes should override it so that kdump can work appropriately.
This patch only provides x86-specific version.

For Xen's PV kernel, just keep the current behavior.

Changes in V4:
- Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
- Rename panic_smp_send_stop to crash_smp_send_stop
- Don't change the behavior for Xen's PV kernel

Changes in V3:
- Revise comments, description, and symbol names

Changes in V2:
- Replace smp_send_stop() call with crash_kexec version which
  saves cpu states and cleans up VMX/SVM
- Drop a fix for Problem 1 at this moment

Reported-by: Daniel Walker <dwalker@fifo99.com>
Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: Eric Biederman <ebiederm@xmission.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Daniel Walker <dwalker@fifo99.com>
Cc: Xunlei Pang <xpang@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Toshi Kani <toshi.kani@hpe.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 arch/x86/include/asm/kexec.h |    1 +
 arch/x86/include/asm/smp.h   |    1 +
 arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
 arch/x86/kernel/smp.c        |    5 ++++
 kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
 5 files changed, 66 insertions(+), 10 deletions(-)

diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
index d2434c1..282630e 100644
--- a/arch/x86/include/asm/kexec.h
+++ b/arch/x86/include/asm/kexec.h
@@ -210,6 +210,7 @@ struct kexec_entry64_regs {
 
 typedef void crash_vmclear_fn(void);
 extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
+extern void kdump_nmi_shootdown_cpus(void);
 
 #endif /* __ASSEMBLY__ */
 
diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
index ebd0c16..f70989c 100644
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -50,6 +50,7 @@ struct smp_ops {
 	void (*smp_cpus_done)(unsigned max_cpus);
 
 	void (*stop_other_cpus)(int wait);
+	void (*crash_stop_other_cpus)(void);
 	void (*smp_send_reschedule)(int cpu);
 
 	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 9616cf7..650830e 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
 	disable_local_APIC();
 }
 
-static void kdump_nmi_shootdown_cpus(void)
+void kdump_nmi_shootdown_cpus(void)
 {
 	nmi_shootdown_cpus(kdump_nmi_callback);
 
 	disable_local_APIC();
 }
 
+/* Override the weak function in kernel/panic.c */
+void crash_smp_send_stop(void)
+{
+	static int cpus_stopped;
+
+	if (cpus_stopped)
+		return;
+
+	if (smp_ops.crash_stop_other_cpus)
+		smp_ops.crash_stop_other_cpus();
+	else
+		smp_send_stop();
+
+	cpus_stopped = 1;
+}
+
 #else
-static void kdump_nmi_shootdown_cpus(void)
+void crash_smp_send_stop(void)
 {
 	/* There are no cpus to shootdown */
 }
@@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	/* The kernel is broken so disable interrupts */
 	local_irq_disable();
 
-	kdump_nmi_shootdown_cpus();
+	crash_smp_send_stop();
 
 	/*
 	 * VMCLEAR VMCSs loaded on this cpu if needed.
diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index 658777c..68f8cc2 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -32,6 +32,8 @@
 #include <asm/nmi.h>
 #include <asm/mce.h>
 #include <asm/trace/irq_vectors.h>
+#include <asm/kexec.h>
+
 /*
  *	Some notes on x86 processor bugs affecting SMP operation:
  *
@@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
 	.smp_cpus_done		= native_smp_cpus_done,
 
 	.stop_other_cpus	= native_stop_other_cpus,
+#if defined(CONFIG_KEXEC_CORE)
+	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
+#endif
 	.smp_send_reschedule	= native_smp_send_reschedule,
 
 	.cpu_up			= native_cpu_up,
diff --git a/kernel/panic.c b/kernel/panic.c
index ca8cea1..e6480e2 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
 	panic_smp_self_stop();
 }
 
+/*
+ * Stop other CPUs in panic.  Architecture dependent code may override this
+ * with more suitable version.  For example, if the architecture supports
+ * crash dump, it should save registers of each stopped CPU and disable
+ * per-CPU features such as virtualization extensions.
+ */
+void __weak crash_smp_send_stop(void)
+{
+	static int cpus_stopped;
+
+	/*
+	 * This function can be called twice in panic path, but obviously
+	 * we execute this only once.
+	 */
+	if (cpus_stopped)
+		return;
+
+	/*
+	 * Note smp_send_stop is the usual smp shutdown function, which
+	 * unfortunately means it may not be hardened to work in a panic
+	 * situation.
+	 */
+	smp_send_stop();
+	cpus_stopped = 1;
+}
+
 atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
 
 /*
@@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
 	if (!_crash_kexec_post_notifiers) {
 		printk_nmi_flush_on_panic();
 		__crash_kexec(NULL);
-	}
 
-	/*
-	 * Note smp_send_stop is the usual smp shutdown function, which
-	 * unfortunately means it may not be hardened to work in a panic
-	 * situation.
-	 */
-	smp_send_stop();
+		/*
+		 * Note smp_send_stop is the usual smp shutdown function, which
+		 * unfortunately means it may not be hardened to work in a
+		 * panic situation.
+		 */
+		smp_send_stop();
+	} else {
+		/*
+		 * If we want to do crash dump after notifier calls and
+		 * kmsg_dump, we will need architecture dependent extra
+		 * works in addition to stopping other CPUs.
+		 */
+		crash_smp_send_stop();
+	}
 
 	/*
 	 * Run any panic handlers, including those that might need to

Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Dave Young]

Hi Hidehiro

Thanks for the update.
On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> Daniel Walker reported problems which happens when
> crash_kexec_post_notifiers kernel option is enabled
> (https://lkml.org/lkml/2015/6/24/44).
> 
> In that case, smp_send_stop() is called before entering kdump routines
> which assume other CPUs are still online.  As the result, for x86,
> kdump routines fail to save other CPUs' registers  and disable
> virtualization extensions.

Seems you simplified the changelog, but I think a little more details
will be helpful to understand the patch. You know sometimes lkml.org
does not work well.

> 
> To fix this problem, call a new kdump friendly function,
> crash_smp_send_stop(), instead of the smp_send_stop() when
> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> weak function, and it just call smp_send_stop().  Architecture
> codes should override it so that kdump can work appropriately.
> This patch only provides x86-specific version.
> 
> For Xen's PV kernel, just keep the current behavior.

Could you explain a bit about above Xen PV kernel behavior?

BTW, this version looks better,  I think I'm fine with this version
besides of the questions about changelog.

> 
> Changes in V4:
> - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
> - Rename panic_smp_send_stop to crash_smp_send_stop
> - Don't change the behavior for Xen's PV kernel
> 
> Changes in V3:
> - Revise comments, description, and symbol names
> 
> Changes in V2:
> - Replace smp_send_stop() call with crash_kexec version which
>   saves cpu states and cleans up VMX/SVM
> - Drop a fix for Problem 1 at this moment
> 
> Reported-by: Daniel Walker <dwalker@fifo99.com>
> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> Cc: Dave Young <dyoung@redhat.com>
> Cc: Baoquan He <bhe@redhat.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> Cc: Eric Biederman <ebiederm@xmission.com>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Daniel Walker <dwalker@fifo99.com>
> Cc: Xunlei Pang <xpang@redhat.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Borislav Petkov <bp@suse.de>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Toshi Kani <toshi.kani@hpe.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> ---
>  arch/x86/include/asm/kexec.h |    1 +
>  arch/x86/include/asm/smp.h   |    1 +
>  arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
>  arch/x86/kernel/smp.c        |    5 ++++
>  kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
>  5 files changed, 66 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
> index d2434c1..282630e 100644
> --- a/arch/x86/include/asm/kexec.h
> +++ b/arch/x86/include/asm/kexec.h
> @@ -210,6 +210,7 @@ struct kexec_entry64_regs {
>  
>  typedef void crash_vmclear_fn(void);
>  extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
> +extern void kdump_nmi_shootdown_cpus(void);
>  
>  #endif /* __ASSEMBLY__ */
>  
> diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
> index ebd0c16..f70989c 100644
> --- a/arch/x86/include/asm/smp.h
> +++ b/arch/x86/include/asm/smp.h
> @@ -50,6 +50,7 @@ struct smp_ops {
>  	void (*smp_cpus_done)(unsigned max_cpus);
>  
>  	void (*stop_other_cpus)(int wait);
> +	void (*crash_stop_other_cpus)(void);
>  	void (*smp_send_reschedule)(int cpu);
>  
>  	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 9616cf7..650830e 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
>  	disable_local_APIC();
>  }
>  
> -static void kdump_nmi_shootdown_cpus(void)
> +void kdump_nmi_shootdown_cpus(void)
>  {
>  	nmi_shootdown_cpus(kdump_nmi_callback);
>  
>  	disable_local_APIC();
>  }
>  
> +/* Override the weak function in kernel/panic.c */
> +void crash_smp_send_stop(void)
> +{
> +	static int cpus_stopped;
> +
> +	if (cpus_stopped)
> +		return;
> +
> +	if (smp_ops.crash_stop_other_cpus)
> +		smp_ops.crash_stop_other_cpus();
> +	else
> +		smp_send_stop();
> +
> +	cpus_stopped = 1;
> +}
> +
>  #else
> -static void kdump_nmi_shootdown_cpus(void)
> +void crash_smp_send_stop(void)
>  {
>  	/* There are no cpus to shootdown */
>  }
> @@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
>  	/* The kernel is broken so disable interrupts */
>  	local_irq_disable();
>  
> -	kdump_nmi_shootdown_cpus();
> +	crash_smp_send_stop();
>  
>  	/*
>  	 * VMCLEAR VMCSs loaded on this cpu if needed.
> diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
> index 658777c..68f8cc2 100644
> --- a/arch/x86/kernel/smp.c
> +++ b/arch/x86/kernel/smp.c
> @@ -32,6 +32,8 @@
>  #include <asm/nmi.h>
>  #include <asm/mce.h>
>  #include <asm/trace/irq_vectors.h>
> +#include <asm/kexec.h>
> +
>  /*
>   *	Some notes on x86 processor bugs affecting SMP operation:
>   *
> @@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
>  	.smp_cpus_done		= native_smp_cpus_done,
>  
>  	.stop_other_cpus	= native_stop_other_cpus,
> +#if defined(CONFIG_KEXEC_CORE)
> +	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
> +#endif
>  	.smp_send_reschedule	= native_smp_send_reschedule,
>  
>  	.cpu_up			= native_cpu_up,
> diff --git a/kernel/panic.c b/kernel/panic.c
> index ca8cea1..e6480e2 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
>  	panic_smp_self_stop();
>  }
>  
> +/*
> + * Stop other CPUs in panic.  Architecture dependent code may override this
> + * with more suitable version.  For example, if the architecture supports
> + * crash dump, it should save registers of each stopped CPU and disable
> + * per-CPU features such as virtualization extensions.
> + */
> +void __weak crash_smp_send_stop(void)
> +{
> +	static int cpus_stopped;
> +
> +	/*
> +	 * This function can be called twice in panic path, but obviously
> +	 * we execute this only once.
> +	 */
> +	if (cpus_stopped)
> +		return;
> +
> +	/*
> +	 * Note smp_send_stop is the usual smp shutdown function, which
> +	 * unfortunately means it may not be hardened to work in a panic
> +	 * situation.
> +	 */
> +	smp_send_stop();
> +	cpus_stopped = 1;
> +}
> +
>  atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
>  
>  /*
> @@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
>  	if (!_crash_kexec_post_notifiers) {
>  		printk_nmi_flush_on_panic();
>  		__crash_kexec(NULL);
> -	}
>  
> -	/*
> -	 * Note smp_send_stop is the usual smp shutdown function, which
> -	 * unfortunately means it may not be hardened to work in a panic
> -	 * situation.
> -	 */
> -	smp_send_stop();
> +		/*
> +		 * Note smp_send_stop is the usual smp shutdown function, which
> +		 * unfortunately means it may not be hardened to work in a
> +		 * panic situation.
> +		 */
> +		smp_send_stop();
> +	} else {
> +		/*
> +		 * If we want to do crash dump after notifier calls and
> +		 * kmsg_dump, we will need architecture dependent extra
> +		 * works in addition to stopping other CPUs.
> +		 */
> +		crash_smp_send_stop();
> +	}
>  
>  	/*
>  	 * Run any panic handlers, including those that might need to
> 
> 

RE: Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

Hi Dave,

Thank you for the review.

> From: Dave Young [mailto:dyoung@redhat.com]
> Sent: Friday, August 12, 2016 12:17 PM
> 
> Thanks for the update.
> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> > Daniel Walker reported problems which happens when
> > crash_kexec_post_notifiers kernel option is enabled
> > (https://lkml.org/lkml/2015/6/24/44).
> >
> > In that case, smp_send_stop() is called before entering kdump routines
> > which assume other CPUs are still online.  As the result, for x86,
> > kdump routines fail to save other CPUs' registers  and disable
> > virtualization extensions.
> 
> Seems you simplified the changelog, but I think a little more details
> will be helpful to understand the patch. You know sometimes lkml.org
> does not work well.

So, I'll try another archives when I post patch set next time.

> > To fix this problem, call a new kdump friendly function,
> > crash_smp_send_stop(), instead of the smp_send_stop() when
> > crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> > weak function, and it just call smp_send_stop().  Architecture
> > codes should override it so that kdump can work appropriately.
> > This patch only provides x86-specific version.
> >
> > For Xen's PV kernel, just keep the current behavior.
> 
> Could you explain a bit about above Xen PV kernel behavior?
> 
> BTW, this version looks better,  I think I'm fine with this version
> besides of the questions about changelog.

As for Dom0 kernel, it doesn't use crash_kexec routines, and
it relies on panic notifier chain.  At the end of the chain,
xen_panic_event is called, and it issues a hypercall which
requests Hypervisor to execute kdump.  This means whether
crash_kexec_panic_notifiers is set or not, panic notifiers
are called after smp_send_stop.  Even if we save registers
in Dom0 kernel, they seem to be ignored (Hypervisor is responsible
for that).  This is why I kept the current behavior for Xen.

For PV DomU kernel, kdump is not supported.  For PV HVM
DomU, I'm not sure what will happen on panic because I
couldn't boot PV HVM DomU and test it.  But I think it will
work similarly to baremetal kernels with extra cleanups
for Hypervisor.

Best regards,

Hidehiro Kawai

> > Changes in V4:
> > - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
> > - Rename panic_smp_send_stop to crash_smp_send_stop
> > - Don't change the behavior for Xen's PV kernel
> >
> > Changes in V3:
> > - Revise comments, description, and symbol names
> >
> > Changes in V2:
> > - Replace smp_send_stop() call with crash_kexec version which
> >   saves cpu states and cleans up VMX/SVM
> > - Drop a fix for Problem 1 at this moment
> >
> > Reported-by: Daniel Walker <dwalker@fifo99.com>
> > Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> > Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> > Cc: Dave Young <dyoung@redhat.com>
> > Cc: Baoquan He <bhe@redhat.com>
> > Cc: Vivek Goyal <vgoyal@redhat.com>
> > Cc: Eric Biederman <ebiederm@xmission.com>
> > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > Cc: Daniel Walker <dwalker@fifo99.com>
> > Cc: Xunlei Pang <xpang@redhat.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: "H. Peter Anvin" <hpa@zytor.com>
> > Cc: Borislav Petkov <bp@suse.de>
> > Cc: David Vrabel <david.vrabel@citrix.com>
> > Cc: Toshi Kani <toshi.kani@hpe.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > ---
> >  arch/x86/include/asm/kexec.h |    1 +
> >  arch/x86/include/asm/smp.h   |    1 +
> >  arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
> >  arch/x86/kernel/smp.c        |    5 ++++
> >  kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
> >  5 files changed, 66 insertions(+), 10 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
> > index d2434c1..282630e 100644
> > --- a/arch/x86/include/asm/kexec.h
> > +++ b/arch/x86/include/asm/kexec.h
> > @@ -210,6 +210,7 @@ struct kexec_entry64_regs {
> >
> >  typedef void crash_vmclear_fn(void);
> >  extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
> > +extern void kdump_nmi_shootdown_cpus(void);
> >
> >  #endif /* __ASSEMBLY__ */
> >
> > diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
> > index ebd0c16..f70989c 100644
> > --- a/arch/x86/include/asm/smp.h
> > +++ b/arch/x86/include/asm/smp.h
> > @@ -50,6 +50,7 @@ struct smp_ops {
> >  	void (*smp_cpus_done)(unsigned max_cpus);
> >
> >  	void (*stop_other_cpus)(int wait);
> > +	void (*crash_stop_other_cpus)(void);
> >  	void (*smp_send_reschedule)(int cpu);
> >
> >  	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
> > diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> > index 9616cf7..650830e 100644
> > --- a/arch/x86/kernel/crash.c
> > +++ b/arch/x86/kernel/crash.c
> > @@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
> >  	disable_local_APIC();
> >  }
> >
> > -static void kdump_nmi_shootdown_cpus(void)
> > +void kdump_nmi_shootdown_cpus(void)
> >  {
> >  	nmi_shootdown_cpus(kdump_nmi_callback);
> >
> >  	disable_local_APIC();
> >  }
> >
> > +/* Override the weak function in kernel/panic.c */
> > +void crash_smp_send_stop(void)
> > +{
> > +	static int cpus_stopped;
> > +
> > +	if (cpus_stopped)
> > +		return;
> > +
> > +	if (smp_ops.crash_stop_other_cpus)
> > +		smp_ops.crash_stop_other_cpus();
> > +	else
> > +		smp_send_stop();
> > +
> > +	cpus_stopped = 1;
> > +}
> > +
> >  #else
> > -static void kdump_nmi_shootdown_cpus(void)
> > +void crash_smp_send_stop(void)
> >  {
> >  	/* There are no cpus to shootdown */
> >  }
> > @@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
> >  	/* The kernel is broken so disable interrupts */
> >  	local_irq_disable();
> >
> > -	kdump_nmi_shootdown_cpus();
> > +	crash_smp_send_stop();
> >
> >  	/*
> >  	 * VMCLEAR VMCSs loaded on this cpu if needed.
> > diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
> > index 658777c..68f8cc2 100644
> > --- a/arch/x86/kernel/smp.c
> > +++ b/arch/x86/kernel/smp.c
> > @@ -32,6 +32,8 @@
> >  #include <asm/nmi.h>
> >  #include <asm/mce.h>
> >  #include <asm/trace/irq_vectors.h>
> > +#include <asm/kexec.h>
> > +
> >  /*
> >   *	Some notes on x86 processor bugs affecting SMP operation:
> >   *
> > @@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
> >  	.smp_cpus_done		= native_smp_cpus_done,
> >
> >  	.stop_other_cpus	= native_stop_other_cpus,
> > +#if defined(CONFIG_KEXEC_CORE)
> > +	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
> > +#endif
> >  	.smp_send_reschedule	= native_smp_send_reschedule,
> >
> >  	.cpu_up			= native_cpu_up,
> > diff --git a/kernel/panic.c b/kernel/panic.c
> > index ca8cea1..e6480e2 100644
> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
> >  	panic_smp_self_stop();
> >  }
> >
> > +/*
> > + * Stop other CPUs in panic.  Architecture dependent code may override this
> > + * with more suitable version.  For example, if the architecture supports
> > + * crash dump, it should save registers of each stopped CPU and disable
> > + * per-CPU features such as virtualization extensions.
> > + */
> > +void __weak crash_smp_send_stop(void)
> > +{
> > +	static int cpus_stopped;
> > +
> > +	/*
> > +	 * This function can be called twice in panic path, but obviously
> > +	 * we execute this only once.
> > +	 */
> > +	if (cpus_stopped)
> > +		return;
> > +
> > +	/*
> > +	 * Note smp_send_stop is the usual smp shutdown function, which
> > +	 * unfortunately means it may not be hardened to work in a panic
> > +	 * situation.
> > +	 */
> > +	smp_send_stop();
> > +	cpus_stopped = 1;
> > +}
> > +
> >  atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
> >
> >  /*
> > @@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
> >  	if (!_crash_kexec_post_notifiers) {
> >  		printk_nmi_flush_on_panic();
> >  		__crash_kexec(NULL);
> > -	}
> >
> > -	/*
> > -	 * Note smp_send_stop is the usual smp shutdown function, which
> > -	 * unfortunately means it may not be hardened to work in a panic
> > -	 * situation.
> > -	 */
> > -	smp_send_stop();
> > +		/*
> > +		 * Note smp_send_stop is the usual smp shutdown function, which
> > +		 * unfortunately means it may not be hardened to work in a
> > +		 * panic situation.
> > +		 */
> > +		smp_send_stop();
> > +	} else {
> > +		/*
> > +		 * If we want to do crash dump after notifier calls and
> > +		 * kmsg_dump, we will need architecture dependent extra
> > +		 * works in addition to stopping other CPUs.
> > +		 */
> > +		crash_smp_send_stop();
> > +	}
> >
> >  	/*
> >  	 * Run any panic handlers, including those that might need to
> >
> >

Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Xunlei Pang]

On 2016/08/15/ at 19:22, Hidehiro Kawai wrote:
> Hi Dave,
>
> Thank you for the review.
>
>> From: Dave Young [mailto:dyoung@redhat.com]
>> Sent: Friday, August 12, 2016 12:17 PM
>>
>> Thanks for the update.
>> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
>>> Daniel Walker reported problems which happens when
>>> crash_kexec_post_notifiers kernel option is enabled
>>> (https://lkml.org/lkml/2015/6/24/44).
>>>
>>> In that case, smp_send_stop() is called before entering kdump routines
>>> which assume other CPUs are still online.  As the result, for x86,
>>> kdump routines fail to save other CPUs' registers  and disable
>>> virtualization extensions.
>> Seems you simplified the changelog, but I think a little more details
>> will be helpful to understand the patch. You know sometimes lkml.org
>> does not work well.
> So, I'll try another archives when I post patch set next time.

Hi Hidehiro Kawai,

What's the status of this patch set, are you going to send an updated version?

Regards,
Xunlei

>>> To fix this problem, call a new kdump friendly function,
>>> crash_smp_send_stop(), instead of the smp_send_stop() when
>>> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
>>> weak function, and it just call smp_send_stop().  Architecture
>>> codes should override it so that kdump can work appropriately.
>>> This patch only provides x86-specific version.
>>>
>>> For Xen's PV kernel, just keep the current behavior.
>> Could you explain a bit about above Xen PV kernel behavior?
>>
>> BTW, this version looks better,  I think I'm fine with this version
>> besides of the questions about changelog.
> As for Dom0 kernel, it doesn't use crash_kexec routines, and
> it relies on panic notifier chain.  At the end of the chain,
> xen_panic_event is called, and it issues a hypercall which
> requests Hypervisor to execute kdump.  This means whether
> crash_kexec_panic_notifiers is set or not, panic notifiers
> are called after smp_send_stop.  Even if we save registers
> in Dom0 kernel, they seem to be ignored (Hypervisor is responsible
> for that).  This is why I kept the current behavior for Xen.
>
> For PV DomU kernel, kdump is not supported.  For PV HVM
> DomU, I'm not sure what will happen on panic because I
> couldn't boot PV HVM DomU and test it.  But I think it will
> work similarly to baremetal kernels with extra cleanups
> for Hypervisor.
>
> Best regards,
>
> Hidehiro Kawai
>
>>> Changes in V4:
>>> - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
>>> - Rename panic_smp_send_stop to crash_smp_send_stop
>>> - Don't change the behavior for Xen's PV kernel
>>>
>>> Changes in V3:
>>> - Revise comments, description, and symbol names
>>>
>>> Changes in V2:
>>> - Replace smp_send_stop() call with crash_kexec version which
>>>   saves cpu states and cleans up VMX/SVM
>>> - Drop a fix for Problem 1 at this moment
>>>
>>> Reported-by: Daniel Walker <dwalker@fifo99.com>
>>> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
>>> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
>>> Cc: Dave Young <dyoung@redhat.com>
>>> Cc: Baoquan He <bhe@redhat.com>
>>> Cc: Vivek Goyal <vgoyal@redhat.com>
>>> Cc: Eric Biederman <ebiederm@xmission.com>
>>> Cc: Masami Hiramatsu <mhiramat@kernel.org>
>>> Cc: Daniel Walker <dwalker@fifo99.com>
>>> Cc: Xunlei Pang <xpang@redhat.com>
>>> Cc: Thomas Gleixner <tglx@linutronix.de>
>>> Cc: Ingo Molnar <mingo@redhat.com>
>>> Cc: "H. Peter Anvin" <hpa@zytor.com>
>>> Cc: Borislav Petkov <bp@suse.de>
>>> Cc: David Vrabel <david.vrabel@citrix.com>
>>> Cc: Toshi Kani <toshi.kani@hpe.com>
>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>> ---
>>>  arch/x86/include/asm/kexec.h |    1 +
>>>  arch/x86/include/asm/smp.h   |    1 +
>>>  arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
>>>  arch/x86/kernel/smp.c        |    5 ++++
>>>  kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
>>>  5 files changed, 66 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
>>> index d2434c1..282630e 100644
>>> --- a/arch/x86/include/asm/kexec.h
>>> +++ b/arch/x86/include/asm/kexec.h
>>> @@ -210,6 +210,7 @@ struct kexec_entry64_regs {
>>>
>>>  typedef void crash_vmclear_fn(void);
>>>  extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
>>> +extern void kdump_nmi_shootdown_cpus(void);
>>>
>>>  #endif /* __ASSEMBLY__ */
>>>
>>> diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
>>> index ebd0c16..f70989c 100644
>>> --- a/arch/x86/include/asm/smp.h
>>> +++ b/arch/x86/include/asm/smp.h
>>> @@ -50,6 +50,7 @@ struct smp_ops {
>>>  	void (*smp_cpus_done)(unsigned max_cpus);
>>>
>>>  	void (*stop_other_cpus)(int wait);
>>> +	void (*crash_stop_other_cpus)(void);
>>>  	void (*smp_send_reschedule)(int cpu);
>>>
>>>  	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
>>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>>> index 9616cf7..650830e 100644
>>> --- a/arch/x86/kernel/crash.c
>>> +++ b/arch/x86/kernel/crash.c
>>> @@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
>>>  	disable_local_APIC();
>>>  }
>>>
>>> -static void kdump_nmi_shootdown_cpus(void)
>>> +void kdump_nmi_shootdown_cpus(void)
>>>  {
>>>  	nmi_shootdown_cpus(kdump_nmi_callback);
>>>
>>>  	disable_local_APIC();
>>>  }
>>>
>>> +/* Override the weak function in kernel/panic.c */
>>> +void crash_smp_send_stop(void)
>>> +{
>>> +	static int cpus_stopped;
>>> +
>>> +	if (cpus_stopped)
>>> +		return;
>>> +
>>> +	if (smp_ops.crash_stop_other_cpus)
>>> +		smp_ops.crash_stop_other_cpus();
>>> +	else
>>> +		smp_send_stop();
>>> +
>>> +	cpus_stopped = 1;
>>> +}
>>> +
>>>  #else
>>> -static void kdump_nmi_shootdown_cpus(void)
>>> +void crash_smp_send_stop(void)
>>>  {
>>>  	/* There are no cpus to shootdown */
>>>  }
>>> @@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
>>>  	/* The kernel is broken so disable interrupts */
>>>  	local_irq_disable();
>>>
>>> -	kdump_nmi_shootdown_cpus();
>>> +	crash_smp_send_stop();
>>>
>>>  	/*
>>>  	 * VMCLEAR VMCSs loaded on this cpu if needed.
>>> diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
>>> index 658777c..68f8cc2 100644
>>> --- a/arch/x86/kernel/smp.c
>>> +++ b/arch/x86/kernel/smp.c
>>> @@ -32,6 +32,8 @@
>>>  #include <asm/nmi.h>
>>>  #include <asm/mce.h>
>>>  #include <asm/trace/irq_vectors.h>
>>> +#include <asm/kexec.h>
>>> +
>>>  /*
>>>   *	Some notes on x86 processor bugs affecting SMP operation:
>>>   *
>>> @@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
>>>  	.smp_cpus_done		= native_smp_cpus_done,
>>>
>>>  	.stop_other_cpus	= native_stop_other_cpus,
>>> +#if defined(CONFIG_KEXEC_CORE)
>>> +	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
>>> +#endif
>>>  	.smp_send_reschedule	= native_smp_send_reschedule,
>>>
>>>  	.cpu_up			= native_cpu_up,
>>> diff --git a/kernel/panic.c b/kernel/panic.c
>>> index ca8cea1..e6480e2 100644
>>> --- a/kernel/panic.c
>>> +++ b/kernel/panic.c
>>> @@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
>>>  	panic_smp_self_stop();
>>>  }
>>>
>>> +/*
>>> + * Stop other CPUs in panic.  Architecture dependent code may override this
>>> + * with more suitable version.  For example, if the architecture supports
>>> + * crash dump, it should save registers of each stopped CPU and disable
>>> + * per-CPU features such as virtualization extensions.
>>> + */
>>> +void __weak crash_smp_send_stop(void)
>>> +{
>>> +	static int cpus_stopped;
>>> +
>>> +	/*
>>> +	 * This function can be called twice in panic path, but obviously
>>> +	 * we execute this only once.
>>> +	 */
>>> +	if (cpus_stopped)
>>> +		return;
>>> +
>>> +	/*
>>> +	 * Note smp_send_stop is the usual smp shutdown function, which
>>> +	 * unfortunately means it may not be hardened to work in a panic
>>> +	 * situation.
>>> +	 */
>>> +	smp_send_stop();
>>> +	cpus_stopped = 1;
>>> +}
>>> +
>>>  atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
>>>
>>>  /*
>>> @@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
>>>  	if (!_crash_kexec_post_notifiers) {
>>>  		printk_nmi_flush_on_panic();
>>>  		__crash_kexec(NULL);
>>> -	}
>>>
>>> -	/*
>>> -	 * Note smp_send_stop is the usual smp shutdown function, which
>>> -	 * unfortunately means it may not be hardened to work in a panic
>>> -	 * situation.
>>> -	 */
>>> -	smp_send_stop();
>>> +		/*
>>> +		 * Note smp_send_stop is the usual smp shutdown function, which
>>> +		 * unfortunately means it may not be hardened to work in a
>>> +		 * panic situation.
>>> +		 */
>>> +		smp_send_stop();
>>> +	} else {
>>> +		/*
>>> +		 * If we want to do crash dump after notifier calls and
>>> +		 * kmsg_dump, we will need architecture dependent extra
>>> +		 * works in addition to stopping other CPUs.
>>> +		 */
>>> +		crash_smp_send_stop();
>>> +	}
>>>
>>>  	/*
>>>  	 * Run any panic handlers, including those that might need to
>>>
>>>

RE: Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

Hi Xunlei,

> From: Xunlei Pang [mailto:xpang@redhat.com]
> Sent: Tuesday, September 20, 2016 4:40 PM
> On 2016/08/15/ at 19:22, Hidehiro Kawai wrote:
> > Hi Dave,
> >
> > Thank you for the review.
> >
> >> From: Dave Young [mailto:dyoung@redhat.com]
> >> Sent: Friday, August 12, 2016 12:17 PM
> >>
> >> Thanks for the update.
> >> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> >>> Daniel Walker reported problems which happens when
> >>> crash_kexec_post_notifiers kernel option is enabled
> >>> (https://lkml.org/lkml/2015/6/24/44).
> >>>
> >>> In that case, smp_send_stop() is called before entering kdump routines
> >>> which assume other CPUs are still online.  As the result, for x86,
> >>> kdump routines fail to save other CPUs' registers  and disable
> >>> virtualization extensions.
> >> Seems you simplified the changelog, but I think a little more details
> >> will be helpful to understand the patch. You know sometimes lkml.org
> >> does not work well.
> > So, I'll try another archives when I post patch set next time.
> 
> Hi Hidehiro Kawai,
> 
> What's the status of this patch set, are you going to send an updated version?

Sorry, I misunderstood what Dave said, and I thought I don't
need to revise the patch set.

Currently, this patch set is in -mm, then -next tree.
What I need to fix is only commit descriptions, so I'm going to
post revised descriptions as replies to this thread, and then
request to Andrew to replace them if there is no objection.
(or should I resend the whole patch set?)

Regards,
Hidehiro Kawai

> >>> To fix this problem, call a new kdump friendly function,
> >>> crash_smp_send_stop(), instead of the smp_send_stop() when
> >>> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> >>> weak function, and it just call smp_send_stop().  Architecture
> >>> codes should override it so that kdump can work appropriately.
> >>> This patch only provides x86-specific version.
> >>>
> >>> For Xen's PV kernel, just keep the current behavior.
> >> Could you explain a bit about above Xen PV kernel behavior?
> >>
> >> BTW, this version looks better,  I think I'm fine with this version
> >> besides of the questions about changelog.
> > As for Dom0 kernel, it doesn't use crash_kexec routines, and
> > it relies on panic notifier chain.  At the end of the chain,
> > xen_panic_event is called, and it issues a hypercall which
> > requests Hypervisor to execute kdump.  This means whether
> > crash_kexec_panic_notifiers is set or not, panic notifiers
> > are called after smp_send_stop.  Even if we save registers
> > in Dom0 kernel, they seem to be ignored (Hypervisor is responsible
> > for that).  This is why I kept the current behavior for Xen.
> >
> > For PV DomU kernel, kdump is not supported.  For PV HVM
> > DomU, I'm not sure what will happen on panic because I
> > couldn't boot PV HVM DomU and test it.  But I think it will
> > work similarly to baremetal kernels with extra cleanups
> > for Hypervisor.
> >
> > Best regards,
> >
> > Hidehiro Kawai
> >
> >>> Changes in V4:
> >>> - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
> >>> - Rename panic_smp_send_stop to crash_smp_send_stop
> >>> - Don't change the behavior for Xen's PV kernel
> >>>
> >>> Changes in V3:
> >>> - Revise comments, description, and symbol names
> >>>
> >>> Changes in V2:
> >>> - Replace smp_send_stop() call with crash_kexec version which
> >>>   saves cpu states and cleans up VMX/SVM
> >>> - Drop a fix for Problem 1 at this moment
> >>>
> >>> Reported-by: Daniel Walker <dwalker@fifo99.com>
> >>> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> >>> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> >>> Cc: Dave Young <dyoung@redhat.com>
> >>> Cc: Baoquan He <bhe@redhat.com>
> >>> Cc: Vivek Goyal <vgoyal@redhat.com>
> >>> Cc: Eric Biederman <ebiederm@xmission.com>
> >>> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> >>> Cc: Daniel Walker <dwalker@fifo99.com>
> >>> Cc: Xunlei Pang <xpang@redhat.com>
> >>> Cc: Thomas Gleixner <tglx@linutronix.de>
> >>> Cc: Ingo Molnar <mingo@redhat.com>
> >>> Cc: "H. Peter Anvin" <hpa@zytor.com>
> >>> Cc: Borislav Petkov <bp@suse.de>
> >>> Cc: David Vrabel <david.vrabel@citrix.com>
> >>> Cc: Toshi Kani <toshi.kani@hpe.com>
> >>> Cc: Andrew Morton <akpm@linux-foundation.org>
> >>> ---
> >>>  arch/x86/include/asm/kexec.h |    1 +
> >>>  arch/x86/include/asm/smp.h   |    1 +
> >>>  arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
> >>>  arch/x86/kernel/smp.c        |    5 ++++
> >>>  kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
> >>>  5 files changed, 66 insertions(+), 10 deletions(-)
> >>>
> >>> diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
> >>> index d2434c1..282630e 100644
> >>> --- a/arch/x86/include/asm/kexec.h
> >>> +++ b/arch/x86/include/asm/kexec.h
> >>> @@ -210,6 +210,7 @@ struct kexec_entry64_regs {
> >>>
> >>>  typedef void crash_vmclear_fn(void);
> >>>  extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
> >>> +extern void kdump_nmi_shootdown_cpus(void);
> >>>
> >>>  #endif /* __ASSEMBLY__ */
> >>>
> >>> diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
> >>> index ebd0c16..f70989c 100644
> >>> --- a/arch/x86/include/asm/smp.h
> >>> +++ b/arch/x86/include/asm/smp.h
> >>> @@ -50,6 +50,7 @@ struct smp_ops {
> >>>  	void (*smp_cpus_done)(unsigned max_cpus);
> >>>
> >>>  	void (*stop_other_cpus)(int wait);
> >>> +	void (*crash_stop_other_cpus)(void);
> >>>  	void (*smp_send_reschedule)(int cpu);
> >>>
> >>>  	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
> >>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> >>> index 9616cf7..650830e 100644
> >>> --- a/arch/x86/kernel/crash.c
> >>> +++ b/arch/x86/kernel/crash.c
> >>> @@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
> >>>  	disable_local_APIC();
> >>>  }
> >>>
> >>> -static void kdump_nmi_shootdown_cpus(void)
> >>> +void kdump_nmi_shootdown_cpus(void)
> >>>  {
> >>>  	nmi_shootdown_cpus(kdump_nmi_callback);
> >>>
> >>>  	disable_local_APIC();
> >>>  }
> >>>
> >>> +/* Override the weak function in kernel/panic.c */
> >>> +void crash_smp_send_stop(void)
> >>> +{
> >>> +	static int cpus_stopped;
> >>> +
> >>> +	if (cpus_stopped)
> >>> +		return;
> >>> +
> >>> +	if (smp_ops.crash_stop_other_cpus)
> >>> +		smp_ops.crash_stop_other_cpus();
> >>> +	else
> >>> +		smp_send_stop();
> >>> +
> >>> +	cpus_stopped = 1;
> >>> +}
> >>> +
> >>>  #else
> >>> -static void kdump_nmi_shootdown_cpus(void)
> >>> +void crash_smp_send_stop(void)
> >>>  {
> >>>  	/* There are no cpus to shootdown */
> >>>  }
> >>> @@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
> >>>  	/* The kernel is broken so disable interrupts */
> >>>  	local_irq_disable();
> >>>
> >>> -	kdump_nmi_shootdown_cpus();
> >>> +	crash_smp_send_stop();
> >>>
> >>>  	/*
> >>>  	 * VMCLEAR VMCSs loaded on this cpu if needed.
> >>> diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
> >>> index 658777c..68f8cc2 100644
> >>> --- a/arch/x86/kernel/smp.c
> >>> +++ b/arch/x86/kernel/smp.c
> >>> @@ -32,6 +32,8 @@
> >>>  #include <asm/nmi.h>
> >>>  #include <asm/mce.h>
> >>>  #include <asm/trace/irq_vectors.h>
> >>> +#include <asm/kexec.h>
> >>> +
> >>>  /*
> >>>   *	Some notes on x86 processor bugs affecting SMP operation:
> >>>   *
> >>> @@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
> >>>  	.smp_cpus_done		= native_smp_cpus_done,
> >>>
> >>>  	.stop_other_cpus	= native_stop_other_cpus,
> >>> +#if defined(CONFIG_KEXEC_CORE)
> >>> +	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
> >>> +#endif
> >>>  	.smp_send_reschedule	= native_smp_send_reschedule,
> >>>
> >>>  	.cpu_up			= native_cpu_up,
> >>> diff --git a/kernel/panic.c b/kernel/panic.c
> >>> index ca8cea1..e6480e2 100644
> >>> --- a/kernel/panic.c
> >>> +++ b/kernel/panic.c
> >>> @@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
> >>>  	panic_smp_self_stop();
> >>>  }
> >>>
> >>> +/*
> >>> + * Stop other CPUs in panic.  Architecture dependent code may override this
> >>> + * with more suitable version.  For example, if the architecture supports
> >>> + * crash dump, it should save registers of each stopped CPU and disable
> >>> + * per-CPU features such as virtualization extensions.
> >>> + */
> >>> +void __weak crash_smp_send_stop(void)
> >>> +{
> >>> +	static int cpus_stopped;
> >>> +
> >>> +	/*
> >>> +	 * This function can be called twice in panic path, but obviously
> >>> +	 * we execute this only once.
> >>> +	 */
> >>> +	if (cpus_stopped)
> >>> +		return;
> >>> +
> >>> +	/*
> >>> +	 * Note smp_send_stop is the usual smp shutdown function, which
> >>> +	 * unfortunately means it may not be hardened to work in a panic
> >>> +	 * situation.
> >>> +	 */
> >>> +	smp_send_stop();
> >>> +	cpus_stopped = 1;
> >>> +}
> >>> +
> >>>  atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
> >>>
> >>>  /*
> >>> @@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
> >>>  	if (!_crash_kexec_post_notifiers) {
> >>>  		printk_nmi_flush_on_panic();
> >>>  		__crash_kexec(NULL);
> >>> -	}
> >>>
> >>> -	/*
> >>> -	 * Note smp_send_stop is the usual smp shutdown function, which
> >>> -	 * unfortunately means it may not be hardened to work in a panic
> >>> -	 * situation.
> >>> -	 */
> >>> -	smp_send_stop();
> >>> +		/*
> >>> +		 * Note smp_send_stop is the usual smp shutdown function, which
> >>> +		 * unfortunately means it may not be hardened to work in a
> >>> +		 * panic situation.
> >>> +		 */
> >>> +		smp_send_stop();
> >>> +	} else {
> >>> +		/*
> >>> +		 * If we want to do crash dump after notifier calls and
> >>> +		 * kmsg_dump, we will need architecture dependent extra
> >>> +		 * works in addition to stopping other CPUs.
> >>> +		 */
> >>> +		crash_smp_send_stop();
> >>> +	}
> >>>
> >>>  	/*
> >>>  	 * Run any panic handlers, including those that might need to
> >>>
> >>>

RE: RE: Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

Here is the revised commit description reflecting Dave's
comment.  Cc list was copied from -mm version.

From: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Subject: x86/panic: replace smp_send_stop() with kdump friendly version in panic path

This patch fixes a problem reported by Daniel Walker
(https://lkml.org/lkml/2015/6/24/44).

When kernel panics with crash_kexec_post_notifiers kernel parameter
enabled, other CPUs are stopped by smp_send_stop() instead of
machine_crash_shutdown() in __crash_kexec() path.

  panic()
    if crash_kexec_post_notifiers == 1
      smp_send_stop()
      atomic_notifier_call_chain()
      kmsg_dump()
    __crash_kexec()
      machine_crash_shutdown()

Different from smp_send_stop(), machine_crash_shutdown() stops other
CPUs with extra works for kdump.  So, if smp_send_stop() stops other
CPUs in advance, these extra works won't be done.  For x86, kdump
routines miss to save other CPUs' registers and disable virtualization
extensions.

To fix this problem, call a new kdump friendly function,
crash_smp_send_stop(), instead of the smp_send_stop() when
crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
weak function, and it just call smp_send_stop().  Architecture
codes should override it so that kdump can work appropriately.
This patch only provides x86-specific version.

For Xen's PV kernel, just keep the current behavior.
As for Dom0, it doesn't use crash_kexec routines, and it relies on
panic notifier chain.  At the end of the chain, a hypercall is
issued which requests the hypervisor to execute kdump.  This means
regardless of crash_kexec_post_notifiers setting, smp_send_stop().
For PV HVM, it would work similarly to baremetal kernels with extra
cleanups for hypervisor.  It doesn't need additional care.

Changes in V4:
- Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
- Rename panic_smp_send_stop to crash_smp_send_stop
- Don't change the behavior for Xen's PV kernel

Changes in V3:
- Revise comments, description, and symbol names

Changes in V2:
- Replace smp_send_stop() call with crash_kexec version which
  saves cpu states and cleans up VMX/SVM
- Drop a fix for Problem 1 at this moment

Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
Link: http://lkml.kernel.org/r/20160810080948.11028.15344.stgit@sysi4-13.yrl.intra.hitachi.co.jp
Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Reported-by: Daniel Walker <dwalker@fifo99.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: Eric Biederman <ebiederm@xmission.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Daniel Walker <dwalker@fifo99.com>
Cc: Xunlei Pang <xpang@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Toshi Kani <toshi.kani@hpe.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: David Daney <david.daney@cavium.com>
Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
Cc: "Steven J. Hill" <steven.hill@cavium.com>
Cc: Corey Minyard <cminyard@mvista.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

> -----Original Message-----
> From: Hidehiro Kawai
> Hi Xunlei,
> 
> > From: Xunlei Pang [mailto:xpang@redhat.com]
> > Sent: Tuesday, September 20, 2016 4:40 PM
> > On 2016/08/15/ at 19:22, Hidehiro Kawai wrote:
> > > Hi Dave,
> > >
> > > Thank you for the review.
> > >
> > >> From: Dave Young [mailto:dyoung@redhat.com]
> > >> Sent: Friday, August 12, 2016 12:17 PM
> > >>
> > >> Thanks for the update.
> > >> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> > >>> Daniel Walker reported problems which happens when
> > >>> crash_kexec_post_notifiers kernel option is enabled
> > >>> (https://lkml.org/lkml/2015/6/24/44).
> > >>>
> > >>> In that case, smp_send_stop() is called before entering kdump routines
> > >>> which assume other CPUs are still online.  As the result, for x86,
> > >>> kdump routines fail to save other CPUs' registers  and disable
> > >>> virtualization extensions.
> > >> Seems you simplified the changelog, but I think a little more details
> > >> will be helpful to understand the patch. You know sometimes lkml.org
> > >> does not work well.
> > > So, I'll try another archives when I post patch set next time.
> >
> > Hi Hidehiro Kawai,
> >
> > What's the status of this patch set, are you going to send an updated version?
> 
> Sorry, I misunderstood what Dave said, and I thought I don't
> need to revise the patch set.
> 
> Currently, this patch set is in -mm, then -next tree.
> What I need to fix is only commit descriptions, so I'm going to
> post revised descriptions as replies to this thread, and then
> request to Andrew to replace them if there is no objection.
> (or should I resend the whole patch set?)
> 
> Regards,
> Hidehiro Kawai
> 
> > >>> To fix this problem, call a new kdump friendly function,
> > >>> crash_smp_send_stop(), instead of the smp_send_stop() when
> > >>> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> > >>> weak function, and it just call smp_send_stop().  Architecture
> > >>> codes should override it so that kdump can work appropriately.
> > >>> This patch only provides x86-specific version.
> > >>>
> > >>> For Xen's PV kernel, just keep the current behavior.
> > >> Could you explain a bit about above Xen PV kernel behavior?
> > >>
> > >> BTW, this version looks better,  I think I'm fine with this version
> > >> besides of the questions about changelog.
> > > As for Dom0 kernel, it doesn't use crash_kexec routines, and
> > > it relies on panic notifier chain.  At the end of the chain,
> > > xen_panic_event is called, and it issues a hypercall which
> > > requests Hypervisor to execute kdump.  This means whether
> > > crash_kexec_panic_notifiers is set or not, panic notifiers
> > > are called after smp_send_stop.  Even if we save registers
> > > in Dom0 kernel, they seem to be ignored (Hypervisor is responsible
> > > for that).  This is why I kept the current behavior for Xen.
> > >
> > > For PV DomU kernel, kdump is not supported.  For PV HVM
> > > DomU, I'm not sure what will happen on panic because I
> > > couldn't boot PV HVM DomU and test it.  But I think it will
> > > work similarly to baremetal kernels with extra cleanups
> > > for Hypervisor.
> > >
> > > Best regards,
> > >
> > > Hidehiro Kawai
> > >
> > >>> Changes in V4:
> > >>> - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
> > >>> - Rename panic_smp_send_stop to crash_smp_send_stop
> > >>> - Don't change the behavior for Xen's PV kernel
> > >>>
> > >>> Changes in V3:
> > >>> - Revise comments, description, and symbol names
> > >>>
> > >>> Changes in V2:
> > >>> - Replace smp_send_stop() call with crash_kexec version which
> > >>>   saves cpu states and cleans up VMX/SVM
> > >>> - Drop a fix for Problem 1 at this moment
> > >>>
> > >>> Reported-by: Daniel Walker <dwalker@fifo99.com>
> > >>> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> > >>> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> > >>> Cc: Dave Young <dyoung@redhat.com>
> > >>> Cc: Baoquan He <bhe@redhat.com>
> > >>> Cc: Vivek Goyal <vgoyal@redhat.com>
> > >>> Cc: Eric Biederman <ebiederm@xmission.com>
> > >>> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > >>> Cc: Daniel Walker <dwalker@fifo99.com>
> > >>> Cc: Xunlei Pang <xpang@redhat.com>
> > >>> Cc: Thomas Gleixner <tglx@linutronix.de>
> > >>> Cc: Ingo Molnar <mingo@redhat.com>
> > >>> Cc: "H. Peter Anvin" <hpa@zytor.com>
> > >>> Cc: Borislav Petkov <bp@suse.de>
> > >>> Cc: David Vrabel <david.vrabel@citrix.com>
> > >>> Cc: Toshi Kani <toshi.kani@hpe.com>
> > >>> Cc: Andrew Morton <akpm@linux-foundation.org>
> > >>> ---
> > >>>  arch/x86/include/asm/kexec.h |    1 +
> > >>>  arch/x86/include/asm/smp.h   |    1 +
> > >>>  arch/x86/kernel/crash.c      |   22 +++++++++++++++++---
> > >>>  arch/x86/kernel/smp.c        |    5 ++++
> > >>>  kernel/panic.c               |   47 ++++++++++++++++++++++++++++++++++++------
> > >>>  5 files changed, 66 insertions(+), 10 deletions(-)
> > >>>
> > >>> diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
> > >>> index d2434c1..282630e 100644
> > >>> --- a/arch/x86/include/asm/kexec.h
> > >>> +++ b/arch/x86/include/asm/kexec.h
> > >>> @@ -210,6 +210,7 @@ struct kexec_entry64_regs {
> > >>>
> > >>>  typedef void crash_vmclear_fn(void);
> > >>>  extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss;
> > >>> +extern void kdump_nmi_shootdown_cpus(void);
> > >>>
> > >>>  #endif /* __ASSEMBLY__ */
> > >>>
> > >>> diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
> > >>> index ebd0c16..f70989c 100644
> > >>> --- a/arch/x86/include/asm/smp.h
> > >>> +++ b/arch/x86/include/asm/smp.h
> > >>> @@ -50,6 +50,7 @@ struct smp_ops {
> > >>>  	void (*smp_cpus_done)(unsigned max_cpus);
> > >>>
> > >>>  	void (*stop_other_cpus)(int wait);
> > >>> +	void (*crash_stop_other_cpus)(void);
> > >>>  	void (*smp_send_reschedule)(int cpu);
> > >>>
> > >>>  	int (*cpu_up)(unsigned cpu, struct task_struct *tidle);
> > >>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> > >>> index 9616cf7..650830e 100644
> > >>> --- a/arch/x86/kernel/crash.c
> > >>> +++ b/arch/x86/kernel/crash.c
> > >>> @@ -133,15 +133,31 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
> > >>>  	disable_local_APIC();
> > >>>  }
> > >>>
> > >>> -static void kdump_nmi_shootdown_cpus(void)
> > >>> +void kdump_nmi_shootdown_cpus(void)
> > >>>  {
> > >>>  	nmi_shootdown_cpus(kdump_nmi_callback);
> > >>>
> > >>>  	disable_local_APIC();
> > >>>  }
> > >>>
> > >>> +/* Override the weak function in kernel/panic.c */
> > >>> +void crash_smp_send_stop(void)
> > >>> +{
> > >>> +	static int cpus_stopped;
> > >>> +
> > >>> +	if (cpus_stopped)
> > >>> +		return;
> > >>> +
> > >>> +	if (smp_ops.crash_stop_other_cpus)
> > >>> +		smp_ops.crash_stop_other_cpus();
> > >>> +	else
> > >>> +		smp_send_stop();
> > >>> +
> > >>> +	cpus_stopped = 1;
> > >>> +}
> > >>> +
> > >>>  #else
> > >>> -static void kdump_nmi_shootdown_cpus(void)
> > >>> +void crash_smp_send_stop(void)
> > >>>  {
> > >>>  	/* There are no cpus to shootdown */
> > >>>  }
> > >>> @@ -160,7 +176,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
> > >>>  	/* The kernel is broken so disable interrupts */
> > >>>  	local_irq_disable();
> > >>>
> > >>> -	kdump_nmi_shootdown_cpus();
> > >>> +	crash_smp_send_stop();
> > >>>
> > >>>  	/*
> > >>>  	 * VMCLEAR VMCSs loaded on this cpu if needed.
> > >>> diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
> > >>> index 658777c..68f8cc2 100644
> > >>> --- a/arch/x86/kernel/smp.c
> > >>> +++ b/arch/x86/kernel/smp.c
> > >>> @@ -32,6 +32,8 @@
> > >>>  #include <asm/nmi.h>
> > >>>  #include <asm/mce.h>
> > >>>  #include <asm/trace/irq_vectors.h>
> > >>> +#include <asm/kexec.h>
> > >>> +
> > >>>  /*
> > >>>   *	Some notes on x86 processor bugs affecting SMP operation:
> > >>>   *
> > >>> @@ -342,6 +344,9 @@ struct smp_ops smp_ops = {
> > >>>  	.smp_cpus_done		= native_smp_cpus_done,
> > >>>
> > >>>  	.stop_other_cpus	= native_stop_other_cpus,
> > >>> +#if defined(CONFIG_KEXEC_CORE)
> > >>> +	.crash_stop_other_cpus	= kdump_nmi_shootdown_cpus,
> > >>> +#endif
> > >>>  	.smp_send_reschedule	= native_smp_send_reschedule,
> > >>>
> > >>>  	.cpu_up			= native_cpu_up,
> > >>> diff --git a/kernel/panic.c b/kernel/panic.c
> > >>> index ca8cea1..e6480e2 100644
> > >>> --- a/kernel/panic.c
> > >>> +++ b/kernel/panic.c
> > >>> @@ -71,6 +71,32 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs)
> > >>>  	panic_smp_self_stop();
> > >>>  }
> > >>>
> > >>> +/*
> > >>> + * Stop other CPUs in panic.  Architecture dependent code may override this
> > >>> + * with more suitable version.  For example, if the architecture supports
> > >>> + * crash dump, it should save registers of each stopped CPU and disable
> > >>> + * per-CPU features such as virtualization extensions.
> > >>> + */
> > >>> +void __weak crash_smp_send_stop(void)
> > >>> +{
> > >>> +	static int cpus_stopped;
> > >>> +
> > >>> +	/*
> > >>> +	 * This function can be called twice in panic path, but obviously
> > >>> +	 * we execute this only once.
> > >>> +	 */
> > >>> +	if (cpus_stopped)
> > >>> +		return;
> > >>> +
> > >>> +	/*
> > >>> +	 * Note smp_send_stop is the usual smp shutdown function, which
> > >>> +	 * unfortunately means it may not be hardened to work in a panic
> > >>> +	 * situation.
> > >>> +	 */
> > >>> +	smp_send_stop();
> > >>> +	cpus_stopped = 1;
> > >>> +}
> > >>> +
> > >>>  atomic_t panic_cpu = ATOMIC_INIT(PANIC_CPU_INVALID);
> > >>>
> > >>>  /*
> > >>> @@ -164,14 +190,21 @@ void panic(const char *fmt, ...)
> > >>>  	if (!_crash_kexec_post_notifiers) {
> > >>>  		printk_nmi_flush_on_panic();
> > >>>  		__crash_kexec(NULL);
> > >>> -	}
> > >>>
> > >>> -	/*
> > >>> -	 * Note smp_send_stop is the usual smp shutdown function, which
> > >>> -	 * unfortunately means it may not be hardened to work in a panic
> > >>> -	 * situation.
> > >>> -	 */
> > >>> -	smp_send_stop();
> > >>> +		/*
> > >>> +		 * Note smp_send_stop is the usual smp shutdown function, which
> > >>> +		 * unfortunately means it may not be hardened to work in a
> > >>> +		 * panic situation.
> > >>> +		 */
> > >>> +		smp_send_stop();
> > >>> +	} else {
> > >>> +		/*
> > >>> +		 * If we want to do crash dump after notifier calls and
> > >>> +		 * kmsg_dump, we will need architecture dependent extra
> > >>> +		 * works in addition to stopping other CPUs.
> > >>> +		 */
> > >>> +		crash_smp_send_stop();
> > >>> +	}
> > >>>
> > >>>  	/*
> > >>>  	 * Run any panic handlers, including those that might need to
> > >>>
> > >>>

Re: [V4 PATCH 1/2] x86/panic: Replace smp_send_stop() with kdump friendly version in panic path

['Dave Young']

Hi, 河合英宏

Thanks for the patch log update, it looks good to me.

Acked-by: Dave Young <dyoung@redhat.com>

On 09/20/16 at 11:22am, 河合英宏 / KAWAI，HIDEHIRO wrote:
> Here is the revised commit description reflecting Dave's
> comment.  Cc list was copied from -mm version.
> 
> From: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> Subject: x86/panic: replace smp_send_stop() with kdump friendly version in panic path
> 
> This patch fixes a problem reported by Daniel Walker
> (https://lkml.org/lkml/2015/6/24/44).
> 
> When kernel panics with crash_kexec_post_notifiers kernel parameter
> enabled, other CPUs are stopped by smp_send_stop() instead of
> machine_crash_shutdown() in __crash_kexec() path.
> 
>   panic()
>     if crash_kexec_post_notifiers == 1
>       smp_send_stop()
>       atomic_notifier_call_chain()
>       kmsg_dump()
>     __crash_kexec()
>       machine_crash_shutdown()
> 
> Different from smp_send_stop(), machine_crash_shutdown() stops other
> CPUs with extra works for kdump.  So, if smp_send_stop() stops other
> CPUs in advance, these extra works won't be done.  For x86, kdump
> routines miss to save other CPUs' registers and disable virtualization
> extensions.
> 
> To fix this problem, call a new kdump friendly function,
> crash_smp_send_stop(), instead of the smp_send_stop() when
> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> weak function, and it just call smp_send_stop().  Architecture
> codes should override it so that kdump can work appropriately.
> This patch only provides x86-specific version.
> 
> For Xen's PV kernel, just keep the current behavior.
> As for Dom0, it doesn't use crash_kexec routines, and it relies on
> panic notifier chain.  At the end of the chain, a hypercall is
> issued which requests the hypervisor to execute kdump.  This means
> regardless of crash_kexec_post_notifiers setting, smp_send_stop().
> For PV HVM, it would work similarly to baremetal kernels with extra
> cleanups for hypervisor.  It doesn't need additional care.
> 
> Changes in V4:
> - Keep to use smp_send_stop if crash_kexec_post_notifiers is not set
> - Rename panic_smp_send_stop to crash_smp_send_stop
> - Don't change the behavior for Xen's PV kernel
> 
> Changes in V3:
> - Revise comments, description, and symbol names
> 
> Changes in V2:
> - Replace smp_send_stop() call with crash_kexec version which
>   saves cpu states and cleans up VMX/SVM
> - Drop a fix for Problem 1 at this moment
> 
> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> Link: http://lkml.kernel.org/r/20160810080948.11028.15344.stgit@sysi4-13.yrl.intra.hitachi.co.jp
> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> Reported-by: Daniel Walker <dwalker@fifo99.com>
> Cc: Dave Young <dyoung@redhat.com>
> Cc: Baoquan He <bhe@redhat.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> Cc: Eric Biederman <ebiederm@xmission.com>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Daniel Walker <dwalker@fifo99.com>
> Cc: Xunlei Pang <xpang@redhat.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Borislav Petkov <bp@suse.de>
> Cc: David Vrabel <david.vrabel@citrix.com>
> Cc: Toshi Kani <toshi.kani@hpe.com>
> Cc: Ralf Baechle <ralf@linux-mips.org>
> Cc: David Daney <david.daney@cavium.com>
> Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
> Cc: "Steven J. Hill" <steven.hill@cavium.com>
> Cc: Corey Minyard <cminyard@mvista.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> 

[snip]

Thanks
Dave

[V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Hidehiro Kawai]

Daniel Walker reported problems which happens when
crash_kexec_post_notifiers kernel option is enabled
(https://lkml.org/lkml/2015/6/24/44).

In that case, smp_send_stop() is called before entering kdump routines
which assume other CPUs are still online.  As the result, kdump
routines fail to save other CPUs' registers.  Additionally for MIPS
OCTEON, it misses to stop the watchdog timer.

To fix this problem, call a new kdump friendly function,
crash_smp_send_stop(), instead of the smp_send_stop() when
crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
weak function, and it just call smp_send_stop().  Architecture
codes should override it so that kdump can work appropriately.
This patch provides MIPS version.

Reported-by: Daniel Walker <dwalker@fifo99.com>
Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: David Daney <david.daney@cavium.com>
Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
Cc: "Steven J. Hill" <steven.hill@cavium.com>
Cc: Corey Minyard <cminyard@mvista.com>

---
I'm not familiar with MIPS, and I don't have a test environment and
just did build tests only.  Please don't apply this patch until
someone does enough tests, otherwise simply drop this patch.
---
 arch/mips/cavium-octeon/setup.c  |   14 ++++++++++++++
 arch/mips/include/asm/kexec.h    |    1 +
 arch/mips/kernel/crash.c         |   18 +++++++++++++++++-
 arch/mips/kernel/machine_kexec.c |    1 +
 4 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c
index cb16fcc..5537f95 100644
--- a/arch/mips/cavium-octeon/setup.c
+++ b/arch/mips/cavium-octeon/setup.c
@@ -267,6 +267,17 @@ static void octeon_crash_shutdown(struct pt_regs *regs)
 	default_machine_crash_shutdown(regs);
 }
 
+#ifdef CONFIG_SMP
+void octeon_crash_smp_send_stop(void)
+{
+	int cpu;
+
+	/* disable watchdogs */
+	for_each_online_cpu(cpu)
+		cvmx_write_csr(CVMX_CIU_WDOGX(cpu_logical_map(cpu)), 0);
+}
+#endif
+
 #endif /* CONFIG_KEXEC */
 
 #ifdef CONFIG_CAVIUM_RESERVE32
@@ -911,6 +922,9 @@ void __init prom_init(void)
 	_machine_kexec_shutdown = octeon_shutdown;
 	_machine_crash_shutdown = octeon_crash_shutdown;
 	_machine_kexec_prepare = octeon_kexec_prepare;
+#ifdef CONFIG_SMP
+	_crash_smp_send_stop = octeon_crash_smp_send_stop;
+#endif
 #endif
 
 	octeon_user_io_init();
diff --git a/arch/mips/include/asm/kexec.h b/arch/mips/include/asm/kexec.h
index ee25ebb..493a3cc 100644
--- a/arch/mips/include/asm/kexec.h
+++ b/arch/mips/include/asm/kexec.h
@@ -45,6 +45,7 @@ extern const unsigned char kexec_smp_wait[];
 extern unsigned long secondary_kexec_args[4];
 extern void (*relocated_kexec_smp_wait) (void *);
 extern atomic_t kexec_ready_to_reboot;
+extern void (*_crash_smp_send_stop)(void);
 #endif
 #endif
 
diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
index 610f0f3..1723b17 100644
--- a/arch/mips/kernel/crash.c
+++ b/arch/mips/kernel/crash.c
@@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
 
 static void crash_kexec_prepare_cpus(void)
 {
+	static int cpus_stopped;
 	unsigned int msecs;
+	unsigned int ncpus;
 
-	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
+	if (cpus_stopped)
+		return;
+
+	ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
 
 	dump_send_ipi(crash_shutdown_secondary);
 	smp_wmb();
@@ -64,6 +69,17 @@ static void crash_kexec_prepare_cpus(void)
 		cpu_relax();
 		mdelay(1);
 	}
+
+	cpus_stopped = 1;
+}
+
+/* Override the weak function in kernel/panic.c */
+void crash_smp_send_stop(void)
+{
+	if (_crash_smp_send_stop)
+		_crash_smp_send_stop();
+
+	crash_kexec_prepare_cpus();
 }
 
 #else /* !defined(CONFIG_SMP)  */
diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
index 50980bf3..5972520 100644
--- a/arch/mips/kernel/machine_kexec.c
+++ b/arch/mips/kernel/machine_kexec.c
@@ -25,6 +25,7 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
 #ifdef CONFIG_SMP
 void (*relocated_kexec_smp_wait) (void *);
 atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
+void (*_crash_smp_send_stop)(void) = NULL;
 #endif
 
 int

Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Dave Young]

On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> Daniel Walker reported problems which happens when
> crash_kexec_post_notifiers kernel option is enabled
> (https://lkml.org/lkml/2015/6/24/44).
> 
> In that case, smp_send_stop() is called before entering kdump routines
> which assume other CPUs are still online.  As the result, kdump
> routines fail to save other CPUs' registers.  Additionally for MIPS
> OCTEON, it misses to stop the watchdog timer.
> 
> To fix this problem, call a new kdump friendly function,
> crash_smp_send_stop(), instead of the smp_send_stop() when
> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> weak function, and it just call smp_send_stop().  Architecture
> codes should override it so that kdump can work appropriately.
> This patch provides MIPS version.
> 
> Reported-by: Daniel Walker <dwalker@fifo99.com>
> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> Cc: Ralf Baechle <ralf@linux-mips.org>
> Cc: David Daney <david.daney@cavium.com>
> Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
> Cc: "Steven J. Hill" <steven.hill@cavium.com>
> Cc: Corey Minyard <cminyard@mvista.com>
> 
> ---
> I'm not familiar with MIPS, and I don't have a test environment and
> just did build tests only.  Please don't apply this patch until
> someone does enough tests, otherwise simply drop this patch.
> ---
>  arch/mips/cavium-octeon/setup.c  |   14 ++++++++++++++
>  arch/mips/include/asm/kexec.h    |    1 +
>  arch/mips/kernel/crash.c         |   18 +++++++++++++++++-
>  arch/mips/kernel/machine_kexec.c |    1 +
>  4 files changed, 33 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c
> index cb16fcc..5537f95 100644
> --- a/arch/mips/cavium-octeon/setup.c
> +++ b/arch/mips/cavium-octeon/setup.c
> @@ -267,6 +267,17 @@ static void octeon_crash_shutdown(struct pt_regs *regs)
>  	default_machine_crash_shutdown(regs);
>  }
>  
> +#ifdef CONFIG_SMP
> +void octeon_crash_smp_send_stop(void)
> +{
> +	int cpu;
> +
> +	/* disable watchdogs */
> +	for_each_online_cpu(cpu)
> +		cvmx_write_csr(CVMX_CIU_WDOGX(cpu_logical_map(cpu)), 0);
> +}
> +#endif
> +
>  #endif /* CONFIG_KEXEC */
>  
>  #ifdef CONFIG_CAVIUM_RESERVE32
> @@ -911,6 +922,9 @@ void __init prom_init(void)
>  	_machine_kexec_shutdown = octeon_shutdown;
>  	_machine_crash_shutdown = octeon_crash_shutdown;
>  	_machine_kexec_prepare = octeon_kexec_prepare;
> +#ifdef CONFIG_SMP
> +	_crash_smp_send_stop = octeon_crash_smp_send_stop;
> +#endif
>  #endif
>  
>  	octeon_user_io_init();
> diff --git a/arch/mips/include/asm/kexec.h b/arch/mips/include/asm/kexec.h
> index ee25ebb..493a3cc 100644
> --- a/arch/mips/include/asm/kexec.h
> +++ b/arch/mips/include/asm/kexec.h
> @@ -45,6 +45,7 @@ extern const unsigned char kexec_smp_wait[];
>  extern unsigned long secondary_kexec_args[4];
>  extern void (*relocated_kexec_smp_wait) (void *);
>  extern atomic_t kexec_ready_to_reboot;
> +extern void (*_crash_smp_send_stop)(void);
>  #endif
>  #endif
>  
> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
> index 610f0f3..1723b17 100644
> --- a/arch/mips/kernel/crash.c
> +++ b/arch/mips/kernel/crash.c
> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
>  
>  static void crash_kexec_prepare_cpus(void)
>  {
> +	static int cpus_stopped;
>  	unsigned int msecs;
> +	unsigned int ncpus;
>  
> -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
> +	if (cpus_stopped)
> +		return;
> +
> +	ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
>  
>  	dump_send_ipi(crash_shutdown_secondary);
>  	smp_wmb();
> @@ -64,6 +69,17 @@ static void crash_kexec_prepare_cpus(void)
>  		cpu_relax();
>  		mdelay(1);
>  	}
> +
> +	cpus_stopped = 1;
> +}
> +
> +/* Override the weak function in kernel/panic.c */
> +void crash_smp_send_stop(void)
> +{
> +	if (_crash_smp_send_stop)
> +		_crash_smp_send_stop();
> +
> +	crash_kexec_prepare_cpus();
>  }
>  
>  #else /* !defined(CONFIG_SMP)  */
> diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
> index 50980bf3..5972520 100644
> --- a/arch/mips/kernel/machine_kexec.c
> +++ b/arch/mips/kernel/machine_kexec.c
> @@ -25,6 +25,7 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
>  #ifdef CONFIG_SMP
>  void (*relocated_kexec_smp_wait) (void *);
>  atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
> +void (*_crash_smp_send_stop)(void) = NULL;
>  #endif
>  
>  int
> 
> 

Can any mips people review this patch and have a test?

Thanks
Dave

Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Corey Minyard]

I'll try to test this, but I have one comment inline...

On 08/11/2016 10:17 PM, Dave Young wrote:
> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
>> Daniel Walker reported problems which happens when
>> crash_kexec_post_notifiers kernel option is enabled
>> (https://lkml.org/lkml/2015/6/24/44).
>>
>> In that case, smp_send_stop() is called before entering kdump routines
>> which assume other CPUs are still online.  As the result, kdump
>> routines fail to save other CPUs' registers.  Additionally for MIPS
>> OCTEON, it misses to stop the watchdog timer.
>>
>> To fix this problem, call a new kdump friendly function,
>> crash_smp_send_stop(), instead of the smp_send_stop() when
>> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
>> weak function, and it just call smp_send_stop().  Architecture
>> codes should override it so that kdump can work appropriately.
>> This patch provides MIPS version.
>>
>> Reported-by: Daniel Walker <dwalker@fifo99.com>
>> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
>> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
>> Cc: Ralf Baechle <ralf@linux-mips.org>
>> Cc: David Daney <david.daney@cavium.com>
>> Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
>> Cc: "Steven J. Hill" <steven.hill@cavium.com>
>> Cc: Corey Minyard <cminyard@mvista.com>
>>
>> ---
>> I'm not familiar with MIPS, and I don't have a test environment and
>> just did build tests only.  Please don't apply this patch until
>> someone does enough tests, otherwise simply drop this patch.
>> ---
>>   arch/mips/cavium-octeon/setup.c  |   14 ++++++++++++++
>>   arch/mips/include/asm/kexec.h    |    1 +
>>   arch/mips/kernel/crash.c         |   18 +++++++++++++++++-
>>   arch/mips/kernel/machine_kexec.c |    1 +
>>   4 files changed, 33 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c
>> index cb16fcc..5537f95 100644
>> --- a/arch/mips/cavium-octeon/setup.c
>> +++ b/arch/mips/cavium-octeon/setup.c
>> @@ -267,6 +267,17 @@ static void octeon_crash_shutdown(struct pt_regs *regs)
>>   	default_machine_crash_shutdown(regs);
>>   }
>>   
>> +#ifdef CONFIG_SMP
>> +void octeon_crash_smp_send_stop(void)
>> +{
>> +	int cpu;
>> +
>> +	/* disable watchdogs */
>> +	for_each_online_cpu(cpu)
>> +		cvmx_write_csr(CVMX_CIU_WDOGX(cpu_logical_map(cpu)), 0);
>> +}
>> +#endif
>> +
>>   #endif /* CONFIG_KEXEC */
>>   
>>   #ifdef CONFIG_CAVIUM_RESERVE32
>> @@ -911,6 +922,9 @@ void __init prom_init(void)
>>   	_machine_kexec_shutdown = octeon_shutdown;
>>   	_machine_crash_shutdown = octeon_crash_shutdown;
>>   	_machine_kexec_prepare = octeon_kexec_prepare;
>> +#ifdef CONFIG_SMP
>> +	_crash_smp_send_stop = octeon_crash_smp_send_stop;
>> +#endif
>>   #endif
>>   
>>   	octeon_user_io_init();
>> diff --git a/arch/mips/include/asm/kexec.h b/arch/mips/include/asm/kexec.h
>> index ee25ebb..493a3cc 100644
>> --- a/arch/mips/include/asm/kexec.h
>> +++ b/arch/mips/include/asm/kexec.h
>> @@ -45,6 +45,7 @@ extern const unsigned char kexec_smp_wait[];
>>   extern unsigned long secondary_kexec_args[4];
>>   extern void (*relocated_kexec_smp_wait) (void *);
>>   extern atomic_t kexec_ready_to_reboot;
>> +extern void (*_crash_smp_send_stop)(void);
>>   #endif
>>   #endif
>>   
>> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
>> index 610f0f3..1723b17 100644
>> --- a/arch/mips/kernel/crash.c
>> +++ b/arch/mips/kernel/crash.c
>> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
>>   
>>   static void crash_kexec_prepare_cpus(void)
>>   {
>> +	static int cpus_stopped;
>>   	unsigned int msecs;
>> +	unsigned int ncpus;
>>   
>> -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
>> +	if (cpus_stopped)
>> +		return;

Wouldn't you want an atomic operation and some special handling here to
ensure that only one CPU does this?  So if a CPU comes in here and
another CPU is already in the process stopping the CPUs it won't result in a
deadlock.

-corey

>> +
>> +	ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
>>   
>>   	dump_send_ipi(crash_shutdown_secondary);
>>   	smp_wmb();
>> @@ -64,6 +69,17 @@ static void crash_kexec_prepare_cpus(void)
>>   		cpu_relax();
>>   		mdelay(1);
>>   	}
>> +
>> +	cpus_stopped = 1;
>> +}
>> +
>> +/* Override the weak function in kernel/panic.c */
>> +void crash_smp_send_stop(void)
>> +{
>> +	if (_crash_smp_send_stop)
>> +		_crash_smp_send_stop();
>> +
>> +	crash_kexec_prepare_cpus();
>>   }
>>   
>>   #else /* !defined(CONFIG_SMP)  */
>> diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
>> index 50980bf3..5972520 100644
>> --- a/arch/mips/kernel/machine_kexec.c
>> +++ b/arch/mips/kernel/machine_kexec.c
>> @@ -25,6 +25,7 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
>>   #ifdef CONFIG_SMP
>>   void (*relocated_kexec_smp_wait) (void *);
>>   atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
>> +void (*_crash_smp_send_stop)(void) = NULL;
>>   #endif
>>   
>>   int
>>
>>
> Can any mips people review this patch and have a test?
>
> Thanks
> Dave
>

RE: Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

Hi Corey,

> From: Corey Minyard [mailto:cminyard@mvista.com]
> Sent: Friday, August 12, 2016 10:56 PM
> I'll try to test this, but I have one comment inline...

Thank you very much!

> On 08/11/2016 10:17 PM, Dave Young wrote:
> > On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
[snip]
> >> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
> >> index 610f0f3..1723b17 100644
> >> --- a/arch/mips/kernel/crash.c
> >> +++ b/arch/mips/kernel/crash.c
> >> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
> >>
> >>   static void crash_kexec_prepare_cpus(void)
> >>   {
> >> +	static int cpus_stopped;
> >>   	unsigned int msecs;
> >> +	unsigned int ncpus;
> >>
> >> -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
> >> +	if (cpus_stopped)
> >> +		return;
> 
> Wouldn't you want an atomic operation and some special handling here to
> ensure that only one CPU does this?  So if a CPU comes in here and
> another CPU is already in the process stopping the CPUs it won't result in a
> deadlock.

Because this function can be called only one panicking CPU,
there is no problem.

There are two paths which crash_kexec_prepare_cpus is called.

Path 1 (panic path):
panic()
  crash_smp_send_stop()
    crash_kexec_prepare_cpus()

Path 2 (oops path):
crash_kexec()
  __crash_kexec()
    machine_crash_shutdown()
      default_machine_crash_shutdown() // for MIPS
        crash_kexec_prepare_cpus()

Here, panic() and crash_kexec() run exclusively via
panic_cpu atomic variable.  So we can use cpus_stopped as
normal variable.

Best regards,

Hidehiro Kawai

Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Corey Minyard]

On 08/15/2016 06:35 AM, 河合英宏 / KAWAI，HIDEHIRO wrote:
> Hi Corey,
>
>> From: Corey Minyard [mailto:cminyard@mvista.com]
>> Sent: Friday, August 12, 2016 10:56 PM
>> I'll try to test this, but I have one comment inline...
> Thank you very much!
>
>> On 08/11/2016 10:17 PM, Dave Young wrote:
>>> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> [snip]
>>>> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
>>>> index 610f0f3..1723b17 100644
>>>> --- a/arch/mips/kernel/crash.c
>>>> +++ b/arch/mips/kernel/crash.c
>>>> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
>>>>
>>>>    static void crash_kexec_prepare_cpus(void)
>>>>    {
>>>> +	static int cpus_stopped;
>>>>    	unsigned int msecs;
>>>> +	unsigned int ncpus;
>>>>
>>>> -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
>>>> +	if (cpus_stopped)
>>>> +		return;
>> Wouldn't you want an atomic operation and some special handling here to
>> ensure that only one CPU does this?  So if a CPU comes in here and
>> another CPU is already in the process stopping the CPUs it won't result in a
>> deadlock.
> Because this function can be called only one panicking CPU,
> there is no problem.
>
> There are two paths which crash_kexec_prepare_cpus is called.
>
> Path 1 (panic path):
> panic()
>    crash_smp_send_stop()
>      crash_kexec_prepare_cpus()
>
> Path 2 (oops path):
> crash_kexec()
>    __crash_kexec()
>      machine_crash_shutdown()
>        default_machine_crash_shutdown() // for MIPS
>          crash_kexec_prepare_cpus()
>
> Here, panic() and crash_kexec() run exclusively via
> panic_cpu atomic variable.  So we can use cpus_stopped as
> normal variable.

Ok, if the code can only be entered once, what's the purpose of 
cpus_stopped?
I guess that's what confused me.  You are right, the panic_cpu atomic should
keep this on a single CPU.

Also, panic() will call panic_smp_self_stop() if it finds another CPU 
has already
called panic, which will just spin with interrupts off by default. I 
didn't see a
definition for it in MIPS, wouldn't it need to be overridden to avoid a 
deadlock?

-corey

>
> Best regards,
>
> Hidehiro Kawai
>

Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Corey Minyard]

On 08/15/2016 12:06 PM, Corey Minyard wrote:
> On 08/15/2016 06:35 AM, 河合英宏 / KAWAI，HIDEHIRO wrote:
>> Hi Corey,
>>
>>> From: Corey Minyard [mailto:cminyard@mvista.com]
>>> Sent: Friday, August 12, 2016 10:56 PM
>>> I'll try to test this, but I have one comment inline...
>> Thank you very much!
>>
>>> On 08/11/2016 10:17 PM, Dave Young wrote:
>>>> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
>> [snip]
>>>>> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
>>>>> index 610f0f3..1723b17 100644
>>>>> --- a/arch/mips/kernel/crash.c
>>>>> +++ b/arch/mips/kernel/crash.c
>>>>> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void 
>>>>> *passed_regs)
>>>>>
>>>>>    static void crash_kexec_prepare_cpus(void)
>>>>>    {
>>>>> +    static int cpus_stopped;
>>>>>        unsigned int msecs;
>>>>> +    unsigned int ncpus;
>>>>>
>>>>> -    unsigned int ncpus = num_online_cpus() - 1;/* Excluding the 
>>>>> panic cpu */
>>>>> +    if (cpus_stopped)
>>>>> +        return;
>>> Wouldn't you want an atomic operation and some special handling here to
>>> ensure that only one CPU does this?  So if a CPU comes in here and
>>> another CPU is already in the process stopping the CPUs it won't 
>>> result in a
>>> deadlock.
>> Because this function can be called only one panicking CPU,
>> there is no problem.
>>
>> There are two paths which crash_kexec_prepare_cpus is called.
>>
>> Path 1 (panic path):
>> panic()
>>    crash_smp_send_stop()
>>      crash_kexec_prepare_cpus()
>>
>> Path 2 (oops path):
>> crash_kexec()
>>    __crash_kexec()
>>      machine_crash_shutdown()
>>        default_machine_crash_shutdown() // for MIPS
>>          crash_kexec_prepare_cpus()
>>
>> Here, panic() and crash_kexec() run exclusively via
>> panic_cpu atomic variable.  So we can use cpus_stopped as
>> normal variable.
>
> Ok, if the code can only be entered once, what's the purpose of 
> cpus_stopped?
> I guess that's what confused me.  You are right, the panic_cpu atomic 
> should
> keep this on a single CPU.

Never mind, I see the path through panic() where that is required. My 
question
below still remains, though.

-corey

>
> Also, panic() will call panic_smp_self_stop() if it finds another CPU 
> has already
> called panic, which will just spin with interrupts off by default. I 
> didn't see a
> definition for it in MIPS, wouldn't it need to be overridden to avoid 
> a deadlock?
>
> -corey
>
>>
>> Best regards,
>>
>> Hidehiro Kawai
>>
>

RE: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

> From: Corey Minyard [mailto:cminyard@mvista.com]
> Sent: Tuesday, August 16, 2016 3:02 AM
> On 08/15/2016 12:06 PM, Corey Minyard wrote:
> > On 08/15/2016 06:35 AM, 河合英宏 / KAWAI，HIDEHIRO wrote:
> >> Hi Corey,
> >>
> >>> From: Corey Minyard [mailto:cminyard@mvista.com]
> >>> Sent: Friday, August 12, 2016 10:56 PM
> >>> I'll try to test this, but I have one comment inline...
> >> Thank you very much!
> >>
> >>> On 08/11/2016 10:17 PM, Dave Young wrote:
> >>>> On 08/10/16 at 05:09pm, Hidehiro Kawai wrote:
> >> [snip]
> >>>>> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
> >>>>> index 610f0f3..1723b17 100644
> >>>>> --- a/arch/mips/kernel/crash.c
> >>>>> +++ b/arch/mips/kernel/crash.c
> >>>>> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void
> >>>>> *passed_regs)
> >>>>>
> >>>>>    static void crash_kexec_prepare_cpus(void)
> >>>>>    {
> >>>>> +    static int cpus_stopped;
> >>>>>        unsigned int msecs;
> >>>>> +    unsigned int ncpus;
> >>>>>
> >>>>> -    unsigned int ncpus = num_online_cpus() - 1;/* Excluding the
> >>>>> panic cpu */
> >>>>> +    if (cpus_stopped)
> >>>>> +        return;
> >>> Wouldn't you want an atomic operation and some special handling here to
> >>> ensure that only one CPU does this?  So if a CPU comes in here and
> >>> another CPU is already in the process stopping the CPUs it won't
> >>> result in a
> >>> deadlock.
> >> Because this function can be called only one panicking CPU,
> >> there is no problem.
> >>
> >> There are two paths which crash_kexec_prepare_cpus is called.
> >>
> >> Path 1 (panic path):
> >> panic()
> >>    crash_smp_send_stop()
> >>      crash_kexec_prepare_cpus()
> >>
> >> Path 2 (oops path):
> >> crash_kexec()
> >>    __crash_kexec()
> >>      machine_crash_shutdown()
> >>        default_machine_crash_shutdown() // for MIPS
> >>          crash_kexec_prepare_cpus()
> >>
> >> Here, panic() and crash_kexec() run exclusively via
> >> panic_cpu atomic variable.  So we can use cpus_stopped as
> >> normal variable.
> >
> > Ok, if the code can only be entered once, what's the purpose of
> > cpus_stopped?
> > I guess that's what confused me.  You are right, the panic_cpu atomic
> > should
> > keep this on a single CPU.
> 
> Never mind, I see the path through panic() where that is required. My
> question
> below still remains, though.
> 
> > Also, panic() will call panic_smp_self_stop() if it finds another CPU
> > has already
> > called panic, which will just spin with interrupts off by default. I
> > didn't see a
> > definition for it in MIPS, wouldn't it need to be overridden to avoid
> > a deadlock?

No deadlock should happen. Panicking CPU calls crash_kexec_prepare_cpus(),
and it issues an IPI and wait for other CPUs handle it.  If some of them
are looping in panic_smp_self_stop() with interrupt disabled, they can't
handle the IPI.  But it's not a severe problem.  crash_kexec_prepare_cpus()
has a timeout mechanism, and it will go out from the wait loop when it
happens.

In that case, of course, their registers are not saved.  This could be
improved, but I'd like to entrust MIPS experts with the improvement.
This is another issue.

Best regards,

Hidehiro Kawai

Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[Corey Minyard]

Sorry this took so long, but I have finally tested this, it seems to 
work fine:

Tested-by: Corey Minyard <cminyard@mvista.com>
Reviewed-by: Corey Minyard <cminyard@mvista.com>

On 08/10/2016 03:09 AM, Hidehiro Kawai wrote:
> Daniel Walker reported problems which happens when
> crash_kexec_post_notifiers kernel option is enabled
> (https://lkml.org/lkml/2015/6/24/44).
>
> In that case, smp_send_stop() is called before entering kdump routines
> which assume other CPUs are still online.  As the result, kdump
> routines fail to save other CPUs' registers.  Additionally for MIPS
> OCTEON, it misses to stop the watchdog timer.
>
> To fix this problem, call a new kdump friendly function,
> crash_smp_send_stop(), instead of the smp_send_stop() when
> crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> weak function, and it just call smp_send_stop().  Architecture
> codes should override it so that kdump can work appropriately.
> This patch provides MIPS version.
>
> Reported-by: Daniel Walker <dwalker@fifo99.com>
> Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> Cc: Ralf Baechle <ralf@linux-mips.org>
> Cc: David Daney <david.daney@cavium.com>
> Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
> Cc: "Steven J. Hill" <steven.hill@cavium.com>
> Cc: Corey Minyard <cminyard@mvista.com>
>
> ---
> I'm not familiar with MIPS, and I don't have a test environment and
> just did build tests only.  Please don't apply this patch until
> someone does enough tests, otherwise simply drop this patch.
> ---
>   arch/mips/cavium-octeon/setup.c  |   14 ++++++++++++++
>   arch/mips/include/asm/kexec.h    |    1 +
>   arch/mips/kernel/crash.c         |   18 +++++++++++++++++-
>   arch/mips/kernel/machine_kexec.c |    1 +
>   4 files changed, 33 insertions(+), 1 deletion(-)
>
> diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c
> index cb16fcc..5537f95 100644
> --- a/arch/mips/cavium-octeon/setup.c
> +++ b/arch/mips/cavium-octeon/setup.c
> @@ -267,6 +267,17 @@ static void octeon_crash_shutdown(struct pt_regs *regs)
>   	default_machine_crash_shutdown(regs);
>   }
>   
> +#ifdef CONFIG_SMP
> +void octeon_crash_smp_send_stop(void)
> +{
> +	int cpu;
> +
> +	/* disable watchdogs */
> +	for_each_online_cpu(cpu)
> +		cvmx_write_csr(CVMX_CIU_WDOGX(cpu_logical_map(cpu)), 0);
> +}
> +#endif
> +
>   #endif /* CONFIG_KEXEC */
>   
>   #ifdef CONFIG_CAVIUM_RESERVE32
> @@ -911,6 +922,9 @@ void __init prom_init(void)
>   	_machine_kexec_shutdown = octeon_shutdown;
>   	_machine_crash_shutdown = octeon_crash_shutdown;
>   	_machine_kexec_prepare = octeon_kexec_prepare;
> +#ifdef CONFIG_SMP
> +	_crash_smp_send_stop = octeon_crash_smp_send_stop;
> +#endif
>   #endif
>   
>   	octeon_user_io_init();
> diff --git a/arch/mips/include/asm/kexec.h b/arch/mips/include/asm/kexec.h
> index ee25ebb..493a3cc 100644
> --- a/arch/mips/include/asm/kexec.h
> +++ b/arch/mips/include/asm/kexec.h
> @@ -45,6 +45,7 @@ extern const unsigned char kexec_smp_wait[];
>   extern unsigned long secondary_kexec_args[4];
>   extern void (*relocated_kexec_smp_wait) (void *);
>   extern atomic_t kexec_ready_to_reboot;
> +extern void (*_crash_smp_send_stop)(void);
>   #endif
>   #endif
>   
> diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
> index 610f0f3..1723b17 100644
> --- a/arch/mips/kernel/crash.c
> +++ b/arch/mips/kernel/crash.c
> @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
>   
>   static void crash_kexec_prepare_cpus(void)
>   {
> +	static int cpus_stopped;
>   	unsigned int msecs;
> +	unsigned int ncpus;
>   
> -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
> +	if (cpus_stopped)
> +		return;
> +
> +	ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
>   
>   	dump_send_ipi(crash_shutdown_secondary);
>   	smp_wmb();
> @@ -64,6 +69,17 @@ static void crash_kexec_prepare_cpus(void)
>   		cpu_relax();
>   		mdelay(1);
>   	}
> +
> +	cpus_stopped = 1;
> +}
> +
> +/* Override the weak function in kernel/panic.c */
> +void crash_smp_send_stop(void)
> +{
> +	if (_crash_smp_send_stop)
> +		_crash_smp_send_stop();
> +
> +	crash_kexec_prepare_cpus();
>   }
>   
>   #else /* !defined(CONFIG_SMP)  */
> diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
> index 50980bf3..5972520 100644
> --- a/arch/mips/kernel/machine_kexec.c
> +++ b/arch/mips/kernel/machine_kexec.c
> @@ -25,6 +25,7 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
>   #ifdef CONFIG_SMP
>   void (*relocated_kexec_smp_wait) (void *);
>   atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
> +void (*_crash_smp_send_stop)(void) = NULL;
>   #endif
>   
>   int
>
>

RE: Re: [V4 PATCH 2/2] mips/panic: Replace smp_send_stop() with kdump friendly version in panic path

[河合英宏 / KAWAI，HIDEHIRO]

Dave Young suggested to me to explain the problem in more detail,
so here is the revised commit description.  The patch is now in -mm,
so I copied Cc list from -mm version.  Also I added Corey Minyard's
Tested-by and Reviewed-by.

From: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Subject: mips/panic: replace smp_send_stop() with kdump friendly version in panic path

This patch fixes the problems reported by Daniel Walker
(https://lkml.org/lkml/2015/6/24/44).

When kernel panics with crash_kexec_post_notifiers kernel parameter
enabled, other CPUs are stopped by smp_send_stop() instead of
machine_crash_shutdown() in __crash_kexec() path.

  panic()
    if crash_kexec_post_notifiers == 1
      smp_send_stop()
      atomic_notifier_call_chain()
      kmsg_dump()
    __crash_kexec()
      machine_crash_shutdown()
        octeon_generic_shutdown() // shutdown watchdog for ONLINE CPUs

Different from smp_send_stop(), machine_crash_shutdown() stops other
CPUs with extra works for kdump.  So, if smp_send_stop() stops other
CPUs in advance, these extra works won't be done.  As the result,
kdump routines miss to save other CPUs' registers.  Additionally for
MIPS OCTEON, it misses to stop the watchdog timer.

To fix this problem, call a new kdump friendly function,
crash_smp_send_stop(), instead of the smp_send_stop() when
crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
weak function, and it just call smp_send_stop().  Architecture
codes should override it so that kdump can work appropriately.
This patch provides MIPS version.

Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
Link: http://lkml.kernel.org/r/20160810080950.11028.28000.stgit@sysi4-13.yrl.intra.hitachi.co.jp
Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Reported-by: Daniel Walker <dwalker@fifo99.com>
Tested-by: Corey Minyard <cminyard@mvista.com>
Reviewed-by: Corey Minyard <cminyard@mvista.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: Eric Biederman <ebiederm@xmission.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Daniel Walker <dwalker@fifo99.com>
Cc: Xunlei Pang <xpang@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Toshi Kani <toshi.kani@hpe.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: David Daney <david.daney@cavium.com>
Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
Cc: "Steven J. Hill" <steven.hill@cavium.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

> From: Corey Minyard [mailto:cminyard@mvista.com]
> Sent: Friday, August 19, 2016 6:18 AM
> Sorry this took so long, but I have finally tested this, it seems to
> work fine:
> 
> Tested-by: Corey Minyard <cminyard@mvista.com>
> Reviewed-by: Corey Minyard <cminyard@mvista.com>
> 
> On 08/10/2016 03:09 AM, Hidehiro Kawai wrote:
> > Daniel Walker reported problems which happens when
> > crash_kexec_post_notifiers kernel option is enabled
> > (https://lkml.org/lkml/2015/6/24/44).
> >
> > In that case, smp_send_stop() is called before entering kdump routines
> > which assume other CPUs are still online.  As the result, kdump
> > routines fail to save other CPUs' registers.  Additionally for MIPS
> > OCTEON, it misses to stop the watchdog timer.
> >
> > To fix this problem, call a new kdump friendly function,
> > crash_smp_send_stop(), instead of the smp_send_stop() when
> > crash_kexec_post_notifiers is enabled.  crash_smp_send_stop() is a
> > weak function, and it just call smp_send_stop().  Architecture
> > codes should override it so that kdump can work appropriately.
> > This patch provides MIPS version.
> >
> > Reported-by: Daniel Walker <dwalker@fifo99.com>
> > Fixes: f06e5153f4ae (kernel/panic.c: add "crash_kexec_post_notifiers" option)
> > Signed-off-by: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
> > Cc: Ralf Baechle <ralf@linux-mips.org>
> > Cc: David Daney <david.daney@cavium.com>
> > Cc: Aaro Koskinen <aaro.koskinen@iki.fi>
> > Cc: "Steven J. Hill" <steven.hill@cavium.com>
> > Cc: Corey Minyard <cminyard@mvista.com>
> >
> > ---
> > I'm not familiar with MIPS, and I don't have a test environment and
> > just did build tests only.  Please don't apply this patch until
> > someone does enough tests, otherwise simply drop this patch.
> > ---
> >   arch/mips/cavium-octeon/setup.c  |   14 ++++++++++++++
> >   arch/mips/include/asm/kexec.h    |    1 +
> >   arch/mips/kernel/crash.c         |   18 +++++++++++++++++-
> >   arch/mips/kernel/machine_kexec.c |    1 +
> >   4 files changed, 33 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c
> > index cb16fcc..5537f95 100644
> > --- a/arch/mips/cavium-octeon/setup.c
> > +++ b/arch/mips/cavium-octeon/setup.c
> > @@ -267,6 +267,17 @@ static void octeon_crash_shutdown(struct pt_regs *regs)
> >   	default_machine_crash_shutdown(regs);
> >   }
> >
> > +#ifdef CONFIG_SMP
> > +void octeon_crash_smp_send_stop(void)
> > +{
> > +	int cpu;
> > +
> > +	/* disable watchdogs */
> > +	for_each_online_cpu(cpu)
> > +		cvmx_write_csr(CVMX_CIU_WDOGX(cpu_logical_map(cpu)), 0);
> > +}
> > +#endif
> > +
> >   #endif /* CONFIG_KEXEC */
> >
> >   #ifdef CONFIG_CAVIUM_RESERVE32
> > @@ -911,6 +922,9 @@ void __init prom_init(void)
> >   	_machine_kexec_shutdown = octeon_shutdown;
> >   	_machine_crash_shutdown = octeon_crash_shutdown;
> >   	_machine_kexec_prepare = octeon_kexec_prepare;
> > +#ifdef CONFIG_SMP
> > +	_crash_smp_send_stop = octeon_crash_smp_send_stop;
> > +#endif
> >   #endif
> >
> >   	octeon_user_io_init();
> > diff --git a/arch/mips/include/asm/kexec.h b/arch/mips/include/asm/kexec.h
> > index ee25ebb..493a3cc 100644
> > --- a/arch/mips/include/asm/kexec.h
> > +++ b/arch/mips/include/asm/kexec.h
> > @@ -45,6 +45,7 @@ extern const unsigned char kexec_smp_wait[];
> >   extern unsigned long secondary_kexec_args[4];
> >   extern void (*relocated_kexec_smp_wait) (void *);
> >   extern atomic_t kexec_ready_to_reboot;
> > +extern void (*_crash_smp_send_stop)(void);
> >   #endif
> >   #endif
> >
> > diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
> > index 610f0f3..1723b17 100644
> > --- a/arch/mips/kernel/crash.c
> > +++ b/arch/mips/kernel/crash.c
> > @@ -47,9 +47,14 @@ static void crash_shutdown_secondary(void *passed_regs)
> >
> >   static void crash_kexec_prepare_cpus(void)
> >   {
> > +	static int cpus_stopped;
> >   	unsigned int msecs;
> > +	unsigned int ncpus;
> >
> > -	unsigned int ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
> > +	if (cpus_stopped)
> > +		return;
> > +
> > +	ncpus = num_online_cpus() - 1;/* Excluding the panic cpu */
> >
> >   	dump_send_ipi(crash_shutdown_secondary);
> >   	smp_wmb();
> > @@ -64,6 +69,17 @@ static void crash_kexec_prepare_cpus(void)
> >   		cpu_relax();
> >   		mdelay(1);
> >   	}
> > +
> > +	cpus_stopped = 1;
> > +}
> > +
> > +/* Override the weak function in kernel/panic.c */
> > +void crash_smp_send_stop(void)
> > +{
> > +	if (_crash_smp_send_stop)
> > +		_crash_smp_send_stop();
> > +
> > +	crash_kexec_prepare_cpus();
> >   }
> >
> >   #else /* !defined(CONFIG_SMP)  */
> > diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
> > index 50980bf3..5972520 100644
> > --- a/arch/mips/kernel/machine_kexec.c
> > +++ b/arch/mips/kernel/machine_kexec.c
> > @@ -25,6 +25,7 @@ void (*_machine_crash_shutdown)(struct pt_regs *regs) = NULL;
> >   #ifdef CONFIG_SMP
> >   void (*relocated_kexec_smp_wait) (void *);
> >   atomic_t kexec_ready_to_reboot = ATOMIC_INIT(0);
> > +void (*_crash_smp_send_stop)(void) = NULL;
> >   #endif
> >
> >   int
> >
> >