[RFC PATCH 0/2] sysctl: Panic on RCU stall and schedule while atomic

[RFC PATCH 0/2] sysctl: Panic on RCU stall and schedule while atomic

[Daniel Bristot de Oliveira]

This patch series implements two kernel.panic_on_* like sysctl:

kernel.panic_on_rcu_stall:
	panic() on RCU Stall detection.

kernel.panic_on_sched_in_atomic:
	panic() on schedule while atomic detection.

These sysctls are useful to capture a vmcore when is not possible
to recompile the kernel to include the panic() call. For instance
when supporting enterprise users.

Daniel Bristot de Oliveira (2):
  rcu: sysctl: Panic on RCU Stall
  sched: sysctl: Panic on scheduling while atomic

 Documentation/sysctl/kernel.txt | 25 +++++++++++++++++++++++++
 include/linux/kernel.h          |  2 ++
 kernel/rcu/tree.c               |  8 ++++++++
 kernel/sched/core.c             |  7 +++++++
 kernel/sysctl.c                 | 20 ++++++++++++++++++++
 5 files changed, 62 insertions(+)

-- 
2.5.5

[RFC PATCH 1/2] rcu: sysctl: Panic on RCU Stall

[Daniel Bristot de Oliveira]

It is not always easy to define the cause of an RCU stall just by
analysing the RCU stall messages, mainly when the problem is caused
by the indirect starvation of rcu threads. For example, when preempt_rcu
is not awakened due to the starvation of a timer softirq.

We have been hard coding panic() in the RCU stall functions for
some time while testing the kernel-rt. But this is not possible in
some scenarios, like when supporting customers.

This patch implements the sysctl kerner.panic_on_rcu_stall. If
set to 1, the system will panic() when an RCU stall takes place,
enabling the capture of a vmcore. The vmcore provides a way to analyze
all kernel/tasks states, helping out to point to the culprit and the
solution for the stall.

The kerner.panic_on_rcu_stall sysctl is disabled by default.

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Reviewed-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Tested-by: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>
---
 Documentation/sysctl/kernel.txt | 12 ++++++++++++
 include/linux/kernel.h          |  1 +
 kernel/rcu/tree.c               |  8 ++++++++
 kernel/sysctl.c                 | 11 +++++++++++
 4 files changed, 32 insertions(+)

diff --git a/Documentation/sysctl/kernel.txt b/Documentation/sysctl/kernel.txt
index a3683ce..29ec4bb 100644
--- a/Documentation/sysctl/kernel.txt
+++ b/Documentation/sysctl/kernel.txt
@@ -58,6 +58,7 @@ show up in /proc/sys/kernel:
 - panic_on_stackoverflow
 - panic_on_unrecovered_nmi
 - panic_on_warn
+- panic_on_rcu_stall
 - perf_cpu_time_max_percent
 - perf_event_paranoid
 - perf_event_max_stack
@@ -618,6 +619,17 @@ a kernel rebuild when attempting to kdump at the location of a WARN().
 
 ==============================================================
 
+panic_on_rcu_stall:
+
+When set to 1, calls panic() after RCU stall detection messages. This
+is useful to define the root cause of RCU stalls using a vmcore.
+
+0: do not panic() when RCU stall takes place, default behavior.
+
+1: panic() after printing RCU stall messages.
+
+==============================================================
+
 perf_cpu_time_max_percent:
 
 Hints to the kernel how much CPU time it should be allowed to
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 94aa10f..c420821 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -451,6 +451,7 @@ extern int panic_on_oops;
 extern int panic_on_unrecovered_nmi;
 extern int panic_on_io_nmi;
 extern int panic_on_warn;
+extern int sysctl_panic_on_rcu_stall;
 extern int sysctl_panic_on_stackoverflow;
 
 extern bool crash_kexec_post_notifiers;
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c7f1bc4..296db5d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -125,6 +125,8 @@ int rcu_num_lvls __read_mostly = RCU_NUM_LVLS;
 /* Number of rcu_nodes at specified level. */
 static int num_rcu_lvl[] = NUM_RCU_LVL_INIT;
 int rcu_num_nodes __read_mostly = NUM_RCU_NODES; /* Total # rcu_nodes in use. */
+/* panic() on RCU Stall sysctl*/
+int sysctl_panic_on_rcu_stall;
 
 /*
  * The rcu_scheduler_active variable transitions from zero to one just
@@ -1390,6 +1392,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum)
 
 	rcu_check_gp_kthread_starvation(rsp);
 
+	if (sysctl_panic_on_rcu_stall)
+		panic("RCU Stall\n");
+
 	force_quiescent_state(rsp);  /* Kick them all. */
 }
 
@@ -1430,6 +1435,9 @@ static void print_cpu_stall(struct rcu_state *rsp)
 			   jiffies + 3 * rcu_jiffies_till_stall_check() + 3);
 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 
+	if (sysctl_panic_on_rcu_stall)
+		panic("RCU Stall\n");
+
 	/*
 	 * Attempt to revive the RCU machinery by forcing a context switch.
 	 *
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 87b2fc3..d3b93aa 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1205,6 +1205,17 @@ static struct ctl_table kern_table[] = {
 		.extra2		= &one,
 	},
 #endif
+#ifdef CONFIG_RCU_STALL_COMMON
+	{
+		.procname	= "panic_on_rcu_stall",
+		.data		= &sysctl_panic_on_rcu_stall,
+		.maxlen		= sizeof(sysctl_panic_on_rcu_stall),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
+#endif
 	{ }
 };
 
-- 
2.5.5

Re: [RFC PATCH 1/2] rcu: sysctl: Panic on RCU Stall

[Josh Triplett]

On Tue, May 31, 2016 at 04:07:32PM -0300, Daniel Bristot de Oliveira wrote:
> It is not always easy to define the cause of an RCU stall just by
> analysing the RCU stall messages, mainly when the problem is caused
> by the indirect starvation of rcu threads. For example, when preempt_rcu
> is not awakened due to the starvation of a timer softirq.
> 
> We have been hard coding panic() in the RCU stall functions for
> some time while testing the kernel-rt. But this is not possible in
> some scenarios, like when supporting customers.
> 
> This patch implements the sysctl kerner.panic_on_rcu_stall. If
> set to 1, the system will panic() when an RCU stall takes place,
> enabling the capture of a vmcore. The vmcore provides a way to analyze
> all kernel/tasks states, helping out to point to the culprit and the
> solution for the stall.
> 
> The kerner.panic_on_rcu_stall sysctl is disabled by default.

s/kerner/kernel/ (here and in the previous paragraph).

Also, even though it's only two lines, please consider creating a static
function wrapping the if and panic, to avoid duplication.

With those changes,
Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Re: [RFC PATCH 1/2] rcu: sysctl: Panic on RCU Stall

[Josh Triplett]

On Tue, May 31, 2016 at 12:18:27PM -0700, Josh Triplett wrote:
> On Tue, May 31, 2016 at 04:07:32PM -0300, Daniel Bristot de Oliveira wrote:
> > It is not always easy to define the cause of an RCU stall just by
> > analysing the RCU stall messages, mainly when the problem is caused
> > by the indirect starvation of rcu threads. For example, when preempt_rcu
> > is not awakened due to the starvation of a timer softirq.
> > 
> > We have been hard coding panic() in the RCU stall functions for
> > some time while testing the kernel-rt. But this is not possible in
> > some scenarios, like when supporting customers.
> > 
> > This patch implements the sysctl kerner.panic_on_rcu_stall. If
> > set to 1, the system will panic() when an RCU stall takes place,
> > enabling the capture of a vmcore. The vmcore provides a way to analyze
> > all kernel/tasks states, helping out to point to the culprit and the
> > solution for the stall.
> > 
> > The kerner.panic_on_rcu_stall sysctl is disabled by default.
> 
> s/kerner/kernel/ (here and in the previous paragraph).
> 
> Also, even though it's only two lines, please consider creating a static
> function wrapping the if and panic, to avoid duplication.
> 
> With those changes,
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Sorry, realized something else a moment after sending: I don't think
this will build if you use the tiny RCU implementation.  That
implementation *does* support tracing, and if you enable tracing,
you'll have CONFIG_RCU_STALL_COMMON=y, but you won't build tree.c where
the variable definition lives.  So, the sysctl code will reference a
variable that doesn't exist.

- Josh Triplett

Re: [RFC PATCH 1/2] rcu: sysctl: Panic on RCU Stall

[Daniel Bristot de Oliveira]

On 05/31/2016 04:23 PM, Josh Triplett wrote:

Hi Josh,

> Sorry, realized something else a moment after sending: I don't think
> this will build if you use the tiny RCU implementation.  That
> implementation *does* support tracing, and if you enable tracing,
> you'll have CONFIG_RCU_STALL_COMMON=y, but you won't build tree.c where
> the variable definition lives.  So, the sysctl code will reference a
> variable that doesn't exist.

Good catch!

[root@f23 linux]# egrep '(TINY_RCU|RCU_TRACE)' .config
CONFIG_TINY_RCU=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_TRACE=y
[root@f23 linux]# make
...
kernel/built-in.o:(.data+0x4688): undefined reference to `sysctl_panic_on_rcu_stall'
Makefile:937: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 1


How about this change in the condition:
-#ifdef CONFIG_RCU_STALL_COMMON
+#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)

It worked fine here, but I may be missing something, so I better
check with the experts :-).

Thanks for reviewing the patch set!

-- Daniel

Re: [RFC PATCH 1/2] rcu: sysctl: Panic on RCU Stall

[Paul E. McKenney]

On Tue, May 31, 2016 at 07:49:41PM -0300, Daniel Bristot de Oliveira wrote:
> On 05/31/2016 04:23 PM, Josh Triplett wrote:
> 
> Hi Josh,
> 
> > Sorry, realized something else a moment after sending: I don't think
> > this will build if you use the tiny RCU implementation.  That
> > implementation *does* support tracing, and if you enable tracing,
> > you'll have CONFIG_RCU_STALL_COMMON=y, but you won't build tree.c where
> > the variable definition lives.  So, the sysctl code will reference a
> > variable that doesn't exist.
> 
> Good catch!
> 
> [root@f23 linux]# egrep '(TINY_RCU|RCU_TRACE)' .config
> CONFIG_TINY_RCU=y
> # CONFIG_TREE_RCU_TRACE is not set
> CONFIG_RCU_TRACE=y
> [root@f23 linux]# make
> ...
> kernel/built-in.o:(.data+0x4688): undefined reference to `sysctl_panic_on_rcu_stall'
> Makefile:937: recipe for target 'vmlinux' failed
> make: *** [vmlinux] Error 1
> 
> 
> How about this change in the condition:
> -#ifdef CONFIG_RCU_STALL_COMMON
> +#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)

Looks plausible to me.  But what I would trust quite a bit more than my
own opinion would be successful builds for all three configurations.  ;-)

							Thanx, Paul

> It worked fine here, but I may be missing something, so I better
> check with the experts :-).
> 
> Thanks for reviewing the patch set!
> 
> -- Daniel
> 

[RFC PATCH 2/2] sched: sysctl: Panic on scheduling while atomic

[Daniel Bristot de Oliveira]

Currently, a schedule while atomic error prints the stack trace to the
kernel log and the system continue running.

Although it is possible to collect the kernel log messages and analyse
it, often more information are needed. Furthermore, keep the system
running is not always the best choice. For example, when the preempt
count underflows the system will not stop to complain about scheduling
while atomic, so the kernel log can wraparound overwriting the first
stack trace, tuning the analysis even more difficult.

This patch implements the kernel.panic_on_sched_in_atomic sysctl to
help out on these more complex situations.

When kernel.panic_on_sched_in_atomic is set to 1, the kernel will
panic() in the schedule while atomic detection.

The default value for the sysctl is 0, maintaining the current behavior.

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>
---
 Documentation/sysctl/kernel.txt | 13 +++++++++++++
 include/linux/kernel.h          |  1 +
 kernel/sched/core.c             |  7 +++++++
 kernel/sysctl.c                 |  9 +++++++++
 4 files changed, 30 insertions(+)

diff --git a/Documentation/sysctl/kernel.txt b/Documentation/sysctl/kernel.txt
index 29ec4bb..0b176bb 100644
--- a/Documentation/sysctl/kernel.txt
+++ b/Documentation/sysctl/kernel.txt
@@ -59,6 +59,7 @@ show up in /proc/sys/kernel:
 - panic_on_unrecovered_nmi
 - panic_on_warn
 - panic_on_rcu_stall
+- panic_on_sched_in_atomic
 - perf_cpu_time_max_percent
 - perf_event_paranoid
 - perf_event_max_stack
@@ -630,6 +631,18 @@ is useful to define the root cause of RCU stalls using kdump.
 
 ==============================================================
 
+panic_on_sched_in_atomic:
+
+When set to 1, calls panic() in the schedule while in atomic detection.
+This is useful get a vmcore to inspect the root cause of a schedule()
+call in the atomic context.
+
+0: do not panic() on scheduling while in atomic, default behavior.
+
+1: panic() after printing schedule while in atomic messages.
+
+==============================================================
+
 perf_cpu_time_max_percent:
 
 Hints to the kernel how much CPU time it should be allowed to
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index c420821..e4d9804 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -452,6 +452,7 @@ extern int panic_on_unrecovered_nmi;
 extern int panic_on_io_nmi;
 extern int panic_on_warn;
 extern int sysctl_panic_on_rcu_stall;
+extern int sysctl_panic_on_sched_in_atomic;
 extern int sysctl_panic_on_stackoverflow;
 
 extern bool crash_kexec_post_notifiers;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 7f2cae4..602f978 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -152,6 +152,11 @@ __read_mostly int scheduler_running;
  */
 int sysctl_sched_rt_runtime = 950000;
 
+/*
+ * panic on scheduling while atomic
+ */
+__read_mostly int sysctl_panic_on_sched_in_atomic = 0;
+
 /* cpus with isolated domains */
 cpumask_var_t cpu_isolated_map;
 
@@ -3146,6 +3151,8 @@ static noinline void __schedule_bug(struct task_struct *prev)
 		pr_cont("\n");
 	}
 #endif
+	if (sysctl_panic_on_sched_in_atomic)
+		panic("scheduling while atomic\n");
 	dump_stack();
 	add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
 }
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index d3b93aa..f0a984c 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1216,6 +1216,15 @@ static struct ctl_table kern_table[] = {
 		.extra2		= &one,
 	},
 #endif
+	{
+		.procname	= "panic_on_sched_in_atomic",
+		.data		= &sysctl_panic_on_sched_in_atomic,
+		.maxlen		= sizeof(sysctl_panic_on_sched_in_atomic),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
 	{ }
 };
 
-- 
2.5.5

Re: [RFC PATCH 2/2] sched: sysctl: Panic on scheduling while atomic

[Josh Triplett]

On Tue, May 31, 2016 at 04:07:33PM -0300, Daniel Bristot de Oliveira wrote:
> Currently, a schedule while atomic error prints the stack trace to the
> kernel log and the system continue running.
> 
> Although it is possible to collect the kernel log messages and analyse
> it, often more information are needed. Furthermore, keep the system
> running is not always the best choice. For example, when the preempt
> count underflows the system will not stop to complain about scheduling
> while atomic, so the kernel log can wraparound overwriting the first
> stack trace, tuning the analysis even more difficult.
> 
> This patch implements the kernel.panic_on_sched_in_atomic sysctl to
> help out on these more complex situations.
> 
> When kernel.panic_on_sched_in_atomic is set to 1, the kernel will
> panic() in the schedule while atomic detection.
> 
> The default value for the sysctl is 0, maintaining the current behavior.
> 
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Reviewed-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> Signed-off-by: Daniel Bristot de Oliveira <bristot@redhat.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  Documentation/sysctl/kernel.txt | 13 +++++++++++++
>  include/linux/kernel.h          |  1 +
>  kernel/sched/core.c             |  7 +++++++
>  kernel/sysctl.c                 |  9 +++++++++
>  4 files changed, 30 insertions(+)
> 
> diff --git a/Documentation/sysctl/kernel.txt b/Documentation/sysctl/kernel.txt
> index 29ec4bb..0b176bb 100644
> --- a/Documentation/sysctl/kernel.txt
> +++ b/Documentation/sysctl/kernel.txt
> @@ -59,6 +59,7 @@ show up in /proc/sys/kernel:
>  - panic_on_unrecovered_nmi
>  - panic_on_warn
>  - panic_on_rcu_stall
> +- panic_on_sched_in_atomic
>  - perf_cpu_time_max_percent
>  - perf_event_paranoid
>  - perf_event_max_stack
> @@ -630,6 +631,18 @@ is useful to define the root cause of RCU stalls using kdump.
>  
>  ==============================================================
>  
> +panic_on_sched_in_atomic:
> +
> +When set to 1, calls panic() in the schedule while in atomic detection.
> +This is useful get a vmcore to inspect the root cause of a schedule()
> +call in the atomic context.
> +
> +0: do not panic() on scheduling while in atomic, default behavior.
> +
> +1: panic() after printing schedule while in atomic messages.
> +
> +==============================================================
> +
>  perf_cpu_time_max_percent:
>  
>  Hints to the kernel how much CPU time it should be allowed to
> diff --git a/include/linux/kernel.h b/include/linux/kernel.h
> index c420821..e4d9804 100644
> --- a/include/linux/kernel.h
> +++ b/include/linux/kernel.h
> @@ -452,6 +452,7 @@ extern int panic_on_unrecovered_nmi;
>  extern int panic_on_io_nmi;
>  extern int panic_on_warn;
>  extern int sysctl_panic_on_rcu_stall;
> +extern int sysctl_panic_on_sched_in_atomic;
>  extern int sysctl_panic_on_stackoverflow;
>  
>  extern bool crash_kexec_post_notifiers;
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 7f2cae4..602f978 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -152,6 +152,11 @@ __read_mostly int scheduler_running;
>   */
>  int sysctl_sched_rt_runtime = 950000;
>  
> +/*
> + * panic on scheduling while atomic
> + */
> +__read_mostly int sysctl_panic_on_sched_in_atomic = 0;
> +
>  /* cpus with isolated domains */
>  cpumask_var_t cpu_isolated_map;
>  
> @@ -3146,6 +3151,8 @@ static noinline void __schedule_bug(struct task_struct *prev)
>  		pr_cont("\n");
>  	}
>  #endif
> +	if (sysctl_panic_on_sched_in_atomic)
> +		panic("scheduling while atomic\n");
>  	dump_stack();
>  	add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
>  }
> diff --git a/kernel/sysctl.c b/kernel/sysctl.c
> index d3b93aa..f0a984c 100644
> --- a/kernel/sysctl.c
> +++ b/kernel/sysctl.c
> @@ -1216,6 +1216,15 @@ static struct ctl_table kern_table[] = {
>  		.extra2		= &one,
>  	},
>  #endif
> +	{
> +		.procname	= "panic_on_sched_in_atomic",
> +		.data		= &sysctl_panic_on_sched_in_atomic,
> +		.maxlen		= sizeof(sysctl_panic_on_sched_in_atomic),
> +		.mode		= 0644,
> +		.proc_handler	= proc_dointvec_minmax,
> +		.extra1		= &zero,
> +		.extra2		= &one,
> +	},
>  	{ }
>  };
>  
> -- 
> 2.5.5
> 

Re: [RFC PATCH 2/2] sched: sysctl: Panic on scheduling while atomic

[Peter Zijlstra]

On Tue, May 31, 2016 at 04:07:33PM -0300, Daniel Bristot de Oliveira wrote:
> Currently, a schedule while atomic error prints the stack trace to the
> kernel log and the system continue running.
> 
> Although it is possible to collect the kernel log messages and analyse
> it, often more information are needed. Furthermore, keep the system
> running is not always the best choice. For example, when the preempt
> count underflows the system will not stop to complain about scheduling
> while atomic, so the kernel log can wraparound overwriting the first
> stack trace, tuning the analysis even more difficult.
> 
> This patch implements the kernel.panic_on_sched_in_atomic sysctl to
> help out on these more complex situations.
> 
> When kernel.panic_on_sched_in_atomic is set to 1, the kernel will
> panic() in the schedule while atomic detection.

Do we really need more panic_on_* knobs? Can't we re-purpose
panic_on_warn for this?

Re: [RFC PATCH 2/2] sched: sysctl: Panic on scheduling while atomic

[Daniel Bristot de Oliveira]

On 06/01/2016 06:45 AM, Peter Zijlstra wrote:
> Do we really need more panic_on_* knobs? Can't we re-purpose
> panic_on_warn for this?

I think this case is very specific, specific enough to deserve its own
sysctl. But I see your point, and the possibilities I can see are:

1) convert the printk(KERN_ERR "BUG:...") to a WARN(1,"BUG:..."), but
this will hide all other information printed in the function, so -1 for
this option;

2) fire a WARN in the end of the function with a message like the
previous printk(); or

3) re-use the panic_on_warn variable to condition the panic,
or to force a WARN.

But, IMHO, I think these are not as polish as use a specific sysctl.

If you think that any of these options are better than what I propose,
I can cook a v2 with the best option :-).

-- Daniel

Re: [RFC PATCH 0/2] sysctl: Panic on RCU stall and schedule while atomic

[Christian Borntraeger]

On 05/31/2016 09:07 PM, Daniel Bristot de Oliveira wrote:
> This patch series implements two kernel.panic_on_* like sysctl:
> 
> kernel.panic_on_rcu_stall:
> 	panic() on RCU Stall detection.
> 
> kernel.panic_on_sched_in_atomic:
> 	panic() on schedule while atomic detection.
> 
> These sysctls are useful to capture a vmcore when is not possible
> to recompile the kernel to include the panic() call. For instance
> when supporting enterprise users.
> 
> Daniel Bristot de Oliveira (2):
>   rcu: sysctl: Panic on RCU Stall
>   sched: sysctl: Panic on scheduling while atomic
> 
>  Documentation/sysctl/kernel.txt | 25 +++++++++++++++++++++++++
>  include/linux/kernel.h          |  2 ++
>  kernel/rcu/tree.c               |  8 ++++++++
>  kernel/sched/core.c             |  7 +++++++
>  kernel/sysctl.c                 | 20 ++++++++++++++++++++
>  5 files changed, 62 insertions(+)

Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>

for both changes. I too often had to add manual panics in a test
kernel and reproduce an issue. This will allow to take a (k)dump at a point
in time where the data structures might still explain what happened.
As this is an opt-in nobody, this should not cause any trouble for
people that do not want to take dumps.

Christian