* [PATCH] panic: Fix a possible deadlock in panic()
@ 2012-06-28 23:43 Vikram Mulukutla
2012-06-29 6:54 ` Michael Holzheu
2012-06-29 20:29 ` Andrew Morton
0 siblings, 2 replies; 3+ messages in thread
From: Vikram Mulukutla @ 2012-06-28 23:43 UTC (permalink / raw)
To: Andrew Morton
Cc: Vikram Mulukutla, Michael Holzheu, Stephen Boyd, linux-kernel
panic_lock is meant to ensure that panic processing takes
place only on one cpu; if any of the other cpus encounter
a panic, they will spin waiting to be shut down.
However, this causes a regression in this scenario:
1. Cpu 0 encounters a panic and acquires the panic_lock
and proceeds with the panic processing.
2. There is an interrupt on cpu 0 that also encounters
an error condition and invokes panic.
3. This second invocation fails to acquire the panic_lock
and enters the infinite while loop in panic_smp_self_stop.
Thus all panic processing is stopped, and the cpu is stuck
for eternity in the while(1) inside panic_smp_self_stop.
To address this, disable local interrupts with
local_irq_disable before acquiring the panic_lock. This will
prevent interrupt handlers from executing during the panic
processing, thus avoiding this particular problem.
Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
---
kernel/panic.c | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/kernel/panic.c b/kernel/panic.c
index d2a5f4e..e1b2822 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
int state = 0;
/*
+ * Disable local interrupts. This will prevent panic_smp_self_stop
+ * from deadlocking the first cpu that invokes the panic, since
+ * there is nothing to prevent an interrupt handler (that runs
+ * after the panic_lock is acquired) from invoking panic again.
+ */
+ local_irq_disable();
+
+ /*
* It's possible to come here directly from a panic-assertion and
* not have preempt disabled. Some functions called from here want
* preempt to be disabled. No point enabling it later though...
--
1.7.8.3
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] panic: Fix a possible deadlock in panic()
2012-06-28 23:43 [PATCH] panic: Fix a possible deadlock in panic() Vikram Mulukutla
@ 2012-06-29 6:54 ` Michael Holzheu
2012-06-29 20:29 ` Andrew Morton
1 sibling, 0 replies; 3+ messages in thread
From: Michael Holzheu @ 2012-06-29 6:54 UTC (permalink / raw)
To: Vikram Mulukutla; +Cc: Andrew Morton, Stephen Boyd, linux-kernel, linux-arch
Hello Vikram,
Putting "linux-arch" on cc...
On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <markivx@codeaurora.org> wrote:
> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
>
> However, this causes a regression in this scenario:
>
> 1. Cpu 0 encounters a panic and acquires the panic_lock
> and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
> an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
> and enters the infinite while loop in panic_smp_self_stop.
>
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
>
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.
Looks good to me.
I re-read the panic lock discussion and in fact one version of my patch
also disabled interrupts:
http://lists.infradead.org/pipermail/kexec/2011-October/005695.html
I think the reason why we later took a version with irqs enabled was
that we did not think about the scenario you described above and
we wanted to make the change as less intrusive as possible. But I am
not really sure about that.
Regarding you patch: Perhaps we could use spin_trylock_irq() instead of
local_irq_disable() and spin_lock().
Michael
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] panic: Fix a possible deadlock in panic()
2012-06-28 23:43 [PATCH] panic: Fix a possible deadlock in panic() Vikram Mulukutla
2012-06-29 6:54 ` Michael Holzheu
@ 2012-06-29 20:29 ` Andrew Morton
1 sibling, 0 replies; 3+ messages in thread
From: Andrew Morton @ 2012-06-29 20:29 UTC (permalink / raw)
To: Vikram Mulukutla; +Cc: Michael Holzheu, Stephen Boyd, linux-kernel
On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <markivx@codeaurora.org> wrote:
> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
>
> However, this causes a regression in this scenario:
>
> 1. Cpu 0 encounters a panic and acquires the panic_lock
> and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
> an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
> and enters the infinite while loop in panic_smp_self_stop.
>
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
>
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.
>
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
> int state = 0;
>
> /*
> + * Disable local interrupts. This will prevent panic_smp_self_stop
> + * from deadlocking the first cpu that invokes the panic, since
> + * there is nothing to prevent an interrupt handler (that runs
> + * after the panic_lock is acquired) from invoking panic again.
> + */
> + local_irq_disable();
> +
> + /*
> * It's possible to come here directly from a panic-assertion and
> * not have preempt disabled. Some functions called from here want
> * preempt to be disabled. No point enabling it later though...
Seems sane. panic() *should* work correctly when called with
interrupts disabled, so there be no bad effects from internally
disabling interrupts. If there are bad effects, we should fix them up.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-06-29 20:29 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-28 23:43 [PATCH] panic: Fix a possible deadlock in panic() Vikram Mulukutla
2012-06-29 6:54 ` Michael Holzheu
2012-06-29 20:29 ` Andrew Morton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.