* [PATCH] printk: fixing the deadlock when calling printk in nmi handle
@ 2012-07-04 13:00 Liu, Chuansheng
2012-07-04 13:22 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Liu, Chuansheng @ 2012-07-04 13:00 UTC (permalink / raw)
To: 'linux-kernel@vger.kernel.org' (linux-kernel@vger.kernel.org)
Cc: a.p.zijlstra, kay, gregkh, mingo
From: liu chuansheng <chuansheng.liu@intel.com>
Subject: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
Current printk implementation can not fully support that
calling it in nmi handler for SMP arch.
There is typical case in nmi handler function arch_trigger_all_cpu_backtrace_handler().
In my platform, there are 2 CPUs, when function arch_trigger_all_cpu_backtrace()
is called, 2 CPUs will recevied the nmi interrupts, and the
arch_trigger_all_cpu_backtrace_handler() will called on 2 CPUs:
case1:
CPU0 CPU1
calling arch_trigger_all_cpu_backtrace() calling printk, and has obtain the logbuf_lock
nmi interrupt received nmi interrupt received
call arch_trigger_all_cpu_backtrace_handler() call arch_trigger_all_cpu_backtrace_handler()
Obtain arch_spin_lock(&lock); Waiting for arch_spin_lock(&lock);
Continue to call printk()
CPU0 will be blocked by logbuf_lock CPU1 is blocked by arch_spin_lock(&lock)
The deadlock will be happening.
case2:
CPU0 CPU1:(run dmesg command)
calling arch_trigger_all_cpu_backtrace() calling do_syslog
Obtaining the logbuf_lock
nmi interrupt received nmi interrupt received
....
The dealock will happen also somtimes.
I just write a simple interface to run the arch_trigger_all_cpu_backtrace_handler() every 5s,
it will trigger dead lock many times.
The solution is when printk is called in nmi handler, we will use trylock instead of lock.
And in nmi handler, do the call the console write function because normal console write function
include many spin locks also. This fix can confirm the traces in nmi handler can be output successfully
almost.
Signed-off-by: liu chuansheng <chuansheng.liu@intel.com>
---
kernel/printk.c | 11 +++++++++--
1 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/kernel/printk.c b/kernel/printk.c
index dba1821..de68e24 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -1275,7 +1275,7 @@ static int console_trylock_for_printk(unsigned int cpu)
{
int retval = 0, wake = 0;
- if (console_trylock()) {
+ if (!in_nmi() && console_trylock()) {
retval = 1;
/*
@@ -1432,7 +1432,13 @@ asmlinkage int vprintk_emit(int facility, int level,
}
lockdep_off();
- raw_spin_lock(&logbuf_lock);
+ if(unlikely(in_nmi())) {
+ if(!raw_spin_trylock(&logbuf_lock))
+ goto out_restore_lockdep_irqs;
+ } else {
+ raw_spin_lock(&logbuf_lock);
+ }
+
logbuf_cpu = this_cpu;
if (recursion_bug) {
@@ -1524,6 +1530,7 @@ asmlinkage int vprintk_emit(int facility, int level,
if (console_trylock_for_printk(this_cpu))
console_unlock();
+out_restore_lockdep_irqs:
lockdep_on();
out_restore_irqs:
local_irq_restore(flags);
--
1.7.0.4
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 13:00 [PATCH] printk: fixing the deadlock when calling printk in nmi handle Liu, Chuansheng
@ 2012-07-04 13:22 ` Peter Zijlstra
2012-07-04 13:30 ` Liu, Chuansheng
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 13:22 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 13:00 +0000, Liu, Chuansheng wrote:
> From: liu chuansheng <chuansheng.liu@intel.com>
> Subject: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
>
> Current printk implementation can not fully support that
> calling it in nmi handler for SMP arch.
>
> There is typical case in nmi handler function arch_trigger_all_cpu_backtrace_handler().
>
> In my platform, there are 2 CPUs, when function arch_trigger_all_cpu_backtrace()
> is called, 2 CPUs will recevied the nmi interrupts, and the
> arch_trigger_all_cpu_backtrace_handler() will called on 2 CPUs:
>
> case1:
> CPU0 CPU1
> calling arch_trigger_all_cpu_backtrace() calling printk, and has obtain the logbuf_lock
> nmi interrupt received nmi interrupt received
> call arch_trigger_all_cpu_backtrace_handler() call arch_trigger_all_cpu_backtrace_handler()
> Obtain arch_spin_lock(&lock); Waiting for arch_spin_lock(&lock);
> Continue to call printk()
> CPU0 will be blocked by logbuf_lock CPU1 is blocked by arch_spin_lock(&lock)
>
> The deadlock will be happening.
>
> case2:
> CPU0 CPU1:(run dmesg command)
> calling arch_trigger_all_cpu_backtrace() calling do_syslog
> Obtaining the logbuf_lock
> nmi interrupt received nmi interrupt received
> ....
> The dealock will happen also somtimes.
>
> I just write a simple interface to run the arch_trigger_all_cpu_backtrace_handler() every 5s,
> it will trigger dead lock many times.
>
> The solution is when printk is called in nmi handler, we will use trylock instead of lock.
> And in nmi handler, do the call the console write function because normal console write function
> include many spin locks also. This fix can confirm the traces in nmi handler can be output successfully
> almost.
>
> Signed-off-by: liu chuansheng <chuansheng.liu@intel.com>
Yuck.. and no. This makes sane things like early 8250 serial console
less reliable.
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 13:22 ` Peter Zijlstra
@ 2012-07-04 13:30 ` Liu, Chuansheng
2012-07-04 13:39 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Liu, Chuansheng @ 2012-07-04 13:30 UTC (permalink / raw)
To: Peter Zijlstra
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 399 bytes --]
> Yuck.. and no. This makes sane things like early 8250 serial console less
> reliable.
Peter, could you share more? I took many time to dig out this issue.
Anyway, in nmi handler, calling printk is dangerous. This patch is for this case.
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 13:30 ` Liu, Chuansheng
@ 2012-07-04 13:39 ` Peter Zijlstra
2012-07-04 14:09 ` Peter Zijlstra
2012-07-04 14:46 ` Liu, Chuansheng
0 siblings, 2 replies; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 13:39 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 13:30 +0000, Liu, Chuansheng wrote:
> > Yuck.. and no. This makes sane things like early 8250 serial console less
> > reliable.
> Peter, could you share more?
You avoid calling console write, this delays writing the output, this
delay might be infinite if the machine is sick enough.
Things like the 8250 early console don't take locks and are fine to
call.
> I took many time to dig out this issue.
Well, it was a known issue..
> Anyway, in nmi handler, calling printk is dangerous. This patch is for this case.
But its ugly and decreases reliability.. Your 'solution' is to
basically ignore serialization and pray, that's not a solution, that's
gambling.
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 13:39 ` Peter Zijlstra
@ 2012-07-04 14:09 ` Peter Zijlstra
2012-07-04 14:12 ` Peter Zijlstra
2012-07-04 14:46 ` Liu, Chuansheng
1 sibling, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 14:09 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 15:39 +0200, Peter Zijlstra wrote:
> > Anyway, in nmi handler, calling printk is dangerous. This patch is for this case.
>
> But its ugly and decreases reliability.. Your 'solution' is to
> basically ignore serialization and pray, that's not a solution, that's
> gambling.
If you want to fix it, do something like:
http://marc.info/?l=linux-kernel&m=132446646923333
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 14:09 ` Peter Zijlstra
@ 2012-07-04 14:12 ` Peter Zijlstra
2012-07-04 14:18 ` Liu, Chuansheng
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 14:12 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 16:09 +0200, Peter Zijlstra wrote:
> On Wed, 2012-07-04 at 15:39 +0200, Peter Zijlstra wrote:
> > > Anyway, in nmi handler, calling printk is dangerous. This patch is for this case.
> >
> > But its ugly and decreases reliability.. Your 'solution' is to
> > basically ignore serialization and pray, that's not a solution, that's
> > gambling.
>
> If you want to fix it, do something like:
>
> http://marc.info/?l=linux-kernel&m=132446646923333
Note that the one 'down' side of this is that the folks who care about
logbuf content might get upset since we don't put all msgs in there
anymore.
These would be the kmsg_dump users..
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 14:12 ` Peter Zijlstra
@ 2012-07-04 14:18 ` Liu, Chuansheng
0 siblings, 0 replies; 11+ messages in thread
From: Liu, Chuansheng @ 2012-07-04 14:18 UTC (permalink / raw)
To: Peter Zijlstra
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 345 bytes --]
> Note that the one 'down' side of this is that the folks who care about logbuf
> content might get upset since we don't put all msgs in there anymore.
Got it, thanks your correction.
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 13:39 ` Peter Zijlstra
2012-07-04 14:09 ` Peter Zijlstra
@ 2012-07-04 14:46 ` Liu, Chuansheng
2012-07-04 14:50 ` Peter Zijlstra
1 sibling, 1 reply; 11+ messages in thread
From: Liu, Chuansheng @ 2012-07-04 14:46 UTC (permalink / raw)
To: Peter Zijlstra
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 408 bytes --]
> > I took many time to dig out this issue.
>
> Well, it was a known issue..
Still feel unfair :)
If this is a known issue that calling printk is not safe in nmi handler,
Why not defense it like this patch before real solution coming out?
Thanks.
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 14:46 ` Liu, Chuansheng
@ 2012-07-04 14:50 ` Peter Zijlstra
2012-07-04 14:52 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 14:50 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 14:46 +0000, Liu, Chuansheng wrote:
> > > I took many time to dig out this issue.
> >
> > Well, it was a known issue..
> Still feel unfair :)
> If this is a known issue that calling printk is not safe in nmi handler,
> Why not defense it like this patch before real solution coming out?
Why replace one broken thing with another broken thing?
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 14:50 ` Peter Zijlstra
@ 2012-07-04 14:52 ` Peter Zijlstra
2012-07-04 15:06 ` Liu, Chuansheng
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2012-07-04 14:52 UTC (permalink / raw)
To: Liu, Chuansheng
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
On Wed, 2012-07-04 at 16:50 +0200, Peter Zijlstra wrote:
> On Wed, 2012-07-04 at 14:46 +0000, Liu, Chuansheng wrote:
> > > > I took many time to dig out this issue.
> > >
> > > Well, it was a known issue..
> > Still feel unfair :)
> > If this is a known issue that calling printk is not safe in nmi handler,
> > Why not defense it like this patch before real solution coming out?
>
> Why replace one broken thing with another broken thing?
Also, I much prefer obviously broken over subtly broken.
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH] printk: fixing the deadlock when calling printk in nmi handle
2012-07-04 14:52 ` Peter Zijlstra
@ 2012-07-04 15:06 ` Liu, Chuansheng
0 siblings, 0 replies; 11+ messages in thread
From: Liu, Chuansheng @ 2012-07-04 15:06 UTC (permalink / raw)
To: Peter Zijlstra
Cc: 'linux-kernel@vger.kernel.org'
(linux-kernel@vger.kernel.org),
kay, gregkh, mingo
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1181 bytes --]
> -----Original Message-----
> From: Peter Zijlstra [mailto:peterz@infradead.org]
> Sent: Wednesday, July 04, 2012 10:52 PM
> To: Liu, Chuansheng
> Cc: 'linux-kernel@vger.kernel.org' (linux-kernel@vger.kernel.org); kay@vrfy.org;
> gregkh@linuxfoundation.org; mingo@elte.hu
> Subject: RE: [PATCH] printk: fixing the deadlock when calling printk in nmi
> handle
>
> On Wed, 2012-07-04 at 16:50 +0200, Peter Zijlstra wrote:
> > On Wed, 2012-07-04 at 14:46 +0000, Liu, Chuansheng wrote:
> > > > > I took many time to dig out this issue.
> > > >
> > > > Well, it was a known issue..
> > > Still feel unfair :)
> > > If this is a known issue that calling printk is not safe in nmi
> > > handler, Why not defense it like this patch before real solution coming out?
> >
> > Why replace one broken thing with another broken thing?
>
> Also, I much prefer obviously broken over subtly broken.
Supporting your opinion totally, but from production view, it is not a good thing that system crash,
I prefer to lose one log instead:)
ÿôèº{.nÇ+·®+%Ëÿ±éݶ\x17¥wÿº{.nÇ+·¥{±þG«éÿ{ayº\x1dÊÚë,j\a¢f£¢·hïêÿêçz_è®\x03(éÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?¨èÚ&£ø§~á¶iOæ¬z·vØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?I¥
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-07-04 15:06 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-07-04 13:00 [PATCH] printk: fixing the deadlock when calling printk in nmi handle Liu, Chuansheng
2012-07-04 13:22 ` Peter Zijlstra
2012-07-04 13:30 ` Liu, Chuansheng
2012-07-04 13:39 ` Peter Zijlstra
2012-07-04 14:09 ` Peter Zijlstra
2012-07-04 14:12 ` Peter Zijlstra
2012-07-04 14:18 ` Liu, Chuansheng
2012-07-04 14:46 ` Liu, Chuansheng
2012-07-04 14:50 ` Peter Zijlstra
2012-07-04 14:52 ` Peter Zijlstra
2012-07-04 15:06 ` Liu, Chuansheng
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.