From: William Weston <weston@sysex.net>
To: Ingo Molnar <mingo@elte.hu>
Cc: "K.R. Foley" <kr@cybsft.com>,
linux-kernel@vger.kernel.org,
"Eugeny S. Mints" <emints@ru.mvista.com>,
Daniel Walker <dwalker@mvista.com>
Subject: Re: [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.48-00
Date: Mon, 27 Jun 2005 20:32:37 -0700 (PDT) [thread overview]
Message-ID: <Pine.LNX.4.58.0506272001190.5720@echo.lysdexia.org> (raw)
In-Reply-To: <20050627081542.GA15096@elte.hu>
On Mon, 27 Jun 2005, Ingo Molnar wrote:
> > I still haven't been able to get any NMI watchdog traces with the
> > lockups induced by VLC and burnP6. Early printk is enabled on the
> > serial console. I have noticed, however, that scheduling performance
> > slowly degrades during the ~1 minute before locking up. Once I was
> > able to get a delayed SysRq response (~30s) from the serial console
> > after the X console became unresponsive. Is this possibly a scheduler
> > starvation issue that affects everything, including the NMI watchdog?
> > Any more suggestions for catching a trace?
>
> hm. Nothing should starve the NMI watchdog. Only a nasty, complete
> kernel crash or a hardware failure can lock up the box in a way that not
> even the NMI watchdog can get some message out to the console. Just in
> case, could you try the latest patch (-50-24 or later), there were a
> couple of bugs fixed.
OK. Running on -50-25 now. The burnP6 starvation doesn't seem to affect
the whole system, but comes close enough to require the reset button every
time. I usually, but not always, lose network, X, the keyboard, mouse,
and serial console. I'm still unable to get any sort of a trace from
these lockups, since it's looking more like a bunch of processes starving
than a kernel crash or a full lockup.
Once, with VLC (viewing a 5mbit/s mcast/udp stream) and two burnP6
instances running, I was able to fire up top on the serial console and
found out that the IRQ thread for the ns83820 nic was using 99% of one
CPU.
Once, two burnP6 instances made the system unresponsive without running
VLC. Everything froze as soon as I fired the second one up. This may be
a good one to try to reproduce on an HT machine, since it doesn't require
VLC and a multicast video stream to make the system unresponsive.
Once, with a normal desktop load and a yum update, this came across on the
serial console:
cat/2100[CPU#1]: BUG in update_out_trace at kernel/latency.c:587
[<c01041b3>] dump_stack+0x23/0x30 (20)
[<c0122e0b>] __WARN_ON+0x6b/0x90 (52)
[<c0140530>] update_out_trace+0x3d0/0x440 (104)
[<c0140881>] l_stax2e1/0x310 (72)
[<c0191eab>] seq_read+0x7b/0x2d0 (60)
[<c016ecd4>] vfs_read+0xd4/0x140 (36)
[<c016efb0>] sys_read+0x50/0x80 (44)
[<c01031c0>] sysenter_past_esp+0x61/0x89 (-8116)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c038dd9f>] .... _raw_spin_lock_irqsave+0x1f/0xb0
.....[<c0122dc7>] .. ( <= __WARN_ON+0x27/0x90)
.. [<c014280d>] .... print_traces+0x1d/0x60
.....[<c01040b2>] .. ( <= show_trace+0xc2/0xf0)
------------------------------
| showing all locks held by: | (cat/2100 [deca9320, 117]):
------------------------------
#001: [d9ac47a4] {(struct semaphore *)(&p->sem)}
... acquired at: seq_read+0x2b/0x2d0
#002: [c03fb264] {out_mutex.lock}
... acquired at: l_start+0x1b/0x310
#003: [c04d1404] {max_mutex.lock}
... acquired at: l_start+0x2dc/0x310
CPU0: 000000abfe71626e (000000abfe71654e) ... #3 (000000abfe71654e) 000000abfe7176be
CPU1: 000000abfe6d3f0e (000000abfe6d4efe) ... #41 (000000abfe70fa46) 000000abfe71227e
CPU1 entries: 41
first stamp: 000000abfe71626e
last stamp: 000000abfe71626e
cat/2101[CPU#1]: BUG in update_out_trace at kernel/latency.c:587
[<c01041b3>] dump_stack+0x23/0x30 (20)
[<c0122e0b>] __WARN_ON+0x6b/0x90 (52)
[<c0140530>] update_out_trace+0x3d0/0x440 (104)
[<c0140881>] l_start+0x2e1/0x310 (72)
[<c0191eab>] seq_read+0x7b/0x2d0 (60)
[<c016ecd4>] vfs_read+0xd4/0x140 (36)
[<c016efb0>] sys_read+0x50/0x80 (44)
[<c01031c0>] sysenter_past_esp+0x61/0x89 (-8116)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c038dd9f>] .... _raw_spin_lock_irqsave+0x1f/0xb0
.....[<c0122dc7>] .. ( <= __WARN_ON+0x27/0x90)
.. [<c014280d>] .... print_traces+0x1d/0x60
.....[<c01040b2>] .. ( <= show_trace+0xc2/0xf0)
------------------------------
| showing all locks held by: | (cat/2101 [deca9320, 116]):
------------------------------
#001: [d9ac47a4] {(struct semaphore *)(&p->sem)}
... acquired at: seq_read+0x2b/0x2d0
#002: [c04d1404] {max_mutex.lock}
... acquired at: l_start+0x2dc/0x310
CPU0: 000000abfe71626e (000000abfe71654e) ... #3 (00000071654e) 000000abfe7176be
CPU1: 000000abfe6d3f0e (000000abfe6d4efe) ... #41 (000000abfe70fa46) 000000abfe71227e
CPU1 entries: 41
first stamp: 000000abfe71626e
last stamp: 000000abfe71626e
Best Regards,
--ww
next prev parent reply other threads:[~2005-06-28 3:35 UTC|newest]
Thread overview: 139+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-06-08 11:28 [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.48-00 Ingo Molnar
2005-06-08 14:18 ` Michal Schmidt
2005-06-08 14:40 ` Keith Owens
2005-06-08 14:41 ` Paulo Marques
2005-06-08 15:04 ` Michal Schmidt
2005-06-08 15:23 ` Keith Owens
2005-06-08 15:56 ` Paulo Marques
2005-06-08 14:32 ` Michal Schmidt
2005-06-08 15:48 ` K.R. Foley
2005-06-08 16:08 ` K.R. Foley
2005-06-08 16:26 ` K.R. Foley
2005-06-08 17:26 ` K.R. Foley
2005-06-08 19:18 ` Ingo Molnar
2005-06-09 11:39 ` Ingo Molnar
2005-06-10 13:34 ` K.R. Foley
2005-06-09 8:47 ` Serge Noiraud
2005-06-09 11:55 ` Ingo Molnar
2005-06-09 15:12 ` Serge Noiraud
2005-06-09 16:34 ` Daniel Walker
2005-06-13 8:53 ` Serge Noiraud
2005-06-13 9:07 ` Ingo Molnar
2005-06-08 17:51 ` Michal Schmidt
2005-06-09 11:45 ` Ingo Molnar
2005-06-09 12:09 ` Michal Schmidt
2005-06-09 12:13 ` Ingo Molnar
2005-06-09 12:48 ` Michal Schmidt
2005-06-10 8:56 ` Esben Nielsen
2005-06-08 19:58 ` Daniel Walker
2005-06-09 21:24 ` Michal Schmidt
2005-06-09 21:34 ` Michal Schmidt
2005-06-11 7:32 ` Ingo Molnar
2005-06-11 8:10 ` Ingo Molnar
2005-06-11 16:35 ` Peter Zijlstra
2005-06-11 18:48 ` Ingo Molnar
2005-06-11 21:14 ` Peter Zijlstra
2005-06-12 1:40 ` Gene Heskett
2005-06-12 6:49 ` Ingo Molnar
2005-06-12 9:02 ` Gene Heskett
2005-06-12 10:35 ` Ingo Molnar
2005-06-12 13:40 ` Gene Heskett
2005-06-12 13:49 ` Ingo Molnar
2005-06-12 14:23 ` Gene Heskett
2005-06-13 2:11 ` Gene Heskett
2005-06-13 6:09 ` Ingo Molnar
2005-06-13 8:37 ` Gene Heskett
2005-06-14 18:24 ` K.R. Foley
2005-06-14 18:54 ` Ingo Molnar
2005-06-14 22:45 ` Daniel Walker
2005-06-15 6:12 ` Ingo Molnar
2005-06-15 9:13 ` Esben Nielsen
2005-06-14 20:37 ` Paul E. McKenney
2005-06-16 3:51 ` K.R. Foley
2005-06-16 7:29 ` Ingo Molnar
2005-06-16 11:22 ` K.R. Foley
2005-06-16 17:32 ` Ingo Molnar
2005-06-16 17:59 ` K.R. Foley
2005-06-16 20:37 ` K.R. Foley
2005-06-17 11:08 ` Ingo Molnar
2005-06-16 20:43 ` Ingo Molnar
2005-06-17 2:06 ` Gene Heskett
2005-06-17 4:53 ` Gene Heskett
2005-06-17 11:18 ` Ingo Molnar
2005-06-17 14:12 ` K.R. Foley
2005-06-17 15:33 ` K.R. Foley
2005-06-17 19:28 ` William Weston
2005-06-17 21:30 ` William Weston
2005-06-18 12:28 ` Ingo Molnar
2005-06-21 1:18 ` William Weston
2005-06-23 2:05 ` William Weston
2005-06-23 7:56 ` Ingo Molnar
2005-06-23 13:45 ` Gene Heskett
2005-06-21 13:10 ` Ingo Molnar
2005-06-21 19:08 ` William Weston
2005-06-21 20:17 ` Benjamin LaHaise
2005-06-21 20:37 ` William Weston
2005-06-21 20:48 ` Benjamin LaHaise
2005-06-22 2:42 ` Gene Heskett
2005-06-22 7:40 ` Ingo Molnar
2005-06-22 13:27 ` Gene Heskett
2005-06-22 13:51 ` Gene Heskett
2005-06-22 14:08 ` Gene Heskett
2005-06-25 4:41 ` Ingo Molnar
2005-06-25 4:47 ` Ingo Molnar
2005-06-25 5:39 ` Gene Heskett
2005-06-25 7:26 ` Gene Heskett
2005-06-25 9:12 ` Ingo Molnar
2005-06-25 13:19 ` Gene Heskett
2005-06-25 14:39 ` Gene Heskett
2005-06-27 19:01 ` Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24 Chuck Harding
2005-06-27 19:42 ` Chuck Harding
2005-06-27 21:08 ` Gene Heskett
2005-06-27 20:09 ` Daniel Walker
2005-06-27 20:28 ` Chuck Harding
2005-06-28 0:50 ` Daniel Walker
2005-06-28 7:53 ` Steven Rostedt
2005-06-28 8:18 ` Ingo Molnar
2005-06-28 8:34 ` Steven Rostedt
2005-06-28 9:12 ` Ingo Molnar
2005-06-28 15:26 ` Michal Schmidt
2005-06-28 15:31 ` Ingo Molnar
2005-06-28 18:50 ` Chuck Harding
2005-06-28 22:16 ` Chuck Harding
2005-06-29 0:32 ` Chuck Harding
2005-06-28 15:54 ` Daniel Walker
2005-06-21 13:12 ` [patch] Real-Time Preemption, -RT-2.6.12-rc6-V0.7.48-00 Ingo Molnar
[not found] ` <Pine.LNX.4.58.0506211228210.16701@echo.lysdexia.org>
2005-06-22 8:03 ` Ingo Molnar
2005-06-22 8:24 ` Ingo Molnar
2005-06-22 10:08 ` Ingo Molnar
2005-06-22 21:53 ` William Weston
2005-06-22 22:00 ` Ingo Molnar
2005-06-22 23:06 ` William Weston
2005-06-23 0:10 ` Ingo Molnar
2005-06-23 22:11 ` William Weston
2005-06-24 1:46 ` William Weston
2005-06-24 7:06 ` Ingo Molnar
2005-06-24 22:31 ` William Weston
2005-06-25 4:14 ` Ingo Molnar
2005-06-27 4:21 ` William Weston
2005-06-27 8:15 ` Ingo Molnar
2005-06-28 3:32 ` William Weston [this message]
2005-06-28 8:10 ` Ingo Molnar
2005-06-28 11:15 ` Ingo Molnar
[not found] ` <Pine.LNX.4.58.0506281745040.10406@echo.lysdexia.org>
2005-06-29 12:54 ` Ingo Molnar
2005-06-30 0:29 ` William Weston
2005-06-27 5:43 ` Gene Heskett
2005-06-27 8:17 ` Ingo Molnar
2005-06-27 17:29 ` Gene Heskett
2005-06-27 19:54 ` Ingo Molnar
2005-06-27 21:17 ` Gene Heskett
2005-06-28 8:28 ` Ingo Molnar
2005-06-08 14:02 kus Kusche Klaus
2005-06-08 14:14 ` Thomas Gleixner
2005-06-08 15:49 ` Daniel Walker
2005-06-08 15:50 ` Ingo Molnar
2005-06-08 16:23 ` Thomas Gleixner
2005-06-08 17:29 ` Daniel Walker
2005-06-08 19:24 ` Ingo Molnar
2005-06-12 9:29 Zoltan Boszormenyi
2005-06-12 10:39 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.58.0506272001190.5720@echo.lysdexia.org \
--to=weston@sysex.net \
--cc=dwalker@mvista.com \
--cc=emints@ru.mvista.com \
--cc=kr@cybsft.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).