[parisc-linux] Re: Expect defunct, kill -9 panics kernel?

[parisc-linux] Re: Expect defunct, kill -9 panics kernel?

[John David Anglin]

> Is this the usual behaviour you see?
> 
> 1. I run the gcc testsuite.
> 2. expect dies, leaving a defunct process.
> 3. Killing another expect panics the kernel.

It similar to the behavior that I see.  I don't usually see this
with expect though.  Possibly, this is because I use my own build
of expect linked tcl8.3.

I see this behavior quite consistently on my c3750 if I

1.  Run the gcc libjava testsuite.
2.  Usually, there a set of processes (e.g., Process_3) left running
    after the testsuite ends.  These processes are not defunct and
    load the processor.  I can kill all but the oldest thread.
3.  Killing the oldest thread panics the kernel.  Sometimes the system
    reboots.  However, the system often hangs doing endless panics.

I suspect a timing issue as the c3750 is the fastest processor that
I test on.  I don't see as many problems with the libjava testsuite
on slower hardware.  At one time, I thought this might be a 32 versus
64-bit issue, but I see the same problems running a 64-bit kernel.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[James Bottomley]

On Sat, 2007-02-10 at 12:16 -0500, Carlos O'Donell wrote:
> At what point in the process life are we in __wake_up and
> __wake_up_common?
> An address of 0x10 is very suspicious.

Almost every internal kernel event or semaphore uses these.

Because of the empty backtrace, I'd be inclined to say it was the
scheduler, possibly.

0x10 looks to be curr->func implying curr is NULL and thus the queue
task_list is corrupt.

That's the best I can do without the kernel to pull apart.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[John David Anglin]

> Right, now here's a bit of really useful detective work:
> 
> In the same piece of disassembly can you see what happens to %r26 ...
> the first argument to __wake_up_common() which is the wait queue?  It
> may be clobbered, but if it isn't by the time we fault we know that
> 0x45f10250 is the address of the wait queue.  If we're incredibly lucky,
> it's a symbol in the vmlinux, can you see if it is (and if it's valid)?

In the code I'm looking at, r26 is copied to r7 near the beginning of
__wake_up_common().  r7 is 0 in the register dump.  Of course, Carlos'
kernel may differ.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[James Bottomley]

On Sun, 2007-02-11 at 15:22 -0500, Carlos O'Donell wrote:
> On 2/11/07, Carlos O'Donell <carlos@systemhalted.org> wrote:
> > The faulting instruction is:
> >   74:   52 82 00 20     ldd 10(r20),rp
> >
> > Which is just before the curr->func call.
> >   78:   e8 40 f0 00     bve,l (rp),rp
> >   7c:   52 9b 00 30     ldd 18(r20),dp
> >
> > So your assumption was correct. The value of curr->func is null.
> > How did the list get corrupted?
> 
> ... to be precise, the faulting instruction is the break at 0x10 that
> we use for null pointer dereferences.

Right, now here's a bit of really useful detective work:

In the same piece of disassembly can you see what happens to %r26 ...
the first argument to __wake_up_common() which is the wait queue?  It
may be clobbered, but if it isn't by the time we fault we know that
0x45f10250 is the address of the wait queue.  If we're incredibly lucky,
it's a symbol in the vmlinux, can you see if it is (and if it's valid)?

Knowing what the wait queue is will tell us (hopefully) with precision
where the fault lies.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[James Bottomley]

On Sun, 2007-02-11 at 12:09 -0500, Carlos O'Donell wrote:
> How do I validate your guess? Look for a null or bogus curr->func when
> scheduling?

Disassemble the piece in vmlinux for __wait_common and check that the
instruction that faulted is where the code gets the curr->func.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[John David Anglin]

> On 2/10/07, James Bottomley <James.Bottomley@steeleye.com> wrote:
> > On Sat, 2007-02-10 at 14:37 -0500, John David Anglin wrote:
> > > > 0x10 looks to be curr->func implying curr is NULL and thus the queue
> > > > task_list is corrupt.
> > >
> > > Do you think it help to add a check in __wake_up for a NULL pointer?
> >
> > I suppose so ... I'd really like someone to validate my guess though,
> > although an additional BUG_ON() can't hurt.
> 
> How do I validate your guess? Look for a null or bogus curr->func when
> scheduling?

I'm trying the change below.  Hasn't triggered yet.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

diff --git a/kernel/sched.c b/kernel/sched.c
index cca93cc..277e426 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -3703,6 +3703,7 @@ void fastcall __wake_up(wait_queue_head_t *q, unsigned int mode,
 {
 	unsigned long flags;
 
+	BUG_ON(!q);
 	spin_lock_irqsave(&q->lock, flags);
 	__wake_up_common(q, mode, nr_exclusive, 0, key);
 	spin_unlock_irqrestore(&q->lock, flags);
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Expect defunct, kill -9 panics kernel?

[James Bottomley]

On Sat, 2007-02-10 at 14:37 -0500, John David Anglin wrote:
> > 0x10 looks to be curr->func implying curr is NULL and thus the queue
> > task_list is corrupt.
> 
> Do you think it help to add a check in __wake_up for a NULL pointer?

I suppose so ... I'd really like someone to validate my guess though,
although an additional BUG_ON() can't hurt.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux