2.6.11: spinlock problem

2.6.11: spinlock problem

[Darren Reed]

Hi,

I'm seeing a spinlock held panic with a kernel stack like this:

spinlock - panic, lock already held
..
__do_softirq
do_softirq
=========
do_IRQ
common_interrupt
spinlock/spinunlock
..

when I load up the system in testing.
The code protected by the spinlock is quite small - counter increment.

I'm using 2.6.11-1.1369_FC4 #1, installed inside of vmware,
running as a guest on a Windows XP box.

Is this
(a) linux allowing the IRQ too early
(b) vmware not doing something right
(c) enivitable
(d) somehow my fault
(e) something else?

Thanks,
Darren

Re: 2.6.11: spinlock problem

[linux-os (Dick Johnson)]

On Fri, 23 Jun 2006, Darren Reed wrote:

> Hi,
>
> I'm seeing a spinlock held panic with a kernel stack like this:
>
> spinlock - panic, lock already held
> ..
> __do_softirq
> do_softirq
> =========
> do_IRQ
> common_interrupt
> spinlock/spinunlock
> ..
>
> when I load up the system in testing.
> The code protected by the spinlock is quite small - counter increment.
>
> I'm using 2.6.11-1.1369_FC4 #1, installed inside of vmware,
> running as a guest on a Windows XP box.
>
> Is this
> (a) linux allowing the IRQ too early
> (b) vmware not doing something right
> (c) enivitable
> (d) somehow my fault
> (e) something else?
>
> Thanks,
> Darren

Where's the code? Also, did you initialize the spin-lock variable
before use?

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.88 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: 2.6.11: spinlock problem

[Darren Reed]

[ Charset ISO-8859-1 unsupported, converting... ]
> On Fri, 23 Jun 2006, Darren Reed wrote:
> 
> > Hi,
> >
> > I'm seeing a spinlock held panic with a kernel stack like this:
> >
> > spinlock - panic, lock already held
> > ..
> > __do_softirq
> > do_softirq
> > =========
> > do_IRQ
> > common_interrupt
> > spinlock/spinunlock
> > ..
> >
> > when I load up the system in testing.
> > The code protected by the spinlock is quite small - counter increment.
> >
> > I'm using 2.6.11-1.1369_FC4 #1, installed inside of vmware,
> > running as a guest on a Windows XP box.
> >
> > Is this
> > (a) linux allowing the IRQ too early
> > (b) vmware not doing something right
> > (c) enivitable
> > (d) somehow my fault
> > (e) something else?
> >
> > Thanks,
> > Darren
> 
> Where's the code? Also, did you initialize the spin-lock variable
> before use?

locks are intialised...code runs for a minute or so before panic'ing

I write my own wrappers to read/write_lock/unlock.

The call stack for the panic is:
panic
ipf_read_enter
..
do_softirq
=====
do_IRQ
common_interrupt
ipf_rw_exit

ipf_read_enter and ipf_rw_exit are being called for different locks.
The panic is occuring in the spin_lock() for the counter increment.
The counter incrememnt/decrement uses the same lock, regardless of
the counter being used.

I believe I'm hitting a race condition of sorts...I just don't know
who owns it yet - vmware or linux and I cant test running linux
natively at present because I only have one computer.

Darren

INLINE void ipf_rw_exit(rwlk)
ipfrwlock_t *rwlk;
{
        if (rwlk->ipf_isw > 0) {
                rwlk->ipf_isw = 0;
                write_unlock(&rwlk->ipf_lk);
        } else if (rwlk->ipf_isr > 0) {
                ATOMIC_DEC32(rwlk->ipf_isr);
                read_unlock(&rwlk->ipf_lk);
        } else {
                panic("rwlk->ipf_isw %d isr %d rwlk %p name [%s]\n",
                      rwlk->ipf_isw, rwlk->ipf_isr, rwlk, rwlk->ipf_lname);
        }
}

INLINE void ipf_read_enter(rwlk)
ipfrwlock_t *rwlk;
{
        read_lock(&rwlk->ipf_lk);
        ATOMIC_INC32(rwlk->ipf_isr);
}

#  define       ATOMIC_INC32(x)         MUTEX_ENTER(&ipf_rw); (x)++; \
                                        MUTEX_EXIT(&ipf_rw)
#  define       ATOMIC_DEC32(x)         MUTEX_ENTER(&ipf_rw); (x)--; \
                                        MUTEX_EXIT(&ipf_rw)
#  define       MUTEX_ENTER(x)          spin_lock(&(x)->ipf_lk)
#  define       MUTEX_EXIT(x)           spin_unlock(&(x)->ipf_lk)

Re: 2.6.11: spinlock problem

[Frederik Deweerdt]

On Sat, Jun 24, 2006 at 12:47:30PM +1000, Darren Reed wrote:
> I believe I'm hitting a race condition of sorts...I just don't know
> who owns it yet - vmware or linux and I cant test running linux
> natively at present because I only have one computer.
> 
You could try ruuning your code in QEMU http://fabrice.bellard.free.fr/qemu/, 
that would help indentifying the culprit.

Regards,
Frederik

Re: 2.6.11: spinlock problem

[Hareesh Nagarajan]

On 6/23/06, Darren Reed <darrenr@reed.wattle.id.au> wrote:
> Is this
> (a) linux allowing the IRQ too early
> (b) vmware not doing something right
> (c) enivitable
> (d) somehow my fault
> (e) something else?

Turn on spin lock debugging.

-- 
./hareesh

Re: 2.6.11: spinlock problem

[Chuck Ebbert]

In-Reply-To: <200606240247.k5O2lU3C009083@firewall.reed.wattle.id.au>

On Sat, 24 Jun 2006 12:47:30 +1000, Darren Reed wrote:

> The call stack for the panic is:
> panic
> ipf_read_enter
> ..
> do_softirq
> =====
> do_IRQ
> common_interrupt
> ipf_rw_exit

If you are using spinlocks fron interrupt context you need to use
spin_lock_irq/spin_lock_irqsave / spin_unlock_irq/spin_unlock_irqrestore.

-- 
Chuck
 "You can't read a newspaper if you can't read."  --George W. Bush