3.0.4 + rt12: deadlock

3.0.4 + rt12: deadlock

[Fernando Lopez-Lezcano]

[-- Attachment #1: Type: text/plain, Size: 2639 bytes --]

Booting 3.0.4 + rt12 on a quadcore workstation (running fc14) gave me this:

----
=============================================
[ INFO: possible recursive locking detected ]
3.0.4-1.rt12.1.fc14.ccrma.i686.rtPAE #1
---------------------------------------------
swapper/0 is trying to acquire lock:
  (&parent->list_lock){+.+...}, at: [<c05054ce>] 
__cache_free.clone.27+0x45/0xc4

but task is already holding lock:
  (&parent->list_lock){+.+...}, at: [<c050662c>] do_tune_cpucache+0xf0/0x2b0

other info that might help us debug this:
  Possible unsafe locking scenario:

        CPU0
        ----
   lock(&parent->list_lock);
   lock(&parent->list_lock);

  *** DEADLOCK ***

  May be due to missing lock nesting notation

3 locks held by swapper/0:
  #0:  (cache_chain_mutex){+.+...}, at: [<c0bedb96>] 
kmem_cache_init_late+0x15/0x61
  #1:  (&per_cpu(slab_lock, __cpu).lock){+.+...}, at: [<c0504a53>] 
__local_lock_irq+0x1e/0x5b
  #2:  (&parent->list_lock){+.+...}, at: [<c050662c>] 
do_tune_cpucache+0xf0/0x2b0

stack backtrace:
Pid: 0, comm: swapper Not tainted 3.0.4-1.rt12.1.fc14.ccrma.i686.rtPAE #1
Call Trace:
  [<c086e6f6>] ? printk+0x2d/0x2f
  [<c0479b36>] __lock_acquire+0x8b3/0xc2f
  [<c086fd96>] ? rt_spin_lock_slowlock+0x67/0x170
  [<c04790ee>] ? mark_lock+0x26/0x1bb
  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
  [<c047a372>] lock_acquire+0xde/0x11d
  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
  [<c08703b4>] rt_spin_lock+0x3d/0x43
  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
  [<c05054ce>] __cache_free.clone.27+0x45/0xc4
  [<c0437c83>] ? test_ti_thread_flag+0x8/0x10
  [<c0505303>] kmem_cache_free+0x73/0xe1
  [<c05053c0>] slab_destroy+0x4f/0x53
  [<c0505458>] free_block+0x94/0xc5
  [<c0506645>] do_tune_cpucache+0x109/0x2b0
  [<c05069c7>] enable_cpucache+0x7b/0xa7
  [<c0bedba7>] kmem_cache_init_late+0x26/0x61
  [<c0bcd68a>] start_kernel+0x24f/0x367
  [<c0bcd1d1>] ? loglevel+0x1a/0x1a
  [<c0bcd13b>] ? reserve_ebda_region+0x70/0x72
  [<c0bcd0c3>] i386_start_kernel+0xb2/0xba
Console: colour VGA+ 80x25
console [tty0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:  8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        8191
... CLASSHASH_SIZE:          4096
... MAX_LOCKDEP_ENTRIES:     32768
... MAX_LOCKDEP_CHAINS:      65536
... CHAINHASH_SIZE:          32768
----

I started working and a little while later the machine froze (jack + 
heavy prioritized udp traffic in eth1 - with the r8169 driver). It 
recognized alt-sysrq boot so it was not completely dead. Nothing left on 
the logs to see.

Full config attached.
-- Fernando

[-- Attachment #2: config.bz2 --]
[-- Type: application/x-bzip, Size: 28822 bytes --]

Re: 3.0.4 + rt12: deadlock

[Thomas Gleixner]

On Wed, 31 Aug 2011, Fernando Lopez-Lezcano wrote:

> Booting 3.0.4 + rt12 on a quadcore workstation (running fc14) gave me this:
> 
> ----
> =============================================
> [ INFO: possible recursive locking detected ]
> 3.0.4-1.rt12.1.fc14.ccrma.i686.rtPAE #1
> ---------------------------------------------
> swapper/0 is trying to acquire lock:
>  (&parent->list_lock){+.+...}, at: [<c05054ce>]
> __cache_free.clone.27+0x45/0xc4
> 
> but task is already holding lock:
>  (&parent->list_lock){+.+...}, at: [<c050662c>] do_tune_cpucache+0xf0/0x2b0
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>        CPU0
>        ----
>   lock(&parent->list_lock);
>   lock(&parent->list_lock);

That's something which has to do with debugging options (debugobjects
IIRC). There was some attempt to fix that, but that might have gone
lost in my vacation and the following futile attempt to take care of
the resulting backlog. Peter ???

Thanks,

	tglx

 
>  *** DEADLOCK ***
> 
>  May be due to missing lock nesting notation
> 
> 3 locks held by swapper/0:
>  #0:  (cache_chain_mutex){+.+...}, at: [<c0bedb96>]
> kmem_cache_init_late+0x15/0x61
>  #1:  (&per_cpu(slab_lock, __cpu).lock){+.+...}, at: [<c0504a53>]
> __local_lock_irq+0x1e/0x5b
>  #2:  (&parent->list_lock){+.+...}, at: [<c050662c>]
> do_tune_cpucache+0xf0/0x2b0
> 
> stack backtrace:
> Pid: 0, comm: swapper Not tainted 3.0.4-1.rt12.1.fc14.ccrma.i686.rtPAE #1
> Call Trace:
>  [<c086e6f6>] ? printk+0x2d/0x2f
>  [<c0479b36>] __lock_acquire+0x8b3/0xc2f
>  [<c086fd96>] ? rt_spin_lock_slowlock+0x67/0x170
>  [<c04790ee>] ? mark_lock+0x26/0x1bb
>  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
>  [<c047a372>] lock_acquire+0xde/0x11d
>  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
>  [<c08703b4>] rt_spin_lock+0x3d/0x43
>  [<c05054ce>] ? __cache_free.clone.27+0x45/0xc4
>  [<c05054ce>] __cache_free.clone.27+0x45/0xc4
>  [<c0437c83>] ? test_ti_thread_flag+0x8/0x10
>  [<c0505303>] kmem_cache_free+0x73/0xe1
>  [<c05053c0>] slab_destroy+0x4f/0x53
>  [<c0505458>] free_block+0x94/0xc5
>  [<c0506645>] do_tune_cpucache+0x109/0x2b0
>  [<c05069c7>] enable_cpucache+0x7b/0xa7
>  [<c0bedba7>] kmem_cache_init_late+0x26/0x61
>  [<c0bcd68a>] start_kernel+0x24f/0x367
>  [<c0bcd1d1>] ? loglevel+0x1a/0x1a
>  [<c0bcd13b>] ? reserve_ebda_region+0x70/0x72
>  [<c0bcd0c3>] i386_start_kernel+0xb2/0xba
> Console: colour VGA+ 80x25
> console [tty0] enabled
> Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> ... MAX_LOCKDEP_SUBCLASSES:  8
> ... MAX_LOCK_DEPTH:          48
> ... MAX_LOCKDEP_KEYS:        8191
> ... CLASSHASH_SIZE:          4096
> ... MAX_LOCKDEP_ENTRIES:     32768
> ... MAX_LOCKDEP_CHAINS:      65536
> ... CHAINHASH_SIZE:          32768
> ----
> 
> I started working and a little while later the machine froze (jack + heavy
> prioritized udp traffic in eth1 - with the r8169 driver). It recognized
> alt-sysrq boot so it was not completely dead. Nothing left on the logs to see.
> 
> Full config attached.
> -- Fernando
> 

Re: 3.0.4 + rt12: deadlock

[Peter Zijlstra]

On Thu, 2011-09-01 at 21:38 +0200, Thomas Gleixner wrote:
> > =============================================
> > [ INFO: possible recursive locking detected ]
> > 3.0.4-1.rt12.1.fc14.ccrma.i686.rtPAE #1
> > ---------------------------------------------
> > swapper/0 is trying to acquire lock:
> >  (&parent->list_lock){+.+...}, at: [<c05054ce>]
> > __cache_free.clone.27+0x45/0xc4
> > 
> > but task is already holding lock:
> >  (&parent->list_lock){+.+...}, at: [<c050662c>] do_tune_cpucache
> +0xf0/0x2b0
> > 
> > other info that might help us debug this:
> >  Possible unsafe locking scenario:
> > 
> >        CPU0
> >        ----
> >   lock(&parent->list_lock);
> >   lock(&parent->list_lock);
> 

> 
>  
> >  *** DEADLOCK ***
> > 
> >  May be due to missing lock nesting notation
> > 
> > 3 locks held by swapper/0:
> >  #0:  (cache_chain_mutex){+.+...}, at: [<c0bedb96>]
> > kmem_cache_init_late+0x15/0x61
> >  #1:  (&per_cpu(slab_lock, __cpu).lock){+.+...}, at: [<c0504a53>]
> > __local_lock_irq+0x1e/0x5b
> >  #2:  (&parent->list_lock){+.+...}, at: [<c050662c>]
> > do_tune_cpucache+0xf0/0x2b0 

> That's something which has to do with debugging options (debugobjects
> IIRC). There was some attempt to fix that, but that might have gone
> lost in my vacation and the following futile attempt to take care of
> the resulting backlog. Peter ???

Looks like the one supposedly cured by:

patches/peter_zijlstra-slab_lockdep-annotate_the_locks_before_using.patch

which should be in -rt12

will have a peek, never reproduced for me though..