All of lore.kernel.org
 help / color / mirror / Atom feed
* [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
@ 2016-03-14 21:00 Luis Claudio R. Goncalves
  2016-03-19  8:33 ` Thomas Gleixner
  0 siblings, 1 reply; 13+ messages in thread
From: Luis Claudio R. Goncalves @ 2016-03-14 21:00 UTC (permalink / raw)
  To: linux-rt-users

Hello,

I just started using 4.4.4-rt11 this afternoon and saw the lockdep splat below.
Is this something new or did I miss a patch floating on the list?

[67547.917898] =============================================
[67547.917898] [ INFO: possible recursive locking detected ]
[67547.917900] 4.4.4-rt11 #4 Not tainted
[67547.917900] ---------------------------------------------
[67547.917901] kswapd0/59 is trying to acquire lock:
[67547.917908]  (&h->lock#2){+.+.-.}, at: [<ffffffff812934dc>] mb_cache_shrink_scan+0xac/0x220
[67547.917908] 
               but task is already holding lock:
[67547.917910]  (&h->lock#2){+.+.-.}, at: [<ffffffff812934ca>] mb_cache_shrink_scan+0x9a/0x220
[67547.917911] 
               other info that might help us debug this:
[67547.917911]  Possible unsafe locking scenario:

[67547.917911]        CPU0
[67547.917911]        ----
[67547.917912]   lock(&h->lock#2);
[67547.917913]   lock(&h->lock#2);
[67547.917913] 
                *** DEADLOCK ***

[67547.917913]  May be due to missing lock nesting notation

[67547.917914] 2 locks held by kswapd0/59:
[67547.917917]  #0:  (shrinker_rwsem){+.+...}, at: [<ffffffff810e77d2>] rt_down_read_trylock+0x22/0x30
[67547.917919]  #1:  (&h->lock#2){+.+.-.}, at: [<ffffffff812934ca>] mb_cache_shrink_scan+0x9a/0x220
[67547.917920] 
               stack backtrace:
[67547.917921] CPU: 0 PID: 59 Comm: kswapd0 Not tainted 4.4.4-rt11 #4
[67547.917922] Hardware name: Hewlett-Packard p7-1512/2ADA, BIOS 8.15 02/05/2013
[67547.917923]  0000000000000086 000000005fca2e71 ffff88018a9fba10 ffffffff813d5412
[67547.917925]  ffffffff828ede50 ffff88018a9f4000 ffff88018a9fbb00 ffffffff810e0d66
[67547.917926]  ffff88018a9fbb08 000000000000008a ffffffff828e58a0 ffffffff81c0bf18
[67547.917926] Call Trace:
[67547.917929]  [<ffffffff813d5412>] dump_stack+0x67/0x95
[67547.917931]  [<ffffffff810e0d66>] __lock_acquire+0x1cf6/0x1d30
[67547.917934]  [<ffffffff813f55d7>] ? debug_smp_processor_id+0x17/0x20
[67547.917936]  [<ffffffff810deb49>] ? mark_held_locks+0x79/0xa0
[67547.917940]  [<ffffffff817dc0cc>] ? _raw_spin_unlock_irqrestore+0x6c/0x80
[67547.917941]  [<ffffffff810e1907>] lock_acquire+0xf7/0x240
[67547.917942]  [<ffffffff812934dc>] ? mb_cache_shrink_scan+0xac/0x220
[67547.917944]  [<ffffffff817dbb11>] _raw_spin_lock+0x41/0x80
[67547.917945]  [<ffffffff812934dc>] ? mb_cache_shrink_scan+0xac/0x220
[67547.917946]  [<ffffffff812934dc>] mb_cache_shrink_scan+0xac/0x220
[67547.917949]  [<ffffffff811bfaaa>] shrink_slab.part.41+0x24a/0x600
[67547.917951]  [<ffffffff811c479a>] shrink_zone+0x2ca/0x2e0
[67547.917953]  [<ffffffff811c5ee1>] kswapd+0x581/0xbc0
[67547.917955]  [<ffffffff811c5960>] ? mem_cgroup_shrink_node_zone+0x3c0/0x3c0
[67547.917957]  [<ffffffff810a9b71>] kthread+0x101/0x120
[67547.917959]  [<ffffffff810a9a70>] ? kthread_create_on_node+0x250/0x250
[67547.917960]  [<ffffffff817dcfff>] ret_from_fork+0x3f/0x70
[67547.917962]  [<ffffffff810a9a70>] ? kthread_create_on_node+0x250/0x250
 
Best regards,
Luis

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-14 21:00 [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan() Luis Claudio R. Goncalves
@ 2016-03-19  8:33 ` Thomas Gleixner
  2016-03-21 13:34   ` Luis Claudio R. Goncalves
  2016-03-21 14:40   ` Josh Cartwright
  0 siblings, 2 replies; 13+ messages in thread
From: Thomas Gleixner @ 2016-03-19  8:33 UTC (permalink / raw)
  To: Luis Claudio R. Goncalves; +Cc: linux-rt-users

On Mon, 14 Mar 2016, Luis Claudio R. Goncalves wrote:

> Hello,
> 
> I just started using 4.4.4-rt11 this afternoon and saw the lockdep splat below.
> Is this something new or did I miss a patch floating on the list?

That's new. Looking into it.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-19  8:33 ` Thomas Gleixner
@ 2016-03-21 13:34   ` Luis Claudio R. Goncalves
  2016-03-21 14:40   ` Josh Cartwright
  1 sibling, 0 replies; 13+ messages in thread
From: Luis Claudio R. Goncalves @ 2016-03-21 13:34 UTC (permalink / raw)
  To: Thomas Gleixner; +Cc: linux-rt-users

On Sat, Mar 19, 2016 at 09:33:34AM +0100, Thomas Gleixner wrote:
| On Mon, 14 Mar 2016, Luis Claudio R. Goncalves wrote:
| 
| > Hello,
| > 
| > I just started using 4.4.4-rt11 this afternoon and saw the lockdep splat below.
| > Is this something new or did I miss a patch floating on the list?
| 
| That's new. Looking into it.

Thanks!

Please let me know if you want me to run any test or trace. I will be more
than glad on helping.

Luis

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-19  8:33 ` Thomas Gleixner
  2016-03-21 13:34   ` Luis Claudio R. Goncalves
@ 2016-03-21 14:40   ` Josh Cartwright
  2016-03-21 20:47     ` Luis Claudio R. Goncalves
  2016-03-29 15:04     ` Sebastian Andrzej Siewior
  1 sibling, 2 replies; 13+ messages in thread
From: Josh Cartwright @ 2016-03-21 14:40 UTC (permalink / raw)
  To: Thomas Gleixner; +Cc: Luis Claudio R. Goncalves, linux-rt-users

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

On Sat, Mar 19, 2016 at 09:33:34AM +0100, Thomas Gleixner wrote:
> On Mon, 14 Mar 2016, Luis Claudio R. Goncalves wrote:
> 
> > Hello,
> > 
> > I just started using 4.4.4-rt11 this afternoon and saw the lockdep splat below.
> > Is this something new or did I miss a patch floating on the list?
> 
> That's new. Looking into it.

It looks like the way that INIT_HLIST_BL_HEAD() is authored, all
instances of the contained raw spinlock will share the same lockdep
class.  That, coupled with the following in mb_cache_shrink(), is
triggering the lockdep violation:

	void
	mb_cache_shrink(struct block_device *bdev)
	{
		[..]
				/*
				 * Prevent any find or get operation on the entry.
				 */
				hlist_bl_lock(ce->e_block_hash_p);
				hlist_bl_lock(ce->e_index_hash_p);

  Josh

diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
index 44f0b55..d13428a 100644
--- a/include/linux/list_bl.h
+++ b/include/linux/list_bl.h
@@ -42,13 +42,18 @@ struct hlist_bl_node {
 	struct hlist_bl_node *next, **pprev;
 };
 
-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
-{
-	h->first = NULL;
 #ifdef CONFIG_PREEMPT_RT_BASE
-	raw_spin_lock_init(&h->lock);
+#define INIT_HLIST_BL_HEAD(h)		\
+do {					\
+	(h)->first = NULL;		\
+	raw_spin_lock_init(&(h)->lock);	\
+} while (0)
+#else
+#define INIT_HLIST_BL_HEAD(h)		\
+do {					\
+	(h)->first = NULL;		\
+} while (0)
 #endif
-}
 
 static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
 {

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-21 14:40   ` Josh Cartwright
@ 2016-03-21 20:47     ` Luis Claudio R. Goncalves
  2016-03-29 15:04     ` Sebastian Andrzej Siewior
  1 sibling, 0 replies; 13+ messages in thread
From: Luis Claudio R. Goncalves @ 2016-03-21 20:47 UTC (permalink / raw)
  To: Josh Cartwright; +Cc: Thomas Gleixner, linux-rt-users

On Mon, Mar 21, 2016 at 09:40:16AM -0500, Josh Cartwright wrote:
| On Sat, Mar 19, 2016 at 09:33:34AM +0100, Thomas Gleixner wrote:
| > On Mon, 14 Mar 2016, Luis Claudio R. Goncalves wrote:
| > 
| > > Hello,
| > > 
| > > I just started using 4.4.4-rt11 this afternoon and saw the lockdep splat below.
| > > Is this something new or did I miss a patch floating on the list?
| > 
| > That's new. Looking into it.
| 
| It looks like the way that INIT_HLIST_BL_HEAD() is authored, all
| instances of the contained raw spinlock will share the same lockdep
| class.  That, coupled with the following in mb_cache_shrink(), is
| triggering the lockdep violation:
| 
| 	void
| 	mb_cache_shrink(struct block_device *bdev)
| 	{
| 		[..]
| 				/*
| 				 * Prevent any find or get operation on the entry.
| 				 */
| 				hlist_bl_lock(ce->e_block_hash_p);
| 				hlist_bl_lock(ce->e_index_hash_p);
| 
|   Josh

Josh, I built a kernel with your patch and the lockdep complaint is gone. 
I was used to see the message within 30 minutes of normal usage. So far,
four hours with that kernel running and no complaints.

Tested-by: Luis Claudio R. Goncalves <lclaudio@uudg.org>

Thanks!
Luis
 
| diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
| index 44f0b55..d13428a 100644
| --- a/include/linux/list_bl.h
| +++ b/include/linux/list_bl.h
| @@ -42,13 +42,18 @@ struct hlist_bl_node {
|  	struct hlist_bl_node *next, **pprev;
|  };
|  
| -static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
| -{
| -	h->first = NULL;
|  #ifdef CONFIG_PREEMPT_RT_BASE
| -	raw_spin_lock_init(&h->lock);
| +#define INIT_HLIST_BL_HEAD(h)		\
| +do {					\
| +	(h)->first = NULL;		\
| +	raw_spin_lock_init(&(h)->lock);	\
| +} while (0)
| +#else
| +#define INIT_HLIST_BL_HEAD(h)		\
| +do {					\
| +	(h)->first = NULL;		\
| +} while (0)
|  #endif
| -}
|  
|  static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
|  {


---end quoted text---

-- 
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-21 14:40   ` Josh Cartwright
  2016-03-21 20:47     ` Luis Claudio R. Goncalves
@ 2016-03-29 15:04     ` Sebastian Andrzej Siewior
  2016-03-29 18:20       ` Josh Cartwright
  1 sibling, 1 reply; 13+ messages in thread
From: Sebastian Andrzej Siewior @ 2016-03-29 15:04 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: Thomas Gleixner, Luis Claudio R. Goncalves, linux-rt-users

* Josh Cartwright | 2016-03-21 09:40:16 [-0500]:

>diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
>index 44f0b55..d13428a 100644
>--- a/include/linux/list_bl.h
>+++ b/include/linux/list_bl.h
>@@ -42,13 +42,18 @@ struct hlist_bl_node {
> 	struct hlist_bl_node *next, **pprev;
> };
> 
>-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
>-{
>-	h->first = NULL;
> #ifdef CONFIG_PREEMPT_RT_BASE
>-	raw_spin_lock_init(&h->lock);
>+#define INIT_HLIST_BL_HEAD(h)		\
>+do {					\
>+	(h)->first = NULL;		\
>+	raw_spin_lock_init(&(h)->lock);	\
>+} while (0)
>+#else
>+#define INIT_HLIST_BL_HEAD(h)		\
>+do {					\
>+	(h)->first = NULL;		\
>+} while (0)
> #endif
>-}

So we use a macro instead a "static inline" to ensure we end up with
another lockdep class? This surprises me because it would mean that the
function wasn't inlined / key classed was re-used.
Looking at the assembly I see:

|ffffffff8121fb41:       48 c7 c2 e0 bb d6 82    mov    $0xffffffff82d6bbe0,%rdx
|ffffffff8121fb48:       48 89 df                mov    %rbx,%rdi
|ffffffff8121fb4b:       48 c7 c6 30 2b a2 81    mov    $0xffffffff81a22b30,%rsi
|ffffffff8121fb52:       e8 a9 0e e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>
…
|ffffffff8121fc12:       48 c7 c2 d0 bb d6 82    mov    $0xffffffff82d6bbd0,%rdx
|ffffffff8121fc19:       48 c7 c6 7b 2e a3 81    mov    $0xffffffff81a32e7b,%rsi
|ffffffff8121fc20:       48 c7 07 00 00 00 00    movq   $0x0,(%rdi)
|ffffffff8121fc27:       48 83 c7 08             add    $0x8,%rdi
|ffffffff8121fc2b:       e8 d0 0d e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>

rdx holds the third parameter and it is different. What do I miss?
Was the rdx argument in Luis' case the same before this path was
applied?

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-29 15:04     ` Sebastian Andrzej Siewior
@ 2016-03-29 18:20       ` Josh Cartwright
  2016-03-29 23:14         ` Josh Cartwright
  0 siblings, 1 reply; 13+ messages in thread
From: Josh Cartwright @ 2016-03-29 18:20 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: Thomas Gleixner, Luis Claudio R. Goncalves, linux-rt-users

On Tue, Mar 29, 2016 at 05:04:44PM +0200, Sebastian Andrzej Siewior wrote:
> * Josh Cartwright | 2016-03-21 09:40:16 [-0500]:
> 
> >diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
> >index 44f0b55..d13428a 100644
> >--- a/include/linux/list_bl.h
> >+++ b/include/linux/list_bl.h
> >@@ -42,13 +42,18 @@ struct hlist_bl_node {
> > 	struct hlist_bl_node *next, **pprev;
> > };
> > 
> >-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
> >-{
> >-	h->first = NULL;
> > #ifdef CONFIG_PREEMPT_RT_BASE
> >-	raw_spin_lock_init(&h->lock);
> >+#define INIT_HLIST_BL_HEAD(h)		\
> >+do {					\
> >+	(h)->first = NULL;		\
> >+	raw_spin_lock_init(&(h)->lock);	\
> >+} while (0)
> >+#else
> >+#define INIT_HLIST_BL_HEAD(h)		\
> >+do {					\
> >+	(h)->first = NULL;		\
> >+} while (0)
> > #endif
> >-}
> 
> So we use a macro instead a "static inline" to ensure we end up with
> another lockdep class?

Yes, that was the idea.

> This surprises me because it would mean that the function wasn't
> inlined / key classed was re-used.

Hmm.  I'm not sure that's what this means.  I think it's possible that
the function _code_ is inlined, but any function static data is shared
between inlined instances.

This simple experiment seems to also confirm this:

	extern void do_something(int *x);
	
	static inline void experiment_static(void)
	{
		static int x;
		do_something(&x);
	}
	
	#define experiment_macro()		\
		do {				\
			static int x;		\
			do_something(&x);	\
		} while (0)
	
	void f(void)
	{
		asm("# static1");
		experiment_static();
		asm("# static2");
		experiment_static();
		asm("# macro1");
		experiment_macro();
		asm("# macro2");
		experiment_macro();
	}

Generated listing (cleaned up):

	f:
		subq	$8, %rsp
		# static1
		movl	$x.1835, %edi
		call	do_something

		# static2
		movl	$x.1835, %edi
		call	do_something

		# macro1
		movl	$x.1839, %edi
		call	do_something

		# macro2
		movl	$x.1840, %edi
		addq	$8, %rsp
		jmp	do_something
	[..]
		.local	x.1835
		.comm	x.1835,4,4
		.local	x.1840
		.comm	x.1840,4,4
		.local	x.1839
		.comm	x.1839,4,4
		.ident	"GCC: (GNU) 5.3.0"

You can see here that both of the static inlined instances shared the
same 'x.1835' data (and the code is inlined), whereas the macro
instances do not share data.

> Looking at the assembly I see:
> 
> |ffffffff8121fb41:       48 c7 c2 e0 bb d6 82    mov    $0xffffffff82d6bbe0,%rdx
> |ffffffff8121fb48:       48 89 df                mov    %rbx,%rdi
> |ffffffff8121fb4b:       48 c7 c6 30 2b a2 81    mov    $0xffffffff81a22b30,%rsi
> |ffffffff8121fb52:       e8 a9 0e e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>
> ???
> |ffffffff8121fc12:       48 c7 c2 d0 bb d6 82    mov    $0xffffffff82d6bbd0,%rdx
> |ffffffff8121fc19:       48 c7 c6 7b 2e a3 81    mov    $0xffffffff81a32e7b,%rsi
> |ffffffff8121fc20:       48 c7 07 00 00 00 00    movq   $0x0,(%rdi)
> |ffffffff8121fc27:       48 83 c7 08             add    $0x8,%rdi
> |ffffffff8121fc2b:       e8 d0 0d e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>
> 
> rdx holds the third parameter and it is different. What do I miss?

Hmm.  I'm not sure what's going on here for your build.

Here's the relevant listing I get for fs/mbcache.c w/ DEBUG_SPINLOCKS &
PREEMPT_RT_FULL, which shows the same thing as the experiment above (the
class object being shared amongst inline instances):

	mb_cache_create:
	[..]
		movq	$__key.15124, %rdx	#,
		movq	$.LC5, %rsi	#,
		movq	$0, (%rdi)	#, _28->first
		addq	$8, %rdi	#, D.27485
		call	__raw_spin_lock_init	#
	[..]
		movq	$__key.15124, %rdx	#,
		movq	$.LC5, %rsi	#,
		movq	$0, (%rdi)	#, _35->first
		addq	$8, %rdi	#, D.27485
		call	__raw_spin_lock_init	#
	[..]
		.local	__key.15124
		.comm	__key.15124,8,8

  Josh

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-29 18:20       ` Josh Cartwright
@ 2016-03-29 23:14         ` Josh Cartwright
  2016-03-30  8:16           ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 13+ messages in thread
From: Josh Cartwright @ 2016-03-29 23:14 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: Thomas Gleixner, Luis Claudio R. Goncalves, linux-rt-users

On Tue, Mar 29, 2016 at 01:20:02PM -0500, Josh Cartwright wrote:
> On Tue, Mar 29, 2016 at 05:04:44PM +0200, Sebastian Andrzej Siewior wrote:
[..]
> > Looking at the assembly I see:
> > 
> > |ffffffff8121fb41:       48 c7 c2 e0 bb d6 82    mov    $0xffffffff82d6bbe0,%rdx
> > |ffffffff8121fb48:       48 89 df                mov    %rbx,%rdi
> > |ffffffff8121fb4b:       48 c7 c6 30 2b a2 81    mov    $0xffffffff81a22b30,%rsi
> > |ffffffff8121fb52:       e8 a9 0e e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>
> > ???
> > |ffffffff8121fc12:       48 c7 c2 d0 bb d6 82    mov    $0xffffffff82d6bbd0,%rdx
> > |ffffffff8121fc19:       48 c7 c6 7b 2e a3 81    mov    $0xffffffff81a32e7b,%rsi
> > |ffffffff8121fc20:       48 c7 07 00 00 00 00    movq   $0x0,(%rdi)
> > |ffffffff8121fc27:       48 83 c7 08             add    $0x8,%rdi
> > |ffffffff8121fc2b:       e8 d0 0d e9 ff          callq  ffffffff810b0a00 <__raw_spin_lock_init>
> > 
> > rdx holds the third parameter and it is different. What do I miss?
> 
> Hmm.  I'm not sure what's going on here for your build.

Ah!  I think the problem is a bit more nuanced than I let on before.

The problem isn't that _all_ instances of the hlist_bl_head will share a
lockdep class...it's that instances which are INIT_HLIST_BL_HEAD()'d _within
the same compilation unit_ will share a lockdep class.

Depending on what you are looking at, perhaps that's what you're seeing?

  Josh

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan()
  2016-03-29 23:14         ` Josh Cartwright
@ 2016-03-30  8:16           ` Sebastian Andrzej Siewior
  2016-03-31  5:04             ` [PATCH] list_bl: fixup bogus lockdep warning Josh Cartwright
  0 siblings, 1 reply; 13+ messages in thread
From: Sebastian Andrzej Siewior @ 2016-03-30  8:16 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: Thomas Gleixner, Luis Claudio R. Goncalves, linux-rt-users

On 03/30/2016 01:14 AM, Josh Cartwright wrote:
> On Tue, Mar 29, 2016 at 01:20:02PM -0500, Josh Cartwright wrote:
>> On Tue, Mar 29, 2016 at 05:04:44PM +0200, Sebastian Andrzej Siewior wrote:
> 
> The problem isn't that _all_ instances of the hlist_bl_head will share a
> lockdep class...it's that instances which are INIT_HLIST_BL_HEAD()'d _within
> the same compilation unit_ will share a lockdep class.

no, that is okay. No, that is actually all good. I must have mixed up
the lock inits. Sorry for the confusion. After looking at it again it
is all good.

Please send a proper patch and I add it to the queue.

> 
> 
>   Josh
> 
Sebastian

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH] list_bl: fixup bogus lockdep warning
  2016-03-30  8:16           ` Sebastian Andrzej Siewior
@ 2016-03-31  5:04             ` Josh Cartwright
  2016-04-01 10:46               ` Sebastian Andrzej Siewior
  2016-04-01 10:49               ` Sebastian Andrzej Siewior
  0 siblings, 2 replies; 13+ messages in thread
From: Josh Cartwright @ 2016-03-31  5:04 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: linux-rt-users, Thomas Gleixner, Luis Claudio R. Goncalves

At first glance, the use of 'static inline' seems appropriate for
INIT_HLIST_BL_HEAD().

However, when a 'static inline' function invocation is inlined by gcc,
all callers share any static local data declared within that inline
function.

This presents a problem for how lockdep classes are setup.  raw_spinlocks, for
example, when CONFIG_DEBUG_SPINLOCK,

	# define raw_spin_lock_init(lock)				\
	do {								\
		static struct lock_class_key __key;			\
									\
		__raw_spin_lock_init((lock), #lock, &__key);		\
	} while (0)

When this macro is expanded into a 'static inline' caller, like
INIT_HLIST_BL_HEAD():

	static inline INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
	{
		h->first = NULL;
		raw_spin_lock_init(&h->lock);
	}

...the static local lock_class_key object is made a function static.

For compilation units which initialize invoke INIT_HLIST_BL_HEAD() more
than once, then, all of the invocations share this same static local
object.

This can lead to some very confusing lockdep splats (example below).
Solve this problem by forcing the INIT_HLIST_BL_HEAD() to be a macro,
which prevents the lockdep class object sharing.

 =============================================
 [ INFO: possible recursive locking detected ]
 4.4.4-rt11 #4 Not tainted
 ---------------------------------------------
 kswapd0/59 is trying to acquire lock:
  (&h->lock#2){+.+.-.}, at: mb_cache_shrink_scan

 but task is already holding lock:
  (&h->lock#2){+.+.-.}, at:  mb_cache_shrink_scan

 other info that might help us debug this:
  Possible unsafe locking scenario:

        CPU0
        ----
   lock(&h->lock#2);
   lock(&h->lock#2);

  *** DEADLOCK ***

  May be due to missing lock nesting notation

 2 locks held by kswapd0/59:
  #0:  (shrinker_rwsem){+.+...}, at: rt_down_read_trylock
  #1:  (&h->lock#2){+.+.-.}, at: mb_cache_shrink_scan

Reported-by: Luis Claudio R. Goncalves <lclaudio@uudg.org>
Tested-by: Luis Claudio R. Goncalves <lclaudio@uudg.org>
Signed-off-by: Josh Cartwright <joshc@ni.com>
---
 include/linux/list_bl.h | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
index 44f0b55..a1bfb3b 100644
--- a/include/linux/list_bl.h
+++ b/include/linux/list_bl.h
@@ -42,13 +42,16 @@ struct hlist_bl_node {
 	struct hlist_bl_node *next, **pprev;
 };
 
-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
-{
-	h->first = NULL;
 #ifdef CONFIG_PREEMPT_RT_BASE
-	raw_spin_lock_init(&h->lock);
+#define INIT_HLIST_BL_HEAD(h)		\
+do {					\
+	(h)->first = NULL;		\
+	raw_spin_lock_init(&(h)->lock);	\
+} while (0)
+#else
+#define INIT_HLIST_BL_HEAD(h)		\
+	(h)->first = NULL;		\
 #endif
-}
 
 static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
 {
-- 
2.7.4


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [PATCH] list_bl: fixup bogus lockdep warning
  2016-03-31  5:04             ` [PATCH] list_bl: fixup bogus lockdep warning Josh Cartwright
@ 2016-04-01 10:46               ` Sebastian Andrzej Siewior
  2016-04-01 10:49               ` Sebastian Andrzej Siewior
  1 sibling, 0 replies; 13+ messages in thread
From: Sebastian Andrzej Siewior @ 2016-04-01 10:46 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: linux-rt-users, Thomas Gleixner, Luis Claudio R. Goncalves

* Josh Cartwright | 2016-03-31 00:04:25 [-0500]:

>At first glance, the use of 'static inline' seems appropriate for
>INIT_HLIST_BL_HEAD().
>
>Reported-by: Luis Claudio R. Goncalves <lclaudio@uudg.org>
>Tested-by: Luis Claudio R. Goncalves <lclaudio@uudg.org>
>Signed-off-by: Josh Cartwright <joshc@ni.com>

Applied, thanks.

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH] list_bl: fixup bogus lockdep warning
  2016-03-31  5:04             ` [PATCH] list_bl: fixup bogus lockdep warning Josh Cartwright
  2016-04-01 10:46               ` Sebastian Andrzej Siewior
@ 2016-04-01 10:49               ` Sebastian Andrzej Siewior
  2016-04-01 17:02                 ` Josh Cartwright
  1 sibling, 1 reply; 13+ messages in thread
From: Sebastian Andrzej Siewior @ 2016-04-01 10:49 UTC (permalink / raw)
  To: Josh Cartwright
  Cc: linux-rt-users, Thomas Gleixner, Luis Claudio R. Goncalves

* Josh Cartwright | 2016-03-31 00:04:25 [-0500]:

>diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
>index 44f0b55..a1bfb3b 100644
>--- a/include/linux/list_bl.h
>+++ b/include/linux/list_bl.h
>@@ -42,13 +42,16 @@ struct hlist_bl_node {
> 	struct hlist_bl_node *next, **pprev;
> };
> 
>-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
>-{
>-	h->first = NULL;
> #ifdef CONFIG_PREEMPT_RT_BASE
>-	raw_spin_lock_init(&h->lock);
>+#define INIT_HLIST_BL_HEAD(h)		\
>+do {					\
>+	(h)->first = NULL;		\
>+	raw_spin_lock_init(&(h)->lock);	\
>+} while (0)
>+#else
>+#define INIT_HLIST_BL_HEAD(h)		\
>+	(h)->first = NULL;		\

I switched the else part to

 #define INIT_HLIST_BL_HEAD(h) (h)->first = NULL

> #endif
>-}

Sebastian

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH] list_bl: fixup bogus lockdep warning
  2016-04-01 10:49               ` Sebastian Andrzej Siewior
@ 2016-04-01 17:02                 ` Josh Cartwright
  0 siblings, 0 replies; 13+ messages in thread
From: Josh Cartwright @ 2016-04-01 17:02 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: linux-rt-users, Thomas Gleixner, Luis Claudio R. Goncalves

On Fri, Apr 01, 2016 at 12:49:46PM +0200, Sebastian Andrzej Siewior wrote:
> * Josh Cartwright | 2016-03-31 00:04:25 [-0500]:
> 
> >diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h
> >index 44f0b55..a1bfb3b 100644
> >--- a/include/linux/list_bl.h
> >+++ b/include/linux/list_bl.h
> >@@ -42,13 +42,16 @@ struct hlist_bl_node {
> > 	struct hlist_bl_node *next, **pprev;
> > };
> > 
> >-static inline void INIT_HLIST_BL_HEAD(struct hlist_bl_head *h)
> >-{
> >-	h->first = NULL;
> > #ifdef CONFIG_PREEMPT_RT_BASE
> >-	raw_spin_lock_init(&h->lock);
> >+#define INIT_HLIST_BL_HEAD(h)		\
> >+do {					\
> >+	(h)->first = NULL;		\
> >+	raw_spin_lock_init(&(h)->lock);	\
> >+} while (0)
> >+#else
> >+#define INIT_HLIST_BL_HEAD(h)		\
> >+	(h)->first = NULL;		\
> 
> I switched the else part to
> 
>  #define INIT_HLIST_BL_HEAD(h) (h)->first = NULL

That works, thanks.  I had it wrapped it in do { } while(0);, but
checkpatch complained.

Anyway, I decided to do some grepping, and this particular case isn't
the only one where there might be unintentional lockdep class sharing.
I hope to follow up on this when I'm not traveling.

  Josh

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2016-04-01 17:02 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-03-14 21:00 [4.4.4-rt11] Possiblie recursive locking detected in kswapd / mb_cache_shrink_scan() Luis Claudio R. Goncalves
2016-03-19  8:33 ` Thomas Gleixner
2016-03-21 13:34   ` Luis Claudio R. Goncalves
2016-03-21 14:40   ` Josh Cartwright
2016-03-21 20:47     ` Luis Claudio R. Goncalves
2016-03-29 15:04     ` Sebastian Andrzej Siewior
2016-03-29 18:20       ` Josh Cartwright
2016-03-29 23:14         ` Josh Cartwright
2016-03-30  8:16           ` Sebastian Andrzej Siewior
2016-03-31  5:04             ` [PATCH] list_bl: fixup bogus lockdep warning Josh Cartwright
2016-04-01 10:46               ` Sebastian Andrzej Siewior
2016-04-01 10:49               ` Sebastian Andrzej Siewior
2016-04-01 17:02                 ` Josh Cartwright

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.