From: "Paul E. McKenney" <paulmck@kernel.org>
To: linux-kernel@vger.kernel.org, linux-mm@kvack.org
Cc: vbabka@suse.cz, akpm@linux-foundation.org, ming.lei@redhat.com,
riel@fb.com, jweiner@fb.com, cl@linux.com, penberg@kernel.org,
rientjes@google.com, iamjoonsoo.kim@lge.com
Subject: [BUG] Shudown-time lockdep splat in next-20210514
Date: Sat, 15 May 2021 13:46:22 -0700 [thread overview]
Message-ID: <20210515204622.GA2672367@paulmck-ThinkPad-P17-Gen-1> (raw)
Hello!
I am seeing the following lockdep splat in next-20210514. It happens
at shutdown time in all rcutorture scenarios that enable lockdep. It
happens consistently on 2-hour runs, and I am trying it on shorter runs.
If it reproduces nicely, I will try bisection.
In the meantime, does this ring a bell for anyone?
Thanx, Paul
=====================================================
WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
5.13.0-rc1-next-20210514 #1587 Not tainted
-----------------------------------------------------
torture_shutdow/112 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
ffffffffaad7ac20 (fs_reclaim){+.+.}-{0:0}, at: fs_reclaim_acquire+0x7f/0xd0
and this task is already holding:
ffffa37a02ab4098 (&n->list_lock){-.-.}-{2:2}, at: free_debug_processing+0x3a/0x230
which would create a new lock dependency:
(&n->list_lock){-.-.}-{2:2} -> (fs_reclaim){+.+.}-{0:0}
but this new dependency connects a HARDIRQ-irq-safe lock:
(&n->list_lock){-.-.}-{2:2}
... which became HARDIRQ-irq-safe at:
lock_acquire+0xb5/0x3a0
_raw_spin_lock+0x27/0x40
deactivate_slab.isra.90+0x3c2/0x5d0
flush_cpu_slab+0x31/0x50
flush_smp_call_function_queue+0x132/0x1d0
__sysvec_call_function_single+0x3e/0x190
sysvec_call_function_single+0x61/0x80
asm_sysvec_call_function_single+0x12/0x20
default_idle+0xb/0x10
default_idle_call+0x66/0x1e0
do_idle+0x204/0x2a0
cpu_startup_entry+0x14/0x20
secondary_startup_64_no_verify+0xc2/0xcb
to a HARDIRQ-irq-unsafe lock:
(fs_reclaim){+.+.}-{0:0}
... which became HARDIRQ-irq-unsafe at:
...
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
kmem_cache_alloc_node_trace+0x2f/0x230
alloc_worker+0x18/0x50
init_rescuer.part.33+0xf/0x90
workqueue_init+0x1fb/0x2db
kernel_init_freeable+0x10c/0x26f
kernel_init+0x5/0xfe
ret_from_fork+0x22/0x30
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(fs_reclaim);
local_irq_disable();
lock(&n->list_lock);
lock(fs_reclaim);
<Interrupt>
lock(&n->list_lock);
*** DEADLOCK ***
1 lock held by torture_shutdow/112:
#0: ffffa37a02ab4098 (&n->list_lock){-.-.}-{2:2}, at: free_debug_processing+0x3a/0x230
the dependencies between HARDIRQ-irq-safe lock and the holding lock:
-> (&n->list_lock){-.-.}-{2:2} {
IN-HARDIRQ-W at:
lock_acquire+0xb5/0x3a0
_raw_spin_lock+0x27/0x40
deactivate_slab.isra.90+0x3c2/0x5d0
flush_cpu_slab+0x31/0x50
flush_smp_call_function_queue+0x132/0x1d0
__sysvec_call_function_single+0x3e/0x190
sysvec_call_function_single+0x61/0x80
asm_sysvec_call_function_single+0x12/0x20
default_idle+0xb/0x10
default_idle_call+0x66/0x1e0
do_idle+0x204/0x2a0
cpu_startup_entry+0x14/0x20
secondary_startup_64_no_verify+0xc2/0xcb
IN-SOFTIRQ-W at:
lock_acquire+0xb5/0x3a0
_raw_spin_lock_irqsave+0x30/0x50
__slab_free+0xc8/0x4a0
kmem_cache_free+0x2a8/0x310
rcu_do_batch+0x18e/0x4d0
rcu_core+0x2c7/0x470
__do_softirq+0xca/0x409
irq_exit_rcu+0xa9/0xb0
sysvec_apic_timer_interrupt+0x66/0x80
asm_sysvec_apic_timer_interrupt+0x12/0x20
default_idle+0xb/0x10
default_idle_call+0x66/0x1e0
do_idle+0x204/0x2a0
cpu_startup_entry+0x14/0x20
start_kernel+0x6ab/0x6d2
secondary_startup_64_no_verify+0xc2/0xcb
INITIAL USE at:
lock_acquire+0xb5/0x3a0
_raw_spin_lock+0x27/0x40
get_partial_node.isra.92.part.93+0x47/0x300
___slab_alloc+0x2a1/0x6e0
kmem_cache_alloc_node+0x1bc/0x230
__kmem_cache_create+0x1b3/0x430
create_boot_cache+0x70/0x93
kmem_cache_init+0xd9/0x15f
start_kernel+0x3e6/0x6d2
secondary_startup_64_no_verify+0xc2/0xcb
}
... key at: [<ffffffffabfa6370>] __key.50785+0x0/0x10
... acquired at:
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
__alloc_pages+0xed/0x330
stack_depot_save+0x3a1/0x470
save_stack_depot_trace.constprop.100+0x3b/0x60
set_track+0x27/0x80
free_debug_processing+0x169/0x230
__slab_free+0x262/0x4a0
kmem_cache_free+0x2a8/0x310
rcu_torture_cleanup.cold.34+0x186/0x41d
torture_shutdown+0x101/0x1f0
kthread+0x122/0x140
ret_from_fork+0x22/0x30
the dependencies between the lock to be acquired
and HARDIRQ-irq-unsafe lock:
-> (fs_reclaim){+.+.}-{0:0} {
HARDIRQ-ON-W at:
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
kmem_cache_alloc_node_trace+0x2f/0x230
alloc_worker+0x18/0x50
init_rescuer.part.33+0xf/0x90
workqueue_init+0x1fb/0x2db
kernel_init_freeable+0x10c/0x26f
kernel_init+0x5/0xfe
ret_from_fork+0x22/0x30
SOFTIRQ-ON-W at:
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
kmem_cache_alloc_node_trace+0x2f/0x230
alloc_worker+0x18/0x50
init_rescuer.part.33+0xf/0x90
workqueue_init+0x1fb/0x2db
kernel_init_freeable+0x10c/0x26f
kernel_init+0x5/0xfe
ret_from_fork+0x22/0x30
INITIAL USE at:
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
kmem_cache_alloc_node_trace+0x2f/0x230
alloc_worker+0x18/0x50
init_rescuer.part.33+0xf/0x90
workqueue_init+0x1fb/0x2db
kernel_init_freeable+0x10c/0x26f
kernel_init+0x5/0xfe
ret_from_fork+0x22/0x30
}
... key at: [<ffffffffaad7ac20>] __fs_reclaim_map+0x0/0x28
... acquired at:
lock_acquire+0xb5/0x3a0
fs_reclaim_acquire+0x9f/0xd0
__alloc_pages+0xed/0x330
stack_depot_save+0x3a1/0x470
save_stack_depot_trace.constprop.100+0x3b/0x60
set_track+0x27/0x80
free_debug_processing+0x169/0x230
__slab_free+0x262/0x4a0
kmem_cache_free+0x2a8/0x310
rcu_torture_cleanup.cold.34+0x186/0x41d
torture_shutdown+0x101/0x1f0
kthread+0x122/0x140
ret_from_fork+0x22/0x30
stack backtrace:
CPU: 0 PID: 112 Comm: torture_shutdow Not tainted 5.13.0-rc1-next-20210514 #1587
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module_el8.4.0+547+a85d02ba 04/01/2014
Call Trace:
dump_stack_lvl+0x6a/0x88
check_irq_usage.cold.80+0x2cd/0x2d2
? trace_event_raw_event_lock+0xf0/0xf0
? check_path.constprop.61+0x24/0x40
? check_noncircular+0x7e/0x110
? __lock_acquire+0x13a0/0x23c0
__lock_acquire+0x13a0/0x23c0
lock_acquire+0xb5/0x3a0
? fs_reclaim_acquire+0x7f/0xd0
? __lock_acquire+0xa8e/0x23c0
? get_stack_info_noinstr+0x12/0xe0
? unwind_next_frame+0x4a3/0x6a0
fs_reclaim_acquire+0x9f/0xd0
? fs_reclaim_acquire+0x7f/0xd0
__alloc_pages+0xed/0x330
stack_depot_save+0x3a1/0x470
? rcu_torture_cleanup.cold.34+0x186/0x41d
save_stack_depot_trace.constprop.100+0x3b/0x60
? kmem_cache_free+0x2a8/0x310
? rcu_torture_cleanup.cold.34+0x186/0x41d
? torture_shutdown+0x101/0x1f0
? kthread+0x122/0x140
? ret_from_fork+0x22/0x30
set_track+0x27/0x80
free_debug_processing+0x169/0x230
? rcu_torture_cleanup.cold.34+0x186/0x41d
? rcu_torture_cleanup.cold.34+0x186/0x41d
__slab_free+0x262/0x4a0
? vprintk_emit+0x89/0x270
? torture_onoff+0x260/0x260
? rcu_torture_cleanup.cold.34+0x186/0x41d
kmem_cache_free+0x2a8/0x310
? torture_onoff+0x260/0x260
rcu_torture_cleanup.cold.34+0x186/0x41d
? wait_woken+0x80/0x80
? torture_onoff+0x260/0x260
torture_shutdown+0x101/0x1f0
kthread+0x122/0x140
? kthread_park+0x80/0x80
ret_from_fork+0x22/0x30
next reply other threads:[~2021-05-15 20:46 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-15 20:46 Paul E. McKenney [this message]
2021-05-15 21:09 ` [BUG] Shudown-time lockdep splat in next-20210514 Vlastimil Babka
2021-05-15 23:19 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210515204622.GA2672367@paulmck-ThinkPad-P17-Gen-1 \
--to=paulmck@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=jweiner@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ming.lei@redhat.com \
--cc=penberg@kernel.org \
--cc=riel@fb.com \
--cc=rientjes@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.