From: Maurizio Lombardi <mlombard@redhat.com>
To: Xin Long <lucien.xin@gmail.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Christoph Lameter <cl@linux.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Andrew Morton <akpm@linux-foundation.org>,
Vlastimil Babka <vbabka@suse.cz>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Antoine Tenart <atenart@kernel.org>
Subject: Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy
Date: Wed, 1 Jun 2022 16:36:06 +0200 [thread overview]
Message-ID: <CAFL455m6BZW-KVNOkJPo3BSeEK39Vq597F-b5XvQJmU3mrPmZA@mail.gmail.com> (raw)
In-Reply-To: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com>
pá 14. 1. 2022 v 15:23 odesílatel Xin Long <lucien.xin@gmail.com> napsal:
>
> cpus_read_lock() is introduced into kmem_cache_destroy() by
> commit 5a836bf6b09f ("mm: slub: move flush_cpu_slab() invocations
> __free_slab() invocations out of IRQ context"), and it could cause
> a deadlock.
FYI,
I received a bug report from one of our customers, he complains that
his system (with nvmefc boot from SAN) hangs when rebooting.
He runs a RHEL-9 kernel based on version 5.14.0.
What is interesting is that, according to him, after reverting commit
5a836bf6b09f
("mm: slub: move flush_cpu_slab() invocations __free_slab()
invocations out of IRQ context")
the reboot operation doesn't hang anymore.
The call trace seems to point to a possible problem due to the fact that
nvme_delete_ctrl_work is allocated with the WQ_MEM_RECLAIM bit set.
[ 453.012078] ------------[ cut here ]------------
[ 453.016744] workqueue: WQ_MEM_RECLAIM
nvme-delete-wq:nvme_delete_ctrl_work [nvme_core] is flushing
!WQ_MEM_RECLAIM events:flush_cpu_slab
[ 453.016789] WARNING: CPU: 37 PID: 410 at kernel/workqueue.c:2637
check_flush_dependency+0x10a/0x120
[...]
[ 453.262125] Call Trace:
[ 453.264582] __flush_work.isra.0+0xbf/0x220
[ 453.268775] ? __queue_work+0x1dc/0x420
[ 453.272623] flush_all_cpus_locked+0xfb/0x120
[ 453.276992] __kmem_cache_shutdown+0x2b/0x320
[ 453.281361] kmem_cache_destroy+0x49/0x100
[ 453.285465] bioset_exit+0x143/0x190
[ 453.289052] blk_release_queue+0xb9/0x100
[ 453.293075] kobject_cleanup+0x37/0x130
[ 453.296922] nvme_fc_ctrl_free+0xc6/0x150 [nvme_fc]
[ 453.302397] nvme_free_ctrl+0x1ac/0x2b0 [nvme_core]
[ 453.307818] device_release+0x31/0x90
[ 453.312005] kobject_cleanup+0x37/0x130
[ 453.316369] process_one_work+0x1e5/0x3c0
[ 453.320895] worker_thread+0x50/0x3b0
[ 453.325074] ? rescuer_thread+0x370/0x370
[ 453.329592] kthread+0x146/0x170
[ 453.333322] ? set_kthread_struct+0x40/0x40
[ 453.338027] ret_from_fork+0x1f/0x30
[ 453.342082] ---[ end trace 8c9cdd85adbbfc4f ]---
Maurizio Lombardi
prev parent reply other threads:[~2022-06-01 14:36 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-14 14:23 [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy Xin Long
2022-01-16 6:35 ` Hyeonggon Yoo
2022-01-17 8:32 ` Xin Long
2022-01-17 9:33 ` Sebastian Andrzej Siewior
2022-01-17 12:40 ` Vlastimil Babka
2022-01-17 13:13 ` Juri Lelli
2022-01-18 8:00 ` Xin Long
2022-01-18 11:07 ` Vlastimil Babka
2022-01-19 8:38 ` Xin Long
2022-06-01 14:36 ` Maurizio Lombardi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFL455m6BZW-KVNOkJPo3BSeEK39Vq597F-b5XvQJmU3mrPmZA@mail.gmail.com \
--to=mlombard@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=atenart@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=cl@linux.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lucien.xin@gmail.com \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).