linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Tejun Heo <tj@kernel.org>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Jens Axboe <axboe@kernel.dk>, Sasha Levin <sashal@kernel.org>
Subject: [PATCH 5.4 18/24] blk-cgroup: Use cond_resched() when destroy blkgs
Date: Thu, 11 Feb 2021 16:02:41 +0100	[thread overview]
Message-ID: <20210211150149.315436519@linuxfoundation.org> (raw)
In-Reply-To: <20210211150148.516371325@linuxfoundation.org>

From: Baolin Wang <baolin.wang@linux.alibaba.com>

[ Upstream commit 6c635caef410aa757befbd8857c1eadde5cc22ed ]

On !PREEMPT kernel, we can get below softlockup when doing stress
testing with creating and destroying block cgroup repeatly. The
reason is it may take a long time to acquire the queue's lock in
the loop of blkcg_destroy_blkgs(), or the system can accumulate a
huge number of blkgs in pathological cases. We can add a need_resched()
check on each loop and release locks and do cond_resched() if true
to avoid this issue, since the blkcg_destroy_blkgs() is not called
from atomic contexts.

[ 4757.010308] watchdog: BUG: soft lockup - CPU#11 stuck for 94s!
[ 4757.010698] Call trace:
[ 4757.010700]  blkcg_destroy_blkgs+0x68/0x150
[ 4757.010701]  cgwb_release_workfn+0x104/0x158
[ 4757.010702]  process_one_work+0x1bc/0x3f0
[ 4757.010704]  worker_thread+0x164/0x468
[ 4757.010705]  kthread+0x108/0x138

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 block/blk-cgroup.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 3d34ac02d76ef..cb3d44d200055 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1089,6 +1089,8 @@ static void blkcg_css_offline(struct cgroup_subsys_state *css)
  */
 void blkcg_destroy_blkgs(struct blkcg *blkcg)
 {
+	might_sleep();
+
 	spin_lock_irq(&blkcg->lock);
 
 	while (!hlist_empty(&blkcg->blkg_list)) {
@@ -1096,14 +1098,20 @@ void blkcg_destroy_blkgs(struct blkcg *blkcg)
 						struct blkcg_gq, blkcg_node);
 		struct request_queue *q = blkg->q;
 
-		if (spin_trylock(&q->queue_lock)) {
-			blkg_destroy(blkg);
-			spin_unlock(&q->queue_lock);
-		} else {
+		if (need_resched() || !spin_trylock(&q->queue_lock)) {
+			/*
+			 * Given that the system can accumulate a huge number
+			 * of blkgs in pathological cases, check to see if we
+			 * need to rescheduling to avoid softlockup.
+			 */
 			spin_unlock_irq(&blkcg->lock);
-			cpu_relax();
+			cond_resched();
 			spin_lock_irq(&blkcg->lock);
+			continue;
 		}
+
+		blkg_destroy(blkg);
+		spin_unlock(&q->queue_lock);
 	}
 
 	spin_unlock_irq(&blkcg->lock);
-- 
2.27.0




  parent reply	other threads:[~2021-02-11 15:49 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-11 15:02 [PATCH 5.4 00/24] 5.4.98-rc1 review Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 01/24] tracing/kprobe: Fix to support kretprobe events on unloaded modules Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 02/24] af_key: relax availability checks for skb size calculation Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 03/24] regulator: core: avoid regulator_resolve_supply() race condition Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 04/24] mac80211: 160MHz with extended NSS BW in CSA Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 05/24] ASoC: Intel: Skylake: Zero snd_ctl_elem_value Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 06/24] chtls: Fix potential resource leak Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 07/24] pNFS/NFSv4: Try to return invalid layout in pnfs_layout_process() Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 08/24] ASoC: ak4458: correct reset polarity Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 09/24] iwlwifi: mvm: skip power command when unbinding vif during CSA Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 10/24] iwlwifi: mvm: take mutex for calling iwl_mvm_get_sync_time() Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 11/24] iwlwifi: pcie: add a NULL check in iwl_pcie_txq_unmap Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 12/24] iwlwifi: pcie: fix context info memory leak Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 13/24] iwlwifi: mvm: invalidate IDs of internal stations at mvm start Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 14/24] iwlwifi: mvm: guard against device removal in reprobe Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 15/24] SUNRPC: Move simple_get_bytes and simple_get_netobj into private header Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 16/24] SUNRPC: Handle 0 length opaque XDR object data properly Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 17/24] i2c: mediatek: Move suspend and resume handling to NOIRQ phase Greg Kroah-Hartman
2021-02-11 15:02 ` Greg Kroah-Hartman [this message]
2021-02-11 15:02 ` [PATCH 5.4 19/24] regulator: Fix lockdep warning resolving supplies Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 20/24] bpf: Fix 32 bit src register truncation on div/mod Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 21/24] Fix unsynchronized access to sev members through svm_register_enc_region Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 22/24] squashfs: add more sanity checks in id lookup Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 23/24] squashfs: add more sanity checks in inode lookup Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.4 24/24] squashfs: add more sanity checks in xattr id lookup Greg Kroah-Hartman
2021-02-12  4:24 ` [PATCH 5.4 00/24] 5.4.98-rc1 review Naresh Kamboju
2021-02-12 15:20 ` Naresh Kamboju
2021-02-12 16:17 ` Shuah Khan
2021-02-12 18:08 ` Guenter Roeck
2021-02-12 19:02 ` Florian Fainelli
2021-02-13  3:17 ` Ross Schmidt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210211150149.315436519@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=sashal@kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).