All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>,
	Tejun Heo <tj@kernel.org>, Jens Axboe <axboe@kernel.dk>,
	Sasha Levin <sashal@kernel.org>,
	cgroups@vger.kernel.org, linux-block@vger.kernel.org
Subject: [PATCH AUTOSEL 5.4 17/17] blk-cgroup: Use cond_resched() when destroy blkgs
Date: Tue,  2 Feb 2021 10:06:51 -0500	[thread overview]
Message-ID: <20210202150651.1864426-17-sashal@kernel.org> (raw)
In-Reply-To: <20210202150651.1864426-1-sashal@kernel.org>

From: Baolin Wang <baolin.wang@linux.alibaba.com>

[ Upstream commit 6c635caef410aa757befbd8857c1eadde5cc22ed ]

On !PREEMPT kernel, we can get below softlockup when doing stress
testing with creating and destroying block cgroup repeatly. The
reason is it may take a long time to acquire the queue's lock in
the loop of blkcg_destroy_blkgs(), or the system can accumulate a
huge number of blkgs in pathological cases. We can add a need_resched()
check on each loop and release locks and do cond_resched() if true
to avoid this issue, since the blkcg_destroy_blkgs() is not called
from atomic contexts.

[ 4757.010308] watchdog: BUG: soft lockup - CPU#11 stuck for 94s!
[ 4757.010698] Call trace:
[ 4757.010700]  blkcg_destroy_blkgs+0x68/0x150
[ 4757.010701]  cgwb_release_workfn+0x104/0x158
[ 4757.010702]  process_one_work+0x1bc/0x3f0
[ 4757.010704]  worker_thread+0x164/0x468
[ 4757.010705]  kthread+0x108/0x138

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 block/blk-cgroup.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 3d34ac02d76ef..cb3d44d200055 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1089,6 +1089,8 @@ static void blkcg_css_offline(struct cgroup_subsys_state *css)
  */
 void blkcg_destroy_blkgs(struct blkcg *blkcg)
 {
+	might_sleep();
+
 	spin_lock_irq(&blkcg->lock);
 
 	while (!hlist_empty(&blkcg->blkg_list)) {
@@ -1096,14 +1098,20 @@ void blkcg_destroy_blkgs(struct blkcg *blkcg)
 						struct blkcg_gq, blkcg_node);
 		struct request_queue *q = blkg->q;
 
-		if (spin_trylock(&q->queue_lock)) {
-			blkg_destroy(blkg);
-			spin_unlock(&q->queue_lock);
-		} else {
+		if (need_resched() || !spin_trylock(&q->queue_lock)) {
+			/*
+			 * Given that the system can accumulate a huge number
+			 * of blkgs in pathological cases, check to see if we
+			 * need to rescheduling to avoid softlockup.
+			 */
 			spin_unlock_irq(&blkcg->lock);
-			cpu_relax();
+			cond_resched();
 			spin_lock_irq(&blkcg->lock);
+			continue;
 		}
+
+		blkg_destroy(blkg);
+		spin_unlock(&q->queue_lock);
 	}
 
 	spin_unlock_irq(&blkcg->lock);
-- 
2.27.0


      parent reply	other threads:[~2021-02-02 15:49 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-02 15:06 [PATCH AUTOSEL 5.4 01/17] af_key: relax availability checks for skb size calculation Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 02/17] regulator: core: avoid regulator_resolve_supply() race condition Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 03/17] mac80211: 160MHz with extended NSS BW in CSA Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 04/17] ASoC: Intel: Skylake: Zero snd_ctl_elem_value Sasha Levin
2021-02-02 15:06   ` Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 05/17] chtls: Fix potential resource leak Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 06/17] pNFS/NFSv4: Try to return invalid layout in pnfs_layout_process() Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 07/17] ASoC: ak4458: correct reset polarity Sasha Levin
2021-02-02 15:06   ` Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 08/17] iwlwifi: mvm: skip power command when unbinding vif during CSA Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 09/17] iwlwifi: mvm: take mutex for calling iwl_mvm_get_sync_time() Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 10/17] iwlwifi: pcie: add a NULL check in iwl_pcie_txq_unmap Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 11/17] iwlwifi: pcie: fix context info memory leak Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 12/17] iwlwifi: mvm: invalidate IDs of internal stations at mvm start Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 13/17] iwlwifi: mvm: guard against device removal in reprobe Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 14/17] SUNRPC: Move simple_get_bytes and simple_get_netobj into private header Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 15/17] SUNRPC: Handle 0 length opaque XDR object data properly Sasha Levin
2021-02-02 15:06 ` [PATCH AUTOSEL 5.4 16/17] i2c: mediatek: Move suspend and resume handling to NOIRQ phase Sasha Levin
2021-02-02 15:06   ` Sasha Levin
2021-02-02 15:06   ` Sasha Levin
2021-02-02 15:06 ` Sasha Levin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210202150651.1864426-17-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=axboe@kernel.dk \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=cgroups@vger.kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.