From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Tejun Heo <tj@kernel.org>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 4.14 38/52] bdi: Move cgroup bdi_writeback to a dedicated low concurrency workqueue
Date: Sun, 24 Jun 2018 23:21:31 +0800 [thread overview]
Message-ID: <20180624142746.732467308@linuxfoundation.org> (raw)
In-Reply-To: <20180624142744.234164867@linuxfoundation.org>
4.14-stable review patch. If anyone has any objections, please let me know.
------------------
From: Tejun Heo <tj@kernel.org>
commit f183464684190bacbfb14623bd3e4e51b7575b4c upstream.
>From 0aa2e9b921d6db71150633ff290199554f0842a8 Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Wed, 23 May 2018 10:29:00 -0700
cgwb_release() punts the actual release to cgwb_release_workfn() on
system_wq. Depending on the number of cgroups or block devices, there
can be a lot of cgwb_release_workfn() in flight at the same time.
We're periodically seeing close to 256 kworkers getting stuck with the
following stack trace and overtime the entire system gets stuck.
[<ffffffff810ee40c>] _synchronize_rcu_expedited.constprop.72+0x2fc/0x330
[<ffffffff810ee634>] synchronize_rcu_expedited+0x24/0x30
[<ffffffff811ccf23>] bdi_unregister+0x53/0x290
[<ffffffff811cd1e9>] release_bdi+0x89/0xc0
[<ffffffff811cd645>] wb_exit+0x85/0xa0
[<ffffffff811cdc84>] cgwb_release_workfn+0x54/0xb0
[<ffffffff810a68d0>] process_one_work+0x150/0x410
[<ffffffff810a71fd>] worker_thread+0x6d/0x520
[<ffffffff810ad3dc>] kthread+0x12c/0x160
[<ffffffff81969019>] ret_from_fork+0x29/0x40
[<ffffffffffffffff>] 0xffffffffffffffff
The events leading to the lockup are...
1. A lot of cgwb_release_workfn() is queued at the same time and all
system_wq kworkers are assigned to execute them.
2. They all end up calling synchronize_rcu_expedited(). One of them
wins and tries to perform the expedited synchronization.
3. However, that invovles queueing rcu_exp_work to system_wq and
waiting for it. Because #1 is holding all available kworkers on
system_wq, rcu_exp_work can't be executed. cgwb_release_workfn()
is waiting for synchronize_rcu_expedited() which in turn is waiting
for cgwb_release_workfn() to free up some of the kworkers.
We shouldn't be scheduling hundreds of cgwb_release_workfn() at the
same time. There's nothing to be gained from that. This patch
updates cgwb release path to use a dedicated percpu workqueue with
@max_active of 1.
While this resolves the problem at hand, it might be a good idea to
isolate rcu_exp_work to its own workqueue too as it can be used from
various paths and is prone to this sort of indirect A-A deadlocks.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
mm/backing-dev.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -409,6 +409,7 @@ static void wb_exit(struct bdi_writeback
* protected.
*/
static DEFINE_SPINLOCK(cgwb_lock);
+static struct workqueue_struct *cgwb_release_wq;
/**
* wb_congested_get_create - get or create a wb_congested
@@ -519,7 +520,7 @@ static void cgwb_release(struct percpu_r
{
struct bdi_writeback *wb = container_of(refcnt, struct bdi_writeback,
refcnt);
- schedule_work(&wb->release_work);
+ queue_work(cgwb_release_wq, &wb->release_work);
}
static void cgwb_kill(struct bdi_writeback *wb)
@@ -783,6 +784,21 @@ static void cgwb_bdi_register(struct bac
spin_unlock_irq(&cgwb_lock);
}
+static int __init cgwb_init(void)
+{
+ /*
+ * There can be many concurrent release work items overwhelming
+ * system_wq. Put them in a separate wq and limit concurrency.
+ * There's no point in executing many of these in parallel.
+ */
+ cgwb_release_wq = alloc_workqueue("cgwb_release", 0, 1);
+ if (!cgwb_release_wq)
+ return -ENOMEM;
+
+ return 0;
+}
+subsys_initcall(cgwb_init);
+
#else /* CONFIG_CGROUP_WRITEBACK */
static int cgwb_bdi_init(struct backing_dev_info *bdi)
next prev parent reply other threads:[~2018-06-24 15:31 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-24 15:20 [PATCH 4.14 00/52] 4.14.52-stable review Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 01/52] bonding: re-evaluate force_primary when the primary slave name changes Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 03/52] ipv6: allow PMTU exceptions to local routes Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 04/52] net: dsa: add error handling for pskb_trim_rcsum Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 05/52] net/sched: act_simple: fix parsing of TCA_DEF_DATA Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 06/52] tcp: verify the checksum of the first data segment in a new connection Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 07/52] socket: close race condition between sock_close() and sockfs_setattr() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 08/52] udp: fix rx queue len reported by diag and proc interface Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 09/52] net: in virtio_net_hdr only add VLAN_HLEN to csum_start if payload holds vlan Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 10/52] hv_netvsc: Fix a network regression after ifdown/ifup Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 11/52] tls: fix use-after-free in tls_push_record Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 12/52] NFSv4.1: Fix up replays of interrupted requests Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 13/52] ext4: fix hole length detection in ext4_ind_map_blocks() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 14/52] ext4: update mtime in ext4_punch_hole even if no blocks are released Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 15/52] ext4: do not allow external inodes for inline data Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 16/52] ext4: bubble errors from ext4_find_inline_data_nolock() up to ext4_iget() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 17/52] ext4: correctly handle a zero-length xattr with a non-zero e_value_offs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 18/52] ext4: fix fencepost error in check for inode count overflow during resize Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 19/52] driver core: Dont ignore class_dir_create_and_add() failure Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 20/52] Btrfs: fix clone vs chattr NODATASUM race Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 21/52] Btrfs: fix memory and mount leak in btrfs_ioctl_rm_dev_v2() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 22/52] btrfs: return error value if create_io_em failed in cow_file_range Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 23/52] btrfs: scrub: Dont use inode pages for device replace Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 24/52] ALSA: hda/realtek - Enable mic-mute hotkey for several Lenovo AIOs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 25/52] ALSA: hda/conexant - Add fixup for HP Z2 G4 workstation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 26/52] ALSA: hda - Handle kzalloc() failure in snd_hda_attach_pcm_stream() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 27/52] ALSA: hda: add dock and led support for HP EliteBook 830 G5 Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 28/52] ALSA: hda: add dock and led support for HP ProBook 640 G4 Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 29/52] x86/MCE: Fix stack out-of-bounds write in mce-inject.c: Flags_read() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 30/52] smb3: fix various xid leaks Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 31/52] smb3: on reconnect set PreviousSessionId field Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 32/52] CIFS: 511c54a2f69195b28afb9dd119f03787b1625bb4 adds a check for session expiry Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 33/52] cifs: For SMB2 security informaion query, check for minimum sized security descriptor instead of sizeof FileAllInformation class Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 34/52] nbd: fix nbd device deletion Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 35/52] nbd: update size when connected Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 36/52] nbd: use bd_set_size when updating disk size Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 37/52] blk-mq: reinit q->tag_set_list entry only after grace period Greg Kroah-Hartman
2018-06-24 15:21 ` Greg Kroah-Hartman [this message]
2018-06-24 15:21 ` [PATCH 4.14 39/52] cpufreq: Fix new policy initialization during limits updates via sysfs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 40/52] cpufreq: governors: Fix long idle detection logic in load calculation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 41/52] libata: zpodd: small read overflow in eject_tray() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 42/52] libata: Drop SanDisk SD7UB3Q*G1001 NOLPM quirk Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 43/52] w1: mxc_w1: Enable clock before calling clk_get_rate() on it Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 44/52] x86/intel_rdt: Enable CMT and MBM on new Skylake stepping Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 45/52] iwlwifi: fw: harden page loading code Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 46/52] orangefs: set i_size on new symlink Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 47/52] orangefs: report attributes_mask and attributes for statx Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 48/52] HID: intel_ish-hid: ipc: register more pm callbacks to support hibernation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 49/52] HID: wacom: Correct logical maximum Y for 2nd-gen Intuos Pro large Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 50/52] vhost: fix info leak due to uninitialized memory Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 51/52] fs/binfmt_misc.c: do not allow offset overflow Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 52/52] mm, page_alloc: do not break __GFP_THISNODE by zonelist reset Greg Kroah-Hartman
2018-06-25 6:43 ` [PATCH 4.14 00/52] 4.14.52-stable review Naresh Kamboju
2018-06-25 17:19 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180624142746.732467308@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=axboe@kernel.dk \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=stable@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).