linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Tejun Heo <tj@kernel.org>,
	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 4.14 38/52] bdi: Move cgroup bdi_writeback to a dedicated low concurrency workqueue
Date: Sun, 24 Jun 2018 23:21:31 +0800	[thread overview]
Message-ID: <20180624142746.732467308@linuxfoundation.org> (raw)
In-Reply-To: <20180624142744.234164867@linuxfoundation.org>

4.14-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Tejun Heo <tj@kernel.org>

commit f183464684190bacbfb14623bd3e4e51b7575b4c upstream.

>From 0aa2e9b921d6db71150633ff290199554f0842a8 Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Wed, 23 May 2018 10:29:00 -0700

cgwb_release() punts the actual release to cgwb_release_workfn() on
system_wq.  Depending on the number of cgroups or block devices, there
can be a lot of cgwb_release_workfn() in flight at the same time.

We're periodically seeing close to 256 kworkers getting stuck with the
following stack trace and overtime the entire system gets stuck.

  [<ffffffff810ee40c>] _synchronize_rcu_expedited.constprop.72+0x2fc/0x330
  [<ffffffff810ee634>] synchronize_rcu_expedited+0x24/0x30
  [<ffffffff811ccf23>] bdi_unregister+0x53/0x290
  [<ffffffff811cd1e9>] release_bdi+0x89/0xc0
  [<ffffffff811cd645>] wb_exit+0x85/0xa0
  [<ffffffff811cdc84>] cgwb_release_workfn+0x54/0xb0
  [<ffffffff810a68d0>] process_one_work+0x150/0x410
  [<ffffffff810a71fd>] worker_thread+0x6d/0x520
  [<ffffffff810ad3dc>] kthread+0x12c/0x160
  [<ffffffff81969019>] ret_from_fork+0x29/0x40
  [<ffffffffffffffff>] 0xffffffffffffffff

The events leading to the lockup are...

1. A lot of cgwb_release_workfn() is queued at the same time and all
   system_wq kworkers are assigned to execute them.

2. They all end up calling synchronize_rcu_expedited().  One of them
   wins and tries to perform the expedited synchronization.

3. However, that invovles queueing rcu_exp_work to system_wq and
   waiting for it.  Because #1 is holding all available kworkers on
   system_wq, rcu_exp_work can't be executed.  cgwb_release_workfn()
   is waiting for synchronize_rcu_expedited() which in turn is waiting
   for cgwb_release_workfn() to free up some of the kworkers.

We shouldn't be scheduling hundreds of cgwb_release_workfn() at the
same time.  There's nothing to be gained from that.  This patch
updates cgwb release path to use a dedicated percpu workqueue with
@max_active of 1.

While this resolves the problem at hand, it might be a good idea to
isolate rcu_exp_work to its own workqueue too as it can be used from
various paths and is prone to this sort of indirect A-A deadlocks.

Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 mm/backing-dev.c |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -409,6 +409,7 @@ static void wb_exit(struct bdi_writeback
  * protected.
  */
 static DEFINE_SPINLOCK(cgwb_lock);
+static struct workqueue_struct *cgwb_release_wq;
 
 /**
  * wb_congested_get_create - get or create a wb_congested
@@ -519,7 +520,7 @@ static void cgwb_release(struct percpu_r
 {
 	struct bdi_writeback *wb = container_of(refcnt, struct bdi_writeback,
 						refcnt);
-	schedule_work(&wb->release_work);
+	queue_work(cgwb_release_wq, &wb->release_work);
 }
 
 static void cgwb_kill(struct bdi_writeback *wb)
@@ -783,6 +784,21 @@ static void cgwb_bdi_register(struct bac
 	spin_unlock_irq(&cgwb_lock);
 }
 
+static int __init cgwb_init(void)
+{
+	/*
+	 * There can be many concurrent release work items overwhelming
+	 * system_wq.  Put them in a separate wq and limit concurrency.
+	 * There's no point in executing many of these in parallel.
+	 */
+	cgwb_release_wq = alloc_workqueue("cgwb_release", 0, 1);
+	if (!cgwb_release_wq)
+		return -ENOMEM;
+
+	return 0;
+}
+subsys_initcall(cgwb_init);
+
 #else	/* CONFIG_CGROUP_WRITEBACK */
 
 static int cgwb_bdi_init(struct backing_dev_info *bdi)



  parent reply	other threads:[~2018-06-24 15:31 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-06-24 15:20 [PATCH 4.14 00/52] 4.14.52-stable review Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 01/52] bonding: re-evaluate force_primary when the primary slave name changes Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 03/52] ipv6: allow PMTU exceptions to local routes Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 04/52] net: dsa: add error handling for pskb_trim_rcsum Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 05/52] net/sched: act_simple: fix parsing of TCA_DEF_DATA Greg Kroah-Hartman
2018-06-24 15:20 ` [PATCH 4.14 06/52] tcp: verify the checksum of the first data segment in a new connection Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 07/52] socket: close race condition between sock_close() and sockfs_setattr() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 08/52] udp: fix rx queue len reported by diag and proc interface Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 09/52] net: in virtio_net_hdr only add VLAN_HLEN to csum_start if payload holds vlan Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 10/52] hv_netvsc: Fix a network regression after ifdown/ifup Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 11/52] tls: fix use-after-free in tls_push_record Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 12/52] NFSv4.1: Fix up replays of interrupted requests Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 13/52] ext4: fix hole length detection in ext4_ind_map_blocks() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 14/52] ext4: update mtime in ext4_punch_hole even if no blocks are released Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 15/52] ext4: do not allow external inodes for inline data Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 16/52] ext4: bubble errors from ext4_find_inline_data_nolock() up to ext4_iget() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 17/52] ext4: correctly handle a zero-length xattr with a non-zero e_value_offs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 18/52] ext4: fix fencepost error in check for inode count overflow during resize Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 19/52] driver core: Dont ignore class_dir_create_and_add() failure Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 20/52] Btrfs: fix clone vs chattr NODATASUM race Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 21/52] Btrfs: fix memory and mount leak in btrfs_ioctl_rm_dev_v2() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 22/52] btrfs: return error value if create_io_em failed in cow_file_range Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 23/52] btrfs: scrub: Dont use inode pages for device replace Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 24/52] ALSA: hda/realtek - Enable mic-mute hotkey for several Lenovo AIOs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 25/52] ALSA: hda/conexant - Add fixup for HP Z2 G4 workstation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 26/52] ALSA: hda - Handle kzalloc() failure in snd_hda_attach_pcm_stream() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 27/52] ALSA: hda: add dock and led support for HP EliteBook 830 G5 Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 28/52] ALSA: hda: add dock and led support for HP ProBook 640 G4 Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 29/52] x86/MCE: Fix stack out-of-bounds write in mce-inject.c: Flags_read() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 30/52] smb3: fix various xid leaks Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 31/52] smb3: on reconnect set PreviousSessionId field Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 32/52] CIFS: 511c54a2f69195b28afb9dd119f03787b1625bb4 adds a check for session expiry Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 33/52] cifs: For SMB2 security informaion query, check for minimum sized security descriptor instead of sizeof FileAllInformation class Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 34/52] nbd: fix nbd device deletion Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 35/52] nbd: update size when connected Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 36/52] nbd: use bd_set_size when updating disk size Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 37/52] blk-mq: reinit q->tag_set_list entry only after grace period Greg Kroah-Hartman
2018-06-24 15:21 ` Greg Kroah-Hartman [this message]
2018-06-24 15:21 ` [PATCH 4.14 39/52] cpufreq: Fix new policy initialization during limits updates via sysfs Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 40/52] cpufreq: governors: Fix long idle detection logic in load calculation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 41/52] libata: zpodd: small read overflow in eject_tray() Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 42/52] libata: Drop SanDisk SD7UB3Q*G1001 NOLPM quirk Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 43/52] w1: mxc_w1: Enable clock before calling clk_get_rate() on it Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 44/52] x86/intel_rdt: Enable CMT and MBM on new Skylake stepping Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 45/52] iwlwifi: fw: harden page loading code Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 46/52] orangefs: set i_size on new symlink Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 47/52] orangefs: report attributes_mask and attributes for statx Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 48/52] HID: intel_ish-hid: ipc: register more pm callbacks to support hibernation Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 49/52] HID: wacom: Correct logical maximum Y for 2nd-gen Intuos Pro large Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 50/52] vhost: fix info leak due to uninitialized memory Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 51/52] fs/binfmt_misc.c: do not allow offset overflow Greg Kroah-Hartman
2018-06-24 15:21 ` [PATCH 4.14 52/52] mm, page_alloc: do not break __GFP_THISNODE by zonelist reset Greg Kroah-Hartman
2018-06-25  6:43 ` [PATCH 4.14 00/52] 4.14.52-stable review Naresh Kamboju
2018-06-25 17:19 ` Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180624142746.732467308@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=linux-kernel@vger.kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).