From: Thierry Reding <thierry.reding@gmail.com>
To: Tejun Heo <tj@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ulrich Obergfell <uobergfe@redhat.com>,
Ingo Molnar <mingo@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
linux-kernel@vger.kernel.org, kernel-team@fb.com,
Jon Hunter <jonathanh@nvidia.com>,
linux-tegra@vger.kernel.org
Subject: Re: [PATCH] workqueue: warn if memory reclaim tries to flush !WQ_MEM_RECLAIM workqueue
Date: Tue, 26 Jan 2016 18:38:43 +0100 [thread overview]
Message-ID: <20160126173843.GA11115@ulmo.nvidia.com> (raw)
In-Reply-To: <20151203192616.GJ27463@mtj.duckdns.org>
[-- Attachment #1: Type: text/plain, Size: 6277 bytes --]
On Thu, Dec 03, 2015 at 02:26:16PM -0500, Tejun Heo wrote:
> Task or work item involved in memory reclaim trying to flush a
> non-WQ_MEM_RECLAIM workqueue or one of its work items can lead to
> deadlock. Trigger WARN_ONCE() if such conditions are detected.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> ---
> Hello,
>
> So, something like this. Seems to work fine here. If there's no
> objection, I'm gonna push it through wq/for-4.5.
>
> Thanks.
>
> kernel/workqueue.c | 35 +++++++++++++++++++++++++++++++++++
> 1 file changed, 35 insertions(+)
>
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -2330,6 +2330,37 @@ repeat:
> goto repeat;
> }
>
> +/**
> + * check_flush_dependency - check for flush dependency sanity
> + * @target_wq: workqueue being flushed
> + * @target_work: work item being flushed (NULL for workqueue flushes)
> + *
> + * %current is trying to flush the whole @target_wq or @target_work on it.
> + * If @target_wq doesn't have %WQ_MEM_RECLAIM, verify that %current is not
> + * reclaiming memory or running on a workqueue which doesn't have
> + * %WQ_MEM_RECLAIM as that can break forward-progress guarantee leading to
> + * a deadlock.
> + */
> +static void check_flush_dependency(struct workqueue_struct *target_wq,
> + struct work_struct *target_work)
> +{
> + work_func_t target_func = target_work ? target_work->func : NULL;
> + struct worker *worker;
> +
> + if (target_wq->flags & WQ_MEM_RECLAIM)
> + return;
> +
> + worker = current_wq_worker();
> +
> + WARN_ONCE(current->flags & PF_MEMALLOC,
> + "workqueue: PF_MEMALLOC task %d(%s) is flushing !WQ_MEM_RECLAIM %s:%pf",
> + current->pid, current->comm, target_wq->name, target_func);
> + WARN_ONCE(worker && (worker->current_pwq->wq->flags & WQ_MEM_RECLAIM),
> + "workqueue: WQ_MEM_RECLAIM %s:%pf is flushing !WQ_MEM_RECLAIM %s:%pf",
> + worker->current_pwq->wq->name, worker->current_func,
> + target_wq->name, target_func);
> +}
> +
> struct wq_barrier {
> struct work_struct work;
> struct completion done;
> @@ -2539,6 +2570,8 @@ void flush_workqueue(struct workqueue_st
> list_add_tail(&this_flusher.list, &wq->flusher_overflow);
> }
>
> + check_flush_dependency(wq, NULL);
> +
> mutex_unlock(&wq->mutex);
>
> wait_for_completion(&this_flusher.done);
> @@ -2711,6 +2744,8 @@ static bool start_flush_work(struct work
> pwq = worker->current_pwq;
> }
>
> + check_flush_dependency(pwq->wq, work);
> +
> insert_wq_barrier(pwq, barr, work, worker);
> spin_unlock_irq(&pool->lock);
>
I've started noticing the following during boot on some of the devices I
work with:
[ 4.723705] WARNING: CPU: 0 PID: 6 at kernel/workqueue.c:2361 check_flush_dependency+0x138/0x144()
[ 4.736818] workqueue: WQ_MEM_RECLAIM deferwq:deferred_probe_work_func is flushing !WQ_MEM_RECLAIM events:lru_add_drain_per_cpu
[ 4.748099] Modules linked in:
[ 4.751342] CPU: 0 PID: 6 Comm: kworker/u8:0 Not tainted 4.5.0-rc1-00018-g420fc292d9c7 #1
[ 4.759504] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[ 4.765762] Workqueue: deferwq deferred_probe_work_func
[ 4.771004] [<c0017acc>] (unwind_backtrace) from [<c0013134>] (show_stack+0x10/0x14)
[ 4.778746] [<c0013134>] (show_stack) from [<c0245f18>] (dump_stack+0x94/0xd4)
[ 4.785966] [<c0245f18>] (dump_stack) from [<c0026f9c>] (warn_slowpath_common+0x80/0xb0)
[ 4.794048] [<c0026f9c>] (warn_slowpath_common) from [<c0026ffc>] (warn_slowpath_fmt+0x30/0x40)
[ 4.802736] [<c0026ffc>] (warn_slowpath_fmt) from [<c00390b8>] (check_flush_dependency+0x138/0x144)
[ 4.811769] [<c00390b8>] (check_flush_dependency) from [<c0039ca0>] (flush_work+0x50/0x15c)
[ 4.820112] [<c0039ca0>] (flush_work) from [<c00c51b0>] (lru_add_drain_all+0x130/0x180)
[ 4.828110] [<c00c51b0>] (lru_add_drain_all) from [<c00f728c>] (migrate_prep+0x8/0x10)
[ 4.836018] [<c00f728c>] (migrate_prep) from [<c00bfbc4>] (alloc_contig_range+0xd8/0x338)
[ 4.844186] [<c00bfbc4>] (alloc_contig_range) from [<c00f8f18>] (cma_alloc+0xe0/0x1ac)
[ 4.852093] [<c00f8f18>] (cma_alloc) from [<c001cac4>] (__alloc_from_contiguous+0x38/0xd8)
[ 4.860346] [<c001cac4>] (__alloc_from_contiguous) from [<c001ceb4>] (__dma_alloc+0x240/0x278)
[ 4.868944] [<c001ceb4>] (__dma_alloc) from [<c001cf78>] (arm_dma_alloc+0x54/0x5c)
[ 4.876506] [<c001cf78>] (arm_dma_alloc) from [<c0355ea4>] (dmam_alloc_coherent+0xc0/0xec)
[ 4.884764] [<c0355ea4>] (dmam_alloc_coherent) from [<c039cc4c>] (ahci_port_start+0x150/0x1dc)
[ 4.893367] [<c039cc4c>] (ahci_port_start) from [<c0384734>] (ata_host_start.part.3+0xc8/0x1c8)
[ 4.902055] [<c0384734>] (ata_host_start.part.3) from [<c03898dc>] (ata_host_activate+0x50/0x148)
[ 4.910919] [<c03898dc>] (ata_host_activate) from [<c039d558>] (ahci_host_activate+0x44/0x114)
[ 4.919523] [<c039d558>] (ahci_host_activate) from [<c039f05c>] (ahci_platform_init_host+0x1d8/0x3c8)
[ 4.928733] [<c039f05c>] (ahci_platform_init_host) from [<c039e6bc>] (tegra_ahci_probe+0x448/0x4e8)
[ 4.937770] [<c039e6bc>] (tegra_ahci_probe) from [<c0347058>] (platform_drv_probe+0x50/0xac)
[ 4.946197] [<c0347058>] (platform_drv_probe) from [<c03458cc>] (driver_probe_device+0x214/0x2c0)
[ 4.955061] [<c03458cc>] (driver_probe_device) from [<c0343cc0>] (bus_for_each_drv+0x60/0x94)
[ 4.963575] [<c0343cc0>] (bus_for_each_drv) from [<c03455d8>] (__device_attach+0xb0/0x114)
[ 4.971828] [<c03455d8>] (__device_attach) from [<c0344ab8>] (bus_probe_device+0x84/0x8c)
[ 4.979994] [<c0344ab8>] (bus_probe_device) from [<c0344f48>] (deferred_probe_work_func+0x68/0x98)
[ 4.988941] [<c0344f48>] (deferred_probe_work_func) from [<c003b738>] (process_one_work+0x120/0x3f8)
[ 4.998062] [<c003b738>] (process_one_work) from [<c003ba48>] (worker_thread+0x38/0x55c)
[ 5.006144] [<c003ba48>] (worker_thread) from [<c0040f14>] (kthread+0xdc/0xf4)
[ 5.013362] [<c0040f14>] (kthread) from [<c000f778>] (ret_from_fork+0x14/0x3c)
This seems to be caused by the interaction of the probe deferral
workqueue with WQ_MEM_RECLAIM workqueue. Any ideas on how to solve this?
Thierry
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
next prev parent reply other threads:[~2016-01-26 17:38 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-03 0:28 [PATCH 1/2] watchdog: introduce touch_softlockup_watchdog_sched() Tejun Heo
2015-12-03 0:28 ` [PATCH 2/2] workqueue: implement lockup detector Tejun Heo
2015-12-03 14:49 ` Tejun Heo
2015-12-03 17:50 ` Don Zickus
2015-12-03 19:43 ` Tejun Heo
2015-12-03 20:12 ` Ulrich Obergfell
2015-12-03 20:54 ` Tejun Heo
2015-12-04 8:02 ` Ingo Molnar
2015-12-04 16:52 ` Don Zickus
2015-12-04 13:19 ` Ulrich Obergfell
2015-12-07 19:06 ` [PATCH v2 " Tejun Heo
2015-12-07 21:38 ` Don Zickus
2015-12-07 21:39 ` Tejun Heo
2015-12-08 16:00 ` Don Zickus
2015-12-08 16:31 ` Tejun Heo
2015-12-03 9:33 ` [PATCH 1/2] watchdog: introduce touch_softlockup_watchdog_sched() Peter Zijlstra
2015-12-03 10:00 ` Peter Zijlstra
2015-12-03 14:48 ` Tejun Heo
2015-12-03 15:04 ` Peter Zijlstra
2015-12-03 15:06 ` Tejun Heo
2015-12-03 19:26 ` [PATCH] workqueue: warn if memory reclaim tries to flush !WQ_MEM_RECLAIM workqueue Tejun Heo
2015-12-03 20:43 ` Peter Zijlstra
2015-12-03 20:56 ` Tejun Heo
2015-12-03 21:09 ` Peter Zijlstra
2015-12-03 22:04 ` Tejun Heo
2015-12-04 12:51 ` Peter Zijlstra
2015-12-07 15:58 ` Tejun Heo
2016-01-26 17:38 ` Thierry Reding [this message]
2016-01-28 10:12 ` Peter Zijlstra
2016-01-28 12:47 ` Thierry Reding
2016-01-28 12:48 ` Thierry Reding
2016-01-29 11:09 ` Tejun Heo
2016-01-29 15:17 ` Peter Zijlstra
2016-01-29 18:28 ` Tejun Heo
2016-01-29 10:59 ` [PATCH wq/for-4.5-fixes] workqueue: skip flush dependency checks for legacy workqueues Tejun Heo
2016-01-29 15:07 ` Thierry Reding
2016-01-29 18:32 ` Tejun Heo
2016-02-02 6:54 ` Archit Taneja
2016-03-10 15:12 ` [PATCH] workqueue: warn if memory reclaim tries to flush !WQ_MEM_RECLAIM workqueue Adrian Hunter
2016-03-11 17:52 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160126173843.GA11115@ulmo.nvidia.com \
--to=thierry.reding@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=jonathanh@nvidia.com \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=tj@kernel.org \
--cc=uobergfe@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).