From: Lai Jiangshan <jiangshanlai@gmail.com>
To: linux-kernel@vger.kernel.org
Cc: Lai Jiangshan <laijs@linux.alibaba.com>,
Valentin Schneider <valentin.schneider@arm.com>,
Qian Cai <cai@redhat.com>, Peter Zijlstra <peterz@infradead.org>,
Vincent Donnefort <vincent.donnefort@arm.com>,
Tejun Heo <tj@kernel.org>, Lai Jiangshan <jiangshanlai@gmail.com>
Subject: [PATCH 10/10] workqueue: Fix affinity of kworkers when attaching into pool
Date: Mon, 14 Dec 2020 23:54:57 +0800 [thread overview]
Message-ID: <20201214155457.3430-11-jiangshanlai@gmail.com> (raw)
In-Reply-To: <20201214155457.3430-1-jiangshanlai@gmail.com>
From: Lai Jiangshan <laijs@linux.alibaba.com>
When worker_attach_to_pool() is called, we should not put the workers
to pool->attrs->cpumask when there is not CPU online in it.
We have to use wq_online_cpumask in worker_attach_to_pool() to check
if pool->attrs->cpumask is valid rather than cpu_online_mask or
cpu_active_mask due to gaps between stages in cpu hot[un]plug.
To use wq_online_cpumask in worker_attach_to_pool(), we need to protect
wq_online_cpumask in wq_pool_attach_mutex and we modify workqueue_online_cpu()
and workqueue_offline_cpu() to enlarge wq_pool_attach_mutex protected
region. We also put updating wq_online_cpumask and [re|un]bind_workers()
in the same wq_pool_attach_mutex protected region to make the update
for percpu workqueue atomically.
Cc: Valentin Schneider <valentin.schneider@arm.com>
Cc: Qian Cai <cai@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vincent Donnefort <vincent.donnefort@arm.com>
Link: https://lore.kernel.org/lkml/20201210163830.21514-3-valentin.schneider@arm.com/
Signed-off-by: Lai Jiangshan <laijs@linux.alibaba.com>
---
kernel/workqueue.c | 32 +++++++++++++++-----------------
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 5ef41c567c2b..7a04cef90c1c 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -310,7 +310,7 @@ static bool workqueue_freezing; /* PL: have wqs started freezing? */
/* PL: allowable cpus for unbound wqs and work items */
static cpumask_var_t wq_unbound_cpumask;
-/* PL: online cpus (cpu_online_mask with the going-down cpu cleared) */
+/* PL&A: online cpus (cpu_online_mask with the going-down cpu cleared) */
static cpumask_var_t wq_online_cpumask;
/* CPU where unbound work was last round robin scheduled from this CPU */
@@ -1848,11 +1848,11 @@ static void worker_attach_to_pool(struct worker *worker,
{
mutex_lock(&wq_pool_attach_mutex);
- /*
- * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any
- * online CPUs. It'll be re-applied when any of the CPUs come up.
- */
- set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask);
+ /* Is there any cpu in pool->attrs->cpumask online? */
+ if (cpumask_any_and(pool->attrs->cpumask, wq_online_cpumask) < nr_cpu_ids)
+ WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask) < 0);
+ else
+ WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
/*
* The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
@@ -5079,13 +5079,12 @@ int workqueue_online_cpu(unsigned int cpu)
int pi;
mutex_lock(&wq_pool_mutex);
- cpumask_set_cpu(cpu, wq_online_cpumask);
- for_each_cpu_worker_pool(pool, cpu) {
- mutex_lock(&wq_pool_attach_mutex);
+ mutex_lock(&wq_pool_attach_mutex);
+ cpumask_set_cpu(cpu, wq_online_cpumask);
+ for_each_cpu_worker_pool(pool, cpu)
rebind_workers(pool);
- mutex_unlock(&wq_pool_attach_mutex);
- }
+ mutex_unlock(&wq_pool_attach_mutex);
/* update CPU affinity of workers of unbound pools */
for_each_pool(pool, pi) {
@@ -5115,14 +5114,13 @@ int workqueue_offline_cpu(unsigned int cpu)
if (WARN_ON(cpu != smp_processor_id()))
return -1;
- for_each_cpu_worker_pool(pool, cpu) {
- mutex_lock(&wq_pool_attach_mutex);
- unbind_workers(pool);
- mutex_unlock(&wq_pool_attach_mutex);
- }
-
mutex_lock(&wq_pool_mutex);
+
+ mutex_lock(&wq_pool_attach_mutex);
cpumask_clear_cpu(cpu, wq_online_cpumask);
+ for_each_cpu_worker_pool(pool, cpu)
+ unbind_workers(pool);
+ mutex_unlock(&wq_pool_attach_mutex);
/* update CPU affinity of workers of unbound pools */
for_each_pool(pool, pi) {
--
2.19.1.6.gb485710b
next prev parent reply other threads:[~2020-12-14 14:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-14 15:54 [PATCH 00/10] workqueue: break affinity initiatively Lai Jiangshan
2020-12-14 15:54 ` [PATCH 01/10] workqueue: restore unbound_workers' cpumask correctly Lai Jiangshan
2020-12-14 15:54 ` [PATCH 02/10] workqueue: use cpu_possible_mask instead of cpu_active_mask to break affinity Lai Jiangshan
2020-12-14 17:25 ` Peter Zijlstra
2020-12-15 8:33 ` Lai Jiangshan
2020-12-15 8:40 ` Peter Zijlstra
2020-12-16 14:32 ` Tejun Heo
2020-12-14 15:54 ` [PATCH 03/10] workqueue: Manually break affinity on pool detachment Lai Jiangshan
2020-12-14 15:54 ` [PATCH 04/10] workqueue: don't set the worker's cpumask when kthread_bind_mask() Lai Jiangshan
2020-12-16 14:39 ` Tejun Heo
2020-12-14 15:54 ` [PATCH 05/10] workqueue: introduce wq_online_cpumask Lai Jiangshan
2020-12-14 15:54 ` [PATCH 06/10] workqueue: use wq_online_cpumask in restore_unbound_workers_cpumask() Lai Jiangshan
2020-12-14 15:54 ` [PATCH 07/10] workqueue: Manually break affinity on hotplug for unbound pool Lai Jiangshan
2020-12-16 14:50 ` Tejun Heo
2020-12-14 15:54 ` [PATCH 08/10] workqueue: reorganize workqueue_online_cpu() Lai Jiangshan
2020-12-14 15:54 ` [PATCH 09/10] workqueue: reorganize workqueue_offline_cpu() unbind_workers() Lai Jiangshan
2020-12-14 15:54 ` Lai Jiangshan [this message]
2020-12-15 15:03 ` [PATCH 10/10] workqueue: Fix affinity of kworkers when attaching into pool Valentin Schneider
2020-12-14 17:36 ` [PATCH 00/10] workqueue: break affinity initiatively Peter Zijlstra
2020-12-15 5:44 ` Lai Jiangshan
2020-12-15 7:50 ` Peter Zijlstra
2020-12-15 8:14 ` Lai Jiangshan
2020-12-15 8:49 ` Peter Zijlstra
2020-12-15 9:46 ` Lai Jiangshan
2020-12-16 14:30 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201214155457.3430-11-jiangshanlai@gmail.com \
--to=jiangshanlai@gmail.com \
--cc=cai@redhat.com \
--cc=laijs@linux.alibaba.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=tj@kernel.org \
--cc=valentin.schneider@arm.com \
--cc=vincent.donnefort@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).