From: Jens Axboe <axboe@kernel.dk>
To: linux-kernel@vger.kernel.org, io-uring@vger.kernel.org
Cc: peterz@infradead.org, oleg@redhat.com, tglx@linutronix.de,
Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 3/3] task_work: use TIF_TASKWORK if available
Date: Thu, 1 Oct 2020 13:42:08 -0600 [thread overview]
Message-ID: <20201001194208.1153522-4-axboe@kernel.dk> (raw)
In-Reply-To: <20201001194208.1153522-1-axboe@kernel.dk>
If the arch supports TIF_TASKWORK, then use that for TWA_SIGNAL as
it's more efficient than using the signal delivery method. This is
especially true on threaded applications, where ->sighand is shared
across threads.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
kernel/task_work.c | 48 ++++++++++++++++++++++++++++++++++------------
1 file changed, 36 insertions(+), 12 deletions(-)
diff --git a/kernel/task_work.c b/kernel/task_work.c
index 613b2d634af8..ae317cfe86b8 100644
--- a/kernel/task_work.c
+++ b/kernel/task_work.c
@@ -5,6 +5,39 @@
static struct callback_head work_exited; /* all we need is ->next == NULL */
+/*
+ * TWA_SIGNAL signaling - use TIF_TASKWORK, if available.
+ */
+static void task_work_signal(struct task_struct *task)
+{
+#ifndef TIF_TASKWORK
+ unsigned long flags;
+
+ /*
+ * Only grab the sighand lock if we don't already have some
+ * task_work pending. This pairs with the smp_store_mb()
+ * in get_signal(), see comment there.
+ */
+ if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+ lock_task_sighand(task, &flags)) {
+ task->jobctl |= JOBCTL_TASK_WORK;
+ signal_wake_up(task, 0);
+ unlock_task_sighand(task, &flags);
+ }
+#else
+ set_tsk_thread_flag(task, TIF_TASKWORK);
+ set_notify_resume(task);
+#endif
+}
+
+static inline void clear_tsk_taskwork(struct task_struct *task)
+{
+#ifdef TIF_TASKWORK
+ if (test_tsk_thread_flag(task, TIF_TASKWORK))
+ clear_tsk_thread_flag(task, TIF_TASKWORK);
+#endif
+}
+
/**
* task_work_add - ask the @task to execute @work->func()
* @task: the task which should run the callback
@@ -28,7 +61,6 @@ int
task_work_add(struct task_struct *task, struct callback_head *work, int notify)
{
struct callback_head *head;
- unsigned long flags;
do {
head = READ_ONCE(task->task_works);
@@ -42,17 +74,7 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
set_notify_resume(task);
break;
case TWA_SIGNAL:
- /*
- * Only grab the sighand lock if we don't already have some
- * task_work pending. This pairs with the smp_store_mb()
- * in get_signal(), see comment there.
- */
- if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
- lock_task_sighand(task, &flags)) {
- task->jobctl |= JOBCTL_TASK_WORK;
- signal_wake_up(task, 0);
- unlock_task_sighand(task, &flags);
- }
+ task_work_signal(task);
break;
}
@@ -110,6 +132,8 @@ void task_work_run(void)
struct task_struct *task = current;
struct callback_head *work, *head, *next;
+ clear_tsk_taskwork(task);
+
for (;;) {
/*
* work->func() can do task_work_add(), do not set
--
2.28.0
next prev parent reply other threads:[~2020-10-01 19:42 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-01 19:42 [PATCHSET RFC 0/3] kernel: decouple TASK_WORK TWA_SIGNAL handling from signals Jens Axboe
2020-10-01 19:42 ` [PATCH 1/3] kernel: add task_sigpending() helper Jens Axboe
2020-10-01 19:42 ` [PATCH 2/3] kernel: decouple TASK_WORK TWA_SIGNAL handling from signals Jens Axboe
2020-10-01 19:42 ` Jens Axboe [this message]
2020-10-02 15:14 ` [PATCH 3/3] task_work: use TIF_TASKWORK if available Oleg Nesterov
2020-10-02 15:31 ` Thomas Gleixner
2020-10-02 15:38 ` Oleg Nesterov
2020-10-02 16:18 ` Jens Axboe
2020-10-03 1:49 ` Thomas Gleixner
2020-10-03 15:35 ` Jens Axboe
2020-10-02 15:52 ` Jens Axboe
2020-10-02 16:42 ` Jens Axboe
2020-10-02 19:10 ` Thomas Gleixner
2020-10-02 20:14 ` Jens Axboe
2020-10-02 15:53 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201001194208.1153522-4-axboe@kernel.dk \
--to=axboe@kernel.dk \
--cc=io-uring@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).