From: Davidlohr Bueso <dave@stgolabs.net>
To: tglx@linutronix.de
Cc: peterz@infradead.org, mingo@kernel.org, rostedt@goodmis.org,
longman@redhat.com, bigeasy@linutronix.de, boqun.feng@gmail.com,
dave@stgolabs.net, linux-kernel@vger.kernel.org,
Davidlohr Bueso <dbueso@suse.de>
Subject: [PATCH 1/2] sched: Move wake_q code below try_to_wake_up()
Date: Wed, 1 Sep 2021 15:28:24 -0700 [thread overview]
Message-ID: <20210901222825.6313-2-dave@stgolabs.net> (raw)
In-Reply-To: <20210901222825.6313-1-dave@stgolabs.net>
Such that wake_up_q() can make use of it instead of
wake_up_process(). No change in code.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
kernel/sched/core.c | 172 ++++++++++++++++++++++----------------------
1 file changed, 86 insertions(+), 86 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c4462c454ab9..7fc3d22bc6d8 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -865,92 +865,6 @@ static bool set_nr_if_polling(struct task_struct *p)
#endif
#endif
-static bool __wake_q_add(struct wake_q_head *head, struct task_struct *task)
-{
- struct wake_q_node *node = &task->wake_q;
-
- /*
- * Atomically grab the task, if ->wake_q is !nil already it means
- * it's already queued (either by us or someone else) and will get the
- * wakeup due to that.
- *
- * In order to ensure that a pending wakeup will observe our pending
- * state, even in the failed case, an explicit smp_mb() must be used.
- */
- smp_mb__before_atomic();
- if (unlikely(cmpxchg_relaxed(&node->next, NULL, WAKE_Q_TAIL)))
- return false;
-
- /*
- * The head is context local, there can be no concurrency.
- */
- *head->lastp = node;
- head->lastp = &node->next;
- return true;
-}
-
-/**
- * wake_q_add() - queue a wakeup for 'later' waking.
- * @head: the wake_q_head to add @task to
- * @task: the task to queue for 'later' wakeup
- *
- * Queue a task for later wakeup, most likely by the wake_up_q() call in the
- * same context, _HOWEVER_ this is not guaranteed, the wakeup can come
- * instantly.
- *
- * This function must be used as-if it were wake_up_process(); IOW the task
- * must be ready to be woken at this location.
- */
-void wake_q_add(struct wake_q_head *head, struct task_struct *task)
-{
- if (__wake_q_add(head, task))
- get_task_struct(task);
-}
-
-/**
- * wake_q_add_safe() - safely queue a wakeup for 'later' waking.
- * @head: the wake_q_head to add @task to
- * @task: the task to queue for 'later' wakeup
- *
- * Queue a task for later wakeup, most likely by the wake_up_q() call in the
- * same context, _HOWEVER_ this is not guaranteed, the wakeup can come
- * instantly.
- *
- * This function must be used as-if it were wake_up_process(); IOW the task
- * must be ready to be woken at this location.
- *
- * This function is essentially a task-safe equivalent to wake_q_add(). Callers
- * that already hold reference to @task can call the 'safe' version and trust
- * wake_q to do the right thing depending whether or not the @task is already
- * queued for wakeup.
- */
-void wake_q_add_safe(struct wake_q_head *head, struct task_struct *task)
-{
- if (!__wake_q_add(head, task))
- put_task_struct(task);
-}
-
-void wake_up_q(struct wake_q_head *head)
-{
- struct wake_q_node *node = head->first;
-
- while (node != WAKE_Q_TAIL) {
- struct task_struct *task;
-
- task = container_of(node, struct task_struct, wake_q);
- /* Task can safely be re-inserted now: */
- node = node->next;
- task->wake_q.next = NULL;
-
- /*
- * wake_up_process() executes a full barrier, which pairs with
- * the queueing in wake_q_add() so as not to miss wakeups.
- */
- wake_up_process(task);
- put_task_struct(task);
- }
-}
-
/*
* resched_curr - mark rq's current task 'to be rescheduled now'.
*
@@ -4172,6 +4086,92 @@ int wake_up_state(struct task_struct *p, unsigned int state)
return try_to_wake_up(p, state, 0);
}
+static bool __wake_q_add(struct wake_q_head *head, struct task_struct *task)
+{
+ struct wake_q_node *node = &task->wake_q;
+
+ /*
+ * Atomically grab the task, if ->wake_q is !nil already it means
+ * it's already queued (either by us or someone else) and will get the
+ * wakeup due to that.
+ *
+ * In order to ensure that a pending wakeup will observe our pending
+ * state, even in the failed case, an explicit smp_mb() must be used.
+ */
+ smp_mb__before_atomic();
+ if (unlikely(cmpxchg_relaxed(&node->next, NULL, WAKE_Q_TAIL)))
+ return false;
+
+ /*
+ * The head is context local, there can be no concurrency.
+ */
+ *head->lastp = node;
+ head->lastp = &node->next;
+ return true;
+}
+
+/**
+ * wake_q_add() - queue a wakeup for 'later' waking.
+ * @head: the wake_q_head to add @task to
+ * @task: the task to queue for 'later' wakeup
+ *
+ * Queue a task for later wakeup, most likely by the wake_up_q() call in the
+ * same context, _HOWEVER_ this is not guaranteed, the wakeup can come
+ * instantly.
+ *
+ * This function must be used as-if it were wake_up_process(); IOW the task
+ * must be ready to be woken at this location.
+ */
+void wake_q_add(struct wake_q_head *head, struct task_struct *task)
+{
+ if (__wake_q_add(head, task))
+ get_task_struct(task);
+}
+
+/**
+ * wake_q_add_safe() - safely queue a wakeup for 'later' waking.
+ * @head: the wake_q_head to add @task to
+ * @task: the task to queue for 'later' wakeup
+ *
+ * Queue a task for later wakeup, most likely by the wake_up_q() call in the
+ * same context, _HOWEVER_ this is not guaranteed, the wakeup can come
+ * instantly.
+ *
+ * This function must be used as-if it were wake_up_process(); IOW the task
+ * must be ready to be woken at this location.
+ *
+ * This function is essentially a task-safe equivalent to wake_q_add(). Callers
+ * that already hold reference to @task can call the 'safe' version and trust
+ * wake_q to do the right thing depending whether or not the @task is already
+ * queued for wakeup.
+ */
+void wake_q_add_safe(struct wake_q_head *head, struct task_struct *task)
+{
+ if (!__wake_q_add(head, task))
+ put_task_struct(task);
+}
+
+void wake_up_q(struct wake_q_head *head)
+{
+ struct wake_q_node *node = head->first;
+
+ while (node != WAKE_Q_TAIL) {
+ struct task_struct *task;
+
+ task = container_of(node, struct task_struct, wake_q);
+ /* Task can safely be re-inserted now: */
+ node = node->next;
+ task->wake_q.next = NULL;
+
+ /*
+ * wake_up_process() executes a full barrier, which pairs with
+ * the queueing in wake_q_add() so as not to miss wakeups.
+ */
+ wake_up_process(task);
+ put_task_struct(task);
+ }
+}
+
/*
* Perform scheduler related setup for a newly forked process p.
* p is forked by current.
--
2.26.2
next prev parent reply other threads:[~2021-09-01 22:29 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-01 22:28 [PATCH 0/2] locking/rwbase_rt: Use wake_q for lockless reader waker Davidlohr Bueso
2021-09-01 22:28 ` Davidlohr Bueso [this message]
2021-09-01 22:28 ` [PATCH 2/2] locking/rwbase_rt: Lockless reader waking up a writer Davidlohr Bueso
2021-09-01 23:03 ` Davidlohr Bueso
2021-09-03 10:55 ` Sebastian Andrzej Siewior
2021-09-13 12:20 ` Thomas Gleixner
2021-09-14 10:42 ` Thomas Gleixner
2021-09-16 17:02 ` Sebastian Andrzej Siewior
2021-09-16 17:27 ` Davidlohr Bueso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210901222825.6313-2-dave@stgolabs.net \
--to=dave@stgolabs.net \
--cc=bigeasy@linutronix.de \
--cc=boqun.feng@gmail.com \
--cc=dbueso@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).