* [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT.
@ 2021-09-28 15:00 Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior
0 siblings, 2 replies; 5+ messages in thread
From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw)
To: linux-kernel
Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long,
Boqun Feng, Thomas Gleixner
This is a follow up to
https://lkml.kernel.org/r/87wnnjcuam.ffs@tglx
While forming the patch, I got currious why the state check is against
TASK_RTLOCK_WAIT and against TASK_NORMAL. So I swapped to to
TASK_RTLOCK_WAIT which should be more robust.
Sebastian
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] rtmutex: Check explicit for TASK_RTLOCK_WAIT.
2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior
@ 2021-09-28 15:00 ` Sebastian Andrzej Siewior
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior
1 sibling, 1 reply; 5+ messages in thread
From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw)
To: linux-kernel
Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long,
Boqun Feng, Thomas Gleixner, Sebastian Andrzej Siewior
rt_mutex_wake_q_add() needs to need to distiguish between sleeping
locks (TASK_RTLOCK_WAIT) and normal locks which use TASK_NORMAL to use
the proper wake mechanism.
Instead of checking for != TASK_NORMAL make it more robust and check
explicit for TASK_RTLOCK_WAIT which is the reason why a different wake
mechanism is used.
No functional change.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
kernel/locking/rtmutex.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 6bb116c559b4a..cafc259ec59d3 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -449,7 +449,7 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p)
static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh,
struct rt_mutex_waiter *w)
{
- if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state != TASK_NORMAL) {
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) {
if (IS_ENABLED(CONFIG_PROVE_LOCKING))
WARN_ON_ONCE(wqh->rtlock_task);
get_task_struct(w->task);
--
2.33.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock.
2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior
@ 2021-09-28 15:00 ` Sebastian Andrzej Siewior
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
1 sibling, 1 reply; 5+ messages in thread
From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw)
To: linux-kernel
Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long,
Boqun Feng, Thomas Gleixner, Davidlohr Bueso,
Sebastian Andrzej Siewior
From: Thomas Gleixner <tglx@linutronix.de>
The rw_semaphore and rwlock_t implementation both wake the waiter while
holding the rt_mutex_base::wait_lock acquired.
This can be optimized by waking the waiter lockless outside of the
locked section to avoid a needless contention on the
rt_mutex_base::wait_lock lock.
Extend rt_mutex_wake_q_add() to also accept task and state and use it in
__rwbase_read_unlock().
Suggested-by: Davidlohr Bueso <dave@stgolabs.net>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
kernel/locking/rtmutex.c | 23 +++++++++++++++--------
kernel/locking/rwbase_rt.c | 6 +++++-
2 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index cafc259ec59d3..0c6a48dfcecb3 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -446,17 +446,24 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p)
}
/* RT mutex specific wake_q wrappers */
+static __always_inline void rt_mutex_wake_q_add_task(struct rt_wake_q_head *wqh,
+ struct task_struct *task,
+ unsigned int wake_state)
+{
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && wake_state == TASK_RTLOCK_WAIT) {
+ if (IS_ENABLED(CONFIG_PROVE_LOCKING))
+ WARN_ON_ONCE(wqh->rtlock_task);
+ get_task_struct(task);
+ wqh->rtlock_task = task;
+ } else {
+ wake_q_add(&wqh->head, task);
+ }
+}
+
static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh,
struct rt_mutex_waiter *w)
{
- if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) {
- if (IS_ENABLED(CONFIG_PROVE_LOCKING))
- WARN_ON_ONCE(wqh->rtlock_task);
- get_task_struct(w->task);
- wqh->rtlock_task = w->task;
- } else {
- wake_q_add(&wqh->head, w->task);
- }
+ rt_mutex_wake_q_add_task(wqh, w->task, w->wake_state);
}
static __always_inline void rt_mutex_wake_up_q(struct rt_wake_q_head *wqh)
diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c
index 88191f6e252c3..15c81100f0e26 100644
--- a/kernel/locking/rwbase_rt.c
+++ b/kernel/locking/rwbase_rt.c
@@ -148,6 +148,7 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb,
{
struct rt_mutex_base *rtm = &rwb->rtmutex;
struct task_struct *owner;
+ DEFINE_RT_WAKE_Q(wqh);
raw_spin_lock_irq(&rtm->wait_lock);
/*
@@ -158,9 +159,12 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb,
*/
owner = rt_mutex_owner(rtm);
if (owner)
- wake_up_state(owner, state);
+ rt_mutex_wake_q_add_task(&wqh, owner, state);
+ /* Pairs with the preempt_enable in rt_mutex_wake_up_q() */
+ preempt_disable();
raw_spin_unlock_irq(&rtm->wait_lock);
+ rt_mutex_wake_up_q(&wqh);
}
static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb,
--
2.33.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [tip: locking/core] rtmutex: Wake up the waiters lockless while dropping the read lock.
2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior
@ 2021-10-01 15:05 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 5+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-10-01 15:05 UTC (permalink / raw)
To: linux-tip-commits
Cc: Davidlohr Bueso, Thomas Gleixner, Sebastian Andrzej Siewior,
Peter Zijlstra (Intel),
x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 9321f8152d9a764208c3f0dad49e0c55f293b7ab
Gitweb: https://git.kernel.org/tip/9321f8152d9a764208c3f0dad49e0c55f293b7ab
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Tue, 28 Sep 2021 17:00:06 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 01 Oct 2021 13:57:52 +02:00
rtmutex: Wake up the waiters lockless while dropping the read lock.
The rw_semaphore and rwlock_t implementation both wake the waiter while
holding the rt_mutex_base::wait_lock acquired.
This can be optimized by waking the waiter lockless outside of the
locked section to avoid a needless contention on the
rt_mutex_base::wait_lock lock.
Extend rt_mutex_wake_q_add() to also accept task and state and use it in
__rwbase_read_unlock().
Suggested-by: Davidlohr Bueso <dave@stgolabs.net>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20210928150006.597310-3-bigeasy@linutronix.de
---
kernel/locking/rtmutex.c | 19 +++++++++++++------
kernel/locking/rwbase_rt.c | 6 +++++-
2 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index cafc259..0c6a48d 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -446,19 +446,26 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p)
}
/* RT mutex specific wake_q wrappers */
-static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh,
- struct rt_mutex_waiter *w)
+static __always_inline void rt_mutex_wake_q_add_task(struct rt_wake_q_head *wqh,
+ struct task_struct *task,
+ unsigned int wake_state)
{
- if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) {
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && wake_state == TASK_RTLOCK_WAIT) {
if (IS_ENABLED(CONFIG_PROVE_LOCKING))
WARN_ON_ONCE(wqh->rtlock_task);
- get_task_struct(w->task);
- wqh->rtlock_task = w->task;
+ get_task_struct(task);
+ wqh->rtlock_task = task;
} else {
- wake_q_add(&wqh->head, w->task);
+ wake_q_add(&wqh->head, task);
}
}
+static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh,
+ struct rt_mutex_waiter *w)
+{
+ rt_mutex_wake_q_add_task(wqh, w->task, w->wake_state);
+}
+
static __always_inline void rt_mutex_wake_up_q(struct rt_wake_q_head *wqh)
{
if (IS_ENABLED(CONFIG_PREEMPT_RT) && wqh->rtlock_task) {
diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c
index 4ba1508..6b143fb 100644
--- a/kernel/locking/rwbase_rt.c
+++ b/kernel/locking/rwbase_rt.c
@@ -141,6 +141,7 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb,
{
struct rt_mutex_base *rtm = &rwb->rtmutex;
struct task_struct *owner;
+ DEFINE_RT_WAKE_Q(wqh);
raw_spin_lock_irq(&rtm->wait_lock);
/*
@@ -151,9 +152,12 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb,
*/
owner = rt_mutex_owner(rtm);
if (owner)
- wake_up_state(owner, state);
+ rt_mutex_wake_q_add_task(&wqh, owner, state);
+ /* Pairs with the preempt_enable in rt_mutex_wake_up_q() */
+ preempt_disable();
raw_spin_unlock_irq(&rtm->wait_lock);
+ rt_mutex_wake_up_q(&wqh);
}
static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb,
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [tip: locking/core] rtmutex: Check explicit for TASK_RTLOCK_WAIT.
2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior
@ 2021-10-01 15:05 ` tip-bot2 for Sebastian Andrzej Siewior
0 siblings, 0 replies; 5+ messages in thread
From: tip-bot2 for Sebastian Andrzej Siewior @ 2021-10-01 15:05 UTC (permalink / raw)
To: linux-tip-commits
Cc: Sebastian Andrzej Siewior, Peter Zijlstra (Intel), x86, linux-kernel
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 8fe46535e10dbfebad68ad9f2f8260e49f5852c9
Gitweb: https://git.kernel.org/tip/8fe46535e10dbfebad68ad9f2f8260e49f5852c9
Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
AuthorDate: Tue, 28 Sep 2021 17:00:05 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 01 Oct 2021 13:57:52 +02:00
rtmutex: Check explicit for TASK_RTLOCK_WAIT.
rt_mutex_wake_q_add() needs to need to distiguish between sleeping
locks (TASK_RTLOCK_WAIT) and normal locks which use TASK_NORMAL to use
the proper wake mechanism.
Instead of checking for != TASK_NORMAL make it more robust and check
explicit for TASK_RTLOCK_WAIT which is the reason why a different wake
mechanism is used.
No functional change.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20210928150006.597310-2-bigeasy@linutronix.de
---
kernel/locking/rtmutex.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 6bb116c..cafc259 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -449,7 +449,7 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p)
static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh,
struct rt_mutex_waiter *w)
{
- if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state != TASK_NORMAL) {
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) {
if (IS_ENABLED(CONFIG_PROVE_LOCKING))
WARN_ON_ONCE(wqh->rtlock_task);
get_task_struct(w->task);
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-01 15:05 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Sebastian Andrzej Siewior
2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior
2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).