* [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT. @ 2021-09-28 15:00 Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior 0 siblings, 2 replies; 5+ messages in thread From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw) To: linux-kernel Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long, Boqun Feng, Thomas Gleixner This is a follow up to https://lkml.kernel.org/r/87wnnjcuam.ffs@tglx While forming the patch, I got currious why the state check is against TASK_RTLOCK_WAIT and against TASK_NORMAL. So I swapped to to TASK_RTLOCK_WAIT which should be more robust. Sebastian ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] rtmutex: Check explicit for TASK_RTLOCK_WAIT. 2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior @ 2021-09-28 15:00 ` Sebastian Andrzej Siewior 2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior 1 sibling, 1 reply; 5+ messages in thread From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw) To: linux-kernel Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long, Boqun Feng, Thomas Gleixner, Sebastian Andrzej Siewior rt_mutex_wake_q_add() needs to need to distiguish between sleeping locks (TASK_RTLOCK_WAIT) and normal locks which use TASK_NORMAL to use the proper wake mechanism. Instead of checking for != TASK_NORMAL make it more robust and check explicit for TASK_RTLOCK_WAIT which is the reason why a different wake mechanism is used. No functional change. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- kernel/locking/rtmutex.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 6bb116c559b4a..cafc259ec59d3 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -449,7 +449,7 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, struct rt_mutex_waiter *w) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state != TASK_NORMAL) { + if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(wqh->rtlock_task); get_task_struct(w->task); -- 2.33.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [tip: locking/core] rtmutex: Check explicit for TASK_RTLOCK_WAIT. 2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior @ 2021-10-01 15:05 ` tip-bot2 for Sebastian Andrzej Siewior 0 siblings, 0 replies; 5+ messages in thread From: tip-bot2 for Sebastian Andrzej Siewior @ 2021-10-01 15:05 UTC (permalink / raw) To: linux-tip-commits Cc: Sebastian Andrzej Siewior, Peter Zijlstra (Intel), x86, linux-kernel The following commit has been merged into the locking/core branch of tip: Commit-ID: 8fe46535e10dbfebad68ad9f2f8260e49f5852c9 Gitweb: https://git.kernel.org/tip/8fe46535e10dbfebad68ad9f2f8260e49f5852c9 Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de> AuthorDate: Tue, 28 Sep 2021 17:00:05 +02:00 Committer: Peter Zijlstra <peterz@infradead.org> CommitterDate: Fri, 01 Oct 2021 13:57:52 +02:00 rtmutex: Check explicit for TASK_RTLOCK_WAIT. rt_mutex_wake_q_add() needs to need to distiguish between sleeping locks (TASK_RTLOCK_WAIT) and normal locks which use TASK_NORMAL to use the proper wake mechanism. Instead of checking for != TASK_NORMAL make it more robust and check explicit for TASK_RTLOCK_WAIT which is the reason why a different wake mechanism is used. No functional change. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20210928150006.597310-2-bigeasy@linutronix.de --- kernel/locking/rtmutex.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 6bb116c..cafc259 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -449,7 +449,7 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, struct rt_mutex_waiter *w) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state != TASK_NORMAL) { + if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(wqh->rtlock_task); get_task_struct(w->task); ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock. 2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior @ 2021-09-28 15:00 ` Sebastian Andrzej Siewior 2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner 1 sibling, 1 reply; 5+ messages in thread From: Sebastian Andrzej Siewior @ 2021-09-28 15:00 UTC (permalink / raw) To: linux-kernel Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Waiman Long, Boqun Feng, Thomas Gleixner, Davidlohr Bueso, Sebastian Andrzej Siewior From: Thomas Gleixner <tglx@linutronix.de> The rw_semaphore and rwlock_t implementation both wake the waiter while holding the rt_mutex_base::wait_lock acquired. This can be optimized by waking the waiter lockless outside of the locked section to avoid a needless contention on the rt_mutex_base::wait_lock lock. Extend rt_mutex_wake_q_add() to also accept task and state and use it in __rwbase_read_unlock(). Suggested-by: Davidlohr Bueso <dave@stgolabs.net> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- kernel/locking/rtmutex.c | 23 +++++++++++++++-------- kernel/locking/rwbase_rt.c | 6 +++++- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index cafc259ec59d3..0c6a48dfcecb3 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -446,17 +446,24 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) } /* RT mutex specific wake_q wrappers */ +static __always_inline void rt_mutex_wake_q_add_task(struct rt_wake_q_head *wqh, + struct task_struct *task, + unsigned int wake_state) +{ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && wake_state == TASK_RTLOCK_WAIT) { + if (IS_ENABLED(CONFIG_PROVE_LOCKING)) + WARN_ON_ONCE(wqh->rtlock_task); + get_task_struct(task); + wqh->rtlock_task = task; + } else { + wake_q_add(&wqh->head, task); + } +} + static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, struct rt_mutex_waiter *w) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { - if (IS_ENABLED(CONFIG_PROVE_LOCKING)) - WARN_ON_ONCE(wqh->rtlock_task); - get_task_struct(w->task); - wqh->rtlock_task = w->task; - } else { - wake_q_add(&wqh->head, w->task); - } + rt_mutex_wake_q_add_task(wqh, w->task, w->wake_state); } static __always_inline void rt_mutex_wake_up_q(struct rt_wake_q_head *wqh) diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c index 88191f6e252c3..15c81100f0e26 100644 --- a/kernel/locking/rwbase_rt.c +++ b/kernel/locking/rwbase_rt.c @@ -148,6 +148,7 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, { struct rt_mutex_base *rtm = &rwb->rtmutex; struct task_struct *owner; + DEFINE_RT_WAKE_Q(wqh); raw_spin_lock_irq(&rtm->wait_lock); /* @@ -158,9 +159,12 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, */ owner = rt_mutex_owner(rtm); if (owner) - wake_up_state(owner, state); + rt_mutex_wake_q_add_task(&wqh, owner, state); + /* Pairs with the preempt_enable in rt_mutex_wake_up_q() */ + preempt_disable(); raw_spin_unlock_irq(&rtm->wait_lock); + rt_mutex_wake_up_q(&wqh); } static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb, -- 2.33.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [tip: locking/core] rtmutex: Wake up the waiters lockless while dropping the read lock. 2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior @ 2021-10-01 15:05 ` tip-bot2 for Thomas Gleixner 0 siblings, 0 replies; 5+ messages in thread From: tip-bot2 for Thomas Gleixner @ 2021-10-01 15:05 UTC (permalink / raw) To: linux-tip-commits Cc: Davidlohr Bueso, Thomas Gleixner, Sebastian Andrzej Siewior, Peter Zijlstra (Intel), x86, linux-kernel The following commit has been merged into the locking/core branch of tip: Commit-ID: 9321f8152d9a764208c3f0dad49e0c55f293b7ab Gitweb: https://git.kernel.org/tip/9321f8152d9a764208c3f0dad49e0c55f293b7ab Author: Thomas Gleixner <tglx@linutronix.de> AuthorDate: Tue, 28 Sep 2021 17:00:06 +02:00 Committer: Peter Zijlstra <peterz@infradead.org> CommitterDate: Fri, 01 Oct 2021 13:57:52 +02:00 rtmutex: Wake up the waiters lockless while dropping the read lock. The rw_semaphore and rwlock_t implementation both wake the waiter while holding the rt_mutex_base::wait_lock acquired. This can be optimized by waking the waiter lockless outside of the locked section to avoid a needless contention on the rt_mutex_base::wait_lock lock. Extend rt_mutex_wake_q_add() to also accept task and state and use it in __rwbase_read_unlock(). Suggested-by: Davidlohr Bueso <dave@stgolabs.net> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20210928150006.597310-3-bigeasy@linutronix.de --- kernel/locking/rtmutex.c | 19 +++++++++++++------ kernel/locking/rwbase_rt.c | 6 +++++- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index cafc259..0c6a48d 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -446,19 +446,26 @@ static __always_inline void rt_mutex_adjust_prio(struct task_struct *p) } /* RT mutex specific wake_q wrappers */ -static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, - struct rt_mutex_waiter *w) +static __always_inline void rt_mutex_wake_q_add_task(struct rt_wake_q_head *wqh, + struct task_struct *task, + unsigned int wake_state) { - if (IS_ENABLED(CONFIG_PREEMPT_RT) && w->wake_state == TASK_RTLOCK_WAIT) { + if (IS_ENABLED(CONFIG_PREEMPT_RT) && wake_state == TASK_RTLOCK_WAIT) { if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(wqh->rtlock_task); - get_task_struct(w->task); - wqh->rtlock_task = w->task; + get_task_struct(task); + wqh->rtlock_task = task; } else { - wake_q_add(&wqh->head, w->task); + wake_q_add(&wqh->head, task); } } +static __always_inline void rt_mutex_wake_q_add(struct rt_wake_q_head *wqh, + struct rt_mutex_waiter *w) +{ + rt_mutex_wake_q_add_task(wqh, w->task, w->wake_state); +} + static __always_inline void rt_mutex_wake_up_q(struct rt_wake_q_head *wqh) { if (IS_ENABLED(CONFIG_PREEMPT_RT) && wqh->rtlock_task) { diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c index 4ba1508..6b143fb 100644 --- a/kernel/locking/rwbase_rt.c +++ b/kernel/locking/rwbase_rt.c @@ -141,6 +141,7 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, { struct rt_mutex_base *rtm = &rwb->rtmutex; struct task_struct *owner; + DEFINE_RT_WAKE_Q(wqh); raw_spin_lock_irq(&rtm->wait_lock); /* @@ -151,9 +152,12 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, */ owner = rt_mutex_owner(rtm); if (owner) - wake_up_state(owner, state); + rt_mutex_wake_q_add_task(&wqh, owner, state); + /* Pairs with the preempt_enable in rt_mutex_wake_up_q() */ + preempt_disable(); raw_spin_unlock_irq(&rtm->wait_lock); + rt_mutex_wake_up_q(&wqh); } static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb, ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-01 15:05 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-28 15:00 [PATCH 0/2] rtmutex: Lockless reader wake + explicit check for TASK_RTLOCK_WAIT Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 1/2] rtmutex: Check explicit " Sebastian Andrzej Siewior 2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Sebastian Andrzej Siewior 2021-09-28 15:00 ` [PATCH 2/2] rtmutex: Wake up the waiters lockless while dropping the read lock Sebastian Andrzej Siewior 2021-10-01 15:05 ` [tip: locking/core] " tip-bot2 for Thomas Gleixner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).