From: Jason Low <jason.low2@hp.com>
To: mingo@redhat.com, peterz@infradead.org,
paulmck@linux.vnet.ibm.com, Waiman.Long@hp.com,
torvalds@linux-foundation.org, tglx@linutronix.de,
jason.low2@hp.com
Cc: linux-kernel@vger.kernel.org, riel@redhat.com,
akpm@linux-foundation.org, davidlohr@hp.com, hpa@zytor.com,
andi@firstfloor.org, aswin@hp.com, scott.norton@hp.com,
chegu_vinod@hp.com
Subject: [RFC][PATCH v2 4/5] mutex: Disable preemtion between modifying lock->owner and locking/unlocking mutex
Date: Tue, 28 Jan 2014 11:13:15 -0800 [thread overview]
Message-ID: <1390936396-3962-5-git-send-email-jason.low2@hp.com> (raw)
In-Reply-To: <1390936396-3962-1-git-send-email-jason.low2@hp.com>
This RFC patch disables preemption between modifying lock->owner and
locking/unlocking the mutex lock. This prevents situations where the owner
can preempt between those 2 operations, which causes optimistic spinners to
be unable to check if lock->owner is not on CPU. As mentioned in the
thread for this v1 patchset, disabling preemption is a cheap operation.
Signed-off-by: Jason Low <jason.low2@hp.com>
---
kernel/locking/mutex.c | 30 ++++++++++++++++++++++++++++--
1 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 6d85b08..cfaaf53 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -98,8 +98,10 @@ void __sched mutex_lock(struct mutex *lock)
* The locking fastpath is the 1->0 transition from
* 'unlocked' into 'locked' state.
*/
+ preempt_disable();
__mutex_fastpath_lock(&lock->count, __mutex_lock_slowpath);
mutex_set_owner(lock);
+ preempt_enable();
}
EXPORT_SYMBOL(mutex_lock);
@@ -253,9 +255,13 @@ void __sched mutex_unlock(struct mutex *lock)
* the slow path will always be taken, and that clears the owner field
* after verifying that it was indeed current.
*/
+ preempt_disable();
mutex_clear_owner(lock);
#endif
__mutex_fastpath_unlock(&lock->count, __mutex_unlock_slowpath);
+#ifndef CONFIG_DEBUG_MUTEXES
+ preempt_enable();
+#endif
}
EXPORT_SYMBOL(mutex_unlock);
@@ -292,9 +298,13 @@ void __sched ww_mutex_unlock(struct ww_mutex *lock)
* the slow path will always be taken, and that clears the owner field
* after verifying that it was indeed current.
*/
+ preempt_disable();
mutex_clear_owner(&lock->base);
#endif
__mutex_fastpath_unlock(&lock->base.count, __mutex_unlock_slowpath);
+#ifndef CONFIG_DEBUG_MUTEXES
+ preempt_enable();
+#endif
}
EXPORT_SYMBOL(ww_mutex_unlock);
@@ -780,12 +790,16 @@ int __sched mutex_lock_interruptible(struct mutex *lock)
int ret;
might_sleep();
+ preempt_disable();
ret = __mutex_fastpath_lock_retval(&lock->count);
if (likely(!ret)) {
mutex_set_owner(lock);
+ preempt_enable();
return 0;
- } else
+ } else {
+ preempt_enable();
return __mutex_lock_interruptible_slowpath(lock);
+ }
}
EXPORT_SYMBOL(mutex_lock_interruptible);
@@ -795,12 +809,16 @@ int __sched mutex_lock_killable(struct mutex *lock)
int ret;
might_sleep();
+ preempt_disable();
ret = __mutex_fastpath_lock_retval(&lock->count);
if (likely(!ret)) {
mutex_set_owner(lock);
+ preempt_enable();
return 0;
- } else
+ } else {
+ preempt_enable();
return __mutex_lock_killable_slowpath(lock);
+ }
}
EXPORT_SYMBOL(mutex_lock_killable);
@@ -889,9 +907,11 @@ int __sched mutex_trylock(struct mutex *lock)
{
int ret;
+ preempt_disable();
ret = __mutex_fastpath_trylock(&lock->count, __mutex_trylock_slowpath);
if (ret)
mutex_set_owner(lock);
+ preempt_enable();
return ret;
}
@@ -904,6 +924,7 @@ __ww_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
int ret;
might_sleep();
+ preempt_disable();
ret = __mutex_fastpath_lock_retval(&lock->base.count);
@@ -912,6 +933,8 @@ __ww_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
mutex_set_owner(&lock->base);
} else
ret = __ww_mutex_lock_slowpath(lock, ctx);
+
+ preempt_enable();
return ret;
}
EXPORT_SYMBOL(__ww_mutex_lock);
@@ -922,6 +945,7 @@ __ww_mutex_lock_interruptible(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
int ret;
might_sleep();
+ preempt_disable();
ret = __mutex_fastpath_lock_retval(&lock->base.count);
@@ -930,6 +954,8 @@ __ww_mutex_lock_interruptible(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
mutex_set_owner(&lock->base);
} else
ret = __ww_mutex_lock_interruptible_slowpath(lock, ctx);
+
+ preempt_enable();
return ret;
}
EXPORT_SYMBOL(__ww_mutex_lock_interruptible);
--
1.7.1
next prev parent reply other threads:[~2014-01-28 19:13 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-01-28 19:13 [PATCH v2 0/5] mutex: Mutex scalability patches Jason Low
2014-01-28 19:13 ` [PATCH v2 1/5] mutex: In mutex_can_spin_on_owner(), return false if task need_resched() Jason Low
2014-01-28 20:20 ` Paul E. McKenney
2014-01-28 22:01 ` Jason Low
2014-01-28 21:09 ` Davidlohr Bueso
2014-03-11 12:41 ` [tip:core/locking] locking/mutexes: Return false if task need_resched() in mutex_can_spin_on_owner() tip-bot for Jason Low
2014-01-28 19:13 ` [PATCH v2 2/5] mutex: Modify the way optimistic spinners are queued Jason Low
2014-01-28 20:23 ` Paul E. McKenney
2014-01-28 20:24 ` Paul E. McKenney
2014-01-28 21:17 ` Davidlohr Bueso
2014-01-28 22:10 ` Jason Low
2014-02-02 21:58 ` Paul E. McKenney
2014-03-11 12:41 ` [tip:core/locking] locking/mutexes: " tip-bot for Jason Low
2014-03-11 15:24 ` Jason Low
2014-03-11 15:33 ` Peter Zijlstra
2014-01-28 19:13 ` [PATCH v2 3/5] mutex: Unlock the mutex without the wait_lock Jason Low
2014-03-11 12:41 ` [tip:core/locking] locking/mutexes: " tip-bot for Jason Low
2014-03-12 12:24 ` Peter Zijlstra
2014-03-12 18:44 ` Jason Low
2014-03-13 7:28 ` [tip:core/locking] locking/mutex: Fix debug checks tip-bot for Peter Zijlstra
2014-01-28 19:13 ` Jason Low [this message]
2014-01-28 20:54 ` [RFC][PATCH v2 4/5] mutex: Disable preemtion between modifying lock->owner and locking/unlocking mutex Peter Zijlstra
2014-01-28 22:17 ` Jason Low
2014-01-28 19:13 ` [RFC][PATCH v2 5/5] mutex: Give spinners a chance to spin_on_owner if need_resched() triggered while queued Jason Low
2014-01-28 21:07 ` Peter Zijlstra
2014-01-28 22:51 ` Jason Low
2014-01-29 11:51 ` Peter Zijlstra
2014-01-31 3:29 ` Jason Low
2014-01-31 14:09 ` Peter Zijlstra
2014-01-31 20:01 ` Jason Low
2014-01-31 20:08 ` Peter Zijlstra
2014-02-02 21:01 ` Jason Low
2014-02-02 21:12 ` Peter Zijlstra
2014-02-03 18:39 ` Jason Low
2014-02-03 19:25 ` Peter Zijlstra
2014-02-03 20:55 ` Jason Low
2014-02-03 21:06 ` Peter Zijlstra
2014-02-03 21:56 ` Jason Low
2014-02-04 7:13 ` Jason Low
2014-02-02 22:02 ` Paul E. McKenney
2014-02-02 20:02 ` Peter Zijlstra
2014-02-05 21:44 ` Waiman Long
2014-02-06 14:04 ` Peter Zijlstra
2014-02-06 18:45 ` Waiman Long
2014-02-06 20:10 ` Norton, Scott J
2014-02-10 17:01 ` Peter Zijlstra
2014-02-06 17:44 ` Jason Low
2014-02-06 18:37 ` Waiman Long
2014-01-28 21:08 ` [PATCH v2 0/5] mutex: Mutex scalability patches Davidlohr Bueso
2014-01-28 23:11 ` Jason Low
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1390936396-3962-5-git-send-email-jason.low2@hp.com \
--to=jason.low2@hp.com \
--cc=Waiman.Long@hp.com \
--cc=akpm@linux-foundation.org \
--cc=andi@firstfloor.org \
--cc=aswin@hp.com \
--cc=chegu_vinod@hp.com \
--cc=davidlohr@hp.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=scott.norton@hp.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).