From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753786AbbBRRMw (ORCPT ); Wed, 18 Feb 2015 12:12:52 -0500 Received: from terminus.zytor.com ([198.137.202.10]:39276 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753761AbbBRRMu (ORCPT ); Wed, 18 Feb 2015 12:12:50 -0500 Date: Wed, 18 Feb 2015 09:12:13 -0800 From: tip-bot for Davidlohr Bueso Message-ID: Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, dave@stgolabs.net, tim.c.chen@linux.intel.com, hpa@zytor.com, tglx@linutronix.de, walken@google.com, peterz@infradead.org, paulmck@linux.vnet.ibm.com, dbueso@suse.de, jason.low2@hp.com, torvalds@linux-foundation.org Reply-To: walken@google.com, peterz@infradead.org, tim.c.chen@linux.intel.com, tglx@linutronix.de, hpa@zytor.com, torvalds@linux-foundation.org, jason.low2@hp.com, dbueso@suse.de, paulmck@linux.vnet.ibm.com, mingo@kernel.org, dave@stgolabs.net, linux-kernel@vger.kernel.org In-Reply-To: <1422609267-15102-5-git-send-email-dave@stgolabs.net> References: <1422609267-15102-5-git-send-email-dave@stgolabs.net> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/rwsem: Avoid deceiving lock spinners Git-Commit-ID: b3fd4f03ca0b9952221f39ae6790e698bf4b39e7 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: b3fd4f03ca0b9952221f39ae6790e698bf4b39e7 Gitweb: http://git.kernel.org/tip/b3fd4f03ca0b9952221f39ae6790e698bf4b39e7 Author: Davidlohr Bueso AuthorDate: Fri, 30 Jan 2015 01:14:26 -0800 Committer: Ingo Molnar CommitDate: Wed, 18 Feb 2015 16:57:16 +0100 locking/rwsem: Avoid deceiving lock spinners When readers hold the semaphore, the ->owner is nil. As such, and unlike mutexes, '!owner' does not necessarily imply that the lock is free. This will cause writers to potentially spin excessively as they've been mislead to thinking they have a chance of acquiring the lock, instead of blocking. This patch therefore enhances the counter check when the owner is not set by the time we've broken out of the loop. Otherwise we can return true as a new owner has the lock and thus we want to continue spinning. While at it, we can make rwsem_spin_on_owner() less ambiguos and return right away under need_resched conditions. Signed-off-by: Davidlohr Bueso Signed-off-by: Peter Zijlstra (Intel) Cc: Jason Low Cc: Linus Torvalds Cc: Michel Lespinasse Cc: Paul E. McKenney Cc: Tim Chen Link: http://lkml.kernel.org/r/1422609267-15102-5-git-send-email-dave@stgolabs.net Signed-off-by: Ingo Molnar --- kernel/locking/rwsem-xadd.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 07713e5..1c0d11e 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -337,21 +337,30 @@ static inline bool owner_running(struct rw_semaphore *sem, static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) { + long count; + rcu_read_lock(); while (owner_running(sem, owner)) { - if (need_resched()) - break; + /* abort spinning when need_resched */ + if (need_resched()) { + rcu_read_unlock(); + return false; + } cpu_relax_lowlatency(); } rcu_read_unlock(); + if (READ_ONCE(sem->owner)) + return true; /* new owner, continue spinning */ + /* - * We break out the loop above on need_resched() or when the - * owner changed, which is a sign for heavy contention. Return - * success only when sem->owner is NULL. + * When the owner is not set, the lock could be free or + * held by readers. Check the counter to verify the + * state. */ - return sem->owner == NULL; + count = READ_ONCE(sem->count); + return (count == 0 || count == RWSEM_WAITING_BIAS); } static bool rwsem_optimistic_spin(struct rw_semaphore *sem)