From: Tim Chen <tim.c.chen@linux.intel.com> To: Ingo Molnar <mingo@elte.hu>, Andrew Morton <akpm@linux-foundation.org>, Thomas Gleixner <tglx@linutronix.de>, "Paul E.McKenney" <paulmck@linux.vnet.ibm.com>, Will Deacon <will.deacon@arm.com> Cc: linux-kernel@vger.kernel.org, linux-mm <linux-mm@kvack.org>, linux-arch@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Waiman Long <waiman.long@hp.com>, Andrea Arcangeli <aarcange@redhat.com>, Alex Shi <alex.shi@linaro.org>, Andi Kleen <andi@firstfloor.org>, Michel Lespinasse <walken@google.com>, Davidlohr Bueso <davidlohr.bueso@hp.com>, Matthew R Wilcox <matthew.r.wilcox@intel.com>, Dave Hansen <dave.hansen@intel.com>, Peter Zijlstra <a.p.zijlstra@chello.nl>, Rik van Riel <riel@redhat.com>, Peter Hurley <peter@hurleysoftware.com>, Tim Chen <tim.c.chen@linux.intel.com>, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>, George Spelvin <linux@horizon.com>, "H. Peter Anvin" <hpa@zytor.com>, Arnd Bergmann <arnd@arndb.de>, Aswin Chandramouleeswaran <aswin@hp.com>, Scott J Norton <scott.norton@hp.com>, "Figo.zhang" <figo1802@gmail.com> Subject: [PATCH v9 1/6] MCS Lock: Barrier corrections Date: Tue, 21 Jan 2014 15:35:53 -0800 [thread overview] Message-ID: <1390347353.3138.62.camel@schen9-DESK> (raw) In-Reply-To: <cover.1390320729.git.tim.c.chen@linux.intel.com> From: Waiman Long <Waiman.Long@hp.com> This patch corrects the way memory barriers are used in the MCS lock with smp_load_acquire and smp_store_release fucnctions. The previous barriers could leak critical sections if mcs lock is used by itself. It is not a problem when mcs lock is embedded in mutex but will be an issue when the mcs_lock is used elsewhere. The patch removes the incorrect barriers and put in correct barriers with the pair of functions smp_load_acquire and smp_store_release. Suggested-by: Michel Lespinasse <walken@google.com> Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Waiman Long <Waiman.Long@hp.com> Signed-off-by: Jason Low <jason.low2@hp.com> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com> --- kernel/locking/mutex.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 4dd6e4c..fbbd2ed 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -136,9 +136,12 @@ void mspin_lock(struct mspin_node **lock, struct mspin_node *node) return; } ACCESS_ONCE(prev->next) = node; - smp_wmb(); - /* Wait until the lock holder passes the lock down */ - while (!ACCESS_ONCE(node->locked)) + /* + * Wait until the lock holder passes the lock down. + * Using smp_load_acquire() provides a memory barrier that + * ensures subsequent operations happen after the lock is acquired. + */ + while (!(smp_load_acquire(&node->locked))) arch_mutex_cpu_relax(); } @@ -156,8 +159,13 @@ static void mspin_unlock(struct mspin_node **lock, struct mspin_node *node) while (!(next = ACCESS_ONCE(node->next))) arch_mutex_cpu_relax(); } - ACCESS_ONCE(next->locked) = 1; - smp_wmb(); + /* + * Pass lock to next waiter. + * smp_store_release() provides a memory barrier to ensure + * all operations in the critical section has been completed + * before unlocking. + */ + smp_store_release(&next->locked, 1); } /* -- 1.7.11.7
WARNING: multiple messages have this Message-ID (diff)
From: Tim Chen <tim.c.chen@linux.intel.com> To: Ingo Molnar <mingo@elte.hu>, Andrew Morton <akpm@linux-foundation.org>, Thomas Gleixner <tglx@linutronix.de>, "Paul E.McKenney" <paulmck@linux.vnet.ibm.com>, Will Deacon <will.deacon@arm.com> Cc: linux-kernel@vger.kernel.org, linux-mm <linux-mm@kvack.org>, linux-arch@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Waiman Long <waiman.long@hp.com>, Andrea Arcangeli <aarcange@redhat.com>, Alex Shi <alex.shi@linaro.org>, Andi Kleen <andi@firstfloor.org>, Michel Lespinasse <walken@google.com>, Davidlohr Bueso <davidlohr.bueso@hp.com>, Matthew R Wilcox <matthew.r.wilcox@intel.com>, Dave Hansen <dave.hansen@intel.com>, Peter Zijlstra <a.p.zijlstra@chello.nl>, Rik van Riel <riel@redhat.com>, Peter Hurley <peter@hurleysoftware.com>, Tim Chen <tim.c.chen@linux.intel.com>, Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>, George Spelvin <linux@horizon.com>, "H. Peter Anvin" <hpa@zytor.com>, Arnd Bergmann <arnd@arndb.de>, Aswin Chandramouleeswaran <aswin@hp.com>, Scott J Norton <scott.norton@hp.com>, "Figo.zhang" <figo1802@gmail.com> Subject: [PATCH v9 1/6] MCS Lock: Barrier corrections Date: Tue, 21 Jan 2014 15:35:53 -0800 [thread overview] Message-ID: <1390347353.3138.62.camel@schen9-DESK> (raw) In-Reply-To: <cover.1390320729.git.tim.c.chen@linux.intel.com> From: Waiman Long <Waiman.Long@hp.com> This patch corrects the way memory barriers are used in the MCS lock with smp_load_acquire and smp_store_release fucnctions. The previous barriers could leak critical sections if mcs lock is used by itself. It is not a problem when mcs lock is embedded in mutex but will be an issue when the mcs_lock is used elsewhere. The patch removes the incorrect barriers and put in correct barriers with the pair of functions smp_load_acquire and smp_store_release. Suggested-by: Michel Lespinasse <walken@google.com> Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Waiman Long <Waiman.Long@hp.com> Signed-off-by: Jason Low <jason.low2@hp.com> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com> --- kernel/locking/mutex.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 4dd6e4c..fbbd2ed 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -136,9 +136,12 @@ void mspin_lock(struct mspin_node **lock, struct mspin_node *node) return; } ACCESS_ONCE(prev->next) = node; - smp_wmb(); - /* Wait until the lock holder passes the lock down */ - while (!ACCESS_ONCE(node->locked)) + /* + * Wait until the lock holder passes the lock down. + * Using smp_load_acquire() provides a memory barrier that + * ensures subsequent operations happen after the lock is acquired. + */ + while (!(smp_load_acquire(&node->locked))) arch_mutex_cpu_relax(); } @@ -156,8 +159,13 @@ static void mspin_unlock(struct mspin_node **lock, struct mspin_node *node) while (!(next = ACCESS_ONCE(node->next))) arch_mutex_cpu_relax(); } - ACCESS_ONCE(next->locked) = 1; - smp_wmb(); + /* + * Pass lock to next waiter. + * smp_store_release() provides a memory barrier to ensure + * all operations in the critical section has been completed + * before unlocking. + */ + smp_store_release(&next->locked, 1); } /* -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2014-01-21 23:36 UTC|newest] Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top [not found] <cover.1390320729.git.tim.c.chen@linux.intel.com> 2014-01-21 23:35 ` [PATCH v9 0/6] MCS Lock: MCS lock code cleanup and optimizations Tim Chen 2014-01-21 23:35 ` Tim Chen 2014-01-21 23:35 ` Tim Chen [this message] 2014-01-21 23:35 ` [PATCH v9 1/6] MCS Lock: Barrier corrections Tim Chen 2014-01-22 10:13 ` Will Deacon 2014-01-22 10:13 ` Will Deacon 2014-01-28 19:22 ` [tip:core/locking] locking/mutexes/mcs: Correct barrier usage tip-bot for Waiman Long 2014-01-21 23:36 ` [PATCH v9 2/6] MCS Lock: Restructure the MCS lock defines and locking code into its own file Tim Chen 2014-01-21 23:36 ` Tim Chen 2014-01-28 19:22 ` [tip:core/locking] locking/mutexes/mcs: " tip-bot for Tim Chen 2014-01-21 23:36 ` [PATCH v9 3/6] MCS Lock: Optimizations and extra comments Tim Chen 2014-01-21 23:36 ` Tim Chen 2014-01-28 19:23 ` [tip:core/locking] locking/mcs: Micro-optimize the MCS code, add " tip-bot for Jason Low 2014-01-21 23:36 ` [PATCH v9 4/6] MCS Lock: Allow architectures to hook in to contended paths Tim Chen 2014-01-21 23:36 ` Tim Chen 2014-01-28 19:23 ` [tip:core/locking] locking/mcs: " tip-bot for Will Deacon 2014-01-21 23:36 ` [PATCH v9 5/6] MCS Lock: Order the header files in Kbuild of each architecture in alphabetical order Tim Chen 2014-01-21 23:36 ` Tim Chen 2014-01-22 13:08 ` Peter Zijlstra 2014-01-22 13:08 ` Peter Zijlstra 2014-01-22 17:41 ` Tim Chen 2014-01-22 17:41 ` Tim Chen 2014-01-22 17:51 ` Peter Zijlstra 2014-01-22 17:51 ` Peter Zijlstra 2014-01-22 18:18 ` Peter Zijlstra 2014-01-22 18:18 ` Peter Zijlstra 2014-01-22 18:25 ` Peter Zijlstra 2014-01-22 18:25 ` Peter Zijlstra 2014-01-22 18:41 ` Tim Chen 2014-01-22 18:41 ` Tim Chen 2014-01-21 23:36 ` [PATCH v9 6/6] MCS Lock: Allow architecture specific asm files to be used for contended case Tim Chen 2014-01-21 23:36 ` Tim Chen 2014-01-22 18:35 ` Peter Zijlstra 2014-01-22 18:35 ` Peter Zijlstra 2014-01-22 18:35 ` Peter Zijlstra 2014-01-24 17:03 ` Tim Chen 2014-01-24 17:03 ` Tim Chen 2014-01-24 17:03 ` Tim Chen 2014-01-24 17:07 ` Peter Zijlstra 2014-01-24 17:07 ` Peter Zijlstra 2014-01-24 17:07 ` Peter Zijlstra 2014-01-24 17:33 ` Linus Torvalds 2014-01-24 17:33 ` Linus Torvalds 2014-01-24 17:37 ` Will Deacon 2014-01-24 17:37 ` Will Deacon 2014-02-10 13:32 ` [tip:core/locking] locking/mcs: " tip-bot for Tim Chen
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1390347353.3138.62.camel@schen9-DESK \ --to=tim.c.chen@linux.intel.com \ --cc=a.p.zijlstra@chello.nl \ --cc=aarcange@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=alex.shi@linaro.org \ --cc=andi@firstfloor.org \ --cc=arnd@arndb.de \ --cc=aswin@hp.com \ --cc=dave.hansen@intel.com \ --cc=davidlohr.bueso@hp.com \ --cc=figo1802@gmail.com \ --cc=hpa@zytor.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux@horizon.com \ --cc=matthew.r.wilcox@intel.com \ --cc=mingo@elte.hu \ --cc=paulmck@linux.vnet.ibm.com \ --cc=peter@hurleysoftware.com \ --cc=raghavendra.kt@linux.vnet.ibm.com \ --cc=riel@redhat.com \ --cc=scott.norton@hp.com \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ --cc=waiman.long@hp.com \ --cc=walken@google.com \ --cc=will.deacon@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.