From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754909Ab3IXWWr (ORCPT ); Tue, 24 Sep 2013 18:22:47 -0400 Received: from mga09.intel.com ([134.134.136.24]:27631 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754636Ab3IXWWo (ORCPT ); Tue, 24 Sep 2013 18:22:44 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.90,974,1371106800"; d="scan'208";a="383033011" Subject: [PATCH v5 4/6] rwsem/wake: check lock before do atomic update From: Tim Chen To: Ingo Molnar , Andrew Morton Cc: Andrea Arcangeli , Alex Shi , Andi Kleen , Michel Lespinasse , Davidlohr Bueso , Matthew R Wilcox , Dave Hansen , Peter Zijlstra , Rik van Riel , Peter Hurley , Tim Chen , linux-kernel@vger.kernel.org, linux-mm In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Tue, 24 Sep 2013 15:22:40 -0700 Message-ID: <1380061360.3467.53.camel@schen9-DESK> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 (2.32.3-1.fc14) Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Atomic update lock and roll back will cause cache bouncing in large machine. A lock status pre-read can relieve this problem Suggested-by: Davidlohr bueso Suggested-by: Tim Chen Signed-off-by: Alex Shi --- lib/rwsem.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/lib/rwsem.c b/lib/rwsem.c index a8055cf..1d6e6e8 100644 --- a/lib/rwsem.c +++ b/lib/rwsem.c @@ -64,7 +64,7 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) struct rwsem_waiter *waiter; struct task_struct *tsk; struct list_head *next; - long oldcount, woken, loop, adjustment; + long woken, loop, adjustment; waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); if (waiter->type == RWSEM_WAITING_FOR_WRITE) { @@ -86,6 +86,12 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) if (wake_type != RWSEM_WAKE_READ_OWNED) { adjustment = RWSEM_ACTIVE_READ_BIAS; while (1) { + long oldcount; + + /* A writer stole the lock. */ + if (unlikely(sem->count < RWSEM_WAITING_BIAS)) + return sem; + oldcount = rwsem_atomic_update(adjustment, sem) - adjustment; if (likely(oldcount >= RWSEM_WAITING_BIAS)) -- 1.7.4.4 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f170.google.com (mail-pd0-f170.google.com [209.85.192.170]) by kanga.kvack.org (Postfix) with ESMTP id 263DD6B0034 for ; Tue, 24 Sep 2013 18:22:48 -0400 (EDT) Received: by mail-pd0-f170.google.com with SMTP id x10so5218783pdj.29 for ; Tue, 24 Sep 2013 15:22:47 -0700 (PDT) Subject: [PATCH v5 4/6] rwsem/wake: check lock before do atomic update From: Tim Chen In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Tue, 24 Sep 2013 15:22:40 -0700 Message-ID: <1380061360.3467.53.camel@schen9-DESK> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Ingo Molnar , Andrew Morton Cc: Andrea Arcangeli , Alex Shi , Andi Kleen , Michel Lespinasse , Davidlohr Bueso , Matthew R Wilcox , Dave Hansen , Peter Zijlstra , Rik van Riel , Peter Hurley , Tim Chen , linux-kernel@vger.kernel.org, linux-mm Atomic update lock and roll back will cause cache bouncing in large machine. A lock status pre-read can relieve this problem Suggested-by: Davidlohr bueso Suggested-by: Tim Chen Signed-off-by: Alex Shi --- lib/rwsem.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/lib/rwsem.c b/lib/rwsem.c index a8055cf..1d6e6e8 100644 --- a/lib/rwsem.c +++ b/lib/rwsem.c @@ -64,7 +64,7 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) struct rwsem_waiter *waiter; struct task_struct *tsk; struct list_head *next; - long oldcount, woken, loop, adjustment; + long woken, loop, adjustment; waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); if (waiter->type == RWSEM_WAITING_FOR_WRITE) { @@ -86,6 +86,12 @@ __rwsem_do_wake(struct rw_semaphore *sem, enum rwsem_wake_type wake_type) if (wake_type != RWSEM_WAKE_READ_OWNED) { adjustment = RWSEM_ACTIVE_READ_BIAS; while (1) { + long oldcount; + + /* A writer stole the lock. */ + if (unlikely(sem->count < RWSEM_WAITING_BIAS)) + return sem; + oldcount = rwsem_atomic_update(adjustment, sem) - adjustment; if (likely(oldcount >= RWSEM_WAITING_BIAS)) -- 1.7.4.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org