From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751935AbcBLRci (ORCPT ); Fri, 12 Feb 2016 12:32:38 -0500 Received: from g1t6223.austin.hp.com ([15.73.96.124]:55642 "EHLO g1t6223.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751585AbcBLRcg (ORCPT ); Fri, 12 Feb 2016 12:32:36 -0500 From: Waiman Long To: Ingo Molnar Cc: Peter Zijlstra , linux-kernel@vger.kernel.org, Linus Torvalds , Ding Tianhong , Jason Low , Davidlohr Bueso , "Paul E. McKenney" , Thomas Gleixner , Will Deacon , Tim Chen , Waiman Long Subject: [PATCH v2 2/4] locking/mutex: Enable optimistic spinning of woken task in wait queue Date: Fri, 12 Feb 2016 12:32:13 -0500 Message-Id: <1455298335-53229-3-git-send-email-Waiman.Long@hpe.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1455298335-53229-1-git-send-email-Waiman.Long@hpe.com> References: <1455298335-53229-1-git-send-email-Waiman.Long@hpe.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Ding Tianhong reported a live-lock situation where a constant stream of incoming optimistic spinners blocked a task in the wait list from getting the mutex. This patch attempts to fix this live-lock condition by enabling the woken task in the wait queue to enter into an optimistic spinning loop itself in parallel with the regular spinners in the OSQ. This should prevent the live-lock condition from happening. Running the AIM7 benchmarks on a 4-socket E7-4820 v3 system (with ext4 filesystem), the additional spinning of the waiter-spinning improved performance for the following workloads at high user count: Workload % Improvement -------- ------------- alltests 3.9% disk 3.4% fserver 2.0% long 3.8% new_fserver 10.5% The other workloads were about the same as before. Signed-off-by: Waiman Long --- kernel/locking/mutex.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 3c41448..29c6d90 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -531,6 +531,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, struct task_struct *task = current; struct mutex_waiter waiter; unsigned long flags; + bool acquired = false; /* True if the lock is acquired */ int ret; preempt_disable(); @@ -561,7 +562,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, lock_contended(&lock->dep_map, ip); - for (;;) { + while (!acquired) { /* * Lets try to take the lock again - this is needed even if * we get here for the first time (shortly after failing to @@ -596,9 +597,18 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, /* didn't get the lock, go to sleep: */ spin_unlock_mutex(&lock->wait_lock, flags); schedule_preempt_disabled(); + + /* + * Optimistically spinning on the mutex without the wait lock + * The state has to be set to running to avoid another waker + * spinning on the on_cpu flag while the woken waiter is + * spinning on the mutex. + */ + __set_task_state(task, TASK_RUNNING); + acquired = mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx, + true); spin_lock_mutex(&lock->wait_lock, flags); } - __set_task_state(task, TASK_RUNNING); mutex_remove_waiter(lock, &waiter, current_thread_info()); /* set it to 0 if there are no waiters left: */ @@ -606,6 +616,9 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, atomic_set(&lock->count, 0); debug_mutex_free_waiter(&waiter); + if (acquired) + goto unlock; + skip_wait: /* got the lock - cleanup and rejoice! */ lock_acquired(&lock->dep_map, ip); @@ -616,6 +629,7 @@ skip_wait: ww_mutex_set_context_slowpath(ww, ww_ctx); } +unlock: spin_unlock_mutex(&lock->wait_lock, flags); preempt_enable(); return 0; -- 1.7.1