From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E69A9C43387 for ; Thu, 20 Dec 2018 09:23:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B176B218C3 for ; Thu, 20 Dec 2018 09:23:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1545297816; bh=nsZCC066HsfUR6heUfkJZaSGv2Vx60T0A5qY5oweNzQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=keN+LIwO86K+tXqdfXOCTzJ/sx4GJwCsmvvxJ5MNTdiRd/9vCUroQmh08zunHVV3C 7nQGEXKoW+fBNAVB/jn7iswIq5W+j34T91QgsNrlTqXQQfGOeJqgNehC5ndmbkRK3/ SFipEg8YfzNm+0Oa5dxlcgSBV4dag0I88LWXGSTY= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731622AbeLTJXg (ORCPT ); Thu, 20 Dec 2018 04:23:36 -0500 Received: from mail.kernel.org ([198.145.29.99]:54474 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731601AbeLTJXd (ORCPT ); Thu, 20 Dec 2018 04:23:33 -0500 Received: from localhost (5356596B.cm-6-7b.dynamic.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8F35F20656; Thu, 20 Dec 2018 09:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1545297812; bh=nsZCC066HsfUR6heUfkJZaSGv2Vx60T0A5qY5oweNzQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJYOpMOUNEhd3NyncOMzFm5HKsmli+e4UXdcYYsqUsxO9wQgkuJoz+UJaPZvqjxUd WXct4uUA1vvlEF/nyjrh80jSh0VK8q/ttOyJ6WTkyDN8umhcnGS0RPPRGGM/GOpAjJ 8RUbdEJhbyrZCUQ1fT2oRg+yvfYHXOg7yN+5aOyk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Will Deacon , "Peter Zijlstra (Intel)" , Waiman Long , Linus Torvalds , Thomas Gleixner , boqun.feng@gmail.com, linux-arm-kernel@lists.infradead.org, paulmck@linux.vnet.ibm.com, Ingo Molnar , Sebastian Andrzej Siewior , Sasha Levin Subject: [PATCH 4.9 26/61] locking/qspinlock: Kill cmpxchg() loop when claiming lock from head of queue Date: Thu, 20 Dec 2018 10:18:26 +0100 Message-Id: <20181220085844.783297892@linuxfoundation.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181220085843.743900603@linuxfoundation.org> References: <20181220085843.743900603@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.9-stable review patch. If anyone has any objections, please let me know. ------------------ commit c61da58d8a9ba9238250a548f00826eaf44af0f7 upstream. When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg() loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg() loop without the loop, so that a failed cmpxchg() implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Signed-off-by: Will Deacon Acked-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Linus Torvalds Cc: Thomas Gleixner Cc: boqun.feng@gmail.com Cc: linux-arm-kernel@lists.infradead.org Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1524738868-31318-7-git-send-email-will.deacon@arm.com Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Sasha Levin --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index e7ab99a1f438..ba5dc86a4d83 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -581,24 +581,21 @@ locked: * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */ -- 2.19.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D78EC43387 for ; Thu, 20 Dec 2018 09:24:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 17E2020656 for ; Thu, 20 Dec 2018 09:24:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="tbLb6NYM"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="UJYOpMOU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 17E2020656 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linuxfoundation.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RgwiWz89bqHV2yNSKGJddjmB4kLGriSwrxVvEH3bZxQ=; b=tbLb6NYMphtKlr 0hE4ZexQqrAXZVcm2LoJGFmNbd8iDxG6NBHUhX01A7zYDq87KOCZ3O6XbyE+DfziJ5XIYeexGbb9+ CxMNrDBC8a8frrhAIcMU9RUHwHz4PusBP44IAZNPUwz45f3qmZulNt099BI2dNv1L0CMbt9otbjI0 RQRGzc0lw7oA137Va/QIi4cvA7UoW0G8JdhkORbHNSNEoF8OPgqASKDwMtZ8vTgKbkfVzJcqCOnAj i/u4wYFHTxL5mpgvFUwqrzLsljErU0ZRxm7zftnTeupqQFd+Ck3f5sgmTPnQydZI9HRTDyLlJPFAI 2GmpA0Qh+kZD5G+KpFMQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gZuZ3-00007F-L0; Thu, 20 Dec 2018 09:24:21 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gZuYO-0007nR-JR for linux-arm-kernel@lists.infradead.org; Thu, 20 Dec 2018 09:23:43 +0000 Received: from localhost (5356596B.cm-6-7b.dynamic.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8F35F20656; Thu, 20 Dec 2018 09:23:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1545297812; bh=nsZCC066HsfUR6heUfkJZaSGv2Vx60T0A5qY5oweNzQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJYOpMOUNEhd3NyncOMzFm5HKsmli+e4UXdcYYsqUsxO9wQgkuJoz+UJaPZvqjxUd WXct4uUA1vvlEF/nyjrh80jSh0VK8q/ttOyJ6WTkyDN8umhcnGS0RPPRGGM/GOpAjJ 8RUbdEJhbyrZCUQ1fT2oRg+yvfYHXOg7yN+5aOyk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Subject: [PATCH 4.9 26/61] locking/qspinlock: Kill cmpxchg() loop when claiming lock from head of queue Date: Thu, 20 Dec 2018 10:18:26 +0100 Message-Id: <20181220085844.783297892@linuxfoundation.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181220085843.743900603@linuxfoundation.org> References: <20181220085843.743900603@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181220_012340_668098_CD9E7496 X-CRM114-Status: GOOD ( 16.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , "Peter Zijlstra \(Intel\)" , Greg Kroah-Hartman , boqun.feng@gmail.com, Will Deacon , stable@vger.kernel.org, Sebastian Andrzej Siewior , Waiman Long , Thomas Gleixner , paulmck@linux.vnet.ibm.com, Linus Torvalds , Ingo Molnar , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org 4.9-stable review patch. If anyone has any objections, please let me know. ------------------ commit c61da58d8a9ba9238250a548f00826eaf44af0f7 upstream. When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg() loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg() loop without the loop, so that a failed cmpxchg() implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Signed-off-by: Will Deacon Acked-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Linus Torvalds Cc: Thomas Gleixner Cc: boqun.feng@gmail.com Cc: linux-arm-kernel@lists.infradead.org Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1524738868-31318-7-git-send-email-will.deacon@arm.com Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Sasha Levin --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index e7ab99a1f438..ba5dc86a4d83 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -581,24 +581,21 @@ locked: * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */ -- 2.19.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel