From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2444DC433E6 for ; Tue, 2 Feb 2021 19:02:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D560D64F5F for ; Tue, 2 Feb 2021 19:02:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239314AbhBBTBu (ORCPT ); Tue, 2 Feb 2021 14:01:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239265AbhBBS7j (ORCPT ); Tue, 2 Feb 2021 13:59:39 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AD45C0617AB for ; Tue, 2 Feb 2021 10:57:50 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id e2so14797400pgg.10 for ; Tue, 02 Feb 2021 10:57:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=2jALOnR02DelYQsUJSUDQ9Zj6pptUDPmbQP0oTGAN5k=; b=Iev1zX72MfK7mD5itgi9KXbRFo17KVmLsFSv3pkVFuJA56XKPonwgx4fyJLFQFFuPQ 1qqgfd5EZyJ8faqztnNf/X34uP3WvAVpLYuKv9yoShz8iwOBTAwHzQDi5WzTnVveprJs RrpoX6LFPZfYUIZ7tqxPpOYwi3i2hMR3gtYumreFEWQLAGsIlKQK7wULg3yjsIVQ/t1v a/kR6kdfBC7uLLm3D7koCOuPg+I5Tbsy00wlJ50eCxZtvliRdbiYFuT22Xro5nrXnJpt newSqu6vAiwQB6Zj3k4hWYYfNYE0vRdzBp5rYeG2ENS1o+/mXCiWd660JKRKVumOrNpK sbLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2jALOnR02DelYQsUJSUDQ9Zj6pptUDPmbQP0oTGAN5k=; b=CPhC03YSzRVVO1oOXeNKNGdOTg+Y2pwmygVVyoH7F22Ef2MpTNSQSZcQGzNWYP3Rf1 tUA9S7QFw6AmxnZJUZJIIjKWJ/ZicecesbG61fBfryOtr/cnm1LAH2ctZ3JDP1uwOgLy pxyCIRH50BZuLwl9dtfuePgSuLk6v/jPKbWXmJN34offp2r21/U1c4lk66Bfzd2VKotB SQbCU+6mznDw5uAjJY8LB2ZFQoPSAmho+veTt/ZjFqO7iFIt6VDLWilDz94/C3wp73Ok UZu12djtKgYnFKpy9x1ACRoU4ygIM37H29wEJIiv39lCflnS9JZqA+hzFotVthYcwdRV T2Qw== X-Gm-Message-State: AOAM531qQDw9B3RtfYHPRIohVFJ+WIgnLrQzkmgnvSq1wN34mFCiL8mR LpjJT/Ke5UXYM5hyCcmrcpspRbv95uNGumW/RUdZZuanGgBg3GJ3kqZ2BBDd51BxstcGFqsEn1I m6U0efzTJms3QRCijr9SjU+JHA2jYQtrKXyPMfmTlUTmQKS4DCNYTfRXZ+e04u6hbP3VsYpb2 X-Google-Smtp-Source: ABdhPJxT7zsP4FMf12L3RkWtR7DMzVl1GMgVhCc5fj2RYAmrngkxUabKWYjAkS0v/TGl4HDYLG4nU9xuFPXy Sender: "bgardon via sendgmr" X-Received: from bgardon.sea.corp.google.com ([2620:15c:100:202:9090:561:5a98:6d47]) (user=bgardon job=sendgmr) by 2002:a17:902:7c88:b029:e1:c7d:c271 with SMTP id y8-20020a1709027c88b02900e10c7dc271mr24095510pll.74.1612292269427; Tue, 02 Feb 2021 10:57:49 -0800 (PST) Date: Tue, 2 Feb 2021 10:57:12 -0800 In-Reply-To: <20210202185734.1680553-1-bgardon@google.com> Message-Id: <20210202185734.1680553-7-bgardon@google.com> Mime-Version: 1.0 References: <20210202185734.1680553-1-bgardon@google.com> X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH v2 06/28] locking/rwlocks: Add contention detection for rwlocks From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , Peter Shier , Peter Feiner , Junaid Shahid , Jim Mattson , Yulei Zhang , Wanpeng Li , Vitaly Kuznetsov , Xiao Guangrong , Ben Gardon , Ingo Molnar , Will Deacon , Peter Zijlstra , Davidlohr Bueso , Waiman Long Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org rwlocks do not currently have any facility to detect contention like spinlocks do. In order to allow users of rwlocks to better manage latency, add contention detection for queued rwlocks. CC: Ingo Molnar CC: Will Deacon Acked-by: Peter Zijlstra Acked-by: Davidlohr Bueso Acked-by: Waiman Long Acked-by: Paolo Bonzini Signed-off-by: Ben Gardon --- include/asm-generic/qrwlock.h | 24 ++++++++++++++++++------ include/linux/rwlock.h | 7 +++++++ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 84ce841ce735..0020d3b820a7 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -14,6 +14,7 @@ #include #include +#include /* * Writer states & reader shift and bias. @@ -116,15 +117,26 @@ static inline void queued_write_unlock(struct qrwlock *lock) smp_store_release(&lock->wlocked, 0); } +/** + * queued_rwlock_is_contended - check if the lock is contended + * @lock : Pointer to queue rwlock structure + * Return: 1 if lock contended, 0 otherwise + */ +static inline int queued_rwlock_is_contended(struct qrwlock *lock) +{ + return arch_spin_is_locked(&lock->wait_lock); +} + /* * Remapping rwlock architecture specific functions to the corresponding * queue rwlock functions. */ -#define arch_read_lock(l) queued_read_lock(l) -#define arch_write_lock(l) queued_write_lock(l) -#define arch_read_trylock(l) queued_read_trylock(l) -#define arch_write_trylock(l) queued_write_trylock(l) -#define arch_read_unlock(l) queued_read_unlock(l) -#define arch_write_unlock(l) queued_write_unlock(l) +#define arch_read_lock(l) queued_read_lock(l) +#define arch_write_lock(l) queued_write_lock(l) +#define arch_read_trylock(l) queued_read_trylock(l) +#define arch_write_trylock(l) queued_write_trylock(l) +#define arch_read_unlock(l) queued_read_unlock(l) +#define arch_write_unlock(l) queued_write_unlock(l) +#define arch_rwlock_is_contended(l) queued_rwlock_is_contended(l) #endif /* __ASM_GENERIC_QRWLOCK_H */ diff --git a/include/linux/rwlock.h b/include/linux/rwlock.h index 3dcd617e65ae..7ce9a51ae5c0 100644 --- a/include/linux/rwlock.h +++ b/include/linux/rwlock.h @@ -128,4 +128,11 @@ do { \ 1 : ({ local_irq_restore(flags); 0; }); \ }) +#ifdef arch_rwlock_is_contended +#define rwlock_is_contended(lock) \ + arch_rwlock_is_contended(&(lock)->raw_lock) +#else +#define rwlock_is_contended(lock) ((void)(lock), 0) +#endif /* arch_rwlock_is_contended */ + #endif /* __LINUX_RWLOCK_H */ -- 2.30.0.365.g02bc693789-goog