From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F4E2C4708A for ; Thu, 27 May 2021 12:51:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 32DC56109F for ; Thu, 27 May 2021 12:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235874AbhE0MxW (ORCPT ); Thu, 27 May 2021 08:53:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235811AbhE0MxN (ORCPT ); Thu, 27 May 2021 08:53:13 -0400 Received: from mail-wr1-x44a.google.com (mail-wr1-x44a.google.com [IPv6:2a00:1450:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12463C061574 for ; Thu, 27 May 2021 05:51:40 -0700 (PDT) Received: by mail-wr1-x44a.google.com with SMTP id h104-20020adf90710000b029010de8455a3aso1700530wrh.12 for ; Thu, 27 May 2021 05:51:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=jVHXS3bvMibkBWS+bH3WSLubeEwo4y03J+vRbsZSsVjoLuBFta5X2GtmfPD5/3Z7QY TA8LUPhrubLg14KwptPsqTA0fyIxxEiMv2j6qPoea7xGJBviUgOJS8LKfAtQ6LRg4xiZ vNH4I54lKLKr8niplcjTAZf0ZeEFp4WpohPKu/ocRKYxchgP0pLtq8yCHW0wOP4GUgyk /wmQI3jlJLAHjRBTEUFDiGi2tu3nCsJijS3psLbSR0F2aqlmeNBMjDWodcEhaMfXQ52i HA4j1aHPpJE4izPecyVmlmp71jx2oZWHOQWqJWSClhiCwre/jt9wfMAJdPwgoSBHyx7U Go4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=hdpAz0UkUeXDm0KozV+nA181Qb0ZMTG5mKBQjxGRtqSLi0obSi7i12np8M1x6diXHO I0kH9m3stgfICt1HAbE6nSICkF5MyDUPrEnhmovyDEJgV8daHreLAGVmvpBQatSidNog XR6KYshYZ7+DEXTulhX9G3CoVLmvWL1iYP4/mBei9LdWLa/WYnCJtQ3JrIrFGtam1+zi gvzkWBWiiV6vyRisK5BmNNchR0oIscYxp40XpkmVGSBj6OGYUjhckbNV8+w5fop0VyOD Ov4e6GvgGpxwsn+NfxB+vbaWMB/8fMNliAxN2rLigcHYwP92gKvc/lPW4+5OS6WwGosh c9RA== X-Gm-Message-State: AOAM531CSV+007cJTAdv4Foog0BXHEIMRHcHMHnWIAjVONwNJ23BwI26 LvGnPOC0W2iGTeBCk9OMW/2+JVd6XnXT X-Google-Smtp-Source: ABdhPJx3hDj3fqzQNROoGJzyF8S2ksFUOMHLT2ocwl3Ym1DkFjpNtt3AG2ucCS5H6j++RJ8/UzZ53q5BHRH4 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:ad3:: with SMTP id c19mr8297036wmr.66.1622119898658; Thu, 27 May 2021 05:51:38 -0700 (PDT) Date: Thu, 27 May 2021 12:51:28 +0000 In-Reply-To: <20210527125134.2116404-1-qperret@google.com> Message-Id: <20210527125134.2116404-2-qperret@google.com> Mime-Version: 1.0 References: <20210527125134.2116404-1-qperret@google.com> X-Mailer: git-send-email 2.31.1.818.g46aad6cb9e-goog Subject: [PATCH 1/7] KVM: arm64: Move hyp_pool locking out of refcount helpers From: Quentin Perret To: maz@kernel.org, will@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, catalin.marinas@arm.com, suzuki.poulose@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The hyp_page refcount helpers currently rely on the hyp_pool lock for serialization. However, this means the refcounts can't be changed from the buddy allocator core as it already holds the lock, which means pages have to go through odd transient states. For example, when a page is freed, its refcount is set to 0, and the lock is transiently released before the page can be attached to a free list in the buddy tree. This is currently harmless as the allocator checks the list node of each page to see if it is available for allocation or not, but it means the page refcount can't be trusted to represent the state of the page even if the pool lock is held. In order to fix this, remove the pool locking from the refcount helpers, and move all the logic to the buddy allocator. This will simplify the removal of the list node from struct hyp_page in a later patch. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 21 ++------------------- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 19 ++++++++----------- 2 files changed, 10 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h index 18a4494337bd..aada4d97de49 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/gfp.h +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -24,37 +24,20 @@ struct hyp_pool { static inline void hyp_page_ref_inc(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); p->refcount++; - hyp_spin_unlock(&pool->lock); } static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - int ret; - - hyp_spin_lock(&pool->lock); p->refcount--; - ret = (p->refcount == 0); - hyp_spin_unlock(&pool->lock); - - return ret; + return (p->refcount == 0); } static inline void hyp_set_page_refcounted(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - if (p->refcount) { - hyp_spin_unlock(&pool->lock); + if (p->refcount) BUG(); - } p->refcount = 1; - hyp_spin_unlock(&pool->lock); } /* Allocation */ diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index 237e03bf0cb1..04573bf35441 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -93,15 +93,6 @@ static void __hyp_attach_page(struct hyp_pool *pool, list_add_tail(&p->node, &pool->free_area[order]); } -static void hyp_attach_page(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - __hyp_attach_page(pool, p); - hyp_spin_unlock(&pool->lock); -} - static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, struct hyp_page *p, unsigned int order) @@ -128,16 +119,22 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, void hyp_put_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); if (hyp_page_ref_dec_and_test(p)) - hyp_attach_page(p); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); } void hyp_get_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); hyp_page_ref_inc(p); + hyp_spin_unlock(&pool->lock); } void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) @@ -159,8 +156,8 @@ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) p = list_first_entry(&pool->free_area[i], struct hyp_page, node); p = __hyp_extract_page(pool, p, order); - hyp_spin_unlock(&pool->lock); hyp_set_page_refcounted(p); + hyp_spin_unlock(&pool->lock); return hyp_page_to_virt(p); } -- 2.31.1.818.g46aad6cb9e-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58ACBC47089 for ; Thu, 27 May 2021 12:51:45 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id EBCD2613D1 for ; Thu, 27 May 2021 12:51:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBCD2613D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 9DF9649F83; Thu, 27 May 2021 08:51:44 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RycsXk51iPoB; Thu, 27 May 2021 08:51:43 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8CB6E4A531; Thu, 27 May 2021 08:51:43 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id AC9B94A126 for ; Thu, 27 May 2021 08:51:41 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7eZB5kUGtAEM for ; Thu, 27 May 2021 08:51:39 -0400 (EDT) Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id BDBA740808 for ; Thu, 27 May 2021 08:51:39 -0400 (EDT) Received: by mail-wr1-f73.google.com with SMTP id x10-20020adfc18a0000b029010d83c83f2aso1704495wre.8 for ; Thu, 27 May 2021 05:51:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=jVHXS3bvMibkBWS+bH3WSLubeEwo4y03J+vRbsZSsVjoLuBFta5X2GtmfPD5/3Z7QY TA8LUPhrubLg14KwptPsqTA0fyIxxEiMv2j6qPoea7xGJBviUgOJS8LKfAtQ6LRg4xiZ vNH4I54lKLKr8niplcjTAZf0ZeEFp4WpohPKu/ocRKYxchgP0pLtq8yCHW0wOP4GUgyk /wmQI3jlJLAHjRBTEUFDiGi2tu3nCsJijS3psLbSR0F2aqlmeNBMjDWodcEhaMfXQ52i HA4j1aHPpJE4izPecyVmlmp71jx2oZWHOQWqJWSClhiCwre/jt9wfMAJdPwgoSBHyx7U Go4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=tf6JKfeHgwGuun0uyJqG+Khc2urp7t8eKRmu0Dcn0iEW8SjWEro4C7605E63y6P1Tm urinbRDqF2JGcUcFsZUzjptSjwuVcTQc1rxTCx4h+i3OjW3GZn0mQOaHI8wc3fneuA7C Q68ey2PKx9jKQTMYQpKWEvE+GJJ/6m/2qgHKCQtPmJdMh4fPtR+tHOxBtxBXmQv0E3nj JiFmFdlwJxB/noGBSgB7oo72unKhXPSrCXU8HtqSy9uNloubBIUWASkwXNrnUcevO6pc c8fL99o+cGTfUY6+3WYCP5n4/LqVxqTmn2LznBudQ9BvYTLCkjTsdMtNL/mrGRIL8FDS aysw== X-Gm-Message-State: AOAM530iZO5iIDWZVjE01nx4aF+Qcd8R3+pmR6zLR+RK1m4Zc6ndUzwo SNhH9EOicIl948/J0EQiZNxFy3F8W4k6 X-Google-Smtp-Source: ABdhPJx3hDj3fqzQNROoGJzyF8S2ksFUOMHLT2ocwl3Ym1DkFjpNtt3AG2ucCS5H6j++RJ8/UzZ53q5BHRH4 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:ad3:: with SMTP id c19mr8297036wmr.66.1622119898658; Thu, 27 May 2021 05:51:38 -0700 (PDT) Date: Thu, 27 May 2021 12:51:28 +0000 In-Reply-To: <20210527125134.2116404-1-qperret@google.com> Message-Id: <20210527125134.2116404-2-qperret@google.com> Mime-Version: 1.0 References: <20210527125134.2116404-1-qperret@google.com> X-Mailer: git-send-email 2.31.1.818.g46aad6cb9e-goog Subject: [PATCH 1/7] KVM: arm64: Move hyp_pool locking out of refcount helpers From: Quentin Perret To: maz@kernel.org, will@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, catalin.marinas@arm.com, suzuki.poulose@arm.com Cc: kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu The hyp_page refcount helpers currently rely on the hyp_pool lock for serialization. However, this means the refcounts can't be changed from the buddy allocator core as it already holds the lock, which means pages have to go through odd transient states. For example, when a page is freed, its refcount is set to 0, and the lock is transiently released before the page can be attached to a free list in the buddy tree. This is currently harmless as the allocator checks the list node of each page to see if it is available for allocation or not, but it means the page refcount can't be trusted to represent the state of the page even if the pool lock is held. In order to fix this, remove the pool locking from the refcount helpers, and move all the logic to the buddy allocator. This will simplify the removal of the list node from struct hyp_page in a later patch. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 21 ++------------------- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 19 ++++++++----------- 2 files changed, 10 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h index 18a4494337bd..aada4d97de49 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/gfp.h +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -24,37 +24,20 @@ struct hyp_pool { static inline void hyp_page_ref_inc(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); p->refcount++; - hyp_spin_unlock(&pool->lock); } static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - int ret; - - hyp_spin_lock(&pool->lock); p->refcount--; - ret = (p->refcount == 0); - hyp_spin_unlock(&pool->lock); - - return ret; + return (p->refcount == 0); } static inline void hyp_set_page_refcounted(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - if (p->refcount) { - hyp_spin_unlock(&pool->lock); + if (p->refcount) BUG(); - } p->refcount = 1; - hyp_spin_unlock(&pool->lock); } /* Allocation */ diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index 237e03bf0cb1..04573bf35441 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -93,15 +93,6 @@ static void __hyp_attach_page(struct hyp_pool *pool, list_add_tail(&p->node, &pool->free_area[order]); } -static void hyp_attach_page(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - __hyp_attach_page(pool, p); - hyp_spin_unlock(&pool->lock); -} - static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, struct hyp_page *p, unsigned int order) @@ -128,16 +119,22 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, void hyp_put_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); if (hyp_page_ref_dec_and_test(p)) - hyp_attach_page(p); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); } void hyp_get_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); hyp_page_ref_inc(p); + hyp_spin_unlock(&pool->lock); } void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) @@ -159,8 +156,8 @@ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) p = list_first_entry(&pool->free_area[i], struct hyp_page, node); p = __hyp_extract_page(pool, p, order); - hyp_spin_unlock(&pool->lock); hyp_set_page_refcounted(p); + hyp_spin_unlock(&pool->lock); return hyp_page_to_virt(p); } -- 2.31.1.818.g46aad6cb9e-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CAF6C4708A for ; Thu, 27 May 2021 12:53:52 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3613B610A8 for ; Thu, 27 May 2021 12:53:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3613B610A8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=DjEkQfJHCpQP1rnu0ERUZ/8n0VaT/GeCf49fx0Ktg8Y=; b=hTLxXetNLsQs8yb20WMqwI+nOP +jgr00o087YV6auV3oo3soImLRQmR6QljSkZ93EDOH6XzKZ7/luEy119Sl5dhizUTfY+kR2wlUHCx xHVzYWHvfvZpTEe1QuWuZhq+LAvsPTDyDOeBoKrrhEa0KF/FJseqb6lrX2GVTX0G+V5xv9SbCoFow 9JMeLL3h6YRJMiY+GOFB7fGAKwZ4ozRraZFmVy8aOxu2BeJ6de3w4ywyLdL13NkBunB0RlGL2XQap pOAEFU3yblaS0bHR4/vqv7cbIlRDRuaCkCJ+aS+eyrm5RsgHeUoIp4ZBZqDvFDh9XLLMVBfdow6x6 T2ZvFADg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lmFUa-005tau-6q; Thu, 27 May 2021 12:52:04 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lmFUC-005tP4-Pt for linux-arm-kernel@lists.infradead.org; Thu, 27 May 2021 12:51:42 +0000 Received: by mail-wm1-x34a.google.com with SMTP id h18-20020a05600c3512b029018434eb1bd8so1458581wmq.2 for ; Thu, 27 May 2021 05:51:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=jVHXS3bvMibkBWS+bH3WSLubeEwo4y03J+vRbsZSsVjoLuBFta5X2GtmfPD5/3Z7QY TA8LUPhrubLg14KwptPsqTA0fyIxxEiMv2j6qPoea7xGJBviUgOJS8LKfAtQ6LRg4xiZ vNH4I54lKLKr8niplcjTAZf0ZeEFp4WpohPKu/ocRKYxchgP0pLtq8yCHW0wOP4GUgyk /wmQI3jlJLAHjRBTEUFDiGi2tu3nCsJijS3psLbSR0F2aqlmeNBMjDWodcEhaMfXQ52i HA4j1aHPpJE4izPecyVmlmp71jx2oZWHOQWqJWSClhiCwre/jt9wfMAJdPwgoSBHyx7U Go4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=59ZrbnR57dXVmexXuU0iZx85Sibct5xhty4f3jqzmUk=; b=UKHZZ64VwyHe0UxlL9TsaYaH9T4IN1+spcew46CkHQ7KIeI2rczsPVS3CtAgmk0U6y hP0S/SkR7GzZWkz27dYWp2YDO0MxSGlDx8Js5D20LQ8MGOkJiGKeOk12JMFIOHDMmqKi Jhx+izvNewjfB9oH3oObrDTuT3tFZvSClq+1kRGN6et/gYBAWRRjt4c266zmrX05MG6W KsqBbgVnJIb0zt/LoWVPIutr18DZaMIlH40NIFQuEJFhrnmSdXDV6UlBvO+ozBPhQgs/ QamkqnBimilFU7jqT1EKutRioU8auzBzAee4xRy5phpU/lTHk6U9k6ASC43TIdR7zByW Onig== X-Gm-Message-State: AOAM532SnVJPuHZC254v9x9qbrCNyObZsahLxSTy5ileW7Nxt+MNRhAO g1XlYajv4INV6u8aDuIXEePj25mUHSzG X-Google-Smtp-Source: ABdhPJx3hDj3fqzQNROoGJzyF8S2ksFUOMHLT2ocwl3Ym1DkFjpNtt3AG2ucCS5H6j++RJ8/UzZ53q5BHRH4 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:ad3:: with SMTP id c19mr8297036wmr.66.1622119898658; Thu, 27 May 2021 05:51:38 -0700 (PDT) Date: Thu, 27 May 2021 12:51:28 +0000 In-Reply-To: <20210527125134.2116404-1-qperret@google.com> Message-Id: <20210527125134.2116404-2-qperret@google.com> Mime-Version: 1.0 References: <20210527125134.2116404-1-qperret@google.com> X-Mailer: git-send-email 2.31.1.818.g46aad6cb9e-goog Subject: [PATCH 1/7] KVM: arm64: Move hyp_pool locking out of refcount helpers From: Quentin Perret To: maz@kernel.org, will@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, catalin.marinas@arm.com, suzuki.poulose@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210527_055140_866493_309DA83A X-CRM114-Status: GOOD ( 19.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hyp_page refcount helpers currently rely on the hyp_pool lock for serialization. However, this means the refcounts can't be changed from the buddy allocator core as it already holds the lock, which means pages have to go through odd transient states. For example, when a page is freed, its refcount is set to 0, and the lock is transiently released before the page can be attached to a free list in the buddy tree. This is currently harmless as the allocator checks the list node of each page to see if it is available for allocation or not, but it means the page refcount can't be trusted to represent the state of the page even if the pool lock is held. In order to fix this, remove the pool locking from the refcount helpers, and move all the logic to the buddy allocator. This will simplify the removal of the list node from struct hyp_page in a later patch. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 21 ++------------------- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 19 ++++++++----------- 2 files changed, 10 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h index 18a4494337bd..aada4d97de49 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/gfp.h +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -24,37 +24,20 @@ struct hyp_pool { static inline void hyp_page_ref_inc(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); p->refcount++; - hyp_spin_unlock(&pool->lock); } static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - int ret; - - hyp_spin_lock(&pool->lock); p->refcount--; - ret = (p->refcount == 0); - hyp_spin_unlock(&pool->lock); - - return ret; + return (p->refcount == 0); } static inline void hyp_set_page_refcounted(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - if (p->refcount) { - hyp_spin_unlock(&pool->lock); + if (p->refcount) BUG(); - } p->refcount = 1; - hyp_spin_unlock(&pool->lock); } /* Allocation */ diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index 237e03bf0cb1..04573bf35441 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -93,15 +93,6 @@ static void __hyp_attach_page(struct hyp_pool *pool, list_add_tail(&p->node, &pool->free_area[order]); } -static void hyp_attach_page(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - __hyp_attach_page(pool, p); - hyp_spin_unlock(&pool->lock); -} - static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, struct hyp_page *p, unsigned int order) @@ -128,16 +119,22 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, void hyp_put_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); if (hyp_page_ref_dec_and_test(p)) - hyp_attach_page(p); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); } void hyp_get_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); hyp_page_ref_inc(p); + hyp_spin_unlock(&pool->lock); } void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) @@ -159,8 +156,8 @@ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) p = list_first_entry(&pool->free_area[i], struct hyp_page, node); p = __hyp_extract_page(pool, p, order); - hyp_spin_unlock(&pool->lock); hyp_set_page_refcounted(p); + hyp_spin_unlock(&pool->lock); return hyp_page_to_virt(p); } -- 2.31.1.818.g46aad6cb9e-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel