From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id C09E0C433FE for ; Thu, 19 May 2022 13:47:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7B5BD4B45F; Thu, 19 May 2022 09:47:59 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g3pSwKSNci2q; Thu, 19 May 2022 09:47:58 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7B9024B3AD; Thu, 19 May 2022 09:47:58 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 96FE64B465 for ; Thu, 19 May 2022 09:47:57 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fy7kdyvzgaoE for ; Thu, 19 May 2022 09:47:56 -0400 (EDT) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 8AAD74B4BD for ; Thu, 19 May 2022 09:47:56 -0400 (EDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BDE81B824B1; Thu, 19 May 2022 13:47:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6ECFC34117; Thu, 19 May 2022 13:47:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652968074; bh=Bb16fB1SmGSKYWwrpfSZcfkvibh8rQKnfPLdNL//f+Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bOg5OFUjv2277j/meQP3O4SaTl6wfeWQ86gbdZeGsaiOHccCxWEfmELroBkaS0d/I +QKCf7moqK4aBN832WtFmuPKh/qvDFmY/+rgSNCfw8aZ14/zC9tmR49iWg450LFf3/ z6GCDdavpRvE+CKcSQUaQVvEK/wz2/SCGU801vf2bylAjEJoAu0Pq46x2bHvW3KjpZ DA86pBK+RVb2wcG5XWS5V2qvuIcONAV6YwqWmNHaVywTjV8KWF830WnofrKs8mzHiM 4w0lD7YfWAqGD3MnSCvx5zmPJeoD4S//atYCW42tMe97HhbzZGvPjCBcD6hg6dJ8Kw JOA3De/F/wF+w== From: Will Deacon To: kvmarm@lists.cs.columbia.edu Subject: [PATCH 83/89] KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE Date: Thu, 19 May 2022 14:41:58 +0100 Message-Id: <20220519134204.5379-84-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220519134204.5379-1-will@kernel.org> References: <20220519134204.5379-1-will@kernel.org> MIME-Version: 1.0 Cc: Marc Zyngier , kernel-team@android.com, kvm@vger.kernel.org, Andy Lutomirski , linux-arm-kernel@lists.infradead.org, Michael Roth , Catalin Marinas , Chao Peng , Will Deacon X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Break-before-make (BBM) can be expensive, as transitioning via an invalid mapping (i.e. the "break" step) requires the completion of TLB invalidation and can also cause other agents to fault concurrently on the invalid mapping. Since BBM is not required when changing only the software bits of a PTE, avoid the sequence in this case and just update the PTE directly. Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2069e6833831..756bbb15c1f3 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -744,6 +744,13 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!stage2_pte_needs_update(old, new)) return -EAGAIN; + /* + * If we're only changing software bits, then we don't need to + * do anything else/ + */ + if (!((old ^ new) & ~KVM_PTE_LEAF_ATTR_HI_SW)) + goto out_set_pte; + stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); } @@ -754,9 +761,11 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - smp_store_release(ptep, new); if (stage2_pte_is_counted(new)) mm_ops->get_page(ptep); + +out_set_pte: + smp_store_release(ptep, new); if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; -- 2.36.1.124.g0e6072fb45-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D159DC433EF for ; Thu, 19 May 2022 13:49:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239046AbiESNtg (ORCPT ); Thu, 19 May 2022 09:49:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239117AbiESNsU (ORCPT ); Thu, 19 May 2022 09:48:20 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E4AF4248F for ; Thu, 19 May 2022 06:48:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F155C617C1 for ; Thu, 19 May 2022 13:47:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6ECFC34117; Thu, 19 May 2022 13:47:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652968074; bh=Bb16fB1SmGSKYWwrpfSZcfkvibh8rQKnfPLdNL//f+Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bOg5OFUjv2277j/meQP3O4SaTl6wfeWQ86gbdZeGsaiOHccCxWEfmELroBkaS0d/I +QKCf7moqK4aBN832WtFmuPKh/qvDFmY/+rgSNCfw8aZ14/zC9tmR49iWg450LFf3/ z6GCDdavpRvE+CKcSQUaQVvEK/wz2/SCGU801vf2bylAjEJoAu0Pq46x2bHvW3KjpZ DA86pBK+RVb2wcG5XWS5V2qvuIcONAV6YwqWmNHaVywTjV8KWF830WnofrKs8mzHiM 4w0lD7YfWAqGD3MnSCvx5zmPJeoD4S//atYCW42tMe97HhbzZGvPjCBcD6hg6dJ8Kw JOA3De/F/wF+w== From: Will Deacon To: kvmarm@lists.cs.columbia.edu Cc: Will Deacon , Ard Biesheuvel , Sean Christopherson , Alexandru Elisei , Andy Lutomirski , Catalin Marinas , James Morse , Chao Peng , Quentin Perret , Suzuki K Poulose , Michael Roth , Mark Rutland , Fuad Tabba , Oliver Upton , Marc Zyngier , kernel-team@android.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH 83/89] KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE Date: Thu, 19 May 2022 14:41:58 +0100 Message-Id: <20220519134204.5379-84-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220519134204.5379-1-will@kernel.org> References: <20220519134204.5379-1-will@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Break-before-make (BBM) can be expensive, as transitioning via an invalid mapping (i.e. the "break" step) requires the completion of TLB invalidation and can also cause other agents to fault concurrently on the invalid mapping. Since BBM is not required when changing only the software bits of a PTE, avoid the sequence in this case and just update the PTE directly. Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2069e6833831..756bbb15c1f3 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -744,6 +744,13 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!stage2_pte_needs_update(old, new)) return -EAGAIN; + /* + * If we're only changing software bits, then we don't need to + * do anything else/ + */ + if (!((old ^ new) & ~KVM_PTE_LEAF_ATTR_HI_SW)) + goto out_set_pte; + stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); } @@ -754,9 +761,11 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - smp_store_release(ptep, new); if (stage2_pte_is_counted(new)) mm_ops->get_page(ptep); + +out_set_pte: + smp_store_release(ptep, new); if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; -- 2.36.1.124.g0e6072fb45-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4AA0FC433EF for ; Thu, 19 May 2022 14:48:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xyMltFDdPRJPyZHpVzfKISl/IM1qMQAMXxwOS6FLjrs=; b=mJFpiAPiVjwLuL g9A/5XsN6QKZjNkGMBWA7L7rLEH94rSWnP8dgc3b1fo17q+LF+pixVaFbLzCvT1ncjc2g1JD+O6g1 QF1KNpTT6QEJ6km+ZY6TvsO7Y15JDOaEGQTRhF9XvIAgYEuvoDcXHnOaw+xQT9EAAMkCcII4wxuWE YGXFcVdFBMV4NvC4/uYrSgbM0EBXSN0qow06K0IS0NjMYJhwk+ksDDRM9peFWcrPM0VANkOd5asyA LYYlVgBOoe8MvMBES9TPKmoWc79ou/WZcakpbETUE59zQ2zzDPunI3wUea8XvdCSAxmQIwRgJLxiT djgvotV5dAH/XqS5AxZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrhQJ-007Ymb-OJ; Thu, 19 May 2022 14:46:43 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrgVR-0077ag-6I for linux-arm-kernel@lists.infradead.org; Thu, 19 May 2022 13:47:59 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BDE81B824B1; Thu, 19 May 2022 13:47:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6ECFC34117; Thu, 19 May 2022 13:47:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1652968074; bh=Bb16fB1SmGSKYWwrpfSZcfkvibh8rQKnfPLdNL//f+Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bOg5OFUjv2277j/meQP3O4SaTl6wfeWQ86gbdZeGsaiOHccCxWEfmELroBkaS0d/I +QKCf7moqK4aBN832WtFmuPKh/qvDFmY/+rgSNCfw8aZ14/zC9tmR49iWg450LFf3/ z6GCDdavpRvE+CKcSQUaQVvEK/wz2/SCGU801vf2bylAjEJoAu0Pq46x2bHvW3KjpZ DA86pBK+RVb2wcG5XWS5V2qvuIcONAV6YwqWmNHaVywTjV8KWF830WnofrKs8mzHiM 4w0lD7YfWAqGD3MnSCvx5zmPJeoD4S//atYCW42tMe97HhbzZGvPjCBcD6hg6dJ8Kw JOA3De/F/wF+w== From: Will Deacon To: kvmarm@lists.cs.columbia.edu Cc: Will Deacon , Ard Biesheuvel , Sean Christopherson , Alexandru Elisei , Andy Lutomirski , Catalin Marinas , James Morse , Chao Peng , Quentin Perret , Suzuki K Poulose , Michael Roth , Mark Rutland , Fuad Tabba , Oliver Upton , Marc Zyngier , kernel-team@android.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH 83/89] KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE Date: Thu, 19 May 2022 14:41:58 +0100 Message-Id: <20220519134204.5379-84-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220519134204.5379-1-will@kernel.org> References: <20220519134204.5379-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220519_064757_486528_F45A56EB X-CRM114-Status: GOOD ( 15.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Break-before-make (BBM) can be expensive, as transitioning via an invalid mapping (i.e. the "break" step) requires the completion of TLB invalidation and can also cause other agents to fault concurrently on the invalid mapping. Since BBM is not required when changing only the software bits of a PTE, avoid the sequence in this case and just update the PTE directly. Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2069e6833831..756bbb15c1f3 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -744,6 +744,13 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!stage2_pte_needs_update(old, new)) return -EAGAIN; + /* + * If we're only changing software bits, then we don't need to + * do anything else/ + */ + if (!((old ^ new) & ~KVM_PTE_LEAF_ATTR_HI_SW)) + goto out_set_pte; + stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); } @@ -754,9 +761,11 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (mm_ops->icache_inval_pou && stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - smp_store_release(ptep, new); if (stage2_pte_is_counted(new)) mm_ops->get_page(ptep); + +out_set_pte: + smp_store_release(ptep, new); if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; -- 2.36.1.124.g0e6072fb45-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel