From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09034C433E6 for ; Mon, 8 Feb 2021 11:44:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B232F64E30 for ; Mon, 8 Feb 2021 11:44:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233283AbhBHLn4 (ORCPT ); Mon, 8 Feb 2021 06:43:56 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:12151 "EHLO szxga04-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233065AbhBHLXv (ORCPT ); Mon, 8 Feb 2021 06:23:51 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DZ3WW1s5Tz165PY; Mon, 8 Feb 2021 19:21:43 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Mon, 8 Feb 2021 19:22:56 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Catalin Marinas" , James Morse , "Julien Thierry" , Suzuki K Poulose , Gavin Shan , Quentin Perret , , , , CC: , , , Yanan Wang Subject: [RFC PATCH 3/4] KVM: arm64: Install the block entry before unmapping the page mappings Date: Mon, 8 Feb 2021 19:22:49 +0800 Message-ID: <20210208112250.163568-4-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210208112250.163568-1-wangyanan55@huawei.com> References: <20210208112250.163568-1-wangyanan55@huawei.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When KVM needs to coalesce the normal page mappings into a block mapping, we currently invalidate the old table entry first followed by invalidation of TLB, then unmap the page mappings, and install the block entry at last. It will cost a long time to unmap the numerous page mappings, which means there will be a long period when the table entry can be found invalid. If other vCPUs access any guest page within the block range and find the table entry invalid, they will all exit from guest with a translation fault which is not necessary. And KVM will make efforts to handle these faults, especially when performing CMOs by block range. So let's quickly install the block entry at first to ensure uninterrupted memory access of the other vCPUs, and then unmap the page mappings after installation. This will reduce most of the time when the table entry is invalid, and avoid most of the unnecessary translation faults. Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 78a560446f80..308c36b9cd21 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -434,6 +434,7 @@ struct stage2_map_data { kvm_pte_t attr; kvm_pte_t *anchor; + kvm_pte_t *follow; struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; @@ -553,15 +554,14 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; - kvm_set_invalid_pte(ptep); - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. + * If we need to coalesce existing table entries into a block here, + * then install the block entry first and the sub-level page mappings + * will be unmapped later. */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); data->anchor = ptep; + data->follow = kvm_pte_follow(*ptep); + stage2_coalesce_tables_into_block(addr, level, ptep, data); return 0; } @@ -614,20 +614,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret = 0; - if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); - put_page(virt_to_page(ptep)); - - if (data->anchor == ptep) { + if (data->anchor != ptep) { + free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(ptep)); + } else { + free_page((unsigned long)data->follow); data->anchor = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); } - return ret; + return 0; } /* -- 2.23.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79492C433E0 for ; Mon, 8 Feb 2021 11:23:16 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id BFE6564E40 for ; Mon, 8 Feb 2021 11:23:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BFE6564E40 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 56B574B48B; Mon, 8 Feb 2021 06:23:15 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O6rSOj4TfpOB; Mon, 8 Feb 2021 06:23:14 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4192C4B483; Mon, 8 Feb 2021 06:23:14 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 760144B48A for ; Mon, 8 Feb 2021 06:23:12 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QNlvFfzmsDRP for ; Mon, 8 Feb 2021 06:23:11 -0500 (EST) Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 02ACC4B471 for ; Mon, 8 Feb 2021 06:23:09 -0500 (EST) Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DZ3WW1s5Tz165PY; Mon, 8 Feb 2021 19:21:43 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Mon, 8 Feb 2021 19:22:56 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Catalin Marinas" , James Morse , "Julien Thierry" , Suzuki K Poulose , Gavin Shan , Quentin Perret , , , , Subject: [RFC PATCH 3/4] KVM: arm64: Install the block entry before unmapping the page mappings Date: Mon, 8 Feb 2021 19:22:49 +0800 Message-ID: <20210208112250.163568-4-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210208112250.163568-1-wangyanan55@huawei.com> References: <20210208112250.163568-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu When KVM needs to coalesce the normal page mappings into a block mapping, we currently invalidate the old table entry first followed by invalidation of TLB, then unmap the page mappings, and install the block entry at last. It will cost a long time to unmap the numerous page mappings, which means there will be a long period when the table entry can be found invalid. If other vCPUs access any guest page within the block range and find the table entry invalid, they will all exit from guest with a translation fault which is not necessary. And KVM will make efforts to handle these faults, especially when performing CMOs by block range. So let's quickly install the block entry at first to ensure uninterrupted memory access of the other vCPUs, and then unmap the page mappings after installation. This will reduce most of the time when the table entry is invalid, and avoid most of the unnecessary translation faults. Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 78a560446f80..308c36b9cd21 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -434,6 +434,7 @@ struct stage2_map_data { kvm_pte_t attr; kvm_pte_t *anchor; + kvm_pte_t *follow; struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; @@ -553,15 +554,14 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; - kvm_set_invalid_pte(ptep); - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. + * If we need to coalesce existing table entries into a block here, + * then install the block entry first and the sub-level page mappings + * will be unmapped later. */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); data->anchor = ptep; + data->follow = kvm_pte_follow(*ptep); + stage2_coalesce_tables_into_block(addr, level, ptep, data); return 0; } @@ -614,20 +614,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret = 0; - if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); - put_page(virt_to_page(ptep)); - - if (data->anchor == ptep) { + if (data->anchor != ptep) { + free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(ptep)); + } else { + free_page((unsigned long)data->follow); data->anchor = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); } - return ret; + return 0; } /* -- 2.23.0 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55229C433E0 for ; Mon, 8 Feb 2021 11:24:27 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DDE5D64DDF for ; Mon, 8 Feb 2021 11:24:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DDE5D64DDF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=B83Mx0lJVOLfdd3pSJ+2vNoLVv1FGZf1vU50rPFNOJI=; b=AyWpM57UyIFA14ZHSgyg/kJ7D dJQ364bKiNXmmkYmASzM0j5fhLILpIeBbEAXIx9jj6jjvAZmTVhS9FbqxWAkRgu5oWtR4XFZ06AsS UbN3L8xDLDHI+2A7awXZPzDm2sxKYKMhTAf2pc4Iq36kmDa6+zSdE2mi7FMtU5vO4dD7rkGSaYsoX 7+KiekNZek/Bro7Ri4be92R91SI0aQ9XpAaZCdzqpaZVbQenBZzO9iRAXfMuoGIWFaRLCEe+jJOdb zqbMY+0jg5K2WNx6PNwMjH9xg/JsBCF63U/6pHwTMXaToQNifnS/nQrtm5FBmB59vautncp49ihc7 EkNQerhKw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l94dU-0006ij-Gy; Mon, 08 Feb 2021 11:23:20 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l94dI-0006eK-L2 for linux-arm-kernel@lists.infradead.org; Mon, 08 Feb 2021 11:23:10 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DZ3WW1s5Tz165PY; Mon, 8 Feb 2021 19:21:43 +0800 (CST) Received: from DESKTOP-TMVL5KK.china.huawei.com (10.174.187.128) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Mon, 8 Feb 2021 19:22:56 +0800 From: Yanan Wang To: Marc Zyngier , Will Deacon , "Catalin Marinas" , James Morse , "Julien Thierry" , Suzuki K Poulose , Gavin Shan , Quentin Perret , , , , Subject: [RFC PATCH 3/4] KVM: arm64: Install the block entry before unmapping the page mappings Date: Mon, 8 Feb 2021 19:22:49 +0800 Message-ID: <20210208112250.163568-4-wangyanan55@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 In-Reply-To: <20210208112250.163568-1-wangyanan55@huawei.com> References: <20210208112250.163568-1-wangyanan55@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.187.128] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210208_062309_689891_6B908C22 X-CRM114-Status: GOOD ( 15.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuzenghui@huawei.com, wanghaibin.wang@huawei.com, Yanan Wang , zhukeqian1@huawei.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM needs to coalesce the normal page mappings into a block mapping, we currently invalidate the old table entry first followed by invalidation of TLB, then unmap the page mappings, and install the block entry at last. It will cost a long time to unmap the numerous page mappings, which means there will be a long period when the table entry can be found invalid. If other vCPUs access any guest page within the block range and find the table entry invalid, they will all exit from guest with a translation fault which is not necessary. And KVM will make efforts to handle these faults, especially when performing CMOs by block range. So let's quickly install the block entry at first to ensure uninterrupted memory access of the other vCPUs, and then unmap the page mappings after installation. This will reduce most of the time when the table entry is invalid, and avoid most of the unnecessary translation faults. Signed-off-by: Yanan Wang --- arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 78a560446f80..308c36b9cd21 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -434,6 +434,7 @@ struct stage2_map_data { kvm_pte_t attr; kvm_pte_t *anchor; + kvm_pte_t *follow; struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; @@ -553,15 +554,14 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; - kvm_set_invalid_pte(ptep); - /* - * Invalidate the whole stage-2, as we may have numerous leaf - * entries below us which would otherwise need invalidating - * individually. + * If we need to coalesce existing table entries into a block here, + * then install the block entry first and the sub-level page mappings + * will be unmapped later. */ - kvm_call_hyp(__kvm_tlb_flush_vmid, data->mmu); data->anchor = ptep; + data->follow = kvm_pte_follow(*ptep); + stage2_coalesce_tables_into_block(addr, level, ptep, data); return 0; } @@ -614,20 +614,18 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret = 0; - if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); - put_page(virt_to_page(ptep)); - - if (data->anchor == ptep) { + if (data->anchor != ptep) { + free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(ptep)); + } else { + free_page((unsigned long)data->follow); data->anchor = NULL; - ret = stage2_map_walk_leaf(addr, end, level, ptep, data); } - return ret; + return 0; } /* -- 2.23.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel