From: Will Deacon <will@kernel.org> To: kvmarm@lists.cs.columbia.edu Cc: kernel-team@android.com, Marc Zyngier <maz@kernel.org>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, linux-arm-kernel@lists.infradead.org Subject: [PATCH v5 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Date: Fri, 11 Sep 2020 14:25:18 +0100 [thread overview] Message-ID: <20200911132529.19844-11-will@kernel.org> (raw) In-Reply-To: <20200911132529.19844-1-will@kernel.org> Add stage-2 mkyoung(), mkold() and is_young() operations to the generic page-table code. Cc: Marc Zyngier <maz@kernel.org> Cc: Quentin Perret <qperret@google.com> Reviewed-by: Gavin Shan <gshan@redhat.com> Signed-off-by: Will Deacon <will@kernel.org> --- arch/arm64/include/asm/kvm_pgtable.h | 44 ++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 86 ++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 895b2238062b..50782128c861 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -186,6 +186,50 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, */ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size); +/** + * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * If there is a valid, leaf page-table entry used to translate @addr, then + * set the access flag in that entry. + * + * Return: The old page-table entry prior to setting the flag, 0 on failure. + */ +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); + +/** + * kvm_pgtable_stage2_mkold() - Clear the access flag in a page-table entry. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * If there is a valid, leaf page-table entry used to translate @addr, then + * clear the access flag in that entry. + * + * Note that it is the caller's responsibility to invalidate the TLB after + * calling this function to ensure that the updated permissions are visible + * to the CPUs. + * + * Return: The old page-table entry prior to clearing the flag, 0 on failure. + */ +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr); + +/** + * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the + * access flag set. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * Return: True if the page-table entry has the access flag set, false otherwise. + */ +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); + /** * kvm_pgtable_walk() - Walk a page-table. * @pgt: Page-table structure initialised by kvm_pgtable_*_init(). diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2f9b872f5355..af60ea8ee29d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -690,6 +690,92 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } +struct stage2_attr_data { + kvm_pte_t attr_set; + kvm_pte_t attr_clr; + kvm_pte_t pte; +}; + +static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + kvm_pte_t pte = *ptep; + struct stage2_attr_data *data = arg; + + if (!kvm_pte_valid(pte)) + return 0; + + data->pte = pte; + pte &= ~data->attr_clr; + pte |= data->attr_set; + + /* + * We may race with the CPU trying to set the access flag here, + * but worst-case the access flag update gets lost and will be + * set on the next access instead. + */ + if (data->pte != pte) + WRITE_ONCE(*ptep, pte); + + return 0; +} + +static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, + u64 size, kvm_pte_t attr_set, + kvm_pte_t attr_clr, kvm_pte_t *orig_pte) +{ + int ret; + kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI; + struct stage2_attr_data data = { + .attr_set = attr_set & attr_mask, + .attr_clr = attr_clr & attr_mask, + }; + struct kvm_pgtable_walker walker = { + .cb = stage2_attr_walker, + .arg = &data, + .flags = KVM_PGTABLE_WALK_LEAF, + }; + + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + if (ret) + return ret; + + if (orig_pte) + *orig_pte = data.pte; + return 0; +} + +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0, + &pte); + dsb(ishst); + return pte; +} + +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF, + &pte); + /* + * "But where's the TLBI?!", you scream. + * "Over in the core code", I sigh. + * + * See the '->clear_flush_young()' callback on the KVM mmu notifier. + */ + return pte; +} + +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte); + return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF; +} + int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) { size_t pgd_sz; -- 2.28.0.618.gf4bc123cb7-goog _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
WARNING: multiple messages have this Message-ID (diff)
From: Will Deacon <will@kernel.org> To: kvmarm@lists.cs.columbia.edu Cc: kernel-team@android.com, Gavin Shan <gshan@redhat.com>, Suzuki Poulose <suzuki.poulose@arm.com>, Marc Zyngier <maz@kernel.org>, Quentin Perret <qperret@google.com>, Alexandru Elisei <alexandru.elisei@arm.com>, James Morse <james.morse@arm.com>, Andrew Scull <ascull@google.com>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, linux-arm-kernel@lists.infradead.org Subject: [PATCH v5 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Date: Fri, 11 Sep 2020 14:25:18 +0100 [thread overview] Message-ID: <20200911132529.19844-11-will@kernel.org> (raw) In-Reply-To: <20200911132529.19844-1-will@kernel.org> Add stage-2 mkyoung(), mkold() and is_young() operations to the generic page-table code. Cc: Marc Zyngier <maz@kernel.org> Cc: Quentin Perret <qperret@google.com> Reviewed-by: Gavin Shan <gshan@redhat.com> Signed-off-by: Will Deacon <will@kernel.org> --- arch/arm64/include/asm/kvm_pgtable.h | 44 ++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 86 ++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 895b2238062b..50782128c861 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -186,6 +186,50 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, */ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size); +/** + * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * If there is a valid, leaf page-table entry used to translate @addr, then + * set the access flag in that entry. + * + * Return: The old page-table entry prior to setting the flag, 0 on failure. + */ +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); + +/** + * kvm_pgtable_stage2_mkold() - Clear the access flag in a page-table entry. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * If there is a valid, leaf page-table entry used to translate @addr, then + * clear the access flag in that entry. + * + * Note that it is the caller's responsibility to invalidate the TLB after + * calling this function to ensure that the updated permissions are visible + * to the CPUs. + * + * Return: The old page-table entry prior to clearing the flag, 0 on failure. + */ +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr); + +/** + * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the + * access flag set. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address to identify the page-table entry. + * + * The offset of @addr within a page is ignored. + * + * Return: True if the page-table entry has the access flag set, false otherwise. + */ +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr); + /** * kvm_pgtable_walk() - Walk a page-table. * @pgt: Page-table structure initialised by kvm_pgtable_*_init(). diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 2f9b872f5355..af60ea8ee29d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -690,6 +690,92 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } +struct stage2_attr_data { + kvm_pte_t attr_set; + kvm_pte_t attr_clr; + kvm_pte_t pte; +}; + +static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + kvm_pte_t pte = *ptep; + struct stage2_attr_data *data = arg; + + if (!kvm_pte_valid(pte)) + return 0; + + data->pte = pte; + pte &= ~data->attr_clr; + pte |= data->attr_set; + + /* + * We may race with the CPU trying to set the access flag here, + * but worst-case the access flag update gets lost and will be + * set on the next access instead. + */ + if (data->pte != pte) + WRITE_ONCE(*ptep, pte); + + return 0; +} + +static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr, + u64 size, kvm_pte_t attr_set, + kvm_pte_t attr_clr, kvm_pte_t *orig_pte) +{ + int ret; + kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI; + struct stage2_attr_data data = { + .attr_set = attr_set & attr_mask, + .attr_clr = attr_clr & attr_mask, + }; + struct kvm_pgtable_walker walker = { + .cb = stage2_attr_walker, + .arg = &data, + .flags = KVM_PGTABLE_WALK_LEAF, + }; + + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + if (ret) + return ret; + + if (orig_pte) + *orig_pte = data.pte; + return 0; +} + +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0, + &pte); + dsb(ishst); + return pte; +} + +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF, + &pte); + /* + * "But where's the TLBI?!", you scream. + * "Over in the core code", I sigh. + * + * See the '->clear_flush_young()' callback on the KVM mmu notifier. + */ + return pte; +} + +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr) +{ + kvm_pte_t pte = 0; + stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte); + return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF; +} + int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) { size_t pgd_sz; -- 2.28.0.618.gf4bc123cb7-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-09-11 13:26 UTC|newest] Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-09-11 13:25 [PATCH v5 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches() Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-15 10:47 ` Alexandru Elisei 2020-09-15 10:47 ` Alexandru Elisei 2020-09-11 13:25 ` [PATCH v5 07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 08/21] KVM: arm64: Convert kvm_set_spte_hva() " Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 09/21] KVM: arm64: Convert unmap_stage2_range() " Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-15 10:57 ` Alexandru Elisei 2020-09-15 10:57 ` Alexandru Elisei 2020-09-11 13:25 ` Will Deacon [this message] 2020-09-11 13:25 ` [PATCH v5 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Will Deacon 2020-09-11 13:25 ` [PATCH v5 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 13/21] KVM: arm64: Convert write-protect operation to generic page-table API Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-15 16:16 ` Alexandru Elisei 2020-09-15 16:16 ` Alexandru Elisei 2020-09-11 13:25 ` [PATCH v5 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 19/21] KVM: arm64: Remove unused page-table code Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu' Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 13:25 ` [PATCH v5 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration Will Deacon 2020-09-11 13:25 ` Will Deacon 2020-09-11 15:04 ` [PATCH v5 00/21] KVM: arm64: Rewrite page-table code and fault handling Marc Zyngier 2020-09-11 15:04 ` Marc Zyngier 2020-10-01 10:21 ` Alexandru Elisei 2020-10-01 10:21 ` Alexandru Elisei 2020-10-01 12:28 ` Will Deacon 2020-10-01 12:28 ` Will Deacon
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200911132529.19844-11-will@kernel.org \ --to=will@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=kernel-team@android.com \ --cc=kvmarm@lists.cs.columbia.edu \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=maz@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.