* [PATCH v4 01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches()
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon
` (21 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
kvm_mmu_free_memory_caches() is only called by kvm_arch_vcpu_destroy(),
so inline the implementation and get rid of the extra function.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_mmu.h | 2 --
arch/arm64/kvm/arm.c | 2 +-
arch/arm64/kvm/mmu.c | 5 -----
3 files changed, 1 insertion(+), 8 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 189839c3706a..0f078b1920ff 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -141,8 +141,6 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
int kvm_handle_guest_abort(struct kvm_vcpu *vcpu);
-void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
-
phys_addr_t kvm_mmu_get_httbr(void);
phys_addr_t kvm_get_idmap_vector(void);
int kvm_mmu_init(void);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 46dc3d75cf13..262a0afbcc27 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -283,7 +283,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
if (vcpu->arch.has_run_once && unlikely(!irqchip_in_kernel(vcpu->kvm)))
static_branch_dec(&userspace_irqchip_in_use);
- kvm_mmu_free_memory_caches(vcpu);
+ kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
kvm_timer_vcpu_terminate(vcpu);
kvm_pmu_vcpu_destroy(vcpu);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index ba00bcc0c884..935f8f689433 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2324,11 +2324,6 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva)
kvm_test_age_hva_handler, NULL);
}
-void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu)
-{
- kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
-}
-
phys_addr_t kvm_mmu_get_httbr(void)
{
if (__kvm_cpu_uses_extended_idmap())
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
2020-09-07 15:23 ` [PATCH v4 01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches() Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 0:03 ` Gavin Shan
2020-09-09 15:29 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables Will Deacon
` (20 subsequent siblings)
22 siblings, 2 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
The KVM page-table code is intricately tied into the kernel page-table
code and re-uses the pte/pmd/pud/p4d/pgd macros directly in an attempt
to reduce code duplication. Unfortunately, the reality is that there is
an awful lot of code required to make this work, and at the end of the
day you're limited to creating page-tables with the same configuration
as the host kernel. Furthermore, lifting the page-table code to run
directly at EL2 on a non-VHE system (as we plan to to do in future
patches) is practically impossible due to the number of dependencies it
has on the core kernel.
Introduce a framework for walking Armv8 page-tables configured
independently from the host kernel.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 101 ++++++++++
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/pgtable.c | 285 +++++++++++++++++++++++++++
3 files changed, 387 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/include/asm/kvm_pgtable.h
create mode 100644 arch/arm64/kvm/hyp/pgtable.c
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
new file mode 100644
index 000000000000..1c5d981e15c3
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2020 Google LLC
+ * Author: Will Deacon <will@kernel.org>
+ */
+
+#ifndef __ARM64_KVM_PGTABLE_H__
+#define __ARM64_KVM_PGTABLE_H__
+
+#include <linux/bits.h>
+#include <linux/kvm_host.h>
+#include <linux/types.h>
+
+typedef u64 kvm_pte_t;
+
+/**
+ * struct kvm_pgtable - KVM page-table.
+ * @ia_bits: Maximum input address size, in bits.
+ * @start_level: Level at which the page-table walk starts.
+ * @pgd: Pointer to the first top-level entry of the page-table.
+ * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables.
+ */
+struct kvm_pgtable {
+ u32 ia_bits;
+ u32 start_level;
+ kvm_pte_t *pgd;
+
+ /* Stage-2 only */
+ struct kvm_s2_mmu *mmu;
+};
+
+/**
+ * enum kvm_pgtable_prot - Page-table permissions and attributes.
+ * @KVM_PGTABLE_PROT_X: Execute permission.
+ * @KVM_PGTABLE_PROT_W: Write permission.
+ * @KVM_PGTABLE_PROT_R: Read permission.
+ * @KVM_PGTABLE_PROT_DEVICE: Device attributes.
+ */
+enum kvm_pgtable_prot {
+ KVM_PGTABLE_PROT_X = BIT(0),
+ KVM_PGTABLE_PROT_W = BIT(1),
+ KVM_PGTABLE_PROT_R = BIT(2),
+
+ KVM_PGTABLE_PROT_DEVICE = BIT(3),
+};
+
+/**
+ * enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
+ * @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
+ * entries.
+ * @KVM_PGTABLE_WALK_TABLE_PRE: Visit table entries before their
+ * children.
+ * @KVM_PGTABLE_WALK_TABLE_POST: Visit table entries after their
+ * children.
+ */
+enum kvm_pgtable_walk_flags {
+ KVM_PGTABLE_WALK_LEAF = BIT(0),
+ KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
+ KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
+};
+
+typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level,
+ kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag,
+ void * const arg);
+
+/**
+ * struct kvm_pgtable_walker - Hook into a page-table walk.
+ * @cb: Callback function to invoke during the walk.
+ * @arg: Argument passed to the callback function.
+ * @flags: Bitwise-OR of flags to identify the entry types on which to
+ * invoke the callback function.
+ */
+struct kvm_pgtable_walker {
+ const kvm_pgtable_visitor_fn_t cb;
+ void * const arg;
+ const enum kvm_pgtable_walk_flags flags;
+};
+
+/**
+ * kvm_pgtable_walk() - Walk a page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_*_init().
+ * @addr: Input address for the start of the walk.
+ * @size: Size of the range to walk.
+ * @walker: Walker callback description.
+ *
+ * The walker will walk the page-table entries corresponding to the input
+ * address range specified, visiting entries according to the walker flags.
+ * Invalid entries are treated as leaf entries. Leaf entries are reloaded
+ * after invoking the walker callback, allowing the walker to descend into
+ * a newly installed table.
+ *
+ * Returning a negative error code from the walker callback function will
+ * terminate the walk immediately with the same error code.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
+ struct kvm_pgtable_walker *walker);
+
+#endif /* __ARM64_KVM_PGTABLE_H__ */
diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
index f54f0e89a71c..607b8a898826 100644
--- a/arch/arm64/kvm/hyp/Makefile
+++ b/arch/arm64/kvm/hyp/Makefile
@@ -10,5 +10,5 @@ subdir-ccflags-y := -I$(incdir) \
-DDISABLE_BRANCH_PROFILING \
$(DISABLE_STACKLEAK_PLUGIN)
-obj-$(CONFIG_KVM) += vhe/ nvhe/
+obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o
obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
new file mode 100644
index 000000000000..3fb9d1949a3f
--- /dev/null
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -0,0 +1,285 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Stand-alone page-table allocator for hyp stage-1 and guest stage-2.
+ * No bombay mix was harmed in the writing of this file.
+ *
+ * Copyright (C) 2020 Google LLC
+ * Author: Will Deacon <will@kernel.org>
+ */
+
+#include <linux/bitfield.h>
+#include <asm/kvm_pgtable.h>
+
+#define KVM_PGTABLE_MAX_LEVELS 4U
+
+#define KVM_PTE_VALID BIT(0)
+
+#define KVM_PTE_TYPE BIT(1)
+#define KVM_PTE_TYPE_BLOCK 0
+#define KVM_PTE_TYPE_PAGE 1
+#define KVM_PTE_TYPE_TABLE 1
+
+#define KVM_PTE_ADDR_MASK GENMASK(47, PAGE_SHIFT)
+#define KVM_PTE_ADDR_51_48 GENMASK(15, 12)
+
+#define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2)
+
+#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
+
+struct kvm_pgtable_walk_data {
+ struct kvm_pgtable *pgt;
+ struct kvm_pgtable_walker *walker;
+
+ u64 addr;
+ u64 end;
+};
+
+static u64 kvm_granule_shift(u32 level)
+{
+ /* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */
+ return ARM64_HW_PGTABLE_LEVEL_SHIFT(level);
+}
+
+static u64 kvm_granule_size(u32 level)
+{
+ return BIT(kvm_granule_shift(level));
+}
+
+static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level)
+{
+ u64 granule = kvm_granule_size(level);
+
+ /*
+ * Reject invalid block mappings and don't bother with 4TB mappings for
+ * 52-bit PAs.
+ */
+ if (level == 0 || (PAGE_SIZE != SZ_4K && level == 1))
+ return false;
+
+ if (granule > (end - addr))
+ return false;
+
+ return IS_ALIGNED(addr, granule) && IS_ALIGNED(phys, granule);
+}
+
+static u32 kvm_pgtable_idx(struct kvm_pgtable_walk_data *data, u32 level)
+{
+ u64 shift = kvm_granule_shift(level);
+ u64 mask = BIT(PAGE_SHIFT - 3) - 1;
+
+ return (data->addr >> shift) & mask;
+}
+
+static u32 __kvm_pgd_page_idx(struct kvm_pgtable *pgt, u64 addr)
+{
+ u64 shift = kvm_granule_shift(pgt->start_level - 1); /* May underflow */
+ u64 mask = BIT(pgt->ia_bits) - 1;
+
+ return (addr & mask) >> shift;
+}
+
+static u32 kvm_pgd_page_idx(struct kvm_pgtable_walk_data *data)
+{
+ return __kvm_pgd_page_idx(data->pgt, data->addr);
+}
+
+static u32 kvm_pgd_pages(u32 ia_bits, u32 start_level)
+{
+ struct kvm_pgtable pgt = {
+ .ia_bits = ia_bits,
+ .start_level = start_level,
+ };
+
+ return __kvm_pgd_page_idx(&pgt, -1ULL) + 1;
+}
+
+static bool kvm_pte_valid(kvm_pte_t pte)
+{
+ return pte & KVM_PTE_VALID;
+}
+
+static bool kvm_pte_table(kvm_pte_t pte, u32 level)
+{
+ if (level == KVM_PGTABLE_MAX_LEVELS - 1)
+ return false;
+
+ if (!kvm_pte_valid(pte))
+ return false;
+
+ return FIELD_GET(KVM_PTE_TYPE, pte) == KVM_PTE_TYPE_TABLE;
+}
+
+static u64 kvm_pte_to_phys(kvm_pte_t pte)
+{
+ u64 pa = pte & KVM_PTE_ADDR_MASK;
+
+ if (PAGE_SHIFT == 16)
+ pa |= FIELD_GET(KVM_PTE_ADDR_51_48, pte) << 48;
+
+ return pa;
+}
+
+static kvm_pte_t kvm_phys_to_pte(u64 pa)
+{
+ kvm_pte_t pte = pa & KVM_PTE_ADDR_MASK;
+
+ if (PAGE_SHIFT == 16)
+ pte |= FIELD_PREP(KVM_PTE_ADDR_51_48, pa >> 48);
+
+ return pte;
+}
+
+static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte)
+{
+ return __va(kvm_pte_to_phys(pte));
+}
+
+static void kvm_set_invalid_pte(kvm_pte_t *ptep)
+{
+ kvm_pte_t pte = *ptep;
+ WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID);
+}
+
+static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp)
+{
+ kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(__pa(childp));
+
+ pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE);
+ pte |= KVM_PTE_VALID;
+
+ WARN_ON(kvm_pte_valid(old));
+ smp_store_release(ptep, pte);
+}
+
+static bool kvm_set_valid_leaf_pte(kvm_pte_t *ptep, u64 pa, kvm_pte_t attr,
+ u32 level)
+{
+ kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(pa);
+ u64 type = (level == KVM_PGTABLE_MAX_LEVELS - 1) ? KVM_PTE_TYPE_PAGE :
+ KVM_PTE_TYPE_BLOCK;
+
+ pte |= attr & (KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI);
+ pte |= FIELD_PREP(KVM_PTE_TYPE, type);
+ pte |= KVM_PTE_VALID;
+
+ /* Tolerate KVM recreating the exact same mapping. */
+ if (kvm_pte_valid(old))
+ return old == pte;
+
+ smp_store_release(ptep, pte);
+ return true;
+}
+
+static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr,
+ u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag)
+{
+ struct kvm_pgtable_walker *walker = data->walker;
+ return walker->cb(addr, data->end, level, ptep, flag, walker->arg);
+}
+
+static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
+ kvm_pte_t *pgtable, u32 level);
+
+static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data,
+ kvm_pte_t *ptep, u32 level)
+{
+ int ret = 0;
+ u64 addr = data->addr;
+ kvm_pte_t *childp, pte = *ptep;
+ bool table = kvm_pte_table(pte, level);
+ enum kvm_pgtable_walk_flags flags = data->walker->flags;
+
+ if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) {
+ ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
+ KVM_PGTABLE_WALK_TABLE_PRE);
+ }
+
+ if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) {
+ ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
+ KVM_PGTABLE_WALK_LEAF);
+ pte = *ptep;
+ table = kvm_pte_table(pte, level);
+ }
+
+ if (ret)
+ goto out;
+
+ if (!table) {
+ data->addr += kvm_granule_size(level);
+ goto out;
+ }
+
+ childp = kvm_pte_follow(pte);
+ ret = __kvm_pgtable_walk(data, childp, level + 1);
+ if (ret)
+ goto out;
+
+ if (flags & KVM_PGTABLE_WALK_TABLE_POST) {
+ ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
+ KVM_PGTABLE_WALK_TABLE_POST);
+ }
+
+out:
+ return ret;
+}
+
+static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
+ kvm_pte_t *pgtable, u32 level)
+{
+ u32 idx;
+ int ret = 0;
+
+ if (WARN_ON_ONCE(level >= KVM_PGTABLE_MAX_LEVELS))
+ return -EINVAL;
+
+ for (idx = kvm_pgtable_idx(data, level); idx < PTRS_PER_PTE; ++idx) {
+ kvm_pte_t *ptep = &pgtable[idx];
+
+ if (data->addr >= data->end)
+ break;
+
+ ret = __kvm_pgtable_visit(data, ptep, level);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data)
+{
+ u32 idx;
+ int ret = 0;
+ struct kvm_pgtable *pgt = data->pgt;
+ u64 limit = BIT(pgt->ia_bits);
+
+ if (data->addr > limit || data->end > limit)
+ return -ERANGE;
+
+ if (!pgt->pgd)
+ return -EINVAL;
+
+ for (idx = kvm_pgd_page_idx(data); data->addr < data->end; ++idx) {
+ kvm_pte_t *ptep = &pgt->pgd[idx * PTRS_PER_PTE];
+
+ ret = __kvm_pgtable_walk(data, ptep, pgt->start_level);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
+ struct kvm_pgtable_walker *walker)
+{
+ struct kvm_pgtable_walk_data walk_data = {
+ .pgt = pgt,
+ .addr = ALIGN_DOWN(addr, PAGE_SIZE),
+ .end = PAGE_ALIGN(walk_data.addr + size),
+ .walker = walker,
+ };
+
+ return _kvm_pgtable_walk(&walk_data);
+}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-07 15:23 ` [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon
@ 2020-09-08 0:03 ` Gavin Shan
2020-09-10 10:57 ` Will Deacon
2020-09-09 15:29 ` Alexandru Elisei
1 sibling, 1 reply; 52+ messages in thread
From: Gavin Shan @ 2020-09-08 0:03 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, linux-arm-kernel
Hi Will,
On 9/8/20 1:23 AM, Will Deacon wrote:
> The KVM page-table code is intricately tied into the kernel page-table
> code and re-uses the pte/pmd/pud/p4d/pgd macros directly in an attempt
> to reduce code duplication. Unfortunately, the reality is that there is
> an awful lot of code required to make this work, and at the end of the
> day you're limited to creating page-tables with the same configuration
> as the host kernel. Furthermore, lifting the page-table code to run
> directly at EL2 on a non-VHE system (as we plan to to do in future
> patches) is practically impossible due to the number of dependencies it
> has on the core kernel.
>
> Introduce a framework for walking Armv8 page-tables configured
> independently from the host kernel.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
It looks good to me. However, If you get a chance to respin, I guess
it would be nice to add some comments about kvm_block_mapping_supported().
More details are provoided below.
Reviewed-by: Gavin Shan <gshan@redhat.com>
> arch/arm64/include/asm/kvm_pgtable.h | 101 ++++++++++
> arch/arm64/kvm/hyp/Makefile | 2 +-
> arch/arm64/kvm/hyp/pgtable.c | 285 +++++++++++++++++++++++++++
> 3 files changed, 387 insertions(+), 1 deletion(-)
> create mode 100644 arch/arm64/include/asm/kvm_pgtable.h
> create mode 100644 arch/arm64/kvm/hyp/pgtable.c
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> new file mode 100644
> index 000000000000..1c5d981e15c3
> --- /dev/null
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -0,0 +1,101 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2020 Google LLC
> + * Author: Will Deacon <will@kernel.org>
> + */
> +
> +#ifndef __ARM64_KVM_PGTABLE_H__
> +#define __ARM64_KVM_PGTABLE_H__
> +
> +#include <linux/bits.h>
> +#include <linux/kvm_host.h>
> +#include <linux/types.h>
> +
> +typedef u64 kvm_pte_t;
> +
> +/**
> + * struct kvm_pgtable - KVM page-table.
> + * @ia_bits: Maximum input address size, in bits.
> + * @start_level: Level at which the page-table walk starts.
> + * @pgd: Pointer to the first top-level entry of the page-table.
> + * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables.
> + */
> +struct kvm_pgtable {
> + u32 ia_bits;
> + u32 start_level;
> + kvm_pte_t *pgd;
> +
> + /* Stage-2 only */
> + struct kvm_s2_mmu *mmu;
> +};
> +
> +/**
> + * enum kvm_pgtable_prot - Page-table permissions and attributes.
> + * @KVM_PGTABLE_PROT_X: Execute permission.
> + * @KVM_PGTABLE_PROT_W: Write permission.
> + * @KVM_PGTABLE_PROT_R: Read permission.
> + * @KVM_PGTABLE_PROT_DEVICE: Device attributes.
> + */
> +enum kvm_pgtable_prot {
> + KVM_PGTABLE_PROT_X = BIT(0),
> + KVM_PGTABLE_PROT_W = BIT(1),
> + KVM_PGTABLE_PROT_R = BIT(2),
> +
> + KVM_PGTABLE_PROT_DEVICE = BIT(3),
> +};
> +
> +/**
> + * enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
> + * @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
> + * entries.
> + * @KVM_PGTABLE_WALK_TABLE_PRE: Visit table entries before their
> + * children.
> + * @KVM_PGTABLE_WALK_TABLE_POST: Visit table entries after their
> + * children.
> + */
> +enum kvm_pgtable_walk_flags {
> + KVM_PGTABLE_WALK_LEAF = BIT(0),
> + KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
> + KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
> +};
> +
> +typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level,
> + kvm_pte_t *ptep,
> + enum kvm_pgtable_walk_flags flag,
> + void * const arg);
> +
> +/**
> + * struct kvm_pgtable_walker - Hook into a page-table walk.
> + * @cb: Callback function to invoke during the walk.
> + * @arg: Argument passed to the callback function.
> + * @flags: Bitwise-OR of flags to identify the entry types on which to
> + * invoke the callback function.
> + */
> +struct kvm_pgtable_walker {
> + const kvm_pgtable_visitor_fn_t cb;
> + void * const arg;
> + const enum kvm_pgtable_walk_flags flags;
> +};
> +
> +/**
> + * kvm_pgtable_walk() - Walk a page-table.
> + * @pgt: Page-table structure initialised by kvm_pgtable_*_init().
> + * @addr: Input address for the start of the walk.
> + * @size: Size of the range to walk.
> + * @walker: Walker callback description.
> + *
> + * The walker will walk the page-table entries corresponding to the input
> + * address range specified, visiting entries according to the walker flags.
> + * Invalid entries are treated as leaf entries. Leaf entries are reloaded
> + * after invoking the walker callback, allowing the walker to descend into
> + * a newly installed table.
> + *
> + * Returning a negative error code from the walker callback function will
> + * terminate the walk immediately with the same error code.
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + struct kvm_pgtable_walker *walker);
> +
> +#endif /* __ARM64_KVM_PGTABLE_H__ */
> diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
> index f54f0e89a71c..607b8a898826 100644
> --- a/arch/arm64/kvm/hyp/Makefile
> +++ b/arch/arm64/kvm/hyp/Makefile
> @@ -10,5 +10,5 @@ subdir-ccflags-y := -I$(incdir) \
> -DDISABLE_BRANCH_PROFILING \
> $(DISABLE_STACKLEAK_PLUGIN)
>
> -obj-$(CONFIG_KVM) += vhe/ nvhe/
> +obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o
> obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> new file mode 100644
> index 000000000000..3fb9d1949a3f
> --- /dev/null
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -0,0 +1,285 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Stand-alone page-table allocator for hyp stage-1 and guest stage-2.
> + * No bombay mix was harmed in the writing of this file.
> + *
> + * Copyright (C) 2020 Google LLC
> + * Author: Will Deacon <will@kernel.org>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <asm/kvm_pgtable.h>
> +
> +#define KVM_PGTABLE_MAX_LEVELS 4U
> +
> +#define KVM_PTE_VALID BIT(0)
> +
> +#define KVM_PTE_TYPE BIT(1)
> +#define KVM_PTE_TYPE_BLOCK 0
> +#define KVM_PTE_TYPE_PAGE 1
> +#define KVM_PTE_TYPE_TABLE 1
> +
> +#define KVM_PTE_ADDR_MASK GENMASK(47, PAGE_SHIFT)
> +#define KVM_PTE_ADDR_51_48 GENMASK(15, 12)
> +
> +#define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2)
> +
> +#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
> +
> +struct kvm_pgtable_walk_data {
> + struct kvm_pgtable *pgt;
> + struct kvm_pgtable_walker *walker;
> +
> + u64 addr;
> + u64 end;
> +};
> +
> +static u64 kvm_granule_shift(u32 level)
> +{
> + /* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */
> + return ARM64_HW_PGTABLE_LEVEL_SHIFT(level);
> +}
> +
> +static u64 kvm_granule_size(u32 level)
> +{
> + return BIT(kvm_granule_shift(level));
> +}
> +
> +static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level)
> +{
> + u64 granule = kvm_granule_size(level);
> +
> + /*
> + * Reject invalid block mappings and don't bother with 4TB mappings for
> + * 52-bit PAs.
> + */
> + if (level == 0 || (PAGE_SIZE != SZ_4K && level == 1))
> + return false;
> +
> + if (granule > (end - addr))
> + return false;
> +
> + return IS_ALIGNED(addr, granule) && IS_ALIGNED(phys, granule);
> +}
> +
If you get a chance to respin, it would be nice to have more comments to
explain why 4TB block mapping is rejected here. I guess it depends on the
fact: hugeTLB/THP doesn't support such kind of huge block mapping yet?
> +static u32 kvm_pgtable_idx(struct kvm_pgtable_walk_data *data, u32 level)
> +{
> + u64 shift = kvm_granule_shift(level);
> + u64 mask = BIT(PAGE_SHIFT - 3) - 1;
> +
> + return (data->addr >> shift) & mask;
> +}
> +
> +static u32 __kvm_pgd_page_idx(struct kvm_pgtable *pgt, u64 addr)
> +{
> + u64 shift = kvm_granule_shift(pgt->start_level - 1); /* May underflow */
> + u64 mask = BIT(pgt->ia_bits) - 1;
> +
> + return (addr & mask) >> shift;
> +}
> +
> +static u32 kvm_pgd_page_idx(struct kvm_pgtable_walk_data *data)
> +{
> + return __kvm_pgd_page_idx(data->pgt, data->addr);
> +}
> +
> +static u32 kvm_pgd_pages(u32 ia_bits, u32 start_level)
> +{
> + struct kvm_pgtable pgt = {
> + .ia_bits = ia_bits,
> + .start_level = start_level,
> + };
> +
> + return __kvm_pgd_page_idx(&pgt, -1ULL) + 1;
> +}
> +
> +static bool kvm_pte_valid(kvm_pte_t pte)
> +{
> + return pte & KVM_PTE_VALID;
> +}
> +
> +static bool kvm_pte_table(kvm_pte_t pte, u32 level)
> +{
> + if (level == KVM_PGTABLE_MAX_LEVELS - 1)
> + return false;
> +
> + if (!kvm_pte_valid(pte))
> + return false;
> +
> + return FIELD_GET(KVM_PTE_TYPE, pte) == KVM_PTE_TYPE_TABLE;
> +}
> +
> +static u64 kvm_pte_to_phys(kvm_pte_t pte)
> +{
> + u64 pa = pte & KVM_PTE_ADDR_MASK;
> +
> + if (PAGE_SHIFT == 16)
> + pa |= FIELD_GET(KVM_PTE_ADDR_51_48, pte) << 48;
> +
> + return pa;
> +}
> +
> +static kvm_pte_t kvm_phys_to_pte(u64 pa)
> +{
> + kvm_pte_t pte = pa & KVM_PTE_ADDR_MASK;
> +
> + if (PAGE_SHIFT == 16)
> + pte |= FIELD_PREP(KVM_PTE_ADDR_51_48, pa >> 48);
> +
> + return pte;
> +}
> +
> +static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte)
> +{
> + return __va(kvm_pte_to_phys(pte));
> +}
> +
> +static void kvm_set_invalid_pte(kvm_pte_t *ptep)
> +{
> + kvm_pte_t pte = *ptep;
> + WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID);
> +}
> +
> +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp)
> +{
> + kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(__pa(childp));
> +
> + pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE);
> + pte |= KVM_PTE_VALID;
> +
> + WARN_ON(kvm_pte_valid(old));
> + smp_store_release(ptep, pte);
> +}
> +
> +static bool kvm_set_valid_leaf_pte(kvm_pte_t *ptep, u64 pa, kvm_pte_t attr,
> + u32 level)
> +{
> + kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(pa);
> + u64 type = (level == KVM_PGTABLE_MAX_LEVELS - 1) ? KVM_PTE_TYPE_PAGE :
> + KVM_PTE_TYPE_BLOCK;
> +
> + pte |= attr & (KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI);
> + pte |= FIELD_PREP(KVM_PTE_TYPE, type);
> + pte |= KVM_PTE_VALID;
> +
> + /* Tolerate KVM recreating the exact same mapping. */
> + if (kvm_pte_valid(old))
> + return old == pte;
> +
> + smp_store_release(ptep, pte);
> + return true;
> +}
> +
> +static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr,
> + u32 level, kvm_pte_t *ptep,
> + enum kvm_pgtable_walk_flags flag)
> +{
> + struct kvm_pgtable_walker *walker = data->walker;
> + return walker->cb(addr, data->end, level, ptep, flag, walker->arg);
> +}
> +
> +static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
> + kvm_pte_t *pgtable, u32 level);
> +
> +static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data,
> + kvm_pte_t *ptep, u32 level)
> +{
> + int ret = 0;
> + u64 addr = data->addr;
> + kvm_pte_t *childp, pte = *ptep;
> + bool table = kvm_pte_table(pte, level);
> + enum kvm_pgtable_walk_flags flags = data->walker->flags;
> +
> + if (table && (flags & KVM_PGTABLE_WALK_TABLE_PRE)) {
> + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
> + KVM_PGTABLE_WALK_TABLE_PRE);
> + }
> +
> + if (!table && (flags & KVM_PGTABLE_WALK_LEAF)) {
> + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
> + KVM_PGTABLE_WALK_LEAF);
> + pte = *ptep;
> + table = kvm_pte_table(pte, level);
> + }
> +
> + if (ret)
> + goto out;
> +
> + if (!table) {
> + data->addr += kvm_granule_size(level);
> + goto out;
> + }
> +
> + childp = kvm_pte_follow(pte);
> + ret = __kvm_pgtable_walk(data, childp, level + 1);
> + if (ret)
> + goto out;
> +
> + if (flags & KVM_PGTABLE_WALK_TABLE_POST) {
> + ret = kvm_pgtable_visitor_cb(data, addr, level, ptep,
> + KVM_PGTABLE_WALK_TABLE_POST);
> + }
> +
> +out:
> + return ret;
> +}
> +
> +static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
> + kvm_pte_t *pgtable, u32 level)
> +{
> + u32 idx;
> + int ret = 0;
> +
> + if (WARN_ON_ONCE(level >= KVM_PGTABLE_MAX_LEVELS))
> + return -EINVAL;
> +
> + for (idx = kvm_pgtable_idx(data, level); idx < PTRS_PER_PTE; ++idx) {
> + kvm_pte_t *ptep = &pgtable[idx];
> +
> + if (data->addr >= data->end)
> + break;
> +
> + ret = __kvm_pgtable_visit(data, ptep, level);
> + if (ret)
> + break;
> + }
> +
> + return ret;
> +}
> +
> +static int _kvm_pgtable_walk(struct kvm_pgtable_walk_data *data)
> +{
> + u32 idx;
> + int ret = 0;
> + struct kvm_pgtable *pgt = data->pgt;
> + u64 limit = BIT(pgt->ia_bits);
> +
> + if (data->addr > limit || data->end > limit)
> + return -ERANGE;
> +
> + if (!pgt->pgd)
> + return -EINVAL;
> +
> + for (idx = kvm_pgd_page_idx(data); data->addr < data->end; ++idx) {
> + kvm_pte_t *ptep = &pgt->pgd[idx * PTRS_PER_PTE];
> +
> + ret = __kvm_pgtable_walk(data, ptep, pgt->start_level);
> + if (ret)
> + break;
> + }
> +
> + return ret;
> +}
> +
> +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + struct kvm_pgtable_walker *walker)
> +{
> + struct kvm_pgtable_walk_data walk_data = {
> + .pgt = pgt,
> + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> + .end = PAGE_ALIGN(walk_data.addr + size),
> + .walker = walker,
> + };
> +
> + return _kvm_pgtable_walk(&walk_data);
> +}
>
Thanks,
Gavin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-08 0:03 ` Gavin Shan
@ 2020-09-10 10:57 ` Will Deacon
0 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-10 10:57 UTC (permalink / raw)
To: Gavin Shan
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, kvmarm,
linux-arm-kernel
On Tue, Sep 08, 2020 at 10:03:30AM +1000, Gavin Shan wrote:
> On 9/8/20 1:23 AM, Will Deacon wrote:
> > The KVM page-table code is intricately tied into the kernel page-table
> > code and re-uses the pte/pmd/pud/p4d/pgd macros directly in an attempt
> > to reduce code duplication. Unfortunately, the reality is that there is
> > an awful lot of code required to make this work, and at the end of the
> > day you're limited to creating page-tables with the same configuration
> > as the host kernel. Furthermore, lifting the page-table code to run
> > directly at EL2 on a non-VHE system (as we plan to to do in future
> > patches) is practically impossible due to the number of dependencies it
> > has on the core kernel.
> >
> > Introduce a framework for walking Armv8 page-tables configured
> > independently from the host kernel.
> >
> > Cc: Marc Zyngier <maz@kernel.org>
> > Cc: Quentin Perret <qperret@google.com>
> > Signed-off-by: Will Deacon <will@kernel.org>
> > ---
>
> It looks good to me. However, If you get a chance to respin, I guess
> it would be nice to add some comments about kvm_block_mapping_supported().
> More details are provoided below.
>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
[...]
> > +static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level)
> > +{
> > + u64 granule = kvm_granule_size(level);
> > +
> > + /*
> > + * Reject invalid block mappings and don't bother with 4TB mappings for
> > + * 52-bit PAs.
> > + */
> > + if (level == 0 || (PAGE_SIZE != SZ_4K && level == 1))
> > + return false;
> > +
> > + if (granule > (end - addr))
> > + return false;
> > +
> > + return IS_ALIGNED(addr, granule) && IS_ALIGNED(phys, granule);
> > +}
> > +
>
> If you get a chance to respin, it would be nice to have more comments to
> explain why 4TB block mapping is rejected here. I guess it depends on the
> fact: hugeTLB/THP doesn't support such kind of huge block mapping yet?
It's just because I'm lazy, really :) We don't know if we're using 52-bit
PAs here, and I couldn't be bothered to propagate that information given
that this doesn't seem like something anybody will be using at the moment.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-07 15:23 ` [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon
2020-09-08 0:03 ` Gavin Shan
@ 2020-09-09 15:29 ` Alexandru Elisei
2020-09-10 12:37 ` Will Deacon
1 sibling, 1 reply; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-09 15:29 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
On 9/7/20 4:23 PM, Will Deacon wrote:
> [..]
> +
> +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + struct kvm_pgtable_walker *walker)
> +{
> + struct kvm_pgtable_walk_data walk_data = {
> + .pgt = pgt,
> + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> + .end = PAGE_ALIGN(walk_data.addr + size),
> + .walker = walker,
> + };
If the caller wants to walk [0x500, 0x1500), for PAGE_SIZE = 0x1000 (4K), the
function walks the range [0x0, 0x1000). Is that intentional?
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-09 15:29 ` Alexandru Elisei
@ 2020-09-10 12:37 ` Will Deacon
2020-09-10 14:21 ` Andrew Scull
0 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-10 12:37 UTC (permalink / raw)
To: Alexandru Elisei
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, kvmarm,
linux-arm-kernel
On Wed, Sep 09, 2020 at 04:29:26PM +0100, Alexandru Elisei wrote:
> On 9/7/20 4:23 PM, Will Deacon wrote:
> > [..]
> > +
> > +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> > + struct kvm_pgtable_walker *walker)
> > +{
> > + struct kvm_pgtable_walk_data walk_data = {
> > + .pgt = pgt,
> > + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> > + .end = PAGE_ALIGN(walk_data.addr + size),
> > + .walker = walker,
> > + };
>
> If the caller wants to walk [0x500, 0x1500), for PAGE_SIZE = 0x1000 (4K), the
> function walks the range [0x0, 0x1000). Is that intentional?
Yes, although the caller specifies a base and a *size*, rather than an end
address. As a concrete example, much of the hypervisor stage-1 mapping
is created using PAGE_SIZE mappings of random ELF symbols, which correspond
to arbitrary addresses. In these cases, we really do want to round-down the
address and perform a PAGE_SIZE mapping.
The alternative would be to return an error if the size is not page-aligned,
but I don't think that's really necessary, especially since we accept an
unaligned base.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-10 12:37 ` Will Deacon
@ 2020-09-10 14:21 ` Andrew Scull
2020-09-11 10:15 ` Will Deacon
0 siblings, 1 reply; 52+ messages in thread
From: Andrew Scull @ 2020-09-10 14:21 UTC (permalink / raw)
To: Will Deacon
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, James Morse, Catalin Marinas, Alexandru Elisei,
kvmarm, linux-arm-kernel
On Thu, Sep 10, 2020 at 01:37:13PM +0100, Will Deacon wrote:
> On Wed, Sep 09, 2020 at 04:29:26PM +0100, Alexandru Elisei wrote:
> > On 9/7/20 4:23 PM, Will Deacon wrote:
> > > [..]
> > > +
> > > +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> > > + struct kvm_pgtable_walker *walker)
> > > +{
> > > + struct kvm_pgtable_walk_data walk_data = {
> > > + .pgt = pgt,
> > > + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> > > + .end = PAGE_ALIGN(walk_data.addr + size),
> > > + .walker = walker,
> > > + };
> >
> > If the caller wants to walk [0x500, 0x1500), for PAGE_SIZE = 0x1000 (4K), the
> > function walks the range [0x0, 0x1000). Is that intentional?
>
> Yes, although the caller specifies a base and a *size*, rather than an end
> address. As a concrete example, much of the hypervisor stage-1 mapping
> is created using PAGE_SIZE mappings of random ELF symbols, which correspond
> to arbitrary addresses. In these cases, we really do want to round-down the
> address and perform a PAGE_SIZE mapping.
I think Alexandru has a point here. Turning his example into something
equivalent that maps a random ELF symbol:
struct some_hyp_state s = { ... };
// &s == 0x500
// sizeof(s) == PAGE_SIZE
kvm_pgtable_walk(&s, sizeof(s), walker);
Given `s` straddles the two pages, the part in the second page won't be
mapped.
Should the end address instead be calculated as:
.end = PAGE_ALIGN(addr + size),
>
> The alternative would be to return an error if the size is not page-aligned,
> but I don't think that's really necessary, especially since we accept an
> unaligned base.
>
> Will
>
> --
> To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com.
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-10 14:21 ` Andrew Scull
@ 2020-09-11 10:15 ` Will Deacon
2020-09-11 11:22 ` Andrew Scull
0 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-11 10:15 UTC (permalink / raw)
To: Andrew Scull
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, James Morse, Catalin Marinas, Alexandru Elisei,
kvmarm, linux-arm-kernel
On Thu, Sep 10, 2020 at 03:21:59PM +0100, Andrew Scull wrote:
> On Thu, Sep 10, 2020 at 01:37:13PM +0100, Will Deacon wrote:
> > On Wed, Sep 09, 2020 at 04:29:26PM +0100, Alexandru Elisei wrote:
> > > On 9/7/20 4:23 PM, Will Deacon wrote:
> > > > [..]
> > > > +
> > > > +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> > > > + struct kvm_pgtable_walker *walker)
> > > > +{
> > > > + struct kvm_pgtable_walk_data walk_data = {
> > > > + .pgt = pgt,
> > > > + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> > > > + .end = PAGE_ALIGN(walk_data.addr + size),
> > > > + .walker = walker,
> > > > + };
> > >
> > > If the caller wants to walk [0x500, 0x1500), for PAGE_SIZE = 0x1000 (4K), the
> > > function walks the range [0x0, 0x1000). Is that intentional?
> >
> > Yes, although the caller specifies a base and a *size*, rather than an end
> > address. As a concrete example, much of the hypervisor stage-1 mapping
> > is created using PAGE_SIZE mappings of random ELF symbols, which correspond
> > to arbitrary addresses. In these cases, we really do want to round-down the
> > address and perform a PAGE_SIZE mapping.
>
> I think Alexandru has a point here. Turning his example into something
> equivalent that maps a random ELF symbol:
>
> struct some_hyp_state s = { ... };
> // &s == 0x500
> // sizeof(s) == PAGE_SIZE
> kvm_pgtable_walk(&s, sizeof(s), walker);
>
> Given `s` straddles the two pages, the part in the second page won't be
> mapped.
>
> Should the end address instead be calculated as:
>
> .end = PAGE_ALIGN(addr + size),
Cheers for the example, and I see what you mean about structures that
straddle a page boundary. However, I think it's important here that the size
parameter accurately reflects the number of pages mapped: if the caller
passes PAGE_SIZE, we better not map more than a page, since the mmu cache
might not have enough pre-allocated pages for that.
So the API is just that the virtual address bits corresponding to the offset
within a page are ignored. Looking back at the code, that works out for the
hyp mappings (it's actually the _physical_ address that is unaligned there,
and we can just round that down), but I think I have a potential bug in the
ioremap code if you place the GIC (v2) somewhere funky on a machine using
64k pages. In this case, the ioctl() handler only enforces 4k alignment,
and so we could end up truncating the mapping in a similar case to what you
describe above. For example, trying to place it from 60k - 68k would result
in only the first page getting mapped.
I've fixed that in the ioremap code (diff below), and I'll update the
kerneldoc to say that the bottom bits of the VA are ignored.
Cheers,
Will
--->8
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 1041be1fafe4..21b70abf65a7 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -505,6 +505,9 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
KVM_PGTABLE_PROT_R |
(writable ? KVM_PGTABLE_PROT_W : 0);
+ size += offset_in_page(guest_ipa);
+ guest_ipa &= PAGE_MASK;
+
for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) {
ret = kvm_mmu_topup_memory_cache(&cache,
kvm_mmu_cache_min_pages(kvm));
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
2020-09-11 10:15 ` Will Deacon
@ 2020-09-11 11:22 ` Andrew Scull
0 siblings, 0 replies; 52+ messages in thread
From: Andrew Scull @ 2020-09-11 11:22 UTC (permalink / raw)
To: Will Deacon
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, James Morse, Catalin Marinas, Alexandru Elisei,
kvmarm, linux-arm-kernel
On Fri, Sep 11, 2020 at 11:15:04AM +0100, Will Deacon wrote:
> On Thu, Sep 10, 2020 at 03:21:59PM +0100, Andrew Scull wrote:
> > On Thu, Sep 10, 2020 at 01:37:13PM +0100, Will Deacon wrote:
> > > On Wed, Sep 09, 2020 at 04:29:26PM +0100, Alexandru Elisei wrote:
> > > > On 9/7/20 4:23 PM, Will Deacon wrote:
> > > > > [..]
> > > > > +
> > > > > +int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
> > > > > + struct kvm_pgtable_walker *walker)
> > > > > +{
> > > > > + struct kvm_pgtable_walk_data walk_data = {
> > > > > + .pgt = pgt,
> > > > > + .addr = ALIGN_DOWN(addr, PAGE_SIZE),
> > > > > + .end = PAGE_ALIGN(walk_data.addr + size),
> > > > > + .walker = walker,
> > > > > + };
> > > >
> > > > If the caller wants to walk [0x500, 0x1500), for PAGE_SIZE = 0x1000 (4K), the
> > > > function walks the range [0x0, 0x1000). Is that intentional?
> > >
> > > Yes, although the caller specifies a base and a *size*, rather than an end
> > > address. As a concrete example, much of the hypervisor stage-1 mapping
> > > is created using PAGE_SIZE mappings of random ELF symbols, which correspond
> > > to arbitrary addresses. In these cases, we really do want to round-down the
> > > address and perform a PAGE_SIZE mapping.
> >
> > I think Alexandru has a point here. Turning his example into something
> > equivalent that maps a random ELF symbol:
> >
> > struct some_hyp_state s = { ... };
> > // &s == 0x500
> > // sizeof(s) == PAGE_SIZE
> > kvm_pgtable_walk(&s, sizeof(s), walker);
> >
> > Given `s` straddles the two pages, the part in the second page won't be
> > mapped.
> >
> > Should the end address instead be calculated as:
> >
> > .end = PAGE_ALIGN(addr + size),
>
> Cheers for the example, and I see what you mean about structures that
> straddle a page boundary. However, I think it's important here that the size
> parameter accurately reflects the number of pages mapped: if the caller
> passes PAGE_SIZE, we better not map more than a page, since the mmu cache
> might not have enough pre-allocated pages for that.
>
> So the API is just that the virtual address bits corresponding to the offset
> within a page are ignored. Looking back at the code, that works out for the
> hyp mappings (it's actually the _physical_ address that is unaligned there,
> and we can just round that down), but I think I have a potential bug in the
> ioremap code if you place the GIC (v2) somewhere funky on a machine using
> 64k pages. In this case, the ioctl() handler only enforces 4k alignment,
> and so we could end up truncating the mapping in a similar case to what you
> describe above. For example, trying to place it from 60k - 68k would result
> in only the first page getting mapped.
>
> I've fixed that in the ioremap code (diff below), and I'll update the
> kerneldoc to say that the bottom bits of the VA are ignored.
You've described kvm_pgtable_walk as taking a start page and page count
so an alternative to the comment could be defining function to reflect
that e.g.
int kvm_pgtable_walk(struct kvm_pgtable *pgt, gfn_t start,
size_t count, kvm_pgtable_walker *walker);
Up to you.
>
> Cheers,
>
> Will
>
> --->8
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 1041be1fafe4..21b70abf65a7 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -505,6 +505,9 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
> KVM_PGTABLE_PROT_R |
> (writable ? KVM_PGTABLE_PROT_W : 0);
>
> + size += offset_in_page(guest_ipa);
> + guest_ipa &= PAGE_MASK;
> +
> for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) {
> ret = kvm_mmu_topup_memory_cache(&cache,
> kvm_mmu_cache_min_pages(kvm));
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
2020-09-07 15:23 ` [PATCH v4 01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches() Will Deacon
2020-09-07 15:23 ` [PATCH v4 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 1:09 ` Gavin Shan
2020-09-07 15:23 ` [PATCH v4 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables Will Deacon
` (19 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
The generic page-table walker is pretty useless as it stands, because it
doesn't understand enough to allocate anything. Teach it about stage-1
page-tables, and hook up an API for allocating these for the hypervisor
at EL2.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 36 ++++++++
arch/arm64/kvm/hyp/pgtable.c | 133 +++++++++++++++++++++++++++
2 files changed, 169 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 1c5d981e15c3..91e364804547 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -77,6 +77,42 @@ struct kvm_pgtable_walker {
const enum kvm_pgtable_walk_flags flags;
};
+/**
+ * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table.
+ * @pgt: Uninitialised page-table structure to initialise.
+ * @va_bits: Maximum virtual address bits.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits);
+
+/**
+ * kvm_pgtable_hyp_destroy() - Destroy an unused hypervisor stage-1 page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_hyp_init().
+ *
+ * The page-table is assumed to be unreachable by any hardware walkers prior
+ * to freeing and therefore no TLB invalidation is performed.
+ */
+void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt);
+
+/**
+ * kvm_pgtable_hyp_map() - Install a mapping in a hypervisor stage-1 page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_hyp_init().
+ * @addr: Virtual address at which to place the mapping.
+ * @size: Size of the mapping.
+ * @phys: Physical address of the memory to map.
+ * @prot: Permissions and attributes for the mapping.
+ *
+ * If device attributes are not explicitly requested in @prot, then the
+ * mapping will be normal, cacheable. Attempts to install a mapping for
+ * a virtual address that is already mapped will be rejected with an error
+ * and a WARN().
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
+ enum kvm_pgtable_prot prot);
+
/**
* kvm_pgtable_walk() - Walk a page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_*_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 3fb9d1949a3f..23a1006aa4ef 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -24,8 +24,18 @@
#define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2)
+#define KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX GENMASK(4, 2)
+#define KVM_PTE_LEAF_ATTR_LO_S1_AP GENMASK(7, 6)
+#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RO 3
+#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RW 1
+#define KVM_PTE_LEAF_ATTR_LO_S1_SH GENMASK(9, 8)
+#define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3
+#define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10)
+
#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
+#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)
+
struct kvm_pgtable_walk_data {
struct kvm_pgtable *pgt;
struct kvm_pgtable_walker *walker;
@@ -283,3 +293,126 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
return _kvm_pgtable_walk(&walk_data);
}
+
+struct hyp_map_data {
+ u64 phys;
+ kvm_pte_t attr;
+};
+
+static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot,
+ struct hyp_map_data *data)
+{
+ bool device = prot & KVM_PGTABLE_PROT_DEVICE;
+ u32 mtype = device ? MT_DEVICE_nGnRE : MT_NORMAL;
+ kvm_pte_t attr = FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX, mtype);
+ u32 sh = KVM_PTE_LEAF_ATTR_LO_S1_SH_IS;
+ u32 ap = (prot & KVM_PGTABLE_PROT_W) ? KVM_PTE_LEAF_ATTR_LO_S1_AP_RW :
+ KVM_PTE_LEAF_ATTR_LO_S1_AP_RO;
+
+ if (!(prot & KVM_PGTABLE_PROT_R))
+ return -EINVAL;
+
+ if (prot & KVM_PGTABLE_PROT_X) {
+ if (prot & KVM_PGTABLE_PROT_W)
+ return -EINVAL;
+
+ if (device)
+ return -EINVAL;
+ } else {
+ attr |= KVM_PTE_LEAF_ATTR_HI_S1_XN;
+ }
+
+ attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_AP, ap);
+ attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_SH, sh);
+ attr |= KVM_PTE_LEAF_ATTR_LO_S1_AF;
+ data->attr = attr;
+ return 0;
+}
+
+static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level,
+ kvm_pte_t *ptep, struct hyp_map_data *data)
+{
+ u64 granule = kvm_granule_size(level), phys = data->phys;
+
+ if (!kvm_block_mapping_supported(addr, end, phys, level))
+ return false;
+
+ WARN_ON(!kvm_set_valid_leaf_pte(ptep, phys, data->attr, level));
+ data->phys += granule;
+ return true;
+}
+
+static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag, void * const arg)
+{
+ kvm_pte_t *childp;
+
+ if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg))
+ return 0;
+
+ if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1))
+ return -EINVAL;
+
+ childp = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL);
+ if (!childp)
+ return -ENOMEM;
+
+ kvm_set_table_pte(ptep, childp);
+ return 0;
+}
+
+int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
+ enum kvm_pgtable_prot prot)
+{
+ int ret;
+ struct hyp_map_data map_data = {
+ .phys = ALIGN_DOWN(phys, PAGE_SIZE),
+ };
+ struct kvm_pgtable_walker walker = {
+ .cb = hyp_map_walker,
+ .flags = KVM_PGTABLE_WALK_LEAF,
+ .arg = &map_data,
+ };
+
+ ret = hyp_map_set_prot_attr(prot, &map_data);
+ if (ret)
+ return ret;
+
+ ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+ dsb(ishst);
+ isb();
+ return ret;
+}
+
+int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits)
+{
+ u64 levels = ARM64_HW_PGTABLE_LEVELS(va_bits);
+
+ pgt->pgd = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL);
+ if (!pgt->pgd)
+ return -ENOMEM;
+
+ pgt->ia_bits = va_bits;
+ pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels;
+ pgt->mmu = NULL;
+ return 0;
+}
+
+static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag, void * const arg)
+{
+ free_page((unsigned long)kvm_pte_follow(*ptep));
+ return 0;
+}
+
+void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
+{
+ struct kvm_pgtable_walker walker = {
+ .cb = hyp_free_walker,
+ .flags = KVM_PGTABLE_WALK_TABLE_POST,
+ };
+
+ WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker));
+ free_page((unsigned long)pgt->pgd);
+ pgt->pgd = NULL;
+}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables
2020-09-07 15:23 ` [PATCH v4 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables Will Deacon
@ 2020-09-08 1:09 ` Gavin Shan
0 siblings, 0 replies; 52+ messages in thread
From: Gavin Shan @ 2020-09-08 1:09 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, linux-arm-kernel
On 9/8/20 1:23 AM, Will Deacon wrote:
> The generic page-table walker is pretty useless as it stands, because it
> doesn't understand enough to allocate anything. Teach it about stage-1
> page-tables, and hook up an API for allocating these for the hypervisor
> at EL2.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
Reviewed-by: Gavin Shan <gshan@redhat.com>
> arch/arm64/include/asm/kvm_pgtable.h | 36 ++++++++
> arch/arm64/kvm/hyp/pgtable.c | 133 +++++++++++++++++++++++++++
> 2 files changed, 169 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> index 1c5d981e15c3..91e364804547 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -77,6 +77,42 @@ struct kvm_pgtable_walker {
> const enum kvm_pgtable_walk_flags flags;
> };
>
> +/**
> + * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table.
> + * @pgt: Uninitialised page-table structure to initialise.
> + * @va_bits: Maximum virtual address bits.
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits);
> +
> +/**
> + * kvm_pgtable_hyp_destroy() - Destroy an unused hypervisor stage-1 page-table.
> + * @pgt: Page-table structure initialised by kvm_pgtable_hyp_init().
> + *
> + * The page-table is assumed to be unreachable by any hardware walkers prior
> + * to freeing and therefore no TLB invalidation is performed.
> + */
> +void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt);
> +
> +/**
> + * kvm_pgtable_hyp_map() - Install a mapping in a hypervisor stage-1 page-table.
> + * @pgt: Page-table structure initialised by kvm_pgtable_hyp_init().
> + * @addr: Virtual address at which to place the mapping.
> + * @size: Size of the mapping.
> + * @phys: Physical address of the memory to map.
> + * @prot: Permissions and attributes for the mapping.
> + *
> + * If device attributes are not explicitly requested in @prot, then the
> + * mapping will be normal, cacheable. Attempts to install a mapping for
> + * a virtual address that is already mapped will be rejected with an error
> + * and a WARN().
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
> + enum kvm_pgtable_prot prot);
> +
> /**
> * kvm_pgtable_walk() - Walk a page-table.
> * @pgt: Page-table structure initialised by kvm_pgtable_*_init().
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index 3fb9d1949a3f..23a1006aa4ef 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -24,8 +24,18 @@
>
> #define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2)
>
> +#define KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX GENMASK(4, 2)
> +#define KVM_PTE_LEAF_ATTR_LO_S1_AP GENMASK(7, 6)
> +#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RO 3
> +#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RW 1
> +#define KVM_PTE_LEAF_ATTR_LO_S1_SH GENMASK(9, 8)
> +#define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3
> +#define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10)
> +
> #define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
>
> +#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)
> +
> struct kvm_pgtable_walk_data {
> struct kvm_pgtable *pgt;
> struct kvm_pgtable_walker *walker;
> @@ -283,3 +293,126 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size,
>
> return _kvm_pgtable_walk(&walk_data);
> }
> +
> +struct hyp_map_data {
> + u64 phys;
> + kvm_pte_t attr;
> +};
> +
> +static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot,
> + struct hyp_map_data *data)
> +{
> + bool device = prot & KVM_PGTABLE_PROT_DEVICE;
> + u32 mtype = device ? MT_DEVICE_nGnRE : MT_NORMAL;
> + kvm_pte_t attr = FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX, mtype);
> + u32 sh = KVM_PTE_LEAF_ATTR_LO_S1_SH_IS;
> + u32 ap = (prot & KVM_PGTABLE_PROT_W) ? KVM_PTE_LEAF_ATTR_LO_S1_AP_RW :
> + KVM_PTE_LEAF_ATTR_LO_S1_AP_RO;
> +
> + if (!(prot & KVM_PGTABLE_PROT_R))
> + return -EINVAL;
> +
> + if (prot & KVM_PGTABLE_PROT_X) {
> + if (prot & KVM_PGTABLE_PROT_W)
> + return -EINVAL;
> +
> + if (device)
> + return -EINVAL;
> + } else {
> + attr |= KVM_PTE_LEAF_ATTR_HI_S1_XN;
> + }
> +
> + attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_AP, ap);
> + attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_SH, sh);
> + attr |= KVM_PTE_LEAF_ATTR_LO_S1_AF;
> + data->attr = attr;
> + return 0;
> +}
> +
> +static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level,
> + kvm_pte_t *ptep, struct hyp_map_data *data)
> +{
> + u64 granule = kvm_granule_size(level), phys = data->phys;
> +
> + if (!kvm_block_mapping_supported(addr, end, phys, level))
> + return false;
> +
> + WARN_ON(!kvm_set_valid_leaf_pte(ptep, phys, data->attr, level));
> + data->phys += granule;
> + return true;
> +}
> +
> +static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
> + enum kvm_pgtable_walk_flags flag, void * const arg)
> +{
> + kvm_pte_t *childp;
> +
> + if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg))
> + return 0;
> +
> + if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1))
> + return -EINVAL;
> +
> + childp = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL);
> + if (!childp)
> + return -ENOMEM;
> +
> + kvm_set_table_pte(ptep, childp);
> + return 0;
> +}
> +
> +int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
> + enum kvm_pgtable_prot prot)
> +{
> + int ret;
> + struct hyp_map_data map_data = {
> + .phys = ALIGN_DOWN(phys, PAGE_SIZE),
> + };
> + struct kvm_pgtable_walker walker = {
> + .cb = hyp_map_walker,
> + .flags = KVM_PGTABLE_WALK_LEAF,
> + .arg = &map_data,
> + };
> +
> + ret = hyp_map_set_prot_attr(prot, &map_data);
> + if (ret)
> + return ret;
> +
> + ret = kvm_pgtable_walk(pgt, addr, size, &walker);
> + dsb(ishst);
> + isb();
> + return ret;
> +}
> +
> +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits)
> +{
> + u64 levels = ARM64_HW_PGTABLE_LEVELS(va_bits);
> +
> + pgt->pgd = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL);
> + if (!pgt->pgd)
> + return -ENOMEM;
> +
> + pgt->ia_bits = va_bits;
> + pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels;
> + pgt->mmu = NULL;
> + return 0;
> +}
> +
> +static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
> + enum kvm_pgtable_walk_flags flag, void * const arg)
> +{
> + free_page((unsigned long)kvm_pte_follow(*ptep));
> + return 0;
> +}
> +
> +void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
> +{
> + struct kvm_pgtable_walker walker = {
> + .cb = hyp_free_walker,
> + .flags = KVM_PGTABLE_WALK_TABLE_POST,
> + };
> +
> + WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker));
> + free_page((unsigned long)pgt->pgd);
> + pgt->pgd = NULL;
> +}
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (2 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 1:03 ` Gavin Shan
2020-09-07 15:23 ` [PATCH v4 05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables Will Deacon
` (18 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Now that we have a shiny new page-table allocator, replace the hyp
page-table code with calls into the new API. This also allows us to
remove the extended idmap code, as we can now simply ensure that the
VA size is large enough to map everything we need.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_mmu.h | 78 +----
arch/arm64/include/asm/kvm_pgtable.h | 5 +
arch/arm64/include/asm/pgtable-hwdef.h | 6 -
arch/arm64/include/asm/pgtable-prot.h | 6 -
arch/arm64/kvm/mmu.c | 414 +++----------------------
5 files changed, 45 insertions(+), 464 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 0f078b1920ff..42fb50cfe0d8 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -43,16 +43,6 @@
* HYP_VA_MIN = 1 << (VA_BITS - 1)
* HYP_VA_MAX = HYP_VA_MIN + (1 << (VA_BITS - 1)) - 1
*
- * This of course assumes that the trampoline page exists within the
- * VA_BITS range. If it doesn't, then it means we're in the odd case
- * where the kernel idmap (as well as HYP) uses more levels than the
- * kernel runtime page tables (as seen when the kernel is configured
- * for 4k pages, 39bits VA, and yet memory lives just above that
- * limit, forcing the idmap to use 4 levels of page tables while the
- * kernel itself only uses 3). In this particular case, it doesn't
- * matter which side of VA_BITS we use, as we're guaranteed not to
- * conflict with anything.
- *
* When using VHE, there are no separate hyp mappings and all KVM
* functionality is already mapped as part of the main kernel
* mappings, and none of this applies in that case.
@@ -123,9 +113,10 @@ static inline bool kvm_page_empty(void *ptr)
return page_count(ptr_page) == 1;
}
+#include <asm/kvm_pgtable.h>
#include <asm/stage2_pgtable.h>
-int create_hyp_mappings(void *from, void *to, pgprot_t prot);
+int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot);
int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
void __iomem **kaddr,
void __iomem **haddr);
@@ -144,8 +135,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu);
phys_addr_t kvm_mmu_get_httbr(void);
phys_addr_t kvm_get_idmap_vector(void);
int kvm_mmu_init(void);
-void kvm_clear_hyp_idmap(void);
-
#define kvm_mk_pmd(ptep) \
__pmd(__phys_to_pmd_val(__pa(ptep)) | PMD_TYPE_TABLE)
#define kvm_mk_pud(pmdp) \
@@ -263,25 +252,6 @@ static inline bool kvm_s2pud_young(pud_t pud)
return pud_young(pud);
}
-#define hyp_pte_table_empty(ptep) kvm_page_empty(ptep)
-
-#ifdef __PAGETABLE_PMD_FOLDED
-#define hyp_pmd_table_empty(pmdp) (0)
-#else
-#define hyp_pmd_table_empty(pmdp) kvm_page_empty(pmdp)
-#endif
-
-#ifdef __PAGETABLE_PUD_FOLDED
-#define hyp_pud_table_empty(pudp) (0)
-#else
-#define hyp_pud_table_empty(pudp) kvm_page_empty(pudp)
-#endif
-
-#ifdef __PAGETABLE_P4D_FOLDED
-#define hyp_p4d_table_empty(p4dp) (0)
-#else
-#define hyp_p4d_table_empty(p4dp) kvm_page_empty(p4dp)
-#endif
struct kvm;
@@ -350,50 +320,6 @@ static inline void __kvm_flush_dcache_pud(pud_t pud)
void kvm_set_way_flush(struct kvm_vcpu *vcpu);
void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
-static inline bool __kvm_cpu_uses_extended_idmap(void)
-{
- return __cpu_uses_extended_idmap_level();
-}
-
-static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
-{
- return idmap_ptrs_per_pgd;
-}
-
-/*
- * Can't use pgd_populate here, because the extended idmap adds an extra level
- * above CONFIG_PGTABLE_LEVELS (which is 2 or 3 if we're using the extended
- * idmap), and pgd_populate is only available if CONFIG_PGTABLE_LEVELS = 4.
- */
-static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
- pgd_t *hyp_pgd,
- pgd_t *merged_hyp_pgd,
- unsigned long hyp_idmap_start)
-{
- int idmap_idx;
- u64 pgd_addr;
-
- /*
- * Use the first entry to access the HYP mappings. It is
- * guaranteed to be free, otherwise we wouldn't use an
- * extended idmap.
- */
- VM_BUG_ON(pgd_val(merged_hyp_pgd[0]));
- pgd_addr = __phys_to_pgd_val(__pa(hyp_pgd));
- merged_hyp_pgd[0] = __pgd(pgd_addr | PMD_TYPE_TABLE);
-
- /*
- * Create another extended level entry that points to the boot HYP map,
- * which contains an ID mapping of the HYP init code. We essentially
- * merge the boot and runtime HYP maps by doing so, but they don't
- * overlap anyway, so this is fine.
- */
- idmap_idx = hyp_idmap_start >> VA_BITS;
- VM_BUG_ON(pgd_val(merged_hyp_pgd[idmap_idx]));
- pgd_addr = __phys_to_pgd_val(__pa(boot_hyp_pgd));
- merged_hyp_pgd[idmap_idx] = __pgd(pgd_addr | PMD_TYPE_TABLE);
-}
-
static inline unsigned int kvm_get_vmid_bits(void)
{
int reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 91e364804547..3e1dd1b9fd93 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -44,6 +44,11 @@ enum kvm_pgtable_prot {
KVM_PGTABLE_PROT_DEVICE = BIT(3),
};
+#define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W)
+#define PAGE_HYP_EXEC (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_X)
+#define PAGE_HYP_RO (KVM_PGTABLE_PROT_R)
+#define PAGE_HYP_DEVICE (PAGE_HYP | KVM_PGTABLE_PROT_DEVICE)
+
/**
* enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
* @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index d400a4d9aee2..1a989353144e 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -194,12 +194,6 @@
*/
#define PTE_S2_MEMATTR(t) (_AT(pteval_t, (t)) << 2)
-/*
- * EL2/HYP PTE/PMD definitions
- */
-#define PMD_HYP PMD_SECT_USER
-#define PTE_HYP PTE_USER
-
/*
* Highest possible physical address supported.
*/
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 4d867c6446c4..88acd7e1cd05 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -56,7 +56,6 @@ extern bool arm64_use_ng_mappings;
#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
-#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT
#define PAGE_KERNEL __pgprot(PROT_NORMAL)
#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)
@@ -64,11 +63,6 @@ extern bool arm64_use_ng_mappings;
#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
#define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)
-#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
-#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
-#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
-#define PAGE_HYP_DEVICE __pgprot(_PROT_DEFAULT | PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_HYP | PTE_HYP_XN)
-
#define PAGE_S2_MEMATTR(attr) \
({ \
u64 __val; \
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 935f8f689433..fabd72b0c8a4 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -14,6 +14,7 @@
#include <asm/cacheflush.h>
#include <asm/kvm_arm.h>
#include <asm/kvm_mmu.h>
+#include <asm/kvm_pgtable.h>
#include <asm/kvm_ras.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_emulate.h>
@@ -21,9 +22,7 @@
#include "trace.h"
-static pgd_t *boot_hyp_pgd;
-static pgd_t *hyp_pgd;
-static pgd_t *merged_hyp_pgd;
+static struct kvm_pgtable *hyp_pgtable;
static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
static unsigned long hyp_idmap_start;
@@ -32,8 +31,6 @@ static phys_addr_t hyp_idmap_vector;
static unsigned long io_map_base;
-#define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
-
#define KVM_S2PTE_FLAG_IS_IOMAP (1UL << 0)
#define KVM_S2_FLAG_LOGGING_ACTIVE (1UL << 1)
@@ -489,338 +486,28 @@ static void stage2_flush_vm(struct kvm *kvm)
srcu_read_unlock(&kvm->srcu, idx);
}
-static void clear_hyp_pgd_entry(pgd_t *pgd)
-{
- p4d_t *p4d_table __maybe_unused = p4d_offset(pgd, 0UL);
- pgd_clear(pgd);
- p4d_free(NULL, p4d_table);
- put_page(virt_to_page(pgd));
-}
-
-static void clear_hyp_p4d_entry(p4d_t *p4d)
-{
- pud_t *pud_table __maybe_unused = pud_offset(p4d, 0UL);
- VM_BUG_ON(p4d_huge(*p4d));
- p4d_clear(p4d);
- pud_free(NULL, pud_table);
- put_page(virt_to_page(p4d));
-}
-
-static void clear_hyp_pud_entry(pud_t *pud)
-{
- pmd_t *pmd_table __maybe_unused = pmd_offset(pud, 0);
- VM_BUG_ON(pud_huge(*pud));
- pud_clear(pud);
- pmd_free(NULL, pmd_table);
- put_page(virt_to_page(pud));
-}
-
-static void clear_hyp_pmd_entry(pmd_t *pmd)
-{
- pte_t *pte_table = pte_offset_kernel(pmd, 0);
- VM_BUG_ON(pmd_thp_or_huge(*pmd));
- pmd_clear(pmd);
- pte_free_kernel(NULL, pte_table);
- put_page(virt_to_page(pmd));
-}
-
-static void unmap_hyp_ptes(pmd_t *pmd, phys_addr_t addr, phys_addr_t end)
-{
- pte_t *pte, *start_pte;
-
- start_pte = pte = pte_offset_kernel(pmd, addr);
- do {
- if (!pte_none(*pte)) {
- kvm_set_pte(pte, __pte(0));
- put_page(virt_to_page(pte));
- }
- } while (pte++, addr += PAGE_SIZE, addr != end);
-
- if (hyp_pte_table_empty(start_pte))
- clear_hyp_pmd_entry(pmd);
-}
-
-static void unmap_hyp_pmds(pud_t *pud, phys_addr_t addr, phys_addr_t end)
-{
- phys_addr_t next;
- pmd_t *pmd, *start_pmd;
-
- start_pmd = pmd = pmd_offset(pud, addr);
- do {
- next = pmd_addr_end(addr, end);
- /* Hyp doesn't use huge pmds */
- if (!pmd_none(*pmd))
- unmap_hyp_ptes(pmd, addr, next);
- } while (pmd++, addr = next, addr != end);
-
- if (hyp_pmd_table_empty(start_pmd))
- clear_hyp_pud_entry(pud);
-}
-
-static void unmap_hyp_puds(p4d_t *p4d, phys_addr_t addr, phys_addr_t end)
-{
- phys_addr_t next;
- pud_t *pud, *start_pud;
-
- start_pud = pud = pud_offset(p4d, addr);
- do {
- next = pud_addr_end(addr, end);
- /* Hyp doesn't use huge puds */
- if (!pud_none(*pud))
- unmap_hyp_pmds(pud, addr, next);
- } while (pud++, addr = next, addr != end);
-
- if (hyp_pud_table_empty(start_pud))
- clear_hyp_p4d_entry(p4d);
-}
-
-static void unmap_hyp_p4ds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
-{
- phys_addr_t next;
- p4d_t *p4d, *start_p4d;
-
- start_p4d = p4d = p4d_offset(pgd, addr);
- do {
- next = p4d_addr_end(addr, end);
- /* Hyp doesn't use huge p4ds */
- if (!p4d_none(*p4d))
- unmap_hyp_puds(p4d, addr, next);
- } while (p4d++, addr = next, addr != end);
-
- if (hyp_p4d_table_empty(start_p4d))
- clear_hyp_pgd_entry(pgd);
-}
-
-static unsigned int kvm_pgd_index(unsigned long addr, unsigned int ptrs_per_pgd)
-{
- return (addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1);
-}
-
-static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
- phys_addr_t start, u64 size)
-{
- pgd_t *pgd;
- phys_addr_t addr = start, end = start + size;
- phys_addr_t next;
-
- /*
- * We don't unmap anything from HYP, except at the hyp tear down.
- * Hence, we don't have to invalidate the TLBs here.
- */
- pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
- do {
- next = pgd_addr_end(addr, end);
- if (!pgd_none(*pgd))
- unmap_hyp_p4ds(pgd, addr, next);
- } while (pgd++, addr = next, addr != end);
-}
-
-static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
-{
- __unmap_hyp_range(pgdp, PTRS_PER_PGD, start, size);
-}
-
-static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
-{
- __unmap_hyp_range(pgdp, __kvm_idmap_ptrs_per_pgd(), start, size);
-}
-
/**
* free_hyp_pgds - free Hyp-mode page tables
- *
- * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
- * therefore contains either mappings in the kernel memory area (above
- * PAGE_OFFSET), or device mappings in the idmap range.
- *
- * boot_hyp_pgd should only map the idmap range, and is only used in
- * the extended idmap case.
*/
void free_hyp_pgds(void)
{
- pgd_t *id_pgd;
-
mutex_lock(&kvm_hyp_pgd_mutex);
-
- id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
-
- if (id_pgd) {
- /* In case we never called hyp_mmu_init() */
- if (!io_map_base)
- io_map_base = hyp_idmap_start;
- unmap_hyp_idmap_range(id_pgd, io_map_base,
- hyp_idmap_start + PAGE_SIZE - io_map_base);
- }
-
- if (boot_hyp_pgd) {
- free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
- boot_hyp_pgd = NULL;
- }
-
- if (hyp_pgd) {
- unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
- (uintptr_t)high_memory - PAGE_OFFSET);
-
- free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
- hyp_pgd = NULL;
+ if (hyp_pgtable) {
+ kvm_pgtable_hyp_destroy(hyp_pgtable);
+ kfree(hyp_pgtable);
}
- if (merged_hyp_pgd) {
- clear_page(merged_hyp_pgd);
- free_page((unsigned long)merged_hyp_pgd);
- merged_hyp_pgd = NULL;
- }
-
mutex_unlock(&kvm_hyp_pgd_mutex);
}
-static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
- unsigned long end, unsigned long pfn,
- pgprot_t prot)
-{
- pte_t *pte;
- unsigned long addr;
-
- addr = start;
- do {
- pte = pte_offset_kernel(pmd, addr);
- kvm_set_pte(pte, kvm_pfn_pte(pfn, prot));
- get_page(virt_to_page(pte));
- pfn++;
- } while (addr += PAGE_SIZE, addr != end);
-}
-
-static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
- unsigned long end, unsigned long pfn,
- pgprot_t prot)
+static int __create_hyp_mappings(unsigned long start, unsigned long size,
+ unsigned long phys, enum kvm_pgtable_prot prot)
{
- pmd_t *pmd;
- pte_t *pte;
- unsigned long addr, next;
-
- addr = start;
- do {
- pmd = pmd_offset(pud, addr);
-
- BUG_ON(pmd_sect(*pmd));
-
- if (pmd_none(*pmd)) {
- pte = pte_alloc_one_kernel(NULL);
- if (!pte) {
- kvm_err("Cannot allocate Hyp pte\n");
- return -ENOMEM;
- }
- kvm_pmd_populate(pmd, pte);
- get_page(virt_to_page(pmd));
- }
-
- next = pmd_addr_end(addr, end);
-
- create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
- pfn += (next - addr) >> PAGE_SHIFT;
- } while (addr = next, addr != end);
-
- return 0;
-}
-
-static int create_hyp_pud_mappings(p4d_t *p4d, unsigned long start,
- unsigned long end, unsigned long pfn,
- pgprot_t prot)
-{
- pud_t *pud;
- pmd_t *pmd;
- unsigned long addr, next;
- int ret;
-
- addr = start;
- do {
- pud = pud_offset(p4d, addr);
-
- if (pud_none_or_clear_bad(pud)) {
- pmd = pmd_alloc_one(NULL, addr);
- if (!pmd) {
- kvm_err("Cannot allocate Hyp pmd\n");
- return -ENOMEM;
- }
- kvm_pud_populate(pud, pmd);
- get_page(virt_to_page(pud));
- }
-
- next = pud_addr_end(addr, end);
- ret = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);
- if (ret)
- return ret;
- pfn += (next - addr) >> PAGE_SHIFT;
- } while (addr = next, addr != end);
-
- return 0;
-}
-
-static int create_hyp_p4d_mappings(pgd_t *pgd, unsigned long start,
- unsigned long end, unsigned long pfn,
- pgprot_t prot)
-{
- p4d_t *p4d;
- pud_t *pud;
- unsigned long addr, next;
- int ret;
-
- addr = start;
- do {
- p4d = p4d_offset(pgd, addr);
-
- if (p4d_none(*p4d)) {
- pud = pud_alloc_one(NULL, addr);
- if (!pud) {
- kvm_err("Cannot allocate Hyp pud\n");
- return -ENOMEM;
- }
- kvm_p4d_populate(p4d, pud);
- get_page(virt_to_page(p4d));
- }
-
- next = p4d_addr_end(addr, end);
- ret = create_hyp_pud_mappings(p4d, addr, next, pfn, prot);
- if (ret)
- return ret;
- pfn += (next - addr) >> PAGE_SHIFT;
- } while (addr = next, addr != end);
-
- return 0;
-}
-
-static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
- unsigned long start, unsigned long end,
- unsigned long pfn, pgprot_t prot)
-{
- pgd_t *pgd;
- p4d_t *p4d;
- unsigned long addr, next;
- int err = 0;
+ int err;
mutex_lock(&kvm_hyp_pgd_mutex);
- addr = start & PAGE_MASK;
- end = PAGE_ALIGN(end);
- do {
- pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
-
- if (pgd_none(*pgd)) {
- p4d = p4d_alloc_one(NULL, addr);
- if (!p4d) {
- kvm_err("Cannot allocate Hyp p4d\n");
- err = -ENOMEM;
- goto out;
- }
- kvm_pgd_populate(pgd, p4d);
- get_page(virt_to_page(pgd));
- }
-
- next = pgd_addr_end(addr, end);
- err = create_hyp_p4d_mappings(pgd, addr, next, pfn, prot);
- if (err)
- goto out;
- pfn += (next - addr) >> PAGE_SHIFT;
- } while (addr = next, addr != end);
-out:
+ err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot);
mutex_unlock(&kvm_hyp_pgd_mutex);
+
return err;
}
@@ -845,7 +532,7 @@ static phys_addr_t kvm_kaddr_to_phys(void *kaddr)
* in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
* physical pages.
*/
-int create_hyp_mappings(void *from, void *to, pgprot_t prot)
+int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot)
{
phys_addr_t phys_addr;
unsigned long virt_addr;
@@ -862,9 +549,7 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
int err;
phys_addr = kvm_kaddr_to_phys(from + virt_addr - start);
- err = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD,
- virt_addr, virt_addr + PAGE_SIZE,
- __phys_to_pfn(phys_addr),
+ err = __create_hyp_mappings(virt_addr, PAGE_SIZE, phys_addr,
prot);
if (err)
return err;
@@ -874,9 +559,9 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
}
static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
- unsigned long *haddr, pgprot_t prot)
+ unsigned long *haddr,
+ enum kvm_pgtable_prot prot)
{
- pgd_t *pgd = hyp_pgd;
unsigned long base;
int ret = 0;
@@ -908,17 +593,11 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
if (ret)
goto out;
- if (__kvm_cpu_uses_extended_idmap())
- pgd = boot_hyp_pgd;
-
- ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
- base, base + size,
- __phys_to_pfn(phys_addr), prot);
+ ret = __create_hyp_mappings(base, size, phys_addr, prot);
if (ret)
goto out;
*haddr = base + offset_in_page(phys_addr);
-
out:
return ret;
}
@@ -2326,10 +2005,7 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva)
phys_addr_t kvm_mmu_get_httbr(void)
{
- if (__kvm_cpu_uses_extended_idmap())
- return virt_to_phys(merged_hyp_pgd);
- else
- return virt_to_phys(hyp_pgd);
+ return __pa(hyp_pgtable->pgd);
}
phys_addr_t kvm_get_idmap_vector(void)
@@ -2337,15 +2013,11 @@ phys_addr_t kvm_get_idmap_vector(void)
return hyp_idmap_vector;
}
-static int kvm_map_idmap_text(pgd_t *pgd)
+static int kvm_map_idmap_text(void)
{
- int err;
-
- /* Create the idmap in the boot page tables */
- err = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
- hyp_idmap_start, hyp_idmap_end,
- __phys_to_pfn(hyp_idmap_start),
- PAGE_HYP_EXEC);
+ unsigned long size = hyp_idmap_end - hyp_idmap_start;
+ int err = __create_hyp_mappings(hyp_idmap_start, size, hyp_idmap_start,
+ PAGE_HYP_EXEC);
if (err)
kvm_err("Failed to idmap %lx-%lx\n",
hyp_idmap_start, hyp_idmap_end);
@@ -2356,6 +2028,7 @@ static int kvm_map_idmap_text(pgd_t *pgd)
int kvm_mmu_init(void)
{
int err;
+ u32 hyp_va_bits;
hyp_idmap_start = __pa_symbol(__hyp_idmap_text_start);
hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
@@ -2369,6 +2042,8 @@ int kvm_mmu_init(void)
*/
BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK);
+ hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET);
+ kvm_debug("Using %u-bit virtual addresses at EL2\n", hyp_va_bits);
kvm_debug("IDMAP page: %lx\n", hyp_idmap_start);
kvm_debug("HYP VA range: %lx:%lx\n",
kern_hyp_va(PAGE_OFFSET),
@@ -2386,43 +2061,30 @@ int kvm_mmu_init(void)
goto out;
}
- hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, hyp_pgd_order);
- if (!hyp_pgd) {
- kvm_err("Hyp mode PGD not allocated\n");
+ hyp_pgtable = kzalloc(sizeof(*hyp_pgtable), GFP_KERNEL);
+ if (!hyp_pgtable) {
+ kvm_err("Hyp mode page-table not allocated\n");
err = -ENOMEM;
goto out;
}
- if (__kvm_cpu_uses_extended_idmap()) {
- boot_hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO,
- hyp_pgd_order);
- if (!boot_hyp_pgd) {
- kvm_err("Hyp boot PGD not allocated\n");
- err = -ENOMEM;
- goto out;
- }
-
- err = kvm_map_idmap_text(boot_hyp_pgd);
- if (err)
- goto out;
+ err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits);
+ if (err)
+ goto out_free_pgtable;
- merged_hyp_pgd = (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
- if (!merged_hyp_pgd) {
- kvm_err("Failed to allocate extra HYP pgd\n");
- goto out;
- }
- __kvm_extend_hypmap(boot_hyp_pgd, hyp_pgd, merged_hyp_pgd,
- hyp_idmap_start);
- } else {
- err = kvm_map_idmap_text(hyp_pgd);
- if (err)
- goto out;
- }
+ err = kvm_map_idmap_text();
+ if (err)
+ goto out_destroy_pgtable;
io_map_base = hyp_idmap_start;
return 0;
+
+out_destroy_pgtable:
+ kvm_pgtable_hyp_destroy(hyp_pgtable);
+out_free_pgtable:
+ kfree(hyp_pgtable);
+ hyp_pgtable = NULL;
out:
- free_hyp_pgds();
return err;
}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables
2020-09-07 15:23 ` [PATCH v4 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables Will Deacon
@ 2020-09-08 1:03 ` Gavin Shan
0 siblings, 0 replies; 52+ messages in thread
From: Gavin Shan @ 2020-09-08 1:03 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, linux-arm-kernel
Hi Will,
On 9/8/20 1:23 AM, Will Deacon wrote:
> Now that we have a shiny new page-table allocator, replace the hyp
> page-table code with calls into the new API. This also allows us to
> remove the extended idmap code, as we can now simply ensure that the
> VA size is large enough to map everything we need.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
With following one question resolved if getting a chance for another
respin:
Reviewed-by: Gavin Shan <gshan@redhat.com>
> arch/arm64/include/asm/kvm_mmu.h | 78 +----
> arch/arm64/include/asm/kvm_pgtable.h | 5 +
> arch/arm64/include/asm/pgtable-hwdef.h | 6 -
> arch/arm64/include/asm/pgtable-prot.h | 6 -
> arch/arm64/kvm/mmu.c | 414 +++----------------------
> 5 files changed, 45 insertions(+), 464 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 0f078b1920ff..42fb50cfe0d8 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -43,16 +43,6 @@
> * HYP_VA_MIN = 1 << (VA_BITS - 1)
> * HYP_VA_MAX = HYP_VA_MIN + (1 << (VA_BITS - 1)) - 1
> *
> - * This of course assumes that the trampoline page exists within the
> - * VA_BITS range. If it doesn't, then it means we're in the odd case
> - * where the kernel idmap (as well as HYP) uses more levels than the
> - * kernel runtime page tables (as seen when the kernel is configured
> - * for 4k pages, 39bits VA, and yet memory lives just above that
> - * limit, forcing the idmap to use 4 levels of page tables while the
> - * kernel itself only uses 3). In this particular case, it doesn't
> - * matter which side of VA_BITS we use, as we're guaranteed not to
> - * conflict with anything.
> - *
> * When using VHE, there are no separate hyp mappings and all KVM
> * functionality is already mapped as part of the main kernel
> * mappings, and none of this applies in that case.
> @@ -123,9 +113,10 @@ static inline bool kvm_page_empty(void *ptr)
> return page_count(ptr_page) == 1;
> }
>
> +#include <asm/kvm_pgtable.h>
> #include <asm/stage2_pgtable.h>
>
> -int create_hyp_mappings(void *from, void *to, pgprot_t prot);
> +int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot);
> int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
> void __iomem **kaddr,
> void __iomem **haddr);
> @@ -144,8 +135,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu);
> phys_addr_t kvm_mmu_get_httbr(void);
> phys_addr_t kvm_get_idmap_vector(void);
> int kvm_mmu_init(void);
> -void kvm_clear_hyp_idmap(void);
> -
> #define kvm_mk_pmd(ptep) \
> __pmd(__phys_to_pmd_val(__pa(ptep)) | PMD_TYPE_TABLE)
> #define kvm_mk_pud(pmdp) \
> @@ -263,25 +252,6 @@ static inline bool kvm_s2pud_young(pud_t pud)
> return pud_young(pud);
> }
>
> -#define hyp_pte_table_empty(ptep) kvm_page_empty(ptep)
> -
> -#ifdef __PAGETABLE_PMD_FOLDED
> -#define hyp_pmd_table_empty(pmdp) (0)
> -#else
> -#define hyp_pmd_table_empty(pmdp) kvm_page_empty(pmdp)
> -#endif
> -
> -#ifdef __PAGETABLE_PUD_FOLDED
> -#define hyp_pud_table_empty(pudp) (0)
> -#else
> -#define hyp_pud_table_empty(pudp) kvm_page_empty(pudp)
> -#endif
> -
> -#ifdef __PAGETABLE_P4D_FOLDED
> -#define hyp_p4d_table_empty(p4dp) (0)
> -#else
> -#define hyp_p4d_table_empty(p4dp) kvm_page_empty(p4dp)
> -#endif
>
> struct kvm;
>
> @@ -350,50 +320,6 @@ static inline void __kvm_flush_dcache_pud(pud_t pud)
> void kvm_set_way_flush(struct kvm_vcpu *vcpu);
> void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
>
> -static inline bool __kvm_cpu_uses_extended_idmap(void)
> -{
> - return __cpu_uses_extended_idmap_level();
> -}
> -
> -static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
> -{
> - return idmap_ptrs_per_pgd;
> -}
> -
> -/*
> - * Can't use pgd_populate here, because the extended idmap adds an extra level
> - * above CONFIG_PGTABLE_LEVELS (which is 2 or 3 if we're using the extended
> - * idmap), and pgd_populate is only available if CONFIG_PGTABLE_LEVELS = 4.
> - */
> -static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
> - pgd_t *hyp_pgd,
> - pgd_t *merged_hyp_pgd,
> - unsigned long hyp_idmap_start)
> -{
> - int idmap_idx;
> - u64 pgd_addr;
> -
> - /*
> - * Use the first entry to access the HYP mappings. It is
> - * guaranteed to be free, otherwise we wouldn't use an
> - * extended idmap.
> - */
> - VM_BUG_ON(pgd_val(merged_hyp_pgd[0]));
> - pgd_addr = __phys_to_pgd_val(__pa(hyp_pgd));
> - merged_hyp_pgd[0] = __pgd(pgd_addr | PMD_TYPE_TABLE);
> -
> - /*
> - * Create another extended level entry that points to the boot HYP map,
> - * which contains an ID mapping of the HYP init code. We essentially
> - * merge the boot and runtime HYP maps by doing so, but they don't
> - * overlap anyway, so this is fine.
> - */
> - idmap_idx = hyp_idmap_start >> VA_BITS;
> - VM_BUG_ON(pgd_val(merged_hyp_pgd[idmap_idx]));
> - pgd_addr = __phys_to_pgd_val(__pa(boot_hyp_pgd));
> - merged_hyp_pgd[idmap_idx] = __pgd(pgd_addr | PMD_TYPE_TABLE);
> -}
> -
> static inline unsigned int kvm_get_vmid_bits(void)
> {
> int reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> index 91e364804547..3e1dd1b9fd93 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -44,6 +44,11 @@ enum kvm_pgtable_prot {
> KVM_PGTABLE_PROT_DEVICE = BIT(3),
> };
>
> +#define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W)
> +#define PAGE_HYP_EXEC (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_X)
> +#define PAGE_HYP_RO (KVM_PGTABLE_PROT_R)
> +#define PAGE_HYP_DEVICE (PAGE_HYP | KVM_PGTABLE_PROT_DEVICE)
> +
> /**
> * enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
> * @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index d400a4d9aee2..1a989353144e 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -194,12 +194,6 @@
> */
> #define PTE_S2_MEMATTR(t) (_AT(pteval_t, (t)) << 2)
>
> -/*
> - * EL2/HYP PTE/PMD definitions
> - */
> -#define PMD_HYP PMD_SECT_USER
> -#define PTE_HYP PTE_USER
> -
> /*
> * Highest possible physical address supported.
> */
There are more stuff to be dropped in arm64/include/asm/pgtable-hwdef.h:
#define PTE_HYP_XN (_AT(pteval_t, 1) << 54) /* HYP XN */
> diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
> index 4d867c6446c4..88acd7e1cd05 100644
> --- a/arch/arm64/include/asm/pgtable-prot.h
> +++ b/arch/arm64/include/asm/pgtable-prot.h
> @@ -56,7 +56,6 @@ extern bool arm64_use_ng_mappings;
> #define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
>
> #define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
> -#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT
>
> #define PAGE_KERNEL __pgprot(PROT_NORMAL)
> #define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)
> @@ -64,11 +63,6 @@ extern bool arm64_use_ng_mappings;
> #define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
> #define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)
>
> -#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
> -#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
> -#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
> -#define PAGE_HYP_DEVICE __pgprot(_PROT_DEFAULT | PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_HYP | PTE_HYP_XN)
> -
> #define PAGE_S2_MEMATTR(attr) \
> ({ \
> u64 __val; \
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 935f8f689433..fabd72b0c8a4 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -14,6 +14,7 @@
> #include <asm/cacheflush.h>
> #include <asm/kvm_arm.h>
> #include <asm/kvm_mmu.h>
> +#include <asm/kvm_pgtable.h>
> #include <asm/kvm_ras.h>
> #include <asm/kvm_asm.h>
> #include <asm/kvm_emulate.h>
> @@ -21,9 +22,7 @@
>
> #include "trace.h"
>
> -static pgd_t *boot_hyp_pgd;
> -static pgd_t *hyp_pgd;
> -static pgd_t *merged_hyp_pgd;
> +static struct kvm_pgtable *hyp_pgtable;
> static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
>
> static unsigned long hyp_idmap_start;
> @@ -32,8 +31,6 @@ static phys_addr_t hyp_idmap_vector;
>
> static unsigned long io_map_base;
>
> -#define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
> -
> #define KVM_S2PTE_FLAG_IS_IOMAP (1UL << 0)
> #define KVM_S2_FLAG_LOGGING_ACTIVE (1UL << 1)
>
> @@ -489,338 +486,28 @@ static void stage2_flush_vm(struct kvm *kvm)
> srcu_read_unlock(&kvm->srcu, idx);
> }
>
> -static void clear_hyp_pgd_entry(pgd_t *pgd)
> -{
> - p4d_t *p4d_table __maybe_unused = p4d_offset(pgd, 0UL);
> - pgd_clear(pgd);
> - p4d_free(NULL, p4d_table);
> - put_page(virt_to_page(pgd));
> -}
> -
> -static void clear_hyp_p4d_entry(p4d_t *p4d)
> -{
> - pud_t *pud_table __maybe_unused = pud_offset(p4d, 0UL);
> - VM_BUG_ON(p4d_huge(*p4d));
> - p4d_clear(p4d);
> - pud_free(NULL, pud_table);
> - put_page(virt_to_page(p4d));
> -}
> -
> -static void clear_hyp_pud_entry(pud_t *pud)
> -{
> - pmd_t *pmd_table __maybe_unused = pmd_offset(pud, 0);
> - VM_BUG_ON(pud_huge(*pud));
> - pud_clear(pud);
> - pmd_free(NULL, pmd_table);
> - put_page(virt_to_page(pud));
> -}
> -
> -static void clear_hyp_pmd_entry(pmd_t *pmd)
> -{
> - pte_t *pte_table = pte_offset_kernel(pmd, 0);
> - VM_BUG_ON(pmd_thp_or_huge(*pmd));
> - pmd_clear(pmd);
> - pte_free_kernel(NULL, pte_table);
> - put_page(virt_to_page(pmd));
> -}
> -
> -static void unmap_hyp_ptes(pmd_t *pmd, phys_addr_t addr, phys_addr_t end)
> -{
> - pte_t *pte, *start_pte;
> -
> - start_pte = pte = pte_offset_kernel(pmd, addr);
> - do {
> - if (!pte_none(*pte)) {
> - kvm_set_pte(pte, __pte(0));
> - put_page(virt_to_page(pte));
> - }
> - } while (pte++, addr += PAGE_SIZE, addr != end);
> -
> - if (hyp_pte_table_empty(start_pte))
> - clear_hyp_pmd_entry(pmd);
> -}
> -
> -static void unmap_hyp_pmds(pud_t *pud, phys_addr_t addr, phys_addr_t end)
> -{
> - phys_addr_t next;
> - pmd_t *pmd, *start_pmd;
> -
> - start_pmd = pmd = pmd_offset(pud, addr);
> - do {
> - next = pmd_addr_end(addr, end);
> - /* Hyp doesn't use huge pmds */
> - if (!pmd_none(*pmd))
> - unmap_hyp_ptes(pmd, addr, next);
> - } while (pmd++, addr = next, addr != end);
> -
> - if (hyp_pmd_table_empty(start_pmd))
> - clear_hyp_pud_entry(pud);
> -}
> -
> -static void unmap_hyp_puds(p4d_t *p4d, phys_addr_t addr, phys_addr_t end)
> -{
> - phys_addr_t next;
> - pud_t *pud, *start_pud;
> -
> - start_pud = pud = pud_offset(p4d, addr);
> - do {
> - next = pud_addr_end(addr, end);
> - /* Hyp doesn't use huge puds */
> - if (!pud_none(*pud))
> - unmap_hyp_pmds(pud, addr, next);
> - } while (pud++, addr = next, addr != end);
> -
> - if (hyp_pud_table_empty(start_pud))
> - clear_hyp_p4d_entry(p4d);
> -}
> -
> -static void unmap_hyp_p4ds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
> -{
> - phys_addr_t next;
> - p4d_t *p4d, *start_p4d;
> -
> - start_p4d = p4d = p4d_offset(pgd, addr);
> - do {
> - next = p4d_addr_end(addr, end);
> - /* Hyp doesn't use huge p4ds */
> - if (!p4d_none(*p4d))
> - unmap_hyp_puds(p4d, addr, next);
> - } while (p4d++, addr = next, addr != end);
> -
> - if (hyp_p4d_table_empty(start_p4d))
> - clear_hyp_pgd_entry(pgd);
> -}
> -
> -static unsigned int kvm_pgd_index(unsigned long addr, unsigned int ptrs_per_pgd)
> -{
> - return (addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1);
> -}
> -
> -static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
> - phys_addr_t start, u64 size)
> -{
> - pgd_t *pgd;
> - phys_addr_t addr = start, end = start + size;
> - phys_addr_t next;
> -
> - /*
> - * We don't unmap anything from HYP, except at the hyp tear down.
> - * Hence, we don't have to invalidate the TLBs here.
> - */
> - pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
> - do {
> - next = pgd_addr_end(addr, end);
> - if (!pgd_none(*pgd))
> - unmap_hyp_p4ds(pgd, addr, next);
> - } while (pgd++, addr = next, addr != end);
> -}
> -
> -static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
> -{
> - __unmap_hyp_range(pgdp, PTRS_PER_PGD, start, size);
> -}
> -
> -static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
> -{
> - __unmap_hyp_range(pgdp, __kvm_idmap_ptrs_per_pgd(), start, size);
> -}
> -
> /**
> * free_hyp_pgds - free Hyp-mode page tables
> - *
> - * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
> - * therefore contains either mappings in the kernel memory area (above
> - * PAGE_OFFSET), or device mappings in the idmap range.
> - *
> - * boot_hyp_pgd should only map the idmap range, and is only used in
> - * the extended idmap case.
> */
> void free_hyp_pgds(void)
> {
> - pgd_t *id_pgd;
> -
> mutex_lock(&kvm_hyp_pgd_mutex);
> -
> - id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
> -
> - if (id_pgd) {
> - /* In case we never called hyp_mmu_init() */
> - if (!io_map_base)
> - io_map_base = hyp_idmap_start;
> - unmap_hyp_idmap_range(id_pgd, io_map_base,
> - hyp_idmap_start + PAGE_SIZE - io_map_base);
> - }
> -
> - if (boot_hyp_pgd) {
> - free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
> - boot_hyp_pgd = NULL;
> - }
> -
> - if (hyp_pgd) {
> - unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
> - (uintptr_t)high_memory - PAGE_OFFSET);
> -
> - free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
> - hyp_pgd = NULL;
> + if (hyp_pgtable) {
> + kvm_pgtable_hyp_destroy(hyp_pgtable);
> + kfree(hyp_pgtable);
> }
> - if (merged_hyp_pgd) {
> - clear_page(merged_hyp_pgd);
> - free_page((unsigned long)merged_hyp_pgd);
> - merged_hyp_pgd = NULL;
> - }
> -
> mutex_unlock(&kvm_hyp_pgd_mutex);
> }
>
> -static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
> - unsigned long end, unsigned long pfn,
> - pgprot_t prot)
> -{
> - pte_t *pte;
> - unsigned long addr;
> -
> - addr = start;
> - do {
> - pte = pte_offset_kernel(pmd, addr);
> - kvm_set_pte(pte, kvm_pfn_pte(pfn, prot));
> - get_page(virt_to_page(pte));
> - pfn++;
> - } while (addr += PAGE_SIZE, addr != end);
> -}
> -
> -static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
> - unsigned long end, unsigned long pfn,
> - pgprot_t prot)
> +static int __create_hyp_mappings(unsigned long start, unsigned long size,
> + unsigned long phys, enum kvm_pgtable_prot prot)
> {
> - pmd_t *pmd;
> - pte_t *pte;
> - unsigned long addr, next;
> -
> - addr = start;
> - do {
> - pmd = pmd_offset(pud, addr);
> -
> - BUG_ON(pmd_sect(*pmd));
> -
> - if (pmd_none(*pmd)) {
> - pte = pte_alloc_one_kernel(NULL);
> - if (!pte) {
> - kvm_err("Cannot allocate Hyp pte\n");
> - return -ENOMEM;
> - }
> - kvm_pmd_populate(pmd, pte);
> - get_page(virt_to_page(pmd));
> - }
> -
> - next = pmd_addr_end(addr, end);
> -
> - create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
> - pfn += (next - addr) >> PAGE_SHIFT;
> - } while (addr = next, addr != end);
> -
> - return 0;
> -}
> -
> -static int create_hyp_pud_mappings(p4d_t *p4d, unsigned long start,
> - unsigned long end, unsigned long pfn,
> - pgprot_t prot)
> -{
> - pud_t *pud;
> - pmd_t *pmd;
> - unsigned long addr, next;
> - int ret;
> -
> - addr = start;
> - do {
> - pud = pud_offset(p4d, addr);
> -
> - if (pud_none_or_clear_bad(pud)) {
> - pmd = pmd_alloc_one(NULL, addr);
> - if (!pmd) {
> - kvm_err("Cannot allocate Hyp pmd\n");
> - return -ENOMEM;
> - }
> - kvm_pud_populate(pud, pmd);
> - get_page(virt_to_page(pud));
> - }
> -
> - next = pud_addr_end(addr, end);
> - ret = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);
> - if (ret)
> - return ret;
> - pfn += (next - addr) >> PAGE_SHIFT;
> - } while (addr = next, addr != end);
> -
> - return 0;
> -}
> -
> -static int create_hyp_p4d_mappings(pgd_t *pgd, unsigned long start,
> - unsigned long end, unsigned long pfn,
> - pgprot_t prot)
> -{
> - p4d_t *p4d;
> - pud_t *pud;
> - unsigned long addr, next;
> - int ret;
> -
> - addr = start;
> - do {
> - p4d = p4d_offset(pgd, addr);
> -
> - if (p4d_none(*p4d)) {
> - pud = pud_alloc_one(NULL, addr);
> - if (!pud) {
> - kvm_err("Cannot allocate Hyp pud\n");
> - return -ENOMEM;
> - }
> - kvm_p4d_populate(p4d, pud);
> - get_page(virt_to_page(p4d));
> - }
> -
> - next = p4d_addr_end(addr, end);
> - ret = create_hyp_pud_mappings(p4d, addr, next, pfn, prot);
> - if (ret)
> - return ret;
> - pfn += (next - addr) >> PAGE_SHIFT;
> - } while (addr = next, addr != end);
> -
> - return 0;
> -}
> -
> -static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
> - unsigned long start, unsigned long end,
> - unsigned long pfn, pgprot_t prot)
> -{
> - pgd_t *pgd;
> - p4d_t *p4d;
> - unsigned long addr, next;
> - int err = 0;
> + int err;
>
> mutex_lock(&kvm_hyp_pgd_mutex);
> - addr = start & PAGE_MASK;
> - end = PAGE_ALIGN(end);
> - do {
> - pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
> -
> - if (pgd_none(*pgd)) {
> - p4d = p4d_alloc_one(NULL, addr);
> - if (!p4d) {
> - kvm_err("Cannot allocate Hyp p4d\n");
> - err = -ENOMEM;
> - goto out;
> - }
> - kvm_pgd_populate(pgd, p4d);
> - get_page(virt_to_page(pgd));
> - }
> -
> - next = pgd_addr_end(addr, end);
> - err = create_hyp_p4d_mappings(pgd, addr, next, pfn, prot);
> - if (err)
> - goto out;
> - pfn += (next - addr) >> PAGE_SHIFT;
> - } while (addr = next, addr != end);
> -out:
> + err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot);
> mutex_unlock(&kvm_hyp_pgd_mutex);
> +
> return err;
> }
>
> @@ -845,7 +532,7 @@ static phys_addr_t kvm_kaddr_to_phys(void *kaddr)
> * in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
> * physical pages.
> */
> -int create_hyp_mappings(void *from, void *to, pgprot_t prot)
> +int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot)
> {
> phys_addr_t phys_addr;
> unsigned long virt_addr;
> @@ -862,9 +549,7 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
> int err;
>
> phys_addr = kvm_kaddr_to_phys(from + virt_addr - start);
> - err = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD,
> - virt_addr, virt_addr + PAGE_SIZE,
> - __phys_to_pfn(phys_addr),
> + err = __create_hyp_mappings(virt_addr, PAGE_SIZE, phys_addr,
> prot);
> if (err)
> return err;
> @@ -874,9 +559,9 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
> }
>
> static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
> - unsigned long *haddr, pgprot_t prot)
> + unsigned long *haddr,
> + enum kvm_pgtable_prot prot)
> {
> - pgd_t *pgd = hyp_pgd;
> unsigned long base;
> int ret = 0;
>
> @@ -908,17 +593,11 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
> if (ret)
> goto out;
>
> - if (__kvm_cpu_uses_extended_idmap())
> - pgd = boot_hyp_pgd;
> -
> - ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
> - base, base + size,
> - __phys_to_pfn(phys_addr), prot);
> + ret = __create_hyp_mappings(base, size, phys_addr, prot);
> if (ret)
> goto out;
>
> *haddr = base + offset_in_page(phys_addr);
> -
> out:
> return ret;
> }
> @@ -2326,10 +2005,7 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva)
>
> phys_addr_t kvm_mmu_get_httbr(void)
> {
> - if (__kvm_cpu_uses_extended_idmap())
> - return virt_to_phys(merged_hyp_pgd);
> - else
> - return virt_to_phys(hyp_pgd);
> + return __pa(hyp_pgtable->pgd);
> }
>
> phys_addr_t kvm_get_idmap_vector(void)
> @@ -2337,15 +2013,11 @@ phys_addr_t kvm_get_idmap_vector(void)
> return hyp_idmap_vector;
> }
>
> -static int kvm_map_idmap_text(pgd_t *pgd)
> +static int kvm_map_idmap_text(void)
> {
> - int err;
> -
> - /* Create the idmap in the boot page tables */
> - err = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
> - hyp_idmap_start, hyp_idmap_end,
> - __phys_to_pfn(hyp_idmap_start),
> - PAGE_HYP_EXEC);
> + unsigned long size = hyp_idmap_end - hyp_idmap_start;
> + int err = __create_hyp_mappings(hyp_idmap_start, size, hyp_idmap_start,
> + PAGE_HYP_EXEC);
> if (err)
> kvm_err("Failed to idmap %lx-%lx\n",
> hyp_idmap_start, hyp_idmap_end);
> @@ -2356,6 +2028,7 @@ static int kvm_map_idmap_text(pgd_t *pgd)
> int kvm_mmu_init(void)
> {
> int err;
> + u32 hyp_va_bits;
>
> hyp_idmap_start = __pa_symbol(__hyp_idmap_text_start);
> hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
> @@ -2369,6 +2042,8 @@ int kvm_mmu_init(void)
> */
> BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK);
>
> + hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET);
> + kvm_debug("Using %u-bit virtual addresses at EL2\n", hyp_va_bits);
> kvm_debug("IDMAP page: %lx\n", hyp_idmap_start);
> kvm_debug("HYP VA range: %lx:%lx\n",
> kern_hyp_va(PAGE_OFFSET),
> @@ -2386,43 +2061,30 @@ int kvm_mmu_init(void)
> goto out;
> }
>
> - hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, hyp_pgd_order);
> - if (!hyp_pgd) {
> - kvm_err("Hyp mode PGD not allocated\n");
> + hyp_pgtable = kzalloc(sizeof(*hyp_pgtable), GFP_KERNEL);
> + if (!hyp_pgtable) {
> + kvm_err("Hyp mode page-table not allocated\n");
> err = -ENOMEM;
> goto out;
> }
>
> - if (__kvm_cpu_uses_extended_idmap()) {
> - boot_hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO,
> - hyp_pgd_order);
> - if (!boot_hyp_pgd) {
> - kvm_err("Hyp boot PGD not allocated\n");
> - err = -ENOMEM;
> - goto out;
> - }
> -
> - err = kvm_map_idmap_text(boot_hyp_pgd);
> - if (err)
> - goto out;
> + err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits);
> + if (err)
> + goto out_free_pgtable;
>
> - merged_hyp_pgd = (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
> - if (!merged_hyp_pgd) {
> - kvm_err("Failed to allocate extra HYP pgd\n");
> - goto out;
> - }
> - __kvm_extend_hypmap(boot_hyp_pgd, hyp_pgd, merged_hyp_pgd,
> - hyp_idmap_start);
> - } else {
> - err = kvm_map_idmap_text(hyp_pgd);
> - if (err)
> - goto out;
> - }
> + err = kvm_map_idmap_text();
> + if (err)
> + goto out_destroy_pgtable;
>
> io_map_base = hyp_idmap_start;
> return 0;
> +
> +out_destroy_pgtable:
> + kvm_pgtable_hyp_destroy(hyp_pgtable);
> +out_free_pgtable:
> + kfree(hyp_pgtable);
> + hyp_pgtable = NULL;
> out:
> - free_hyp_pgds();
> return err;
> }
>
Thanks,
Gavin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (3 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table Will Deacon
` (17 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Introduce alloc() and free() functions to the generic page-table code
for guest stage-2 page-tables and plumb these into the existing KVM
page-table allocator. Subsequent patches will convert other operations
within the KVM allocator over to the generic code.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_host.h | 1 +
arch/arm64/include/asm/kvm_pgtable.h | 18 +++++++++
arch/arm64/kvm/hyp/pgtable.c | 51 ++++++++++++++++++++++++++
arch/arm64/kvm/mmu.c | 55 +++++++++++++++-------------
4 files changed, 99 insertions(+), 26 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index e52c927aade5..0b7c702b2151 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -81,6 +81,7 @@ struct kvm_s2_mmu {
*/
pgd_t *pgd;
phys_addr_t pgd_phys;
+ struct kvm_pgtable *pgt;
/* The last vcpu id that ran on each physical CPU */
int __percpu *last_vcpu_ran;
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 3e1dd1b9fd93..85078bb632bb 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -118,6 +118,24 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt);
int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
enum kvm_pgtable_prot prot);
+/**
+ * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table.
+ * @pgt: Uninitialised page-table structure to initialise.
+ * @kvm: KVM structure representing the guest virtual machine.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm);
+
+/**
+ * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ *
+ * The page-table is assumed to be unreachable by any hardware walkers prior
+ * to freeing and therefore no TLB invalidation is performed.
+ */
+void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
+
/**
* kvm_pgtable_walk() - Walk a page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_*_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 23a1006aa4ef..96e21017830b 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -416,3 +416,54 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
free_page((unsigned long)pgt->pgd);
pgt->pgd = NULL;
}
+
+int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm)
+{
+ size_t pgd_sz;
+ u64 vtcr = kvm->arch.vtcr;
+ u32 ia_bits = VTCR_EL2_IPA(vtcr);
+ u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
+ u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;
+
+ pgd_sz = kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE;
+ pgt->pgd = alloc_pages_exact(pgd_sz, GFP_KERNEL | __GFP_ZERO);
+ if (!pgt->pgd)
+ return -ENOMEM;
+
+ pgt->ia_bits = ia_bits;
+ pgt->start_level = start_level;
+ pgt->mmu = &kvm->arch.mmu;
+ return 0;
+}
+
+static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag,
+ void * const arg)
+{
+ kvm_pte_t pte = *ptep;
+
+ if (!kvm_pte_valid(pte))
+ return 0;
+
+ put_page(virt_to_page(ptep));
+
+ if (kvm_pte_table(pte, level))
+ free_page((unsigned long)kvm_pte_follow(pte));
+
+ return 0;
+}
+
+void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt)
+{
+ size_t pgd_sz;
+ struct kvm_pgtable_walker walker = {
+ .cb = stage2_free_walker,
+ .flags = KVM_PGTABLE_WALK_LEAF |
+ KVM_PGTABLE_WALK_TABLE_POST,
+ };
+
+ WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker));
+ pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE;
+ free_pages_exact(pgt->pgd, pgd_sz);
+ pgt->pgd = NULL;
+}
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index fabd72b0c8a4..4607e9ca60a2 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -668,47 +668,49 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
* @kvm: The pointer to the KVM structure
* @mmu: The pointer to the s2 MMU structure
*
- * Allocates only the stage-2 HW PGD level table(s) of size defined by
- * stage2_pgd_size(mmu->kvm).
- *
+ * Allocates only the stage-2 HW PGD level table(s).
* Note we don't need locking here as this is only called when the VM is
* created, which can only be done once.
*/
int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
{
- phys_addr_t pgd_phys;
- pgd_t *pgd;
- int cpu;
+ int cpu, err;
+ struct kvm_pgtable *pgt;
- if (mmu->pgd != NULL) {
+ if (mmu->pgt != NULL) {
kvm_err("kvm_arch already initialized?\n");
return -EINVAL;
}
- /* Allocate the HW PGD, making sure that each page gets its own refcount */
- pgd = alloc_pages_exact(stage2_pgd_size(kvm), GFP_KERNEL | __GFP_ZERO);
- if (!pgd)
+ pgt = kzalloc(sizeof(*pgt), GFP_KERNEL);
+ if (!pgt)
return -ENOMEM;
- pgd_phys = virt_to_phys(pgd);
- if (WARN_ON(pgd_phys & ~kvm_vttbr_baddr_mask(kvm)))
- return -EINVAL;
+ err = kvm_pgtable_stage2_init(pgt, kvm);
+ if (err)
+ goto out_free_pgtable;
mmu->last_vcpu_ran = alloc_percpu(typeof(*mmu->last_vcpu_ran));
if (!mmu->last_vcpu_ran) {
- free_pages_exact(pgd, stage2_pgd_size(kvm));
- return -ENOMEM;
+ err = -ENOMEM;
+ goto out_destroy_pgtable;
}
for_each_possible_cpu(cpu)
*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
mmu->kvm = kvm;
- mmu->pgd = pgd;
- mmu->pgd_phys = pgd_phys;
+ mmu->pgt = pgt;
+ mmu->pgd_phys = __pa(pgt->pgd);
+ mmu->pgd = (void *)pgt->pgd;
mmu->vmid.vmid_gen = 0;
-
return 0;
+
+out_destroy_pgtable:
+ kvm_pgtable_stage2_destroy(pgt);
+out_free_pgtable:
+ kfree(pgt);
+ return err;
}
static void stage2_unmap_memslot(struct kvm *kvm,
@@ -781,20 +783,21 @@ void stage2_unmap_vm(struct kvm *kvm)
void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
{
struct kvm *kvm = mmu->kvm;
- void *pgd = NULL;
+ struct kvm_pgtable *pgt = NULL;
spin_lock(&kvm->mmu_lock);
- if (mmu->pgd) {
- unmap_stage2_range(mmu, 0, kvm_phys_size(kvm));
- pgd = READ_ONCE(mmu->pgd);
+ pgt = mmu->pgt;
+ if (pgt) {
mmu->pgd = NULL;
+ mmu->pgd_phys = 0;
+ mmu->pgt = NULL;
+ free_percpu(mmu->last_vcpu_ran);
}
spin_unlock(&kvm->mmu_lock);
- /* Free the HW pgd, one page at a time */
- if (pgd) {
- free_pages_exact(pgd, stage2_pgd_size(kvm));
- free_percpu(mmu->last_vcpu_ran);
+ if (pgt) {
+ kvm_pgtable_stage2_destroy(pgt);
+ kfree(pgt);
}
}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (4 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-10 11:20 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API Will Deacon
` (16 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Add stage-2 map() and unmap() operations to the generic page-table code.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 39 ++++
arch/arm64/kvm/hyp/pgtable.c | 273 +++++++++++++++++++++++++++
2 files changed, 312 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 85078bb632bb..7258966d3daa 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -136,6 +136,45 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm);
*/
void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
+/**
+ * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address at which to place the mapping.
+ * @size: Size of the mapping.
+ * @phys: Physical address of the memory to map.
+ * @prot: Permissions and attributes for the mapping.
+ * @mc: Cache of pre-allocated GFP_PGTABLE_USER memory from which to
+ * allocate page-table pages.
+ *
+ * If device attributes are not explicitly requested in @prot, then the
+ * mapping will be normal, cacheable.
+ *
+ * Note that this function will both coalesce existing table entries and split
+ * existing block mappings, relying on page-faults to fault back areas outside
+ * of the new mapping lazily.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
+ u64 phys, enum kvm_pgtable_prot prot,
+ struct kvm_mmu_memory_cache *mc);
+
+/**
+ * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address from which to remove the mapping.
+ * @size: Size of the mapping.
+ *
+ * TLB invalidation is performed for each page-table entry cleared during the
+ * unmapping operation and the reference count for the page-table page
+ * containing the cleared entry is decremented, with unreferenced pages being
+ * freed. Unmapping a cacheable page will ensure that it is clean to the PoC if
+ * FWB is not supported by the CPU.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
+
/**
* kvm_pgtable_walk() - Walk a page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_*_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 96e21017830b..4623380cf9de 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -32,10 +32,19 @@
#define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3
#define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10)
+#define KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR GENMASK(5, 2)
+#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R BIT(6)
+#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W BIT(7)
+#define KVM_PTE_LEAF_ATTR_LO_S2_SH GENMASK(9, 8)
+#define KVM_PTE_LEAF_ATTR_LO_S2_SH_IS 3
+#define KVM_PTE_LEAF_ATTR_LO_S2_AF BIT(10)
+
#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)
+#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54)
+
struct kvm_pgtable_walk_data {
struct kvm_pgtable *pgt;
struct kvm_pgtable_walker *walker;
@@ -417,6 +426,270 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
pgt->pgd = NULL;
}
+struct stage2_map_data {
+ u64 phys;
+ kvm_pte_t attr;
+
+ kvm_pte_t *anchor;
+
+ struct kvm_s2_mmu *mmu;
+ struct kvm_mmu_memory_cache *memcache;
+};
+
+static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot,
+ struct stage2_map_data *data)
+{
+ bool device = prot & KVM_PGTABLE_PROT_DEVICE;
+ kvm_pte_t attr = device ? PAGE_S2_MEMATTR(DEVICE_nGnRE) :
+ PAGE_S2_MEMATTR(NORMAL);
+ u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS;
+
+ if (!(prot & KVM_PGTABLE_PROT_X))
+ attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
+ else if (device)
+ return -EINVAL;
+
+ if (prot & KVM_PGTABLE_PROT_R)
+ attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
+
+ if (prot & KVM_PGTABLE_PROT_W)
+ attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
+
+ attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh);
+ attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF;
+ data->attr = attr;
+ return 0;
+}
+
+static bool stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level,
+ kvm_pte_t *ptep,
+ struct stage2_map_data *data)
+{
+ u64 granule = kvm_granule_size(level), phys = data->phys;
+
+ if (!kvm_block_mapping_supported(addr, end, phys, level))
+ return false;
+
+ if (kvm_set_valid_leaf_pte(ptep, phys, data->attr, level))
+ goto out;
+
+ /* There's an existing valid leaf entry, so perform break-before-make */
+ kvm_set_invalid_pte(ptep);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
+ kvm_set_valid_leaf_pte(ptep, phys, data->attr, level);
+out:
+ data->phys += granule;
+ return true;
+}
+
+static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level,
+ kvm_pte_t *ptep,
+ struct stage2_map_data *data)
+{
+ if (data->anchor)
+ return 0;
+
+ if (!kvm_block_mapping_supported(addr, end, data->phys, level))
+ return 0;
+
+ kvm_set_invalid_pte(ptep);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, 0);
+ data->anchor = ptep;
+ return 0;
+}
+
+static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ struct stage2_map_data *data)
+{
+ kvm_pte_t *childp, pte = *ptep;
+ struct page *page = virt_to_page(ptep);
+
+ if (data->anchor) {
+ if (kvm_pte_valid(pte))
+ put_page(page);
+
+ return 0;
+ }
+
+ if (stage2_map_walker_try_leaf(addr, end, level, ptep, data))
+ goto out_get_page;
+
+ if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1))
+ return -EINVAL;
+
+ if (!data->memcache)
+ return -ENOMEM;
+
+ childp = kvm_mmu_memory_cache_alloc(data->memcache);
+ if (!childp)
+ return -ENOMEM;
+
+ /*
+ * If we've run into an existing block mapping then replace it with
+ * a table. Accesses beyond 'end' that fall within the new table
+ * will be mapped lazily.
+ */
+ if (kvm_pte_valid(pte)) {
+ kvm_set_invalid_pte(ptep);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
+ put_page(page);
+ }
+
+ kvm_set_table_pte(ptep, childp);
+
+out_get_page:
+ get_page(page);
+ return 0;
+}
+
+static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level,
+ kvm_pte_t *ptep,
+ struct stage2_map_data *data)
+{
+ int ret = 0;
+
+ if (!data->anchor)
+ return 0;
+
+ free_page((unsigned long)kvm_pte_follow(*ptep));
+ put_page(virt_to_page(ptep));
+
+ if (data->anchor == ptep) {
+ data->anchor = NULL;
+ ret = stage2_map_walk_leaf(addr, end, level, ptep, data);
+ }
+
+ return ret;
+}
+
+/*
+ * This is a little fiddly, as we use all three of the walk flags. The idea
+ * is that the TABLE_PRE callback runs for table entries on the way down,
+ * looking for table entries which we could conceivably replace with a
+ * block entry for this mapping. If it finds one, then it sets the 'anchor'
+ * field in 'struct stage2_map_data' to point at the table entry, before
+ * clearing the entry to zero and descending into the now detached table.
+ *
+ * The behaviour of the LEAF callback then depends on whether or not the
+ * anchor has been set. If not, then we're not using a block mapping higher
+ * up the table and we perform the mapping at the existing leaves instead.
+ * If, on the other hand, the anchor _is_ set, then we drop references to
+ * all valid leaves so that the pages beneath the anchor can be freed.
+ *
+ * Finally, the TABLE_POST callback does nothing if the anchor has not
+ * been set, but otherwise frees the page-table pages while walking back up
+ * the page-table, installing the block entry when it revisits the anchor
+ * pointer and clearing the anchor to NULL.
+ */
+static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag, void * const arg)
+{
+ struct stage2_map_data *data = arg;
+
+ switch (flag) {
+ case KVM_PGTABLE_WALK_TABLE_PRE:
+ return stage2_map_walk_table_pre(addr, end, level, ptep, data);
+ case KVM_PGTABLE_WALK_LEAF:
+ return stage2_map_walk_leaf(addr, end, level, ptep, data);
+ case KVM_PGTABLE_WALK_TABLE_POST:
+ return stage2_map_walk_table_post(addr, end, level, ptep, data);
+ }
+
+ return -EINVAL;
+}
+
+int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
+ u64 phys, enum kvm_pgtable_prot prot,
+ struct kvm_mmu_memory_cache *mc)
+{
+ int ret;
+ struct stage2_map_data map_data = {
+ .phys = ALIGN_DOWN(phys, PAGE_SIZE),
+ .mmu = pgt->mmu,
+ .memcache = mc,
+ };
+ struct kvm_pgtable_walker walker = {
+ .cb = stage2_map_walker,
+ .flags = KVM_PGTABLE_WALK_TABLE_PRE |
+ KVM_PGTABLE_WALK_LEAF |
+ KVM_PGTABLE_WALK_TABLE_POST,
+ .arg = &map_data,
+ };
+
+ ret = stage2_map_set_prot_attr(prot, &map_data);
+ if (ret)
+ return ret;
+
+ ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+ dsb(ishst);
+ return ret;
+}
+
+static void stage2_flush_dcache(void *addr, u64 size)
+{
+ if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB))
+ return;
+
+ __flush_dcache_area(addr, size);
+}
+
+static bool stage2_pte_cacheable(kvm_pte_t pte)
+{
+ u64 memattr = FIELD_GET(KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR, pte);
+ return memattr == PAGE_S2_MEMATTR(NORMAL);
+}
+
+static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag,
+ void * const arg)
+{
+ struct kvm_s2_mmu *mmu = arg;
+ kvm_pte_t pte = *ptep, *childp = NULL;
+ bool need_flush = false;
+
+ if (!kvm_pte_valid(pte))
+ return 0;
+
+ if (kvm_pte_table(pte, level)) {
+ childp = kvm_pte_follow(pte);
+
+ if (page_count(virt_to_page(childp)) != 1)
+ return 0;
+ } else if (stage2_pte_cacheable(pte)) {
+ need_flush = true;
+ }
+
+ /*
+ * This is similar to the map() path in that we unmap the entire
+ * block entry and rely on the remaining portions being faulted
+ * back lazily.
+ */
+ kvm_set_invalid_pte(ptep);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, addr, level);
+ put_page(virt_to_page(ptep));
+
+ if (need_flush) {
+ stage2_flush_dcache(kvm_pte_follow(pte),
+ kvm_granule_size(level));
+ }
+
+ if (childp)
+ free_page((unsigned long)childp);
+
+ return 0;
+}
+
+int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+ struct kvm_pgtable_walker walker = {
+ .cb = stage2_unmap_walker,
+ .arg = pgt->mmu,
+ .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
+ };
+
+ return kvm_pgtable_walk(pgt, addr, size, &walker);
+}
+
int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm)
{
size_t pgd_sz;
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table
2020-09-07 15:23 ` [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table Will Deacon
@ 2020-09-10 11:20 ` Alexandru Elisei
2020-09-10 12:34 ` Will Deacon
0 siblings, 1 reply; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-10 11:20 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
On 9/7/20 4:23 PM, Will Deacon wrote:
> Add stage-2 map() and unmap() operations to the generic page-table code.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/include/asm/kvm_pgtable.h | 39 ++++
> arch/arm64/kvm/hyp/pgtable.c | 273 +++++++++++++++++++++++++++
> 2 files changed, 312 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> index 85078bb632bb..7258966d3daa 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -136,6 +136,45 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm);
> */
> void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
>
> +/**
> + * kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address at which to place the mapping.
> + * @size: Size of the mapping.
> + * @phys: Physical address of the memory to map.
> + * @prot: Permissions and attributes for the mapping.
> + * @mc: Cache of pre-allocated GFP_PGTABLE_USER memory from which to
> + * allocate page-table pages.
> + *
> + * If device attributes are not explicitly requested in @prot, then the
> + * mapping will be normal, cacheable.
> + *
> + * Note that this function will both coalesce existing table entries and split
> + * existing block mappings, relying on page-faults to fault back areas outside
> + * of the new mapping lazily.
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
> + u64 phys, enum kvm_pgtable_prot prot,
> + struct kvm_mmu_memory_cache *mc);
> +
> +/**
> + * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address from which to remove the mapping.
> + * @size: Size of the mapping.
> + *
> + * TLB invalidation is performed for each page-table entry cleared during the
> + * unmapping operation and the reference count for the page-table page
> + * containing the cleared entry is decremented, with unreferenced pages being
> + * freed. Unmapping a cacheable page will ensure that it is clean to the PoC if
> + * FWB is not supported by the CPU.
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
> +
> /**
> * kvm_pgtable_walk() - Walk a page-table.
> * @pgt: Page-table structure initialised by kvm_pgtable_*_init().
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index 96e21017830b..4623380cf9de 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -32,10 +32,19 @@
> #define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3
> #define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10)
>
> +#define KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR GENMASK(5, 2)
> +#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R BIT(6)
> +#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W BIT(7)
> +#define KVM_PTE_LEAF_ATTR_LO_S2_SH GENMASK(9, 8)
> +#define KVM_PTE_LEAF_ATTR_LO_S2_SH_IS 3
> +#define KVM_PTE_LEAF_ATTR_LO_S2_AF BIT(10)
> +
> #define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51)
>
> #define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)
>
> +#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54)
> +
> struct kvm_pgtable_walk_data {
> struct kvm_pgtable *pgt;
> struct kvm_pgtable_walker *walker;
> @@ -417,6 +426,270 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt)
> pgt->pgd = NULL;
> }
>
> +struct stage2_map_data {
> + u64 phys;
> + kvm_pte_t attr;
> +
> + kvm_pte_t *anchor;
> +
> + struct kvm_s2_mmu *mmu;
> + struct kvm_mmu_memory_cache *memcache;
> +};
> +
> +static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot,
> + struct stage2_map_data *data)
> +{
> + bool device = prot & KVM_PGTABLE_PROT_DEVICE;
> + kvm_pte_t attr = device ? PAGE_S2_MEMATTR(DEVICE_nGnRE) :
> + PAGE_S2_MEMATTR(NORMAL);
> + u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS;
> +
> + if (!(prot & KVM_PGTABLE_PROT_X))
> + attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
> + else if (device)
> + return -EINVAL;
> +
> + if (prot & KVM_PGTABLE_PROT_R)
> + attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
> +
> + if (prot & KVM_PGTABLE_PROT_W)
> + attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
> +
> + attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh);
> + attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF;
> + data->attr = attr;
> + return 0;
> +}
> +
> +static bool stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level,
> + kvm_pte_t *ptep,
> + struct stage2_map_data *data)
> +{
> + u64 granule = kvm_granule_size(level), phys = data->phys;
> +
> + if (!kvm_block_mapping_supported(addr, end, phys, level))
> + return false;
> +
> + if (kvm_set_valid_leaf_pte(ptep, phys, data->attr, level))
> + goto out;
> +
> + /* There's an existing valid leaf entry, so perform break-before-make */
> + kvm_set_invalid_pte(ptep);
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
> + kvm_set_valid_leaf_pte(ptep, phys, data->attr, level);
> +out:
> + data->phys += granule;
> + return true;
> +}
> +
> +static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level,
> + kvm_pte_t *ptep,
> + struct stage2_map_data *data)
> +{
> + if (data->anchor)
> + return 0;
> +
> + if (!kvm_block_mapping_supported(addr, end, data->phys, level))
> + return 0;
> +
> + kvm_set_invalid_pte(ptep);
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, 0);
> + data->anchor = ptep;
> + return 0;
> +}
> +
> +static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
> + struct stage2_map_data *data)
> +{
> + kvm_pte_t *childp, pte = *ptep;
> + struct page *page = virt_to_page(ptep);
> +
> + if (data->anchor) {
> + if (kvm_pte_valid(pte))
> + put_page(page);
> +
> + return 0;
> + }
> +
> + if (stage2_map_walker_try_leaf(addr, end, level, ptep, data))
> + goto out_get_page;
> +
> + if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1))
> + return -EINVAL;
> +
> + if (!data->memcache)
> + return -ENOMEM;
> +
> + childp = kvm_mmu_memory_cache_alloc(data->memcache);
> + if (!childp)
> + return -ENOMEM;
> +
> + /*
> + * If we've run into an existing block mapping then replace it with
> + * a table. Accesses beyond 'end' that fall within the new table
> + * will be mapped lazily.
> + */
> + if (kvm_pte_valid(pte)) {
> + kvm_set_invalid_pte(ptep);
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level);
> + put_page(page);
> + }
> +
> + kvm_set_table_pte(ptep, childp);
> +
> +out_get_page:
> + get_page(page);
> + return 0;
> +}
> +
> +static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level,
> + kvm_pte_t *ptep,
> + struct stage2_map_data *data)
> +{
> + int ret = 0;
> +
> + if (!data->anchor)
> + return 0;
> +
> + free_page((unsigned long)kvm_pte_follow(*ptep));
> + put_page(virt_to_page(ptep));
> +
> + if (data->anchor == ptep) {
> + data->anchor = NULL;
> + ret = stage2_map_walk_leaf(addr, end, level, ptep, data);
> + }
I had another look at this function. If we're back to the anchor entry, then that
means that we know from the pre-order visitor that 1. the mapping is supported at
this level and 2. that the pte was invalidated. This means that
kvm_set_valid_leaf_pte() will succeed in changing the entry. How about instead of
calling stage2_map_walk_leaf() -> stage2_map_walker_try_leaf() ->
kvm_set_valid_leaf_pte() we call kvm_set_valid_leaf_pte() directly, followed by
get_page(virt_to_page(ptep)? It would make the code a lot easier to follow
(stage2_map_walk_leaf() is pretty complicated, imo, but that can't really be
avoided), and also slightly faster.
> +
> + return ret;
> +}
> +
> +/*
> + * This is a little fiddly, as we use all three of the walk flags. The idea
> + * is that the TABLE_PRE callback runs for table entries on the way down,
> + * looking for table entries which we could conceivably replace with a
> + * block entry for this mapping. If it finds one, then it sets the 'anchor'
> + * field in 'struct stage2_map_data' to point at the table entry, before
> + * clearing the entry to zero and descending into the now detached table.
> + *
> + * The behaviour of the LEAF callback then depends on whether or not the
> + * anchor has been set. If not, then we're not using a block mapping higher
> + * up the table and we perform the mapping at the existing leaves instead.
> + * If, on the other hand, the anchor _is_ set, then we drop references to
> + * all valid leaves so that the pages beneath the anchor can be freed.
> + *
> + * Finally, the TABLE_POST callback does nothing if the anchor has not
> + * been set, but otherwise frees the page-table pages while walking back up
> + * the page-table, installing the block entry when it revisits the anchor
> + * pointer and clearing the anchor to NULL.
> + */
The comment does wonders at explaining what is going on, thank you.
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table
2020-09-10 11:20 ` Alexandru Elisei
@ 2020-09-10 12:34 ` Will Deacon
2020-09-10 13:55 ` Alexandru Elisei
0 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-10 12:34 UTC (permalink / raw)
To: Alexandru Elisei
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, kvmarm,
linux-arm-kernel
On Thu, Sep 10, 2020 at 12:20:42PM +0100, Alexandru Elisei wrote:
> On 9/7/20 4:23 PM, Will Deacon wrote:
> > +static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level,
> > + kvm_pte_t *ptep,
> > + struct stage2_map_data *data)
> > +{
> > + int ret = 0;
> > +
> > + if (!data->anchor)
> > + return 0;
> > +
> > + free_page((unsigned long)kvm_pte_follow(*ptep));
> > + put_page(virt_to_page(ptep));
> > +
> > + if (data->anchor == ptep) {
> > + data->anchor = NULL;
> > + ret = stage2_map_walk_leaf(addr, end, level, ptep, data);
> > + }
>
> I had another look at this function. If we're back to the anchor entry, then that
> means that we know from the pre-order visitor that 1. the mapping is supported at
> this level and 2. that the pte was invalidated. This means that
> kvm_set_valid_leaf_pte() will succeed in changing the entry. How about instead of
> calling stage2_map_walk_leaf() -> stage2_map_walker_try_leaf() ->
> kvm_set_valid_leaf_pte() we call kvm_set_valid_leaf_pte() directly, followed by
> get_page(virt_to_page(ptep)? It would make the code a lot easier to follow
> (stage2_map_walk_leaf() is pretty complicated, imo, but that can't really be
> avoided), and also slightly faster.
I'm not sure I agree. I would consider kvm_set_valid_leaf_pte() to be lower
level, and not the right function to be calling here. Rather,
stage2_map_walk_leaf() is what would have been called if we hadn't spotted
the potential for a block mapping anyway, so I much prefer that symmetry. It
also means that if stage2_map_walk_leaf() grows some extra logic in future
that we need to take into account here, we won't miss anything.
> > +/*
> > + * This is a little fiddly, as we use all three of the walk flags. The idea
> > + * is that the TABLE_PRE callback runs for table entries on the way down,
> > + * looking for table entries which we could conceivably replace with a
> > + * block entry for this mapping. If it finds one, then it sets the 'anchor'
> > + * field in 'struct stage2_map_data' to point at the table entry, before
> > + * clearing the entry to zero and descending into the now detached table.
> > + *
> > + * The behaviour of the LEAF callback then depends on whether or not the
> > + * anchor has been set. If not, then we're not using a block mapping higher
> > + * up the table and we perform the mapping at the existing leaves instead.
> > + * If, on the other hand, the anchor _is_ set, then we drop references to
> > + * all valid leaves so that the pages beneath the anchor can be freed.
> > + *
> > + * Finally, the TABLE_POST callback does nothing if the anchor has not
> > + * been set, but otherwise frees the page-table pages while walking back up
> > + * the page-table, installing the block entry when it revisits the anchor
> > + * pointer and clearing the anchor to NULL.
> > + */
>
> The comment does wonders at explaining what is going on, thank you.
Glad it helps!
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table
2020-09-10 12:34 ` Will Deacon
@ 2020-09-10 13:55 ` Alexandru Elisei
0 siblings, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-10 13:55 UTC (permalink / raw)
To: Will Deacon
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, kvmarm,
linux-arm-kernel
Hi Will,
On 9/10/20 1:34 PM, Will Deacon wrote:
> On Thu, Sep 10, 2020 at 12:20:42PM +0100, Alexandru Elisei wrote:
>> On 9/7/20 4:23 PM, Will Deacon wrote:
>>> +static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level,
>>> + kvm_pte_t *ptep,
>>> + struct stage2_map_data *data)
>>> +{
>>> + int ret = 0;
>>> +
>>> + if (!data->anchor)
>>> + return 0;
>>> +
>>> + free_page((unsigned long)kvm_pte_follow(*ptep));
>>> + put_page(virt_to_page(ptep));
>>> +
>>> + if (data->anchor == ptep) {
>>> + data->anchor = NULL;
>>> + ret = stage2_map_walk_leaf(addr, end, level, ptep, data);
>>> + }
>> I had another look at this function. If we're back to the anchor entry, then that
>> means that we know from the pre-order visitor that 1. the mapping is supported at
>> this level and 2. that the pte was invalidated. This means that
>> kvm_set_valid_leaf_pte() will succeed in changing the entry. How about instead of
>> calling stage2_map_walk_leaf() -> stage2_map_walker_try_leaf() ->
>> kvm_set_valid_leaf_pte() we call kvm_set_valid_leaf_pte() directly, followed by
>> get_page(virt_to_page(ptep)? It would make the code a lot easier to follow
>> (stage2_map_walk_leaf() is pretty complicated, imo, but that can't really be
>> avoided), and also slightly faster.
> I'm not sure I agree. I would consider kvm_set_valid_leaf_pte() to be lower
> level, and not the right function to be calling here. Rather,
> stage2_map_walk_leaf() is what would have been called if we hadn't spotted
> the potential for a block mapping anyway, so I much prefer that symmetry. It
> also means that if stage2_map_walk_leaf() grows some extra logic in future
> that we need to take into account here, we won't miss anything.
Sure, that makes sense.
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (5 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 08/21] KVM: arm64: Convert kvm_set_spte_hva() " Will Deacon
` (15 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Convert kvm_phys_addr_ioremap() to use kvm_pgtable_stage2_map() instead
of stage2_set_pte().
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 29 ++++++++++++-----------------
1 file changed, 12 insertions(+), 17 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 4607e9ca60a2..33146d3dc93a 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1154,35 +1154,30 @@ static int stage2_pudp_test_and_clear_young(pud_t *pud)
int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
phys_addr_t pa, unsigned long size, bool writable)
{
- phys_addr_t addr, end;
+ phys_addr_t addr;
int ret = 0;
- unsigned long pfn;
struct kvm_mmu_memory_cache cache = { 0, __GFP_ZERO, NULL, };
+ struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
+ enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE |
+ KVM_PGTABLE_PROT_R |
+ (writable ? KVM_PGTABLE_PROT_W : 0);
- end = (guest_ipa + size + PAGE_SIZE - 1) & PAGE_MASK;
- pfn = __phys_to_pfn(pa);
-
- for (addr = guest_ipa; addr < end; addr += PAGE_SIZE) {
- pte_t pte = kvm_pfn_pte(pfn, PAGE_S2_DEVICE);
-
- if (writable)
- pte = kvm_s2pte_mkwrite(pte);
-
+ for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) {
ret = kvm_mmu_topup_memory_cache(&cache,
kvm_mmu_cache_min_pages(kvm));
if (ret)
- goto out;
+ break;
+
spin_lock(&kvm->mmu_lock);
- ret = stage2_set_pte(&kvm->arch.mmu, &cache, addr, &pte,
- KVM_S2PTE_FLAG_IS_IOMAP);
+ ret = kvm_pgtable_stage2_map(pgt, addr, PAGE_SIZE, pa, prot,
+ &cache);
spin_unlock(&kvm->mmu_lock);
if (ret)
- goto out;
+ break;
- pfn++;
+ pa += PAGE_SIZE;
}
-out:
kvm_mmu_free_memory_cache(&cache);
return ret;
}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 08/21] KVM: arm64: Convert kvm_set_spte_hva() to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (6 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 09/21] KVM: arm64: Convert unmap_stage2_range() " Will Deacon
` (14 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Convert kvm_set_spte_hva() to use kvm_pgtable_stage2_map() instead
of stage2_set_pte().
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 23 ++++++++++-------------
1 file changed, 10 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 33146d3dc93a..704b471a48ce 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1911,28 +1911,27 @@ int kvm_unmap_hva_range(struct kvm *kvm,
static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
{
- pte_t *pte = (pte_t *)data;
+ kvm_pfn_t *pfn = (kvm_pfn_t *)data;
WARN_ON(size != PAGE_SIZE);
+
/*
- * We can always call stage2_set_pte with KVM_S2PTE_FLAG_LOGGING_ACTIVE
- * flag clear because MMU notifiers will have unmapped a huge PMD before
- * calling ->change_pte() (which in turn calls kvm_set_spte_hva()) and
- * therefore stage2_set_pte() never needs to clear out a huge PMD
- * through this calling path.
+ * The MMU notifiers will have unmapped a huge PMD before calling
+ * ->change_pte() (which in turn calls kvm_set_spte_hva()) and
+ * therefore we never need to clear out a huge PMD through this
+ * calling path and a memcache is not required.
*/
- stage2_set_pte(&kvm->arch.mmu, NULL, gpa, pte, 0);
+ kvm_pgtable_stage2_map(kvm->arch.mmu.pgt, gpa, PAGE_SIZE,
+ __pfn_to_phys(*pfn), KVM_PGTABLE_PROT_R, NULL);
return 0;
}
-
int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
{
unsigned long end = hva + PAGE_SIZE;
kvm_pfn_t pfn = pte_pfn(pte);
- pte_t stage2_pte;
- if (!kvm->arch.mmu.pgd)
+ if (!kvm->arch.mmu.pgt)
return 0;
trace_kvm_set_spte_hva(hva);
@@ -1942,9 +1941,7 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
* just like a translation fault and clean the cache to the PoC.
*/
clean_dcache_guest_page(pfn, PAGE_SIZE);
- stage2_pte = kvm_pfn_pte(pfn, PAGE_S2);
- handle_hva_to_gpa(kvm, hva, end, &kvm_set_spte_handler, &stage2_pte);
-
+ handle_hva_to_gpa(kvm, hva, end, &kvm_set_spte_handler, &pfn);
return 0;
}
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 09/21] KVM: arm64: Convert unmap_stage2_range() to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (7 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 08/21] KVM: arm64: Convert kvm_set_spte_hva() " Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Will Deacon
` (13 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Convert unmap_stage2_range() to use kvm_pgtable_stage2_unmap() instead
of walking the page-table directly.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 62 ++++++++++++++++++++++++++------------------
1 file changed, 37 insertions(+), 25 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 704b471a48ce..a7021509231c 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -39,6 +39,38 @@ static bool is_iomap(unsigned long flags)
return flags & KVM_S2PTE_FLAG_IS_IOMAP;
}
+/*
+ * Release kvm_mmu_lock periodically if the memory region is large. Otherwise,
+ * we may see kernel panics with CONFIG_DETECT_HUNG_TASK,
+ * CONFIG_LOCKUP_DETECTOR, CONFIG_LOCKDEP. Additionally, holding the lock too
+ * long will also starve other vCPUs. We have to also make sure that the page
+ * tables are not freed while we released the lock.
+ */
+static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,
+ phys_addr_t end,
+ int (*fn)(struct kvm_pgtable *, u64, u64),
+ bool resched)
+{
+ int ret;
+ u64 next;
+
+ do {
+ struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
+ if (!pgt)
+ return -EINVAL;
+
+ next = stage2_pgd_addr_end(kvm, addr, end);
+ ret = fn(pgt, addr, next - addr);
+ if (ret)
+ break;
+
+ if (resched && next != end)
+ cond_resched_lock(&kvm->mmu_lock);
+ } while (addr = next, addr != end);
+
+ return ret;
+}
+
static bool memslot_is_logging(struct kvm_memory_slot *memslot)
{
return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY);
@@ -220,8 +252,8 @@ static inline void kvm_pgd_populate(pgd_t *pgdp, p4d_t *p4dp)
* end up writing old data to disk.
*
* This is why right after unmapping a page/section and invalidating
- * the corresponding TLBs, we call kvm_flush_dcache_p*() to make sure
- * the IO subsystem will never hit in the cache.
+ * the corresponding TLBs, we flush to make sure the IO subsystem will
+ * never hit in the cache.
*
* This is all avoided on systems that have ARM64_HAS_STAGE2_FWB, as
* we then fully enforce cacheability of RAM, no matter what the guest
@@ -344,32 +376,12 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
bool may_block)
{
struct kvm *kvm = mmu->kvm;
- pgd_t *pgd;
- phys_addr_t addr = start, end = start + size;
- phys_addr_t next;
+ phys_addr_t end = start + size;
assert_spin_locked(&kvm->mmu_lock);
WARN_ON(size & ~PAGE_MASK);
-
- pgd = mmu->pgd + stage2_pgd_index(kvm, addr);
- do {
- /*
- * Make sure the page table is still active, as another thread
- * could have possibly freed the page table, while we released
- * the lock.
- */
- if (!READ_ONCE(mmu->pgd))
- break;
- next = stage2_pgd_addr_end(kvm, addr, end);
- if (!stage2_pgd_none(kvm, *pgd))
- unmap_stage2_p4ds(mmu, pgd, addr, next);
- /*
- * If the range is too large, release the kvm->mmu_lock
- * to prevent starvation and lockup detector warnings.
- */
- if (may_block && next != end)
- cond_resched_lock(&kvm->mmu_lock);
- } while (pgd++, addr = next, addr != end);
+ WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap,
+ may_block));
}
static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (8 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 09/21] KVM: arm64: Convert unmap_stage2_range() " Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 15:30 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API Will Deacon
` (12 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Add stage-2 mkyoung(), mkold() and is_young() operations to the generic
page-table code.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 38 ++++++++++++
arch/arm64/kvm/hyp/pgtable.c | 86 ++++++++++++++++++++++++++++
2 files changed, 124 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 7258966d3daa..01cad6bbc81b 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -175,6 +175,44 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
*/
int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
+/**
+ * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address to identify the page-table entry.
+ *
+ * If there is a valid, leaf page-table entry used to translate @addr, then
+ * set the access flag in that entry.
+ *
+ * Return: The old page-table entry prior to setting the flag, 0 on failure.
+ */
+kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);
+
+/**
+ * kvm_pgtable_stage2_mkold() - Clear the access flag in a page-table entry.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address to identify the page-table entry.
+ *
+ * If there is a valid, leaf page-table entry used to translate @addr, then
+ * clear the access flag in that entry.
+ *
+ * Note that it is the caller's responsibility to invalidate the TLB after
+ * calling this function to ensure that the updated permissions are visible
+ * to the CPUs.
+ *
+ * Return: The old page-table entry prior to clearing the flag, 0 on failure.
+ */
+kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr);
+
+/**
+ * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the
+ * access flag set.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address to identify the page-table entry.
+ *
+ * Return: True if the page-table entry has the access flag set, false otherwise.
+ */
+bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
+
/**
* kvm_pgtable_walk() - Walk a page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_*_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 4623380cf9de..87ff56d8bcb5 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -690,6 +690,92 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
return kvm_pgtable_walk(pgt, addr, size, &walker);
}
+struct stage2_attr_data {
+ kvm_pte_t attr_set;
+ kvm_pte_t attr_clr;
+ kvm_pte_t pte;
+};
+
+static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag,
+ void * const arg)
+{
+ kvm_pte_t pte = *ptep;
+ struct stage2_attr_data *data = arg;
+
+ if (!kvm_pte_valid(pte))
+ return 0;
+
+ data->pte = pte;
+ pte &= ~data->attr_clr;
+ pte |= data->attr_set;
+
+ /*
+ * We may race with the CPU trying to set the access flag here,
+ * but worst-case the access flag update gets lost and will be
+ * set on the next access instead.
+ */
+ if (data->pte != pte)
+ WRITE_ONCE(*ptep, pte);
+
+ return 0;
+}
+
+static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr,
+ u64 size, kvm_pte_t attr_set,
+ kvm_pte_t attr_clr, kvm_pte_t *orig_pte)
+{
+ int ret;
+ kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI;
+ struct stage2_attr_data data = {
+ .attr_set = attr_set & attr_mask,
+ .attr_clr = attr_clr & attr_mask,
+ };
+ struct kvm_pgtable_walker walker = {
+ .cb = stage2_attr_walker,
+ .arg = &data,
+ .flags = KVM_PGTABLE_WALK_LEAF,
+ };
+
+ ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+ if (ret)
+ return ret;
+
+ if (orig_pte)
+ *orig_pte = data.pte;
+ return 0;
+}
+
+kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
+{
+ kvm_pte_t pte = 0;
+ stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
+ &pte);
+ dsb(ishst);
+ return pte;
+}
+
+kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr)
+{
+ kvm_pte_t pte = 0;
+ stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF,
+ &pte);
+ /*
+ * "But where's the TLBI?!", you scream.
+ * "Over in the core code", I sigh.
+ *
+ * See the '->clear_flush_young()' callback on the KVM mmu notifier.
+ */
+ return pte;
+}
+
+bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr)
+{
+ kvm_pte_t pte = 0;
+ stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte);
+ return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF;
+}
+
int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm)
{
size_t pgd_sz;
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table
2020-09-07 15:23 ` [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Will Deacon
@ 2020-09-08 15:30 ` Alexandru Elisei
2020-09-10 12:42 ` Will Deacon
0 siblings, 1 reply; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-08 15:30 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
The patch looks good to me, I have a question below.
On 9/7/20 4:23 PM, Will Deacon wrote:
> Add stage-2 mkyoung(), mkold() and is_young() operations to the generic
> page-table code.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/include/asm/kvm_pgtable.h | 38 ++++++++++++
> arch/arm64/kvm/hyp/pgtable.c | 86 ++++++++++++++++++++++++++++
> 2 files changed, 124 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> index 7258966d3daa..01cad6bbc81b 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -175,6 +175,44 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
> */
> int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
>
> +/**
> + * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address to identify the page-table entry.
> + *
> + * If there is a valid, leaf page-table entry used to translate @addr, then
> + * set the access flag in that entry.
> + *
> + * Return: The old page-table entry prior to setting the flag, 0 on failure.
> + */
> +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);
> +
> +/**
> + * kvm_pgtable_stage2_mkold() - Clear the access flag in a page-table entry.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address to identify the page-table entry.
> + *
> + * If there is a valid, leaf page-table entry used to translate @addr, then
> + * clear the access flag in that entry.
> + *
> + * Note that it is the caller's responsibility to invalidate the TLB after
> + * calling this function to ensure that the updated permissions are visible
> + * to the CPUs.
> + *
> + * Return: The old page-table entry prior to clearing the flag, 0 on failure.
> + */
> +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr);
> +
> +/**
> + * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the
> + * access flag set.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address to identify the page-table entry.
> + *
> + * Return: True if the page-table entry has the access flag set, false otherwise.
> + */
> +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
> +
> /**
> * kvm_pgtable_walk() - Walk a page-table.
> * @pgt: Page-table structure initialised by kvm_pgtable_*_init().
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index 4623380cf9de..87ff56d8bcb5 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -690,6 +690,92 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
> return kvm_pgtable_walk(pgt, addr, size, &walker);
> }
>
> +struct stage2_attr_data {
> + kvm_pte_t attr_set;
> + kvm_pte_t attr_clr;
> + kvm_pte_t pte;
> +};
> +
> +static int stage2_attr_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
> + enum kvm_pgtable_walk_flags flag,
> + void * const arg)
> +{
> + kvm_pte_t pte = *ptep;
> + struct stage2_attr_data *data = arg;
> +
> + if (!kvm_pte_valid(pte))
> + return 0;
> +
> + data->pte = pte;
> + pte &= ~data->attr_clr;
> + pte |= data->attr_set;
> +
> + /*
> + * We may race with the CPU trying to set the access flag here,
> + * but worst-case the access flag update gets lost and will be
> + * set on the next access instead.
> + */
> + if (data->pte != pte)
> + WRITE_ONCE(*ptep, pte);
> +
> + return 0;
> +}
> +
> +static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr,
> + u64 size, kvm_pte_t attr_set,
> + kvm_pte_t attr_clr, kvm_pte_t *orig_pte)
> +{
> + int ret;
> + kvm_pte_t attr_mask = KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI;
> + struct stage2_attr_data data = {
> + .attr_set = attr_set & attr_mask,
> + .attr_clr = attr_clr & attr_mask,
> + };
> + struct kvm_pgtable_walker walker = {
> + .cb = stage2_attr_walker,
> + .arg = &data,
> + .flags = KVM_PGTABLE_WALK_LEAF,
> + };
> +
> + ret = kvm_pgtable_walk(pgt, addr, size, &walker);
> + if (ret)
> + return ret;
> +
> + if (orig_pte)
> + *orig_pte = data.pte;
> + return 0;
> +}
> +
> +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
> +{
> + kvm_pte_t pte = 0;
> + stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
> + &pte);
> + dsb(ishst);
I am curious about the DSB above. We don't do it when we clear the AF bit, because
in the clear_flush_young() mmu notifier, kvm_age_hva() is followed by
kvm_flush_remote_tlbs() -> kvm_flush_remote_tlbs(), which does a DSB ISHST.
When AF is zero, the entry is not allowed to be stored in a TLB, and that's why we
don't need to issue a TLBI instruction. Do we do the DSB here to make sure all PEs
in the inner shareability domain see the updated translation tables? Or there's
another reason I'm not seeing?
Thanks,
Alex
> + return pte;
> +}
> +
> +kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr)
> +{
> + kvm_pte_t pte = 0;
> + stage2_update_leaf_attrs(pgt, addr, 1, 0, KVM_PTE_LEAF_ATTR_LO_S2_AF,
> + &pte);
> + /*
> + * "But where's the TLBI?!", you scream.
> + * "Over in the core code", I sigh.
> + *
> + * See the '->clear_flush_young()' callback on the KVM mmu notifier.
> + */
> + return pte;
> +}
> +
> +bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr)
> +{
> + kvm_pte_t pte = 0;
> + stage2_update_leaf_attrs(pgt, addr, 1, 0, 0, &pte);
> + return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF;
> +}
> +
> int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm)
> {
> size_t pgd_sz;
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table
2020-09-08 15:30 ` Alexandru Elisei
@ 2020-09-10 12:42 ` Will Deacon
0 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-10 12:42 UTC (permalink / raw)
To: Alexandru Elisei
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, kvmarm,
linux-arm-kernel
On Tue, Sep 08, 2020 at 04:30:15PM +0100, Alexandru Elisei wrote:
> On 9/7/20 4:23 PM, Will Deacon wrote:
> > +kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
> > +{
> > + kvm_pte_t pte = 0;
> > + stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
> > + &pte);
> > + dsb(ishst);
>
> I am curious about the DSB above. We don't do it when we clear the AF bit, because
> in the clear_flush_young() mmu notifier, kvm_age_hva() is followed by
> kvm_flush_remote_tlbs() -> kvm_flush_remote_tlbs(), which does a DSB ISHST.
>
> When AF is zero, the entry is not allowed to be stored in a TLB, and that's why we
> don't need to issue a TLBI instruction. Do we do the DSB here to make sure all PEs
> in the inner shareability domain see the updated translation tables? Or there's
> another reason I'm not seeing?
You're right: the DSB is to ensure that the updated PTE is visible to the
page-table walker, as I'm not sure how well we'll handle the spurious access
fault in this case (particularly as is_young() would return true).
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (9 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 15:39 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table Will Deacon
` (11 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Convert the page-aging functions and access fault handler to use the
generic page-table code instead of walking the page-table directly.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 74 ++++++++++----------------------------------
1 file changed, 16 insertions(+), 58 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index a7021509231c..a036aac3ed24 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1703,46 +1703,23 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
return ret;
}
-/*
- * Resolve the access fault by making the page young again.
- * Note that because the faulting entry is guaranteed not to be
- * cached in the TLB, we don't need to invalidate anything.
- * Only the HW Access Flag updates are supported for Stage 2 (no DBM),
- * so there is no need for atomic (pte|pmd)_mkyoung operations.
- */
+/* Resolve the access fault by making the page young again. */
static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
{
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
- kvm_pfn_t pfn;
- bool pfn_valid = false;
+ pte_t pte;
+ kvm_pte_t kpte;
+ struct kvm_s2_mmu *mmu;
trace_kvm_access_fault(fault_ipa);
spin_lock(&vcpu->kvm->mmu_lock);
-
- if (!stage2_get_leaf_entry(vcpu->arch.hw_mmu, fault_ipa, &pud, &pmd, &pte))
- goto out;
-
- if (pud) { /* HugeTLB */
- *pud = kvm_s2pud_mkyoung(*pud);
- pfn = kvm_pud_pfn(*pud);
- pfn_valid = true;
- } else if (pmd) { /* THP, HugeTLB */
- *pmd = pmd_mkyoung(*pmd);
- pfn = pmd_pfn(*pmd);
- pfn_valid = true;
- } else {
- *pte = pte_mkyoung(*pte); /* Just a page... */
- pfn = pte_pfn(*pte);
- pfn_valid = true;
- }
-
-out:
+ mmu = vcpu->arch.hw_mmu;
+ kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
spin_unlock(&vcpu->kvm->mmu_lock);
- if (pfn_valid)
- kvm_set_pfn_accessed(pfn);
+
+ pte = __pte(kpte);
+ if (pte_valid(pte))
+ kvm_set_pfn_accessed(pte_pfn(pte));
}
/**
@@ -1959,38 +1936,19 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
{
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
+ pte_t pte;
+ kvm_pte_t kpte;
WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
- if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte))
- return 0;
-
- if (pud)
- return stage2_pudp_test_and_clear_young(pud);
- else if (pmd)
- return stage2_pmdp_test_and_clear_young(pmd);
- else
- return stage2_ptep_test_and_clear_young(pte);
+ kpte = kvm_pgtable_stage2_mkold(kvm->arch.mmu.pgt, gpa);
+ pte = __pte(kpte);
+ return pte_valid(pte) && pte_young(pte);
}
static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
{
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
-
WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
- if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte))
- return 0;
-
- if (pud)
- return kvm_s2pud_young(*pud);
- else if (pmd)
- return pmd_young(*pmd);
- else
- return pte_young(*pte);
+ return kvm_pgtable_stage2_is_young(kvm->arch.mmu.pgt, gpa);
}
int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end)
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API
2020-09-07 15:23 ` [PATCH v4 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API Will Deacon
@ 2020-09-08 15:39 ` Alexandru Elisei
0 siblings, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-08 15:39 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
On 9/7/20 4:23 PM, Will Deacon wrote:
> Convert the page-aging functions and access fault handler to use the
> generic page-table code instead of walking the page-table directly.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/kvm/mmu.c | 74 ++++++++++----------------------------------
> 1 file changed, 16 insertions(+), 58 deletions(-)
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index a7021509231c..a036aac3ed24 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1703,46 +1703,23 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> return ret;
> }
>
> -/*
> - * Resolve the access fault by making the page young again.
> - * Note that because the faulting entry is guaranteed not to be
> - * cached in the TLB, we don't need to invalidate anything.
> - * Only the HW Access Flag updates are supported for Stage 2 (no DBM),
> - * so there is no need for atomic (pte|pmd)_mkyoung operations.
> - */
> +/* Resolve the access fault by making the page young again. */
> static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
> {
> - pud_t *pud;
> - pmd_t *pmd;
> - pte_t *pte;
> - kvm_pfn_t pfn;
> - bool pfn_valid = false;
> + pte_t pte;
> + kvm_pte_t kpte;
> + struct kvm_s2_mmu *mmu;
>
> trace_kvm_access_fault(fault_ipa);
>
> spin_lock(&vcpu->kvm->mmu_lock);
> -
> - if (!stage2_get_leaf_entry(vcpu->arch.hw_mmu, fault_ipa, &pud, &pmd, &pte))
> - goto out;
> -
> - if (pud) { /* HugeTLB */
> - *pud = kvm_s2pud_mkyoung(*pud);
> - pfn = kvm_pud_pfn(*pud);
> - pfn_valid = true;
> - } else if (pmd) { /* THP, HugeTLB */
> - *pmd = pmd_mkyoung(*pmd);
> - pfn = pmd_pfn(*pmd);
> - pfn_valid = true;
> - } else {
> - *pte = pte_mkyoung(*pte); /* Just a page... */
> - pfn = pte_pfn(*pte);
> - pfn_valid = true;
> - }
> -
> -out:
> + mmu = vcpu->arch.hw_mmu;
> + kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
> spin_unlock(&vcpu->kvm->mmu_lock);
> - if (pfn_valid)
> - kvm_set_pfn_accessed(pfn);
> +
> + pte = __pte(kpte);
> + if (pte_valid(pte))
> + kvm_set_pfn_accessed(pte_pfn(pte));
> }
>
> /**
> @@ -1959,38 +1936,19 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
>
> static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
> {
> - pud_t *pud;
> - pmd_t *pmd;
> - pte_t *pte;
> + pte_t pte;
> + kvm_pte_t kpte;
>
> WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
> - if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte))
> - return 0;
> -
> - if (pud)
> - return stage2_pudp_test_and_clear_young(pud);
> - else if (pmd)
> - return stage2_pmdp_test_and_clear_young(pmd);
> - else
> - return stage2_ptep_test_and_clear_young(pte);
> + kpte = kvm_pgtable_stage2_mkold(kvm->arch.mmu.pgt, gpa);
> + pte = __pte(kpte);
> + return pte_valid(pte) && pte_young(pte);
I checked that the semantics of the return value are preserved. The old version of
the function returned 1 if stage2_get_leaf_entry() returned true (entry was found
and it was valid) and the entry was young, which is equivalent to pte_valid(pte)
&& pte_young(pte).
> }
>
> static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
> {
> - pud_t *pud;
> - pmd_t *pmd;
> - pte_t *pte;
> -
> WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
> - if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte))
> - return 0;
> -
> - if (pud)
> - return kvm_s2pud_young(*pud);
> - else if (pmd)
> - return pmd_young(*pmd);
> - else
> - return pte_young(*pte);
> + return kvm_pgtable_stage2_is_young(kvm->arch.mmu.pgt, gpa);
> }
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Thanks,
Alex
>
> int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end)
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (10 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 13/21] KVM: arm64: Convert write-protect operation to generic page-table API Will Deacon
` (10 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
From: Quentin Perret <qperret@google.com>
Add a stage-2 wrprotect() operation to the generic page-table code.
Cc: Marc Zyngier <maz@kernel.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 15 +++++++++++++++
arch/arm64/kvm/hyp/pgtable.c | 6 ++++++
2 files changed, 21 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 01cad6bbc81b..e9491c91bd06 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -175,6 +175,21 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
*/
int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
+/**
+ * kvm_pgtable_stage2_wrprotect() - Write-protect guest stage-2 address range
+ * without TLB invalidation.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address from which to write-protect,
+ * @size: Size of the range.
+ *
+ * Note that it is the caller's responsibility to invalidate the TLB after
+ * calling this function to ensure that the updated permissions are visible
+ * to the CPUs.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size);
+
/**
* kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry.
* @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 87ff56d8bcb5..fee282f8690d 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -746,6 +746,12 @@ static int stage2_update_leaf_attrs(struct kvm_pgtable *pgt, u64 addr,
return 0;
}
+int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+ return stage2_update_leaf_attrs(pgt, addr, size, 0,
+ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W, NULL);
+}
+
kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
{
kvm_pte_t pte = 0;
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 13/21] KVM: arm64: Convert write-protect operation to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (11 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table Will Deacon
` (9 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
From: Quentin Perret <qperret@google.com>
Convert stage2_wp_range() to call the kvm_pgtable_stage2_wrprotect()
function of the generic page-table code instead of walking the page-table
directly.
Cc: Marc Zyngier <maz@kernel.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 25 ++++---------------------
1 file changed, 4 insertions(+), 21 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index a036aac3ed24..bbcd632fc5b1 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -71,6 +71,9 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,
return ret;
}
+#define stage2_apply_range_resched(kvm, addr, end, fn) \
+ stage2_apply_range(kvm, addr, end, fn, true)
+
static bool memslot_is_logging(struct kvm_memory_slot *memslot)
{
return memslot->dirty_bitmap && !(memslot->flags & KVM_MEM_READONLY);
@@ -1299,27 +1302,7 @@ static void stage2_wp_p4ds(struct kvm_s2_mmu *mmu, pgd_t *pgd,
static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
{
struct kvm *kvm = mmu->kvm;
- pgd_t *pgd;
- phys_addr_t next;
-
- pgd = mmu->pgd + stage2_pgd_index(kvm, addr);
- do {
- /*
- * Release kvm_mmu_lock periodically if the memory region is
- * large. Otherwise, we may see kernel panics with
- * CONFIG_DETECT_HUNG_TASK, CONFIG_LOCKUP_DETECTOR,
- * CONFIG_LOCKDEP. Additionally, holding the lock too long
- * will also starve other vCPUs. We have to also make sure
- * that the page tables are not freed while we released
- * the lock.
- */
- cond_resched_lock(&kvm->mmu_lock);
- if (!READ_ONCE(mmu->pgd))
- break;
- next = stage2_pgd_addr_end(kvm, addr, end);
- if (stage2_pgd_present(kvm, *pgd))
- stage2_wp_p4ds(mmu, pgd, addr, next);
- } while (pgd++, addr = next, addr != end);
+ stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect);
}
/**
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (12 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 13/21] KVM: arm64: Convert write-protect operation to generic page-table API Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API Will Deacon
` (8 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
From: Quentin Perret <qperret@google.com>
Add support for cache flushing a range of the stage-2 address space to
the generic page-table code.
Cc: Marc Zyngier <maz@kernel.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 12 ++++++++++++
arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index e9491c91bd06..a776bcf74316 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -228,6 +228,18 @@ kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr);
*/
bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr);
+/**
+ * kvm_pgtable_stage2_flush_range() - Clean and invalidate data cache to Point
+ * of Coherency for guest stage-2 address
+ * range.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address from which to flush.
+ * @size: Size of the range.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
+
/**
* kvm_pgtable_walk() - Walk a page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_*_init().
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index fee282f8690d..ad18ce469041 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -782,6 +782,32 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr)
return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF;
}
+static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
+ enum kvm_pgtable_walk_flags flag,
+ void * const arg)
+{
+ kvm_pte_t pte = *ptep;
+
+ if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pte))
+ return 0;
+
+ stage2_flush_dcache(kvm_pte_follow(pte), kvm_granule_size(level));
+ return 0;
+}
+
+int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+ struct kvm_pgtable_walker walker = {
+ .cb = stage2_flush_walker,
+ .flags = KVM_PGTABLE_WALK_LEAF,
+ };
+
+ if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB))
+ return 0;
+
+ return kvm_pgtable_walk(pgt, addr, size, &walker);
+}
+
int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm)
{
size_t pgd_sz;
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (13 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code Will Deacon
` (7 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
From: Quentin Perret <qperret@google.com>
Convert stage2_flush_memslot() to call the kvm_pgtable_stage2_flush()
function of the generic page-table code instead of walking the page-table
directly.
Cc: Marc Zyngier <maz@kernel.org>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 13 +------------
1 file changed, 1 insertion(+), 12 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index bbcd632fc5b1..0af48f35c8dd 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -460,21 +460,10 @@ static void stage2_flush_p4ds(struct kvm_s2_mmu *mmu, pgd_t *pgd,
static void stage2_flush_memslot(struct kvm *kvm,
struct kvm_memory_slot *memslot)
{
- struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;
phys_addr_t end = addr + PAGE_SIZE * memslot->npages;
- phys_addr_t next;
- pgd_t *pgd;
- pgd = mmu->pgd + stage2_pgd_index(kvm, addr);
- do {
- next = stage2_pgd_addr_end(kvm, addr, end);
- if (!stage2_pgd_none(kvm, *pgd))
- stage2_flush_p4ds(mmu, pgd, addr, next);
-
- if (next != end)
- cond_resched_lock(&kvm->mmu_lock);
- } while (pgd++, addr = next, addr != end);
+ stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush);
}
/**
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (14 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 16:37 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API Will Deacon
` (6 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Add support for relaxing the permissions of a stage-2 mapping (i.e.
adding additional permissions) to the generic page-table code.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_pgtable.h | 17 +++++++++++++++++
arch/arm64/kvm/hyp/pgtable.c | 20 ++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index a776bcf74316..5ab4f852a463 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -218,6 +218,23 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);
*/
kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr);
+/**
+ * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a
+ * page-table entry.
+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
+ * @addr: Intermediate physical address to identify the page-table entry.
+ * @prot: Additional permissions to grant for the mapping.
+ *
+ * If there is a valid, leaf page-table entry used to translate @addr, then
+ * relax the permissions in that entry according to the read, write and
+ * execute permissions specified by @prot. No permissions are removed, and
+ * TLB invalidation is performed after updating the entry.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
+ enum kvm_pgtable_prot prot);
+
/**
* kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the
* access flag set.
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index ad18ce469041..df705533dc3f 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -782,6 +782,26 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr)
return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF;
}
+int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
+ enum kvm_pgtable_prot prot)
+{
+ int ret;
+ kvm_pte_t set = 0, clr = 0;
+
+ if (prot & KVM_PGTABLE_PROT_R)
+ set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
+
+ if (prot & KVM_PGTABLE_PROT_W)
+ set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
+
+ if (prot & KVM_PGTABLE_PROT_X)
+ clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
+
+ ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL);
+ kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, 0);
+ return ret;
+}
+
static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
enum kvm_pgtable_walk_flags flag,
void * const arg)
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code
2020-09-07 15:23 ` [PATCH v4 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code Will Deacon
@ 2020-09-08 16:37 ` Alexandru Elisei
0 siblings, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-08 16:37 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
Minor nitpick below.
On 9/7/20 4:23 PM, Will Deacon wrote:
> Add support for relaxing the permissions of a stage-2 mapping (i.e.
> adding additional permissions) to the generic page-table code.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/include/asm/kvm_pgtable.h | 17 +++++++++++++++++
> arch/arm64/kvm/hyp/pgtable.c | 20 ++++++++++++++++++++
> 2 files changed, 37 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
> index a776bcf74316..5ab4f852a463 100644
> --- a/arch/arm64/include/asm/kvm_pgtable.h
> +++ b/arch/arm64/include/asm/kvm_pgtable.h
> @@ -218,6 +218,23 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr);
> */
> kvm_pte_t kvm_pgtable_stage2_mkold(struct kvm_pgtable *pgt, u64 addr);
>
> +/**
> + * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a
> + * page-table entry.
> + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init().
> + * @addr: Intermediate physical address to identify the page-table entry.
> + * @prot: Additional permissions to grant for the mapping.
> + *
> + * If there is a valid, leaf page-table entry used to translate @addr, then
> + * relax the permissions in that entry according to the read, write and
> + * execute permissions specified by @prot. No permissions are removed, and
> + * TLB invalidation is performed after updating the entry.
> + *
> + * Return: 0 on success, negative error code on failure.
> + */
> +int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
> + enum kvm_pgtable_prot prot);
> +
> /**
> * kvm_pgtable_stage2_is_young() - Test whether a page-table entry has the
> * access flag set.
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index ad18ce469041..df705533dc3f 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -782,6 +782,26 @@ bool kvm_pgtable_stage2_is_young(struct kvm_pgtable *pgt, u64 addr)
> return pte & KVM_PTE_LEAF_ATTR_LO_S2_AF;
> }
>
> +int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
> + enum kvm_pgtable_prot prot)
> +{
> + int ret;
> + kvm_pte_t set = 0, clr = 0;
> +
> + if (prot & KVM_PGTABLE_PROT_R)
> + set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
> +
> + if (prot & KVM_PGTABLE_PROT_W)
> + set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
> +
> + if (prot & KVM_PGTABLE_PROT_X)
> + clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
> +
> + ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL);
> + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, 0);
The level 0 argument could be replaced by S2_NO_LEVEL_HINT, which is self
explanatory. However, that means the file needs to include stage2_pgtable.h.
Thanks,
Alex
> + return ret;
> +}
> +
> static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep,
> enum kvm_pgtable_walk_flags flag,
> void * const arg)
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (15 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-09 14:20 ` Alexandru Elisei
2020-09-07 15:23 ` [PATCH v4 18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table Will Deacon
` (5 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Convert user_mem_abort() to call kvm_pgtable_stage2_relax_perms() when
handling a stage-2 permission fault and kvm_pgtable_stage2_map() when
handling a stage-2 translation fault, rather than walking the page-table
manually.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 124 +++++++++++++++----------------------------
1 file changed, 44 insertions(+), 80 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 0af48f35c8dd..dc923e873dad 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1496,18 +1496,19 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
{
int ret;
bool write_fault, writable, force_pte = false;
- bool exec_fault, needs_exec;
+ bool exec_fault;
+ bool device = false;
unsigned long mmu_seq;
- gfn_t gfn = fault_ipa >> PAGE_SHIFT;
struct kvm *kvm = vcpu->kvm;
struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache;
struct vm_area_struct *vma;
short vma_shift;
+ gfn_t gfn;
kvm_pfn_t pfn;
- pgprot_t mem_type = PAGE_S2;
bool logging_active = memslot_is_logging(memslot);
- unsigned long vma_pagesize, flags = 0;
- struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu;
+ unsigned long vma_pagesize;
+ enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
+ struct kvm_pgtable *pgt;
write_fault = kvm_is_write_fault(vcpu);
exec_fault = kvm_vcpu_trap_is_iabt(vcpu);
@@ -1540,22 +1541,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
vma_pagesize = PAGE_SIZE;
}
- /*
- * The stage2 has a minimum of 2 level table (For arm64 see
- * kvm_arm_setup_stage2()). Hence, we are guaranteed that we can
- * use PMD_SIZE huge mappings (even when the PMD is folded into PGD).
- * As for PUD huge maps, we must make sure that we have at least
- * 3 levels, i.e, PMD is not folded.
- */
- if (vma_pagesize == PMD_SIZE ||
- (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm)))
- gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
+ if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
+ fault_ipa &= huge_page_mask(hstate_vma(vma));
+
+ gfn = fault_ipa >> PAGE_SHIFT;
mmap_read_unlock(current->mm);
- /* We need minimum second+third level pages */
- ret = kvm_mmu_topup_memory_cache(memcache, kvm_mmu_cache_min_pages(kvm));
- if (ret)
- return ret;
+ /*
+ * Permission faults just need to update the existing leaf entry,
+ * and so normally don't require allocations from the memcache. The
+ * only exception to this is when dirty logging is enabled at runtime
+ * and a write fault needs to collapse a block entry into a table.
+ */
+ if (fault_status != FSC_PERM || (logging_active && write_fault)) {
+ ret = kvm_mmu_topup_memory_cache(memcache,
+ kvm_mmu_cache_min_pages(kvm));
+ if (ret)
+ return ret;
+ }
mmu_seq = vcpu->kvm->mmu_notifier_seq;
/*
@@ -1578,28 +1581,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
return -EFAULT;
if (kvm_is_device_pfn(pfn)) {
- mem_type = PAGE_S2_DEVICE;
- flags |= KVM_S2PTE_FLAG_IS_IOMAP;
- } else if (logging_active) {
- /*
- * Faults on pages in a memslot with logging enabled
- * should not be mapped with huge pages (it introduces churn
- * and performance degradation), so force a pte mapping.
- */
- flags |= KVM_S2_FLAG_LOGGING_ACTIVE;
-
+ device = true;
+ } else if (logging_active && !write_fault) {
/*
* Only actually map the page as writable if this was a write
* fault.
*/
- if (!write_fault)
- writable = false;
+ writable = false;
}
- if (exec_fault && is_iomap(flags))
+ if (exec_fault && device)
return -ENOEXEC;
spin_lock(&kvm->mmu_lock);
+ pgt = vcpu->arch.hw_mmu->pgt;
if (mmu_notifier_retry(kvm, mmu_seq))
goto out_unlock;
@@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
if (vma_pagesize == PAGE_SIZE && !force_pte)
vma_pagesize = transparent_hugepage_adjust(memslot, hva,
&pfn, &fault_ipa);
- if (writable)
+ if (writable) {
+ prot |= KVM_PGTABLE_PROT_W;
kvm_set_pfn_dirty(pfn);
+ mark_page_dirty(kvm, gfn);
+ }
- if (fault_status != FSC_PERM && !is_iomap(flags))
+ if (fault_status != FSC_PERM && !device)
clean_dcache_guest_page(pfn, vma_pagesize);
- if (exec_fault)
+ if (exec_fault) {
+ prot |= KVM_PGTABLE_PROT_X;
invalidate_icache_guest_page(pfn, vma_pagesize);
+ }
- /*
- * If we took an execution fault we have made the
- * icache/dcache coherent above and should now let the s2
- * mapping be executable.
- *
- * Write faults (!exec_fault && FSC_PERM) are orthogonal to
- * execute permissions, and we preserve whatever we have.
- */
- needs_exec = exec_fault ||
- (fault_status == FSC_PERM &&
- stage2_is_exec(mmu, fault_ipa, vma_pagesize));
-
- if (vma_pagesize == PUD_SIZE) {
- pud_t new_pud = kvm_pfn_pud(pfn, mem_type);
-
- new_pud = kvm_pud_mkhuge(new_pud);
- if (writable)
- new_pud = kvm_s2pud_mkwrite(new_pud);
-
- if (needs_exec)
- new_pud = kvm_s2pud_mkexec(new_pud);
-
- ret = stage2_set_pud_huge(mmu, memcache, fault_ipa, &new_pud);
- } else if (vma_pagesize == PMD_SIZE) {
- pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type);
-
- new_pmd = kvm_pmd_mkhuge(new_pmd);
-
- if (writable)
- new_pmd = kvm_s2pmd_mkwrite(new_pmd);
-
- if (needs_exec)
- new_pmd = kvm_s2pmd_mkexec(new_pmd);
+ if (device)
+ prot |= KVM_PGTABLE_PROT_DEVICE;
+ else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
+ prot |= KVM_PGTABLE_PROT_X;
- ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd);
+ if (fault_status == FSC_PERM && !(logging_active && writable)) {
+ ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot);
} else {
- pte_t new_pte = kvm_pfn_pte(pfn, mem_type);
-
- if (writable) {
- new_pte = kvm_s2pte_mkwrite(new_pte);
- mark_page_dirty(kvm, gfn);
- }
-
- if (needs_exec)
- new_pte = kvm_s2pte_mkexec(new_pte);
-
- ret = stage2_set_pte(mmu, memcache, fault_ipa, &new_pte, flags);
+ ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,
+ __pfn_to_phys(pfn), prot,
+ memcache);
}
out_unlock:
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-07 15:23 ` [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API Will Deacon
@ 2020-09-09 14:20 ` Alexandru Elisei
2020-09-09 17:12 ` Marc Zyngier
0 siblings, 1 reply; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-09 14:20 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
On 9/7/20 4:23 PM, Will Deacon wrote:
> Convert user_mem_abort() to call kvm_pgtable_stage2_relax_perms() when
> handling a stage-2 permission fault and kvm_pgtable_stage2_map() when
> handling a stage-2 translation fault, rather than walking the page-table
> manually.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/kvm/mmu.c | 124 +++++++++++++++----------------------------
> 1 file changed, 44 insertions(+), 80 deletions(-)
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 0af48f35c8dd..dc923e873dad 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1496,18 +1496,19 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> {
> int ret;
> bool write_fault, writable, force_pte = false;
> - bool exec_fault, needs_exec;
> + bool exec_fault;
> + bool device = false;
> unsigned long mmu_seq;
> - gfn_t gfn = fault_ipa >> PAGE_SHIFT;
> struct kvm *kvm = vcpu->kvm;
> struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache;
> struct vm_area_struct *vma;
> short vma_shift;
> + gfn_t gfn;
> kvm_pfn_t pfn;
> - pgprot_t mem_type = PAGE_S2;
> bool logging_active = memslot_is_logging(memslot);
> - unsigned long vma_pagesize, flags = 0;
> - struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu;
> + unsigned long vma_pagesize;
> + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
> + struct kvm_pgtable *pgt;
>
> write_fault = kvm_is_write_fault(vcpu);
> exec_fault = kvm_vcpu_trap_is_iabt(vcpu);
> @@ -1540,22 +1541,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> vma_pagesize = PAGE_SIZE;
> }
>
> - /*
> - * The stage2 has a minimum of 2 level table (For arm64 see
> - * kvm_arm_setup_stage2()). Hence, we are guaranteed that we can
> - * use PMD_SIZE huge mappings (even when the PMD is folded into PGD).
> - * As for PUD huge maps, we must make sure that we have at least
> - * 3 levels, i.e, PMD is not folded.
> - */
> - if (vma_pagesize == PMD_SIZE ||
> - (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm)))
> - gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
> + if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
> + fault_ipa &= huge_page_mask(hstate_vma(vma));
This looks correct to me - if !kvm_stage2_has_pmd(), then PMD is folded onto PUD
and PGD, and PMD_SIZE == PUD_SIZE. Also I like the fact that we update gfn **and**
fault_ipa, the previous version updated only gfn, which made gfn != (fault_ipa >>
PAGE_SHIFT).
> +
> + gfn = fault_ipa >> PAGE_SHIFT;
> mmap_read_unlock(current->mm);
>
> - /* We need minimum second+third level pages */
> - ret = kvm_mmu_topup_memory_cache(memcache, kvm_mmu_cache_min_pages(kvm));
> - if (ret)
> - return ret;
> + /*
> + * Permission faults just need to update the existing leaf entry,
> + * and so normally don't require allocations from the memcache. The
> + * only exception to this is when dirty logging is enabled at runtime
> + * and a write fault needs to collapse a block entry into a table.
> + */
> + if (fault_status != FSC_PERM || (logging_active && write_fault)) {
> + ret = kvm_mmu_topup_memory_cache(memcache,
> + kvm_mmu_cache_min_pages(kvm));
> + if (ret)
> + return ret;
> + }
I'm not 100% sure about this.
I don't think we gain much over the previous code - if we had allocated cache
objects which we hadn't used, we would have used them next time user_mem_abort()
is called (kvm_mmu_topup_memory_cache() checks if we have the required number of
objects in the cache and returns early).
I'm not sure the condition is entirely correct either - if stage 2 already has a
mapping for the IPA and we only need to set write permissions, according to the
condition above we still try to topup the cache, even though we don't strictly
need to.
>
> mmu_seq = vcpu->kvm->mmu_notifier_seq;
> /*
> @@ -1578,28 +1581,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> return -EFAULT;
>
> if (kvm_is_device_pfn(pfn)) {
> - mem_type = PAGE_S2_DEVICE;
> - flags |= KVM_S2PTE_FLAG_IS_IOMAP;
> - } else if (logging_active) {
> - /*
> - * Faults on pages in a memslot with logging enabled
> - * should not be mapped with huge pages (it introduces churn
> - * and performance degradation), so force a pte mapping.
> - */
> - flags |= KVM_S2_FLAG_LOGGING_ACTIVE;
> -
> + device = true;
> + } else if (logging_active && !write_fault) {
> /*
> * Only actually map the page as writable if this was a write
> * fault.
> */
> - if (!write_fault)
> - writable = false;
> + writable = false;
> }
>
> - if (exec_fault && is_iomap(flags))
> + if (exec_fault && device)
> return -ENOEXEC;
>
> spin_lock(&kvm->mmu_lock);
> + pgt = vcpu->arch.hw_mmu->pgt;
> if (mmu_notifier_retry(kvm, mmu_seq))
> goto out_unlock;
>
> @@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> if (vma_pagesize == PAGE_SIZE && !force_pte)
> vma_pagesize = transparent_hugepage_adjust(memslot, hva,
> &pfn, &fault_ipa);
> - if (writable)
> + if (writable) {
> + prot |= KVM_PGTABLE_PROT_W;
> kvm_set_pfn_dirty(pfn);
> + mark_page_dirty(kvm, gfn);
The previous code called mark_page_dirty() only if the vma_pagesize == PAGE_SIZE
(and writable was true, obviously). Is this supposed to fix a bug?
> + }
>
> - if (fault_status != FSC_PERM && !is_iomap(flags))
> + if (fault_status != FSC_PERM && !device)
> clean_dcache_guest_page(pfn, vma_pagesize);
>
> - if (exec_fault)
> + if (exec_fault) {
> + prot |= KVM_PGTABLE_PROT_X;
> invalidate_icache_guest_page(pfn, vma_pagesize);
> + }
>
> - /*
> - * If we took an execution fault we have made the
> - * icache/dcache coherent above and should now let the s2
> - * mapping be executable.
> - *
> - * Write faults (!exec_fault && FSC_PERM) are orthogonal to
> - * execute permissions, and we preserve whatever we have.
> - */
> - needs_exec = exec_fault ||
> - (fault_status == FSC_PERM &&
> - stage2_is_exec(mmu, fault_ipa, vma_pagesize));
> -
> - if (vma_pagesize == PUD_SIZE) {
> - pud_t new_pud = kvm_pfn_pud(pfn, mem_type);
> -
> - new_pud = kvm_pud_mkhuge(new_pud);
> - if (writable)
> - new_pud = kvm_s2pud_mkwrite(new_pud);
> -
> - if (needs_exec)
> - new_pud = kvm_s2pud_mkexec(new_pud);
> -
> - ret = stage2_set_pud_huge(mmu, memcache, fault_ipa, &new_pud);
> - } else if (vma_pagesize == PMD_SIZE) {
> - pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type);
> -
> - new_pmd = kvm_pmd_mkhuge(new_pmd);
> -
> - if (writable)
> - new_pmd = kvm_s2pmd_mkwrite(new_pmd);
> -
> - if (needs_exec)
> - new_pmd = kvm_s2pmd_mkexec(new_pmd);
> + if (device)
> + prot |= KVM_PGTABLE_PROT_DEVICE;
> + else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
> + prot |= KVM_PGTABLE_PROT_X;
>
> - ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd);
> + if (fault_status == FSC_PERM && !(logging_active && writable)) {
I don't understand the second part of the condition (!(logging_active &&
writable)). With logging active, when we get a fault because of a missing stage 2
entry, we map the IPA as read-only at stage 2. If I understand this code
correctly, when the guest then tries to write to the same IPA, writable == true
and we map the IPA again instead of relaxing the permissions. Why is that?
Thanks,
Alex
> + ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot);
> } else {
> - pte_t new_pte = kvm_pfn_pte(pfn, mem_type);
> -
> - if (writable) {
> - new_pte = kvm_s2pte_mkwrite(new_pte);
> - mark_page_dirty(kvm, gfn);
> - }
> -
> - if (needs_exec)
> - new_pte = kvm_s2pte_mkexec(new_pte);
> -
> - ret = stage2_set_pte(mmu, memcache, fault_ipa, &new_pte, flags);
> + ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,
> + __pfn_to_phys(pfn), prot,
> + memcache);
> }
>
> out_unlock:
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-09 14:20 ` Alexandru Elisei
@ 2020-09-09 17:12 ` Marc Zyngier
2020-09-10 10:51 ` Will Deacon
2020-09-10 13:20 ` Alexandru Elisei
0 siblings, 2 replies; 52+ messages in thread
From: Marc Zyngier @ 2020-09-09 17:12 UTC (permalink / raw)
To: Alexandru Elisei
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Catalin Marinas,
Quentin Perret, James Morse, Will Deacon, kvmarm,
linux-arm-kernel
Hi Alex,
On 2020-09-09 15:20, Alexandru Elisei wrote:
> Hi Will,
>
> On 9/7/20 4:23 PM, Will Deacon wrote:
>> Convert user_mem_abort() to call kvm_pgtable_stage2_relax_perms() when
>> handling a stage-2 permission fault and kvm_pgtable_stage2_map() when
>> handling a stage-2 translation fault, rather than walking the
>> page-table
>> manually.
>>
>> Cc: Marc Zyngier <maz@kernel.org>
>> Cc: Quentin Perret <qperret@google.com>
>> Reviewed-by: Gavin Shan <gshan@redhat.com>
>> Signed-off-by: Will Deacon <will@kernel.org>
>> ---
>> arch/arm64/kvm/mmu.c | 124
>> +++++++++++++++----------------------------
>> 1 file changed, 44 insertions(+), 80 deletions(-)
>>
>> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
>> index 0af48f35c8dd..dc923e873dad 100644
>> --- a/arch/arm64/kvm/mmu.c
>> +++ b/arch/arm64/kvm/mmu.c
>> @@ -1496,18 +1496,19 @@ static int user_mem_abort(struct kvm_vcpu
>> *vcpu, phys_addr_t fault_ipa,
>> {
>> int ret;
>> bool write_fault, writable, force_pte = false;
>> - bool exec_fault, needs_exec;
>> + bool exec_fault;
>> + bool device = false;
>> unsigned long mmu_seq;
>> - gfn_t gfn = fault_ipa >> PAGE_SHIFT;
>> struct kvm *kvm = vcpu->kvm;
>> struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache;
>> struct vm_area_struct *vma;
>> short vma_shift;
>> + gfn_t gfn;
>> kvm_pfn_t pfn;
>> - pgprot_t mem_type = PAGE_S2;
>> bool logging_active = memslot_is_logging(memslot);
>> - unsigned long vma_pagesize, flags = 0;
>> - struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu;
>> + unsigned long vma_pagesize;
>> + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
>> + struct kvm_pgtable *pgt;
>>
>> write_fault = kvm_is_write_fault(vcpu);
>> exec_fault = kvm_vcpu_trap_is_iabt(vcpu);
>> @@ -1540,22 +1541,24 @@ static int user_mem_abort(struct kvm_vcpu
>> *vcpu, phys_addr_t fault_ipa,
>> vma_pagesize = PAGE_SIZE;
>> }
>>
>> - /*
>> - * The stage2 has a minimum of 2 level table (For arm64 see
>> - * kvm_arm_setup_stage2()). Hence, we are guaranteed that we can
>> - * use PMD_SIZE huge mappings (even when the PMD is folded into
>> PGD).
>> - * As for PUD huge maps, we must make sure that we have at least
>> - * 3 levels, i.e, PMD is not folded.
>> - */
>> - if (vma_pagesize == PMD_SIZE ||
>> - (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm)))
>> - gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
>> + if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
>> + fault_ipa &= huge_page_mask(hstate_vma(vma));
>
> This looks correct to me - if !kvm_stage2_has_pmd(), then PMD is folded
> onto PUD
> and PGD, and PMD_SIZE == PUD_SIZE. Also I like the fact that we update
> gfn **and**
> fault_ipa, the previous version updated only gfn, which made gfn !=
> (fault_ipa >>
> PAGE_SHIFT).
>
>> +
>> + gfn = fault_ipa >> PAGE_SHIFT;
>> mmap_read_unlock(current->mm);
>>
>> - /* We need minimum second+third level pages */
>> - ret = kvm_mmu_topup_memory_cache(memcache,
>> kvm_mmu_cache_min_pages(kvm));
>> - if (ret)
>> - return ret;
>> + /*
>> + * Permission faults just need to update the existing leaf entry,
>> + * and so normally don't require allocations from the memcache. The
>> + * only exception to this is when dirty logging is enabled at
>> runtime
>> + * and a write fault needs to collapse a block entry into a table.
>> + */
>> + if (fault_status != FSC_PERM || (logging_active && write_fault)) {
>> + ret = kvm_mmu_topup_memory_cache(memcache,
>> + kvm_mmu_cache_min_pages(kvm));
>> + if (ret)
>> + return ret;
>> + }
>
> I'm not 100% sure about this.
>
> I don't think we gain much over the previous code - if we had allocated
> cache
> objects which we hadn't used, we would have used them next time
> user_mem_abort()
> is called (kvm_mmu_topup_memory_cache() checks if we have the required
> number of
> objects in the cache and returns early).
>
> I'm not sure the condition is entirely correct either - if stage 2
> already has a
> mapping for the IPA and we only need to set write permissions,
> according to the
> condition above we still try to topup the cache, even though we don't
> strictly
> need to.
That's because if you are logging, you may have to split an existing
block
mapping and map a single page instead. This requires (at least) an extra
level, and that's why you need to top-up the cache in this case.
>
>>
>> mmu_seq = vcpu->kvm->mmu_notifier_seq;
>> /*
>> @@ -1578,28 +1581,20 @@ static int user_mem_abort(struct kvm_vcpu
>> *vcpu, phys_addr_t fault_ipa,
>> return -EFAULT;
>>
>> if (kvm_is_device_pfn(pfn)) {
>> - mem_type = PAGE_S2_DEVICE;
>> - flags |= KVM_S2PTE_FLAG_IS_IOMAP;
>> - } else if (logging_active) {
>> - /*
>> - * Faults on pages in a memslot with logging enabled
>> - * should not be mapped with huge pages (it introduces churn
>> - * and performance degradation), so force a pte mapping.
>> - */
>> - flags |= KVM_S2_FLAG_LOGGING_ACTIVE;
>> -
>> + device = true;
>> + } else if (logging_active && !write_fault) {
>> /*
>> * Only actually map the page as writable if this was a write
>> * fault.
>> */
>> - if (!write_fault)
>> - writable = false;
>> + writable = false;
>> }
>>
>> - if (exec_fault && is_iomap(flags))
>> + if (exec_fault && device)
>> return -ENOEXEC;
>>
>> spin_lock(&kvm->mmu_lock);
>> + pgt = vcpu->arch.hw_mmu->pgt;
>> if (mmu_notifier_retry(kvm, mmu_seq))
>> goto out_unlock;
>>
>> @@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu
>> *vcpu, phys_addr_t fault_ipa,
>> if (vma_pagesize == PAGE_SIZE && !force_pte)
>> vma_pagesize = transparent_hugepage_adjust(memslot, hva,
>> &pfn, &fault_ipa);
>> - if (writable)
>> + if (writable) {
>> + prot |= KVM_PGTABLE_PROT_W;
>> kvm_set_pfn_dirty(pfn);
>> + mark_page_dirty(kvm, gfn);
>
> The previous code called mark_page_dirty() only if the vma_pagesize ==
> PAGE_SIZE
> (and writable was true, obviously). Is this supposed to fix a bug?
No, this is actually introducing one. mark_page_dirty() checks that
there is an
associated bitmap, and thus only happens when writing to a single page,
but we
shouldn't do it for R/O memslots, which the current code avoids. It
should be
guarded by logging_active.
>
>> + }
>>
>> - if (fault_status != FSC_PERM && !is_iomap(flags))
>> + if (fault_status != FSC_PERM && !device)
>> clean_dcache_guest_page(pfn, vma_pagesize);
>>
>> - if (exec_fault)
>> + if (exec_fault) {
>> + prot |= KVM_PGTABLE_PROT_X;
>> invalidate_icache_guest_page(pfn, vma_pagesize);
>> + }
>>
>> - /*
>> - * If we took an execution fault we have made the
>> - * icache/dcache coherent above and should now let the s2
>> - * mapping be executable.
>> - *
>> - * Write faults (!exec_fault && FSC_PERM) are orthogonal to
>> - * execute permissions, and we preserve whatever we have.
>> - */
>> - needs_exec = exec_fault ||
>> - (fault_status == FSC_PERM &&
>> - stage2_is_exec(mmu, fault_ipa, vma_pagesize));
>> -
>> - if (vma_pagesize == PUD_SIZE) {
>> - pud_t new_pud = kvm_pfn_pud(pfn, mem_type);
>> -
>> - new_pud = kvm_pud_mkhuge(new_pud);
>> - if (writable)
>> - new_pud = kvm_s2pud_mkwrite(new_pud);
>> -
>> - if (needs_exec)
>> - new_pud = kvm_s2pud_mkexec(new_pud);
>> -
>> - ret = stage2_set_pud_huge(mmu, memcache, fault_ipa, &new_pud);
>> - } else if (vma_pagesize == PMD_SIZE) {
>> - pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type);
>> -
>> - new_pmd = kvm_pmd_mkhuge(new_pmd);
>> -
>> - if (writable)
>> - new_pmd = kvm_s2pmd_mkwrite(new_pmd);
>> -
>> - if (needs_exec)
>> - new_pmd = kvm_s2pmd_mkexec(new_pmd);
>> + if (device)
>> + prot |= KVM_PGTABLE_PROT_DEVICE;
>> + else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
>> + prot |= KVM_PGTABLE_PROT_X;
>>
>> - ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd);
>> + if (fault_status == FSC_PERM && !(logging_active && writable)) {
>
> I don't understand the second part of the condition (!(logging_active
> &&
> writable)). With logging active, when we get a fault because of a
> missing stage 2
> entry, we map the IPA as read-only at stage 2. If I understand this
> code
> correctly, when the guest then tries to write to the same IPA, writable
> == true
> and we map the IPA again instead of relaxing the permissions. Why is
> that?
See my reply above: logging means potentially adding a new level, so we
treat it as a new mapping altogether (break the block mapping, TLBI,
install
the new mapping one level down).
All the other cases are happily handled by just relaxing the
permissions.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-09 17:12 ` Marc Zyngier
@ 2020-09-10 10:51 ` Will Deacon
2020-09-10 10:58 ` Marc Zyngier
2020-09-10 13:10 ` Alexandru Elisei
2020-09-10 13:20 ` Alexandru Elisei
1 sibling, 2 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-10 10:51 UTC (permalink / raw)
To: Marc Zyngier
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Catalin Marinas,
Quentin Perret, James Morse, Alexandru Elisei, kvmarm,
linux-arm-kernel
On Wed, Sep 09, 2020 at 06:12:29PM +0100, Marc Zyngier wrote:
> On 2020-09-09 15:20, Alexandru Elisei wrote:
> > On 9/7/20 4:23 PM, Will Deacon wrote:
> > > @@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu
> > > *vcpu, phys_addr_t fault_ipa,
> > > if (vma_pagesize == PAGE_SIZE && !force_pte)
> > > vma_pagesize = transparent_hugepage_adjust(memslot, hva,
> > > &pfn, &fault_ipa);
> > > - if (writable)
> > > + if (writable) {
> > > + prot |= KVM_PGTABLE_PROT_W;
> > > kvm_set_pfn_dirty(pfn);
> > > + mark_page_dirty(kvm, gfn);
> >
> > The previous code called mark_page_dirty() only if the vma_pagesize ==
> > PAGE_SIZE
> > (and writable was true, obviously). Is this supposed to fix a bug?
>
> No, this is actually introducing one. mark_page_dirty() checks that there is
> an
> associated bitmap, and thus only happens when writing to a single page, but
> we
> shouldn't do it for R/O memslots, which the current code avoids. It should
> be
> guarded by logging_active.
gfn_to_pfn_prot() will set "writable" to false for R/O memslots, so I think
we're good here.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-10 10:51 ` Will Deacon
@ 2020-09-10 10:58 ` Marc Zyngier
2020-09-10 13:10 ` Alexandru Elisei
1 sibling, 0 replies; 52+ messages in thread
From: Marc Zyngier @ 2020-09-10 10:58 UTC (permalink / raw)
To: Will Deacon
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Catalin Marinas,
Quentin Perret, James Morse, Alexandru Elisei, kvmarm,
linux-arm-kernel
On 2020-09-10 11:51, Will Deacon wrote:
> On Wed, Sep 09, 2020 at 06:12:29PM +0100, Marc Zyngier wrote:
>> On 2020-09-09 15:20, Alexandru Elisei wrote:
>> > On 9/7/20 4:23 PM, Will Deacon wrote:
>> > > @@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu
>> > > *vcpu, phys_addr_t fault_ipa,
>> > > if (vma_pagesize == PAGE_SIZE && !force_pte)
>> > > vma_pagesize = transparent_hugepage_adjust(memslot, hva,
>> > > &pfn, &fault_ipa);
>> > > - if (writable)
>> > > + if (writable) {
>> > > + prot |= KVM_PGTABLE_PROT_W;
>> > > kvm_set_pfn_dirty(pfn);
>> > > + mark_page_dirty(kvm, gfn);
>> >
>> > The previous code called mark_page_dirty() only if the vma_pagesize ==
>> > PAGE_SIZE
>> > (and writable was true, obviously). Is this supposed to fix a bug?
>>
>> No, this is actually introducing one. mark_page_dirty() checks that
>> there is
>> an
>> associated bitmap, and thus only happens when writing to a single
>> page, but
>> we
>> shouldn't do it for R/O memslots, which the current code avoids. It
>> should
>> be
>> guarded by logging_active.
>
> gfn_to_pfn_prot() will set "writable" to false for R/O memslots, so I
> think
> we're good here.
Ah, good point.
M.
--
Jazz is not dead. It just smells funny...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-10 10:51 ` Will Deacon
2020-09-10 10:58 ` Marc Zyngier
@ 2020-09-10 13:10 ` Alexandru Elisei
1 sibling, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-10 13:10 UTC (permalink / raw)
To: Will Deacon, Marc Zyngier
Cc: Gavin Shan, Suzuki Poulose, Catalin Marinas, Quentin Perret,
James Morse, kernel-team, kvmarm, linux-arm-kernel
Hi,
On 9/10/20 11:51 AM, Will Deacon wrote:
> On Wed, Sep 09, 2020 at 06:12:29PM +0100, Marc Zyngier wrote:
>> On 2020-09-09 15:20, Alexandru Elisei wrote:
>>> On 9/7/20 4:23 PM, Will Deacon wrote:
>>>> @@ -1610,62 +1605,31 @@ static int user_mem_abort(struct kvm_vcpu
>>>> *vcpu, phys_addr_t fault_ipa,
>>>> if (vma_pagesize == PAGE_SIZE && !force_pte)
>>>> vma_pagesize = transparent_hugepage_adjust(memslot, hva,
>>>> &pfn, &fault_ipa);
>>>> - if (writable)
>>>> + if (writable) {
>>>> + prot |= KVM_PGTABLE_PROT_W;
>>>> kvm_set_pfn_dirty(pfn);
>>>> + mark_page_dirty(kvm, gfn);
>>> The previous code called mark_page_dirty() only if the vma_pagesize ==
>>> PAGE_SIZE
>>> (and writable was true, obviously). Is this supposed to fix a bug?
>> No, this is actually introducing one. mark_page_dirty() checks that there is
>> an
>> associated bitmap, and thus only happens when writing to a single page, but
>> we
>> shouldn't do it for R/O memslots, which the current code avoids. It should
>> be
>> guarded by logging_active.
> gfn_to_pfn_prot() will set "writable" to false for R/O memslots, so I think
> we're good here.
That also looks correct to me, gfn_to_pfn_prot() -> __gfn_to_pfn_memslot() sets
*writable to false if the memslot is readonly.
Marc is also right, mark_page_dirty() first checks if there's a dirty bitmap
associated with the memslot, and memslot->dirty_bitmap gets allocated only if
KVM_SET_USER_MEMORY_REGION was called with the KVM_MEM_LOG_DIRTY_PAGES flag set
(should've checked for that before asking).
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
2020-09-09 17:12 ` Marc Zyngier
2020-09-10 10:51 ` Will Deacon
@ 2020-09-10 13:20 ` Alexandru Elisei
1 sibling, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-10 13:20 UTC (permalink / raw)
To: Marc Zyngier
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Catalin Marinas,
Quentin Perret, James Morse, Will Deacon, kvmarm,
linux-arm-kernel
Hi Marc,
On 9/9/20 6:12 PM, Marc Zyngier wrote:
> Hi Alex,
>
> On 2020-09-09 15:20, Alexandru Elisei wrote:
>> Hi Will,
>>
>> On 9/7/20 4:23 PM, Will Deacon wrote:
>>> Convert user_mem_abort() to call kvm_pgtable_stage2_relax_perms() when
>>> handling a stage-2 permission fault and kvm_pgtable_stage2_map() when
>>> handling a stage-2 translation fault, rather than walking the page-table
>>> manually.
>>>
>>> Cc: Marc Zyngier <maz@kernel.org>
>>> Cc: Quentin Perret <qperret@google.com>
>>> Reviewed-by: Gavin Shan <gshan@redhat.com>
>>> Signed-off-by: Will Deacon <will@kernel.org>
>>> ---
>>> arch/arm64/kvm/mmu.c | 124 +++++++++++++++----------------------------
>>> 1 file changed, 44 insertions(+), 80 deletions(-)
>>>
>>> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
>>> index 0af48f35c8dd..dc923e873dad 100644
>>> --- a/arch/arm64/kvm/mmu.c
>>> +++ b/arch/arm64/kvm/mmu.c
>>> @@ -1496,18 +1496,19 @@ static int user_mem_abort(struct kvm_vcpu *vcpu,
>>> phys_addr_t fault_ipa,
>>> {
>>> int ret;
>>> bool write_fault, writable, force_pte = false;
>>> - bool exec_fault, needs_exec;
>>> + bool exec_fault;
>>> + bool device = false;
>>> unsigned long mmu_seq;
>>> - gfn_t gfn = fault_ipa >> PAGE_SHIFT;
>>> struct kvm *kvm = vcpu->kvm;
>>> struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache;
>>> struct vm_area_struct *vma;
>>> short vma_shift;
>>> + gfn_t gfn;
>>> kvm_pfn_t pfn;
>>> - pgprot_t mem_type = PAGE_S2;
>>> bool logging_active = memslot_is_logging(memslot);
>>> - unsigned long vma_pagesize, flags = 0;
>>> - struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu;
>>> + unsigned long vma_pagesize;
>>> + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
>>> + struct kvm_pgtable *pgt;
>>>
>>> write_fault = kvm_is_write_fault(vcpu);
>>> exec_fault = kvm_vcpu_trap_is_iabt(vcpu);
>>> @@ -1540,22 +1541,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu,
>>> phys_addr_t fault_ipa,
>>> vma_pagesize = PAGE_SIZE;
>>> }
>>>
>>> - /*
>>> - * The stage2 has a minimum of 2 level table (For arm64 see
>>> - * kvm_arm_setup_stage2()). Hence, we are guaranteed that we can
>>> - * use PMD_SIZE huge mappings (even when the PMD is folded into PGD).
>>> - * As for PUD huge maps, we must make sure that we have at least
>>> - * 3 levels, i.e, PMD is not folded.
>>> - */
>>> - if (vma_pagesize == PMD_SIZE ||
>>> - (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm)))
>>> - gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
>>> + if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
>>> + fault_ipa &= huge_page_mask(hstate_vma(vma));
>>
>> This looks correct to me - if !kvm_stage2_has_pmd(), then PMD is folded onto PUD
>> and PGD, and PMD_SIZE == PUD_SIZE. Also I like the fact that we update
>> gfn **and**
>> fault_ipa, the previous version updated only gfn, which made gfn !=
>> (fault_ipa >>
>> PAGE_SHIFT).
>>
>>> +
>>> + gfn = fault_ipa >> PAGE_SHIFT;
>>> mmap_read_unlock(current->mm);
>>>
>>> - /* We need minimum second+third level pages */
>>> - ret = kvm_mmu_topup_memory_cache(memcache, kvm_mmu_cache_min_pages(kvm));
>>> - if (ret)
>>> - return ret;
>>> + /*
>>> + * Permission faults just need to update the existing leaf entry,
>>> + * and so normally don't require allocations from the memcache. The
>>> + * only exception to this is when dirty logging is enabled at runtime
>>> + * and a write fault needs to collapse a block entry into a table.
>>> + */
>>> + if (fault_status != FSC_PERM || (logging_active && write_fault)) {
>>> + ret = kvm_mmu_topup_memory_cache(memcache,
>>> + kvm_mmu_cache_min_pages(kvm));
>>> + if (ret)
>>> + return ret;
>>> + }
>>
>> I'm not 100% sure about this.
>>
>> I don't think we gain much over the previous code - if we had allocated cache
>> objects which we hadn't used, we would have used them next time user_mem_abort()
>> is called (kvm_mmu_topup_memory_cache() checks if we have the required number of
>> objects in the cache and returns early).
>>
>> I'm not sure the condition is entirely correct either - if stage 2 already has a
>> mapping for the IPA and we only need to set write permissions, according to the
>> condition above we still try to topup the cache, even though we don't strictly
>> need to.
>
> That's because if you are logging, you may have to split an existing block
> mapping and map a single page instead. This requires (at least) an extra
> level, and that's why you need to top-up the cache in this case.
>
>>
>>> [..]
>>>
>>> -
>>> - if (needs_exec)
>>> - new_pmd = kvm_s2pmd_mkexec(new_pmd);
>>> + if (device)
>>> + prot |= KVM_PGTABLE_PROT_DEVICE;
>>> + else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
>>> + prot |= KVM_PGTABLE_PROT_X;
>>>
>>> - ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd);
>>> + if (fault_status == FSC_PERM && !(logging_active && writable)) {
>>
>> I don't understand the second part of the condition (!(logging_active &&
>> writable)). With logging active, when we get a fault because of a
>> missing stage 2
>> entry, we map the IPA as read-only at stage 2. If I understand this code
>> correctly, when the guest then tries to write to the same IPA, writable == true
>> and we map the IPA again instead of relaxing the permissions. Why is that?
>
> See my reply above: logging means potentially adding a new level, so we
> treat it as a new mapping altogether (break the block mapping, TLBI, install
> the new mapping one level down).
>
> All the other cases are happily handled by just relaxing the permissions.
I think I understand now, I didn't realize that we never check if the IPA is
mapped when we get a write fault with a dirty page logging memslot. We
unconditionally map a new page marked writable, regardless if the IPA was mapped
or not, or if it was mapped using a block mapping or a page.
The code makes sense with that in mind. For what it's worth:
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (16 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 19/21] KVM: arm64: Remove unused page-table code Will Deacon
` (4 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
In preparation for removing the 'pgd' field of 'struct kvm_s2_mmu',
update the few remaining users to check the 'pgt' field instead.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/mmu.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index dc923e873dad..548161795dd9 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1826,7 +1826,7 @@ static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *dat
int kvm_unmap_hva_range(struct kvm *kvm,
unsigned long start, unsigned long end, unsigned flags)
{
- if (!kvm->arch.mmu.pgd)
+ if (!kvm->arch.mmu.pgt)
return 0;
trace_kvm_unmap_hva_range(start, end);
@@ -1889,7 +1889,7 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *
int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end)
{
- if (!kvm->arch.mmu.pgd)
+ if (!kvm->arch.mmu.pgt)
return 0;
trace_kvm_age_hva(start, end);
return handle_hva_to_gpa(kvm, start, end, kvm_age_hva_handler, NULL);
@@ -1897,7 +1897,7 @@ int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end)
int kvm_test_age_hva(struct kvm *kvm, unsigned long hva)
{
- if (!kvm->arch.mmu.pgd)
+ if (!kvm->arch.mmu.pgt)
return 0;
trace_kvm_test_age_hva(hva);
return handle_hva_to_gpa(kvm, hva, hva + PAGE_SIZE,
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 19/21] KVM: arm64: Remove unused page-table code
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (17 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-08 10:33 ` Marc Zyngier
2020-09-07 15:23 ` [PATCH v4 20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu' Will Deacon
` (3 subsequent siblings)
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Now that KVM is using the generic page-table code to manage the guest
stage-2 page-tables, we can remove a bunch of unused macros, #defines
and static inline functions from the old implementation.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_mmu.h | 171 ------
arch/arm64/include/asm/pgtable-hwdef.h | 17 -
arch/arm64/include/asm/pgtable-prot.h | 13 -
arch/arm64/include/asm/stage2_pgtable.h | 215 -------
arch/arm64/kvm/mmu.c | 755 ------------------------
5 files changed, 1171 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 42fb50cfe0d8..c490fe8089b3 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -107,12 +107,6 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
#define kvm_phys_size(kvm) (_AC(1, ULL) << kvm_phys_shift(kvm))
#define kvm_phys_mask(kvm) (kvm_phys_size(kvm) - _AC(1, ULL))
-static inline bool kvm_page_empty(void *ptr)
-{
- struct page *ptr_page = virt_to_page(ptr);
- return page_count(ptr_page) == 1;
-}
-
#include <asm/kvm_pgtable.h>
#include <asm/stage2_pgtable.h>
@@ -135,123 +129,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu);
phys_addr_t kvm_mmu_get_httbr(void);
phys_addr_t kvm_get_idmap_vector(void);
int kvm_mmu_init(void);
-#define kvm_mk_pmd(ptep) \
- __pmd(__phys_to_pmd_val(__pa(ptep)) | PMD_TYPE_TABLE)
-#define kvm_mk_pud(pmdp) \
- __pud(__phys_to_pud_val(__pa(pmdp)) | PMD_TYPE_TABLE)
-#define kvm_mk_p4d(pmdp) \
- __p4d(__phys_to_p4d_val(__pa(pmdp)) | PUD_TYPE_TABLE)
-
-#define kvm_set_pud(pudp, pud) set_pud(pudp, pud)
-
-#define kvm_pfn_pte(pfn, prot) pfn_pte(pfn, prot)
-#define kvm_pfn_pmd(pfn, prot) pfn_pmd(pfn, prot)
-#define kvm_pfn_pud(pfn, prot) pfn_pud(pfn, prot)
-
-#define kvm_pud_pfn(pud) pud_pfn(pud)
-
-#define kvm_pmd_mkhuge(pmd) pmd_mkhuge(pmd)
-#define kvm_pud_mkhuge(pud) pud_mkhuge(pud)
-
-static inline pte_t kvm_s2pte_mkwrite(pte_t pte)
-{
- pte_val(pte) |= PTE_S2_RDWR;
- return pte;
-}
-
-static inline pmd_t kvm_s2pmd_mkwrite(pmd_t pmd)
-{
- pmd_val(pmd) |= PMD_S2_RDWR;
- return pmd;
-}
-
-static inline pud_t kvm_s2pud_mkwrite(pud_t pud)
-{
- pud_val(pud) |= PUD_S2_RDWR;
- return pud;
-}
-
-static inline pte_t kvm_s2pte_mkexec(pte_t pte)
-{
- pte_val(pte) &= ~PTE_S2_XN;
- return pte;
-}
-
-static inline pmd_t kvm_s2pmd_mkexec(pmd_t pmd)
-{
- pmd_val(pmd) &= ~PMD_S2_XN;
- return pmd;
-}
-
-static inline pud_t kvm_s2pud_mkexec(pud_t pud)
-{
- pud_val(pud) &= ~PUD_S2_XN;
- return pud;
-}
-
-static inline void kvm_set_s2pte_readonly(pte_t *ptep)
-{
- pteval_t old_pteval, pteval;
-
- pteval = READ_ONCE(pte_val(*ptep));
- do {
- old_pteval = pteval;
- pteval &= ~PTE_S2_RDWR;
- pteval |= PTE_S2_RDONLY;
- pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
- } while (pteval != old_pteval);
-}
-
-static inline bool kvm_s2pte_readonly(pte_t *ptep)
-{
- return (READ_ONCE(pte_val(*ptep)) & PTE_S2_RDWR) == PTE_S2_RDONLY;
-}
-
-static inline bool kvm_s2pte_exec(pte_t *ptep)
-{
- return !(READ_ONCE(pte_val(*ptep)) & PTE_S2_XN);
-}
-
-static inline void kvm_set_s2pmd_readonly(pmd_t *pmdp)
-{
- kvm_set_s2pte_readonly((pte_t *)pmdp);
-}
-
-static inline bool kvm_s2pmd_readonly(pmd_t *pmdp)
-{
- return kvm_s2pte_readonly((pte_t *)pmdp);
-}
-
-static inline bool kvm_s2pmd_exec(pmd_t *pmdp)
-{
- return !(READ_ONCE(pmd_val(*pmdp)) & PMD_S2_XN);
-}
-
-static inline void kvm_set_s2pud_readonly(pud_t *pudp)
-{
- kvm_set_s2pte_readonly((pte_t *)pudp);
-}
-
-static inline bool kvm_s2pud_readonly(pud_t *pudp)
-{
- return kvm_s2pte_readonly((pte_t *)pudp);
-}
-
-static inline bool kvm_s2pud_exec(pud_t *pudp)
-{
- return !(READ_ONCE(pud_val(*pudp)) & PUD_S2_XN);
-}
-
-static inline pud_t kvm_s2pud_mkyoung(pud_t pud)
-{
- return pud_mkyoung(pud);
-}
-
-static inline bool kvm_s2pud_young(pud_t pud)
-{
- return pud_young(pud);
-}
-
struct kvm;
@@ -293,30 +170,6 @@ static inline void __invalidate_icache_guest_page(kvm_pfn_t pfn,
}
}
-static inline void __kvm_flush_dcache_pte(pte_t pte)
-{
- if (!cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) {
- struct page *page = pte_page(pte);
- kvm_flush_dcache_to_poc(page_address(page), PAGE_SIZE);
- }
-}
-
-static inline void __kvm_flush_dcache_pmd(pmd_t pmd)
-{
- if (!cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) {
- struct page *page = pmd_page(pmd);
- kvm_flush_dcache_to_poc(page_address(page), PMD_SIZE);
- }
-}
-
-static inline void __kvm_flush_dcache_pud(pud_t pud)
-{
- if (!cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) {
- struct page *page = pud_page(pud);
- kvm_flush_dcache_to_poc(page_address(page), PUD_SIZE);
- }
-}
-
void kvm_set_way_flush(struct kvm_vcpu *vcpu);
void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
@@ -477,30 +330,6 @@ static inline int hyp_map_aux_data(void)
#define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr)
-/*
- * Get the magic number 'x' for VTTBR:BADDR of this KVM instance.
- * With v8.2 LVA extensions, 'x' should be a minimum of 6 with
- * 52bit IPS.
- */
-static inline int arm64_vttbr_x(u32 ipa_shift, u32 levels)
-{
- int x = ARM64_VTTBR_X(ipa_shift, levels);
-
- return (IS_ENABLED(CONFIG_ARM64_PA_BITS_52) && x < 6) ? 6 : x;
-}
-
-static inline u64 vttbr_baddr_mask(u32 ipa_shift, u32 levels)
-{
- unsigned int x = arm64_vttbr_x(ipa_shift, levels);
-
- return GENMASK_ULL(PHYS_MASK_SHIFT - 1, x);
-}
-
-static inline u64 kvm_vttbr_baddr_mask(struct kvm *kvm)
-{
- return vttbr_baddr_mask(kvm_phys_shift(kvm), kvm_stage2_levels(kvm));
-}
-
static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu)
{
struct kvm_vmid *vmid = &mmu->vmid;
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 1a989353144e..bb97d464f42b 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -172,23 +172,6 @@
#define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2)
#define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2)
-/*
- * 2nd stage PTE definitions
- */
-#define PTE_S2_RDONLY (_AT(pteval_t, 1) << 6) /* HAP[2:1] */
-#define PTE_S2_RDWR (_AT(pteval_t, 3) << 6) /* HAP[2:1] */
-#define PTE_S2_XN (_AT(pteval_t, 2) << 53) /* XN[1:0] */
-#define PTE_S2_SW_RESVD (_AT(pteval_t, 15) << 55) /* Reserved for SW */
-
-#define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6) /* HAP[2:1] */
-#define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6) /* HAP[2:1] */
-#define PMD_S2_XN (_AT(pmdval_t, 2) << 53) /* XN[1:0] */
-#define PMD_S2_SW_RESVD (_AT(pmdval_t, 15) << 55) /* Reserved for SW */
-
-#define PUD_S2_RDONLY (_AT(pudval_t, 1) << 6) /* HAP[2:1] */
-#define PUD_S2_RDWR (_AT(pudval_t, 3) << 6) /* HAP[2:1] */
-#define PUD_S2_XN (_AT(pudval_t, 2) << 53) /* XN[1:0] */
-
/*
* Memory Attribute override for Stage-2 (MemAttr[3:0])
*/
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 88acd7e1cd05..8f094c43072a 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -73,19 +73,6 @@ extern bool arm64_use_ng_mappings;
__val; \
})
-#define PAGE_S2_XN \
- ({ \
- u64 __val; \
- if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC)) \
- __val = 0; \
- else \
- __val = PTE_S2_XN; \
- __val; \
- })
-
-#define PAGE_S2 __pgprot(_PROT_DEFAULT | PAGE_S2_MEMATTR(NORMAL) | PTE_S2_RDONLY | PAGE_S2_XN)
-#define PAGE_S2_DEVICE __pgprot(_PROT_DEFAULT | PAGE_S2_MEMATTR(DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_S2_XN)
-
#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN)
/* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */
#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h
index 996bf98f0cab..fe341a6578c3 100644
--- a/arch/arm64/include/asm/stage2_pgtable.h
+++ b/arch/arm64/include/asm/stage2_pgtable.h
@@ -8,7 +8,6 @@
#ifndef __ARM64_S2_PGTABLE_H_
#define __ARM64_S2_PGTABLE_H_
-#include <linux/hugetlb.h>
#include <linux/pgtable.h>
/*
@@ -36,21 +35,6 @@
#define stage2_pgdir_size(kvm) (1ULL << stage2_pgdir_shift(kvm))
#define stage2_pgdir_mask(kvm) ~(stage2_pgdir_size(kvm) - 1)
-/*
- * The number of PTRS across all concatenated stage2 tables given by the
- * number of bits resolved at the initial level.
- * If we force more levels than necessary, we may have (stage2_pgdir_shift > IPA),
- * in which case, stage2_pgd_ptrs will have one entry.
- */
-#define pgd_ptrs_shift(ipa, pgdir_shift) \
- ((ipa) > (pgdir_shift) ? ((ipa) - (pgdir_shift)) : 0)
-#define __s2_pgd_ptrs(ipa, lvls) \
- (1 << (pgd_ptrs_shift((ipa), pt_levels_pgdir_shift(lvls))))
-#define __s2_pgd_size(ipa, lvls) (__s2_pgd_ptrs((ipa), (lvls)) * sizeof(pgd_t))
-
-#define stage2_pgd_ptrs(kvm) __s2_pgd_ptrs(kvm_phys_shift(kvm), kvm_stage2_levels(kvm))
-#define stage2_pgd_size(kvm) __s2_pgd_size(kvm_phys_shift(kvm), kvm_stage2_levels(kvm))
-
/*
* kvm_mmmu_cache_min_pages() is the number of pages required to install
* a stage-2 translation. We pre-allocate the entry level page table at
@@ -58,196 +42,6 @@
*/
#define kvm_mmu_cache_min_pages(kvm) (kvm_stage2_levels(kvm) - 1)
-/* Stage2 PUD definitions when the level is present */
-static inline bool kvm_stage2_has_pud(struct kvm *kvm)
-{
- return (CONFIG_PGTABLE_LEVELS > 3) && (kvm_stage2_levels(kvm) > 3);
-}
-
-#define S2_PUD_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(1)
-#define S2_PUD_SIZE (1UL << S2_PUD_SHIFT)
-#define S2_PUD_MASK (~(S2_PUD_SIZE - 1))
-
-#define stage2_pgd_none(kvm, pgd) pgd_none(pgd)
-#define stage2_pgd_clear(kvm, pgd) pgd_clear(pgd)
-#define stage2_pgd_present(kvm, pgd) pgd_present(pgd)
-#define stage2_pgd_populate(kvm, pgd, p4d) pgd_populate(NULL, pgd, p4d)
-
-static inline p4d_t *stage2_p4d_offset(struct kvm *kvm,
- pgd_t *pgd, unsigned long address)
-{
- return p4d_offset(pgd, address);
-}
-
-static inline void stage2_p4d_free(struct kvm *kvm, p4d_t *p4d)
-{
-}
-
-static inline bool stage2_p4d_table_empty(struct kvm *kvm, p4d_t *p4dp)
-{
- return false;
-}
-
-static inline phys_addr_t stage2_p4d_addr_end(struct kvm *kvm,
- phys_addr_t addr, phys_addr_t end)
-{
- return end;
-}
-
-static inline bool stage2_p4d_none(struct kvm *kvm, p4d_t p4d)
-{
- if (kvm_stage2_has_pud(kvm))
- return p4d_none(p4d);
- else
- return 0;
-}
-
-static inline void stage2_p4d_clear(struct kvm *kvm, p4d_t *p4dp)
-{
- if (kvm_stage2_has_pud(kvm))
- p4d_clear(p4dp);
-}
-
-static inline bool stage2_p4d_present(struct kvm *kvm, p4d_t p4d)
-{
- if (kvm_stage2_has_pud(kvm))
- return p4d_present(p4d);
- else
- return 1;
-}
-
-static inline void stage2_p4d_populate(struct kvm *kvm, p4d_t *p4d, pud_t *pud)
-{
- if (kvm_stage2_has_pud(kvm))
- p4d_populate(NULL, p4d, pud);
-}
-
-static inline pud_t *stage2_pud_offset(struct kvm *kvm,
- p4d_t *p4d, unsigned long address)
-{
- if (kvm_stage2_has_pud(kvm))
- return pud_offset(p4d, address);
- else
- return (pud_t *)p4d;
-}
-
-static inline void stage2_pud_free(struct kvm *kvm, pud_t *pud)
-{
- if (kvm_stage2_has_pud(kvm))
- free_page((unsigned long)pud);
-}
-
-static inline bool stage2_pud_table_empty(struct kvm *kvm, pud_t *pudp)
-{
- if (kvm_stage2_has_pud(kvm))
- return kvm_page_empty(pudp);
- else
- return false;
-}
-
-static inline phys_addr_t
-stage2_pud_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)
-{
- if (kvm_stage2_has_pud(kvm)) {
- phys_addr_t boundary = (addr + S2_PUD_SIZE) & S2_PUD_MASK;
-
- return (boundary - 1 < end - 1) ? boundary : end;
- } else {
- return end;
- }
-}
-
-/* Stage2 PMD definitions when the level is present */
-static inline bool kvm_stage2_has_pmd(struct kvm *kvm)
-{
- return (CONFIG_PGTABLE_LEVELS > 2) && (kvm_stage2_levels(kvm) > 2);
-}
-
-#define S2_PMD_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(2)
-#define S2_PMD_SIZE (1UL << S2_PMD_SHIFT)
-#define S2_PMD_MASK (~(S2_PMD_SIZE - 1))
-
-static inline bool stage2_pud_none(struct kvm *kvm, pud_t pud)
-{
- if (kvm_stage2_has_pmd(kvm))
- return pud_none(pud);
- else
- return 0;
-}
-
-static inline void stage2_pud_clear(struct kvm *kvm, pud_t *pud)
-{
- if (kvm_stage2_has_pmd(kvm))
- pud_clear(pud);
-}
-
-static inline bool stage2_pud_present(struct kvm *kvm, pud_t pud)
-{
- if (kvm_stage2_has_pmd(kvm))
- return pud_present(pud);
- else
- return 1;
-}
-
-static inline void stage2_pud_populate(struct kvm *kvm, pud_t *pud, pmd_t *pmd)
-{
- if (kvm_stage2_has_pmd(kvm))
- pud_populate(NULL, pud, pmd);
-}
-
-static inline pmd_t *stage2_pmd_offset(struct kvm *kvm,
- pud_t *pud, unsigned long address)
-{
- if (kvm_stage2_has_pmd(kvm))
- return pmd_offset(pud, address);
- else
- return (pmd_t *)pud;
-}
-
-static inline void stage2_pmd_free(struct kvm *kvm, pmd_t *pmd)
-{
- if (kvm_stage2_has_pmd(kvm))
- free_page((unsigned long)pmd);
-}
-
-static inline bool stage2_pud_huge(struct kvm *kvm, pud_t pud)
-{
- if (kvm_stage2_has_pmd(kvm))
- return pud_huge(pud);
- else
- return 0;
-}
-
-static inline bool stage2_pmd_table_empty(struct kvm *kvm, pmd_t *pmdp)
-{
- if (kvm_stage2_has_pmd(kvm))
- return kvm_page_empty(pmdp);
- else
- return 0;
-}
-
-static inline phys_addr_t
-stage2_pmd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)
-{
- if (kvm_stage2_has_pmd(kvm)) {
- phys_addr_t boundary = (addr + S2_PMD_SIZE) & S2_PMD_MASK;
-
- return (boundary - 1 < end - 1) ? boundary : end;
- } else {
- return end;
- }
-}
-
-static inline bool stage2_pte_table_empty(struct kvm *kvm, pte_t *ptep)
-{
- return kvm_page_empty(ptep);
-}
-
-static inline unsigned long stage2_pgd_index(struct kvm *kvm, phys_addr_t addr)
-{
- return (((addr) >> stage2_pgdir_shift(kvm)) & (stage2_pgd_ptrs(kvm) - 1));
-}
-
static inline phys_addr_t
stage2_pgd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)
{
@@ -256,13 +50,4 @@ stage2_pgd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)
return (boundary - 1 < end - 1) ? boundary : end;
}
-/*
- * Level values for the ARMv8.4-TTL extension, mapping PUD/PMD/PTE and
- * the architectural page-table level.
- */
-#define S2_NO_LEVEL_HINT 0
-#define S2_PUD_LEVEL 1
-#define S2_PMD_LEVEL 2
-#define S2_PTE_LEVEL 3
-
#endif /* __ARM64_S2_PGTABLE_H_ */
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 548161795dd9..ba27eb537e9d 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -31,13 +31,6 @@ static phys_addr_t hyp_idmap_vector;
static unsigned long io_map_base;
-#define KVM_S2PTE_FLAG_IS_IOMAP (1UL << 0)
-#define KVM_S2_FLAG_LOGGING_ACTIVE (1UL << 1)
-
-static bool is_iomap(unsigned long flags)
-{
- return flags & KVM_S2PTE_FLAG_IS_IOMAP;
-}
/*
* Release kvm_mmu_lock periodically if the memory region is large. Otherwise,
@@ -90,154 +83,11 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu);
}
-static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
- int level)
-{
- kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa, level);
-}
-
-/*
- * D-Cache management functions. They take the page table entries by
- * value, as they are flushing the cache using the kernel mapping (or
- * kmap on 32bit).
- */
-static void kvm_flush_dcache_pte(pte_t pte)
-{
- __kvm_flush_dcache_pte(pte);
-}
-
-static void kvm_flush_dcache_pmd(pmd_t pmd)
-{
- __kvm_flush_dcache_pmd(pmd);
-}
-
-static void kvm_flush_dcache_pud(pud_t pud)
-{
- __kvm_flush_dcache_pud(pud);
-}
-
static bool kvm_is_device_pfn(unsigned long pfn)
{
return !pfn_valid(pfn);
}
-/**
- * stage2_dissolve_pmd() - clear and flush huge PMD entry
- * @mmu: pointer to mmu structure to operate on
- * @addr: IPA
- * @pmd: pmd pointer for IPA
- *
- * Function clears a PMD entry, flushes addr 1st and 2nd stage TLBs.
- */
-static void stage2_dissolve_pmd(struct kvm_s2_mmu *mmu, phys_addr_t addr, pmd_t *pmd)
-{
- if (!pmd_thp_or_huge(*pmd))
- return;
-
- pmd_clear(pmd);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL);
- put_page(virt_to_page(pmd));
-}
-
-/**
- * stage2_dissolve_pud() - clear and flush huge PUD entry
- * @mmu: pointer to mmu structure to operate on
- * @addr: IPA
- * @pud: pud pointer for IPA
- *
- * Function clears a PUD entry, flushes addr 1st and 2nd stage TLBs.
- */
-static void stage2_dissolve_pud(struct kvm_s2_mmu *mmu, phys_addr_t addr, pud_t *pudp)
-{
- struct kvm *kvm = mmu->kvm;
-
- if (!stage2_pud_huge(kvm, *pudp))
- return;
-
- stage2_pud_clear(kvm, pudp);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL);
- put_page(virt_to_page(pudp));
-}
-
-static void clear_stage2_pgd_entry(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- p4d_t *p4d_table __maybe_unused = stage2_p4d_offset(kvm, pgd, 0UL);
- stage2_pgd_clear(kvm, pgd);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT);
- stage2_p4d_free(kvm, p4d_table);
- put_page(virt_to_page(pgd));
-}
-
-static void clear_stage2_p4d_entry(struct kvm_s2_mmu *mmu, p4d_t *p4d, phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pud_table __maybe_unused = stage2_pud_offset(kvm, p4d, 0);
- stage2_p4d_clear(kvm, p4d);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT);
- stage2_pud_free(kvm, pud_table);
- put_page(virt_to_page(p4d));
-}
-
-static void clear_stage2_pud_entry(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- pmd_t *pmd_table __maybe_unused = stage2_pmd_offset(kvm, pud, 0);
-
- VM_BUG_ON(stage2_pud_huge(kvm, *pud));
- stage2_pud_clear(kvm, pud);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT);
- stage2_pmd_free(kvm, pmd_table);
- put_page(virt_to_page(pud));
-}
-
-static void clear_stage2_pmd_entry(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr_t addr)
-{
- pte_t *pte_table = pte_offset_kernel(pmd, 0);
- VM_BUG_ON(pmd_thp_or_huge(*pmd));
- pmd_clear(pmd);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT);
- free_page((unsigned long)pte_table);
- put_page(virt_to_page(pmd));
-}
-
-static inline void kvm_set_pte(pte_t *ptep, pte_t new_pte)
-{
- WRITE_ONCE(*ptep, new_pte);
- dsb(ishst);
-}
-
-static inline void kvm_set_pmd(pmd_t *pmdp, pmd_t new_pmd)
-{
- WRITE_ONCE(*pmdp, new_pmd);
- dsb(ishst);
-}
-
-static inline void kvm_pmd_populate(pmd_t *pmdp, pte_t *ptep)
-{
- kvm_set_pmd(pmdp, kvm_mk_pmd(ptep));
-}
-
-static inline void kvm_pud_populate(pud_t *pudp, pmd_t *pmdp)
-{
- WRITE_ONCE(*pudp, kvm_mk_pud(pmdp));
- dsb(ishst);
-}
-
-static inline void kvm_p4d_populate(p4d_t *p4dp, pud_t *pudp)
-{
- WRITE_ONCE(*p4dp, kvm_mk_p4d(pudp));
- dsb(ishst);
-}
-
-static inline void kvm_pgd_populate(pgd_t *pgdp, p4d_t *p4dp)
-{
-#ifndef __PAGETABLE_P4D_FOLDED
- WRITE_ONCE(*pgdp, kvm_mk_pgd(p4dp));
- dsb(ishst);
-#endif
-}
-
/*
* Unmapping vs dcache management:
*
@@ -262,108 +112,6 @@ static inline void kvm_pgd_populate(pgd_t *pgdp, p4d_t *p4dp)
* we then fully enforce cacheability of RAM, no matter what the guest
* does.
*/
-static void unmap_stage2_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd,
- phys_addr_t addr, phys_addr_t end)
-{
- phys_addr_t start_addr = addr;
- pte_t *pte, *start_pte;
-
- start_pte = pte = pte_offset_kernel(pmd, addr);
- do {
- if (!pte_none(*pte)) {
- pte_t old_pte = *pte;
-
- kvm_set_pte(pte, __pte(0));
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PTE_LEVEL);
-
- /* No need to invalidate the cache for device mappings */
- if (!kvm_is_device_pfn(pte_pfn(old_pte)))
- kvm_flush_dcache_pte(old_pte);
-
- put_page(virt_to_page(pte));
- }
- } while (pte++, addr += PAGE_SIZE, addr != end);
-
- if (stage2_pte_table_empty(mmu->kvm, start_pte))
- clear_stage2_pmd_entry(mmu, pmd, start_addr);
-}
-
-static void unmap_stage2_pmds(struct kvm_s2_mmu *mmu, pud_t *pud,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- phys_addr_t next, start_addr = addr;
- pmd_t *pmd, *start_pmd;
-
- start_pmd = pmd = stage2_pmd_offset(kvm, pud, addr);
- do {
- next = stage2_pmd_addr_end(kvm, addr, end);
- if (!pmd_none(*pmd)) {
- if (pmd_thp_or_huge(*pmd)) {
- pmd_t old_pmd = *pmd;
-
- pmd_clear(pmd);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL);
-
- kvm_flush_dcache_pmd(old_pmd);
-
- put_page(virt_to_page(pmd));
- } else {
- unmap_stage2_ptes(mmu, pmd, addr, next);
- }
- }
- } while (pmd++, addr = next, addr != end);
-
- if (stage2_pmd_table_empty(kvm, start_pmd))
- clear_stage2_pud_entry(mmu, pud, start_addr);
-}
-
-static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, p4d_t *p4d,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- phys_addr_t next, start_addr = addr;
- pud_t *pud, *start_pud;
-
- start_pud = pud = stage2_pud_offset(kvm, p4d, addr);
- do {
- next = stage2_pud_addr_end(kvm, addr, end);
- if (!stage2_pud_none(kvm, *pud)) {
- if (stage2_pud_huge(kvm, *pud)) {
- pud_t old_pud = *pud;
-
- stage2_pud_clear(kvm, pud);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL);
- kvm_flush_dcache_pud(old_pud);
- put_page(virt_to_page(pud));
- } else {
- unmap_stage2_pmds(mmu, pud, addr, next);
- }
- }
- } while (pud++, addr = next, addr != end);
-
- if (stage2_pud_table_empty(kvm, start_pud))
- clear_stage2_p4d_entry(mmu, p4d, start_addr);
-}
-
-static void unmap_stage2_p4ds(struct kvm_s2_mmu *mmu, pgd_t *pgd,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- phys_addr_t next, start_addr = addr;
- p4d_t *p4d, *start_p4d;
-
- start_p4d = p4d = stage2_p4d_offset(kvm, pgd, addr);
- do {
- next = stage2_p4d_addr_end(kvm, addr, end);
- if (!stage2_p4d_none(kvm, *p4d))
- unmap_stage2_puds(mmu, p4d, addr, next);
- } while (p4d++, addr = next, addr != end);
-
- if (stage2_p4d_table_empty(kvm, start_p4d))
- clear_stage2_pgd_entry(mmu, pgd, start_addr);
-}
-
/**
* unmap_stage2_range -- Clear stage2 page table entries to unmap a range
* @kvm: The VM pointer
@@ -392,71 +140,6 @@ static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 si
__unmap_stage2_range(mmu, start, size, true);
}
-static void stage2_flush_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd,
- phys_addr_t addr, phys_addr_t end)
-{
- pte_t *pte;
-
- pte = pte_offset_kernel(pmd, addr);
- do {
- if (!pte_none(*pte) && !kvm_is_device_pfn(pte_pfn(*pte)))
- kvm_flush_dcache_pte(*pte);
- } while (pte++, addr += PAGE_SIZE, addr != end);
-}
-
-static void stage2_flush_pmds(struct kvm_s2_mmu *mmu, pud_t *pud,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- pmd_t *pmd;
- phys_addr_t next;
-
- pmd = stage2_pmd_offset(kvm, pud, addr);
- do {
- next = stage2_pmd_addr_end(kvm, addr, end);
- if (!pmd_none(*pmd)) {
- if (pmd_thp_or_huge(*pmd))
- kvm_flush_dcache_pmd(*pmd);
- else
- stage2_flush_ptes(mmu, pmd, addr, next);
- }
- } while (pmd++, addr = next, addr != end);
-}
-
-static void stage2_flush_puds(struct kvm_s2_mmu *mmu, p4d_t *p4d,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pud;
- phys_addr_t next;
-
- pud = stage2_pud_offset(kvm, p4d, addr);
- do {
- next = stage2_pud_addr_end(kvm, addr, end);
- if (!stage2_pud_none(kvm, *pud)) {
- if (stage2_pud_huge(kvm, *pud))
- kvm_flush_dcache_pud(*pud);
- else
- stage2_flush_pmds(mmu, pud, addr, next);
- }
- } while (pud++, addr = next, addr != end);
-}
-
-static void stage2_flush_p4ds(struct kvm_s2_mmu *mmu, pgd_t *pgd,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- p4d_t *p4d;
- phys_addr_t next;
-
- p4d = stage2_p4d_offset(kvm, pgd, addr);
- do {
- next = stage2_p4d_addr_end(kvm, addr, end);
- if (!stage2_p4d_none(kvm, *p4d))
- stage2_flush_puds(mmu, p4d, addr, next);
- } while (p4d++, addr = next, addr != end);
-}
-
static void stage2_flush_memslot(struct kvm *kvm,
struct kvm_memory_slot *memslot)
{
@@ -805,348 +488,6 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
}
}
-static p4d_t *stage2_get_p4d(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- pgd_t *pgd;
- p4d_t *p4d;
-
- pgd = mmu->pgd + stage2_pgd_index(kvm, addr);
- if (stage2_pgd_none(kvm, *pgd)) {
- if (!cache)
- return NULL;
- p4d = kvm_mmu_memory_cache_alloc(cache);
- stage2_pgd_populate(kvm, pgd, p4d);
- get_page(virt_to_page(pgd));
- }
-
- return stage2_p4d_offset(kvm, pgd, addr);
-}
-
-static pud_t *stage2_get_pud(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- p4d_t *p4d;
- pud_t *pud;
-
- p4d = stage2_get_p4d(mmu, cache, addr);
- if (stage2_p4d_none(kvm, *p4d)) {
- if (!cache)
- return NULL;
- pud = kvm_mmu_memory_cache_alloc(cache);
- stage2_p4d_populate(kvm, p4d, pud);
- get_page(virt_to_page(p4d));
- }
-
- return stage2_pud_offset(kvm, p4d, addr);
-}
-
-static pmd_t *stage2_get_pmd(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pud;
- pmd_t *pmd;
-
- pud = stage2_get_pud(mmu, cache, addr);
- if (!pud || stage2_pud_huge(kvm, *pud))
- return NULL;
-
- if (stage2_pud_none(kvm, *pud)) {
- if (!cache)
- return NULL;
- pmd = kvm_mmu_memory_cache_alloc(cache);
- stage2_pud_populate(kvm, pud, pmd);
- get_page(virt_to_page(pud));
- }
-
- return stage2_pmd_offset(kvm, pud, addr);
-}
-
-static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu,
- struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr, const pmd_t *new_pmd)
-{
- pmd_t *pmd, old_pmd;
-
-retry:
- pmd = stage2_get_pmd(mmu, cache, addr);
- VM_BUG_ON(!pmd);
-
- old_pmd = *pmd;
- /*
- * Multiple vcpus faulting on the same PMD entry, can
- * lead to them sequentially updating the PMD with the
- * same value. Following the break-before-make
- * (pmd_clear() followed by tlb_flush()) process can
- * hinder forward progress due to refaults generated
- * on missing translations.
- *
- * Skip updating the page table if the entry is
- * unchanged.
- */
- if (pmd_val(old_pmd) == pmd_val(*new_pmd))
- return 0;
-
- if (pmd_present(old_pmd)) {
- /*
- * If we already have PTE level mapping for this block,
- * we must unmap it to avoid inconsistent TLB state and
- * leaking the table page. We could end up in this situation
- * if the memory slot was marked for dirty logging and was
- * reverted, leaving PTE level mappings for the pages accessed
- * during the period. So, unmap the PTE level mapping for this
- * block and retry, as we could have released the upper level
- * table in the process.
- *
- * Normal THP split/merge follows mmu_notifier callbacks and do
- * get handled accordingly.
- */
- if (!pmd_thp_or_huge(old_pmd)) {
- unmap_stage2_range(mmu, addr & S2_PMD_MASK, S2_PMD_SIZE);
- goto retry;
- }
- /*
- * Mapping in huge pages should only happen through a
- * fault. If a page is merged into a transparent huge
- * page, the individual subpages of that huge page
- * should be unmapped through MMU notifiers before we
- * get here.
- *
- * Merging of CompoundPages is not supported; they
- * should become splitting first, unmapped, merged,
- * and mapped back in on-demand.
- */
- WARN_ON_ONCE(pmd_pfn(old_pmd) != pmd_pfn(*new_pmd));
- pmd_clear(pmd);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL);
- } else {
- get_page(virt_to_page(pmd));
- }
-
- kvm_set_pmd(pmd, *new_pmd);
- return 0;
-}
-
-static int stage2_set_pud_huge(struct kvm_s2_mmu *mmu,
- struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr, const pud_t *new_pudp)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pudp, old_pud;
-
-retry:
- pudp = stage2_get_pud(mmu, cache, addr);
- VM_BUG_ON(!pudp);
-
- old_pud = *pudp;
-
- /*
- * A large number of vcpus faulting on the same stage 2 entry,
- * can lead to a refault due to the stage2_pud_clear()/tlb_flush().
- * Skip updating the page tables if there is no change.
- */
- if (pud_val(old_pud) == pud_val(*new_pudp))
- return 0;
-
- if (stage2_pud_present(kvm, old_pud)) {
- /*
- * If we already have table level mapping for this block, unmap
- * the range for this block and retry.
- */
- if (!stage2_pud_huge(kvm, old_pud)) {
- unmap_stage2_range(mmu, addr & S2_PUD_MASK, S2_PUD_SIZE);
- goto retry;
- }
-
- WARN_ON_ONCE(kvm_pud_pfn(old_pud) != kvm_pud_pfn(*new_pudp));
- stage2_pud_clear(kvm, pudp);
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL);
- } else {
- get_page(virt_to_page(pudp));
- }
-
- kvm_set_pud(pudp, *new_pudp);
- return 0;
-}
-
-/*
- * stage2_get_leaf_entry - walk the stage2 VM page tables and return
- * true if a valid and present leaf-entry is found. A pointer to the
- * leaf-entry is returned in the appropriate level variable - pudpp,
- * pmdpp, ptepp.
- */
-static bool stage2_get_leaf_entry(struct kvm_s2_mmu *mmu, phys_addr_t addr,
- pud_t **pudpp, pmd_t **pmdpp, pte_t **ptepp)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t *ptep;
-
- *pudpp = NULL;
- *pmdpp = NULL;
- *ptepp = NULL;
-
- pudp = stage2_get_pud(mmu, NULL, addr);
- if (!pudp || stage2_pud_none(kvm, *pudp) || !stage2_pud_present(kvm, *pudp))
- return false;
-
- if (stage2_pud_huge(kvm, *pudp)) {
- *pudpp = pudp;
- return true;
- }
-
- pmdp = stage2_pmd_offset(kvm, pudp, addr);
- if (!pmdp || pmd_none(*pmdp) || !pmd_present(*pmdp))
- return false;
-
- if (pmd_thp_or_huge(*pmdp)) {
- *pmdpp = pmdp;
- return true;
- }
-
- ptep = pte_offset_kernel(pmdp, addr);
- if (!ptep || pte_none(*ptep) || !pte_present(*ptep))
- return false;
-
- *ptepp = ptep;
- return true;
-}
-
-static bool stage2_is_exec(struct kvm_s2_mmu *mmu, phys_addr_t addr, unsigned long sz)
-{
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t *ptep;
- bool found;
-
- found = stage2_get_leaf_entry(mmu, addr, &pudp, &pmdp, &ptep);
- if (!found)
- return false;
-
- if (pudp)
- return sz <= PUD_SIZE && kvm_s2pud_exec(pudp);
- else if (pmdp)
- return sz <= PMD_SIZE && kvm_s2pmd_exec(pmdp);
- else
- return sz == PAGE_SIZE && kvm_s2pte_exec(ptep);
-}
-
-static int stage2_set_pte(struct kvm_s2_mmu *mmu,
- struct kvm_mmu_memory_cache *cache,
- phys_addr_t addr, const pte_t *new_pte,
- unsigned long flags)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte, old_pte;
- bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP;
- bool logging_active = flags & KVM_S2_FLAG_LOGGING_ACTIVE;
-
- VM_BUG_ON(logging_active && !cache);
-
- /* Create stage-2 page table mapping - Levels 0 and 1 */
- pud = stage2_get_pud(mmu, cache, addr);
- if (!pud) {
- /*
- * Ignore calls from kvm_set_spte_hva for unallocated
- * address ranges.
- */
- return 0;
- }
-
- /*
- * While dirty page logging - dissolve huge PUD, then continue
- * on to allocate page.
- */
- if (logging_active)
- stage2_dissolve_pud(mmu, addr, pud);
-
- if (stage2_pud_none(kvm, *pud)) {
- if (!cache)
- return 0; /* ignore calls from kvm_set_spte_hva */
- pmd = kvm_mmu_memory_cache_alloc(cache);
- stage2_pud_populate(kvm, pud, pmd);
- get_page(virt_to_page(pud));
- }
-
- pmd = stage2_pmd_offset(kvm, pud, addr);
- if (!pmd) {
- /*
- * Ignore calls from kvm_set_spte_hva for unallocated
- * address ranges.
- */
- return 0;
- }
-
- /*
- * While dirty page logging - dissolve huge PMD, then continue on to
- * allocate page.
- */
- if (logging_active)
- stage2_dissolve_pmd(mmu, addr, pmd);
-
- /* Create stage-2 page mappings - Level 2 */
- if (pmd_none(*pmd)) {
- if (!cache)
- return 0; /* ignore calls from kvm_set_spte_hva */
- pte = kvm_mmu_memory_cache_alloc(cache);
- kvm_pmd_populate(pmd, pte);
- get_page(virt_to_page(pmd));
- }
-
- pte = pte_offset_kernel(pmd, addr);
-
- if (iomap && pte_present(*pte))
- return -EFAULT;
-
- /* Create 2nd stage page table mapping - Level 3 */
- old_pte = *pte;
- if (pte_present(old_pte)) {
- /* Skip page table update if there is no change */
- if (pte_val(old_pte) == pte_val(*new_pte))
- return 0;
-
- kvm_set_pte(pte, __pte(0));
- kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PTE_LEVEL);
- } else {
- get_page(virt_to_page(pte));
- }
-
- kvm_set_pte(pte, *new_pte);
- return 0;
-}
-
-#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
-static int stage2_ptep_test_and_clear_young(pte_t *pte)
-{
- if (pte_young(*pte)) {
- *pte = pte_mkold(*pte);
- return 1;
- }
- return 0;
-}
-#else
-static int stage2_ptep_test_and_clear_young(pte_t *pte)
-{
- return __ptep_test_and_clear_young(pte);
-}
-#endif
-
-static int stage2_pmdp_test_and_clear_young(pmd_t *pmd)
-{
- return stage2_ptep_test_and_clear_young((pte_t *)pmd);
-}
-
-static int stage2_pudp_test_and_clear_young(pud_t *pud)
-{
- return stage2_ptep_test_and_clear_young((pte_t *)pud);
-}
-
/**
* kvm_phys_addr_ioremap - map a device range to guest IPA
*
@@ -1186,102 +527,6 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
return ret;
}
-/**
- * stage2_wp_ptes - write protect PMD range
- * @pmd: pointer to pmd entry
- * @addr: range start address
- * @end: range end address
- */
-static void stage2_wp_ptes(pmd_t *pmd, phys_addr_t addr, phys_addr_t end)
-{
- pte_t *pte;
-
- pte = pte_offset_kernel(pmd, addr);
- do {
- if (!pte_none(*pte)) {
- if (!kvm_s2pte_readonly(pte))
- kvm_set_s2pte_readonly(pte);
- }
- } while (pte++, addr += PAGE_SIZE, addr != end);
-}
-
-/**
- * stage2_wp_pmds - write protect PUD range
- * kvm: kvm instance for the VM
- * @pud: pointer to pud entry
- * @addr: range start address
- * @end: range end address
- */
-static void stage2_wp_pmds(struct kvm_s2_mmu *mmu, pud_t *pud,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- pmd_t *pmd;
- phys_addr_t next;
-
- pmd = stage2_pmd_offset(kvm, pud, addr);
-
- do {
- next = stage2_pmd_addr_end(kvm, addr, end);
- if (!pmd_none(*pmd)) {
- if (pmd_thp_or_huge(*pmd)) {
- if (!kvm_s2pmd_readonly(pmd))
- kvm_set_s2pmd_readonly(pmd);
- } else {
- stage2_wp_ptes(pmd, addr, next);
- }
- }
- } while (pmd++, addr = next, addr != end);
-}
-
-/**
- * stage2_wp_puds - write protect P4D range
- * @p4d: pointer to p4d entry
- * @addr: range start address
- * @end: range end address
- */
-static void stage2_wp_puds(struct kvm_s2_mmu *mmu, p4d_t *p4d,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- pud_t *pud;
- phys_addr_t next;
-
- pud = stage2_pud_offset(kvm, p4d, addr);
- do {
- next = stage2_pud_addr_end(kvm, addr, end);
- if (!stage2_pud_none(kvm, *pud)) {
- if (stage2_pud_huge(kvm, *pud)) {
- if (!kvm_s2pud_readonly(pud))
- kvm_set_s2pud_readonly(pud);
- } else {
- stage2_wp_pmds(mmu, pud, addr, next);
- }
- }
- } while (pud++, addr = next, addr != end);
-}
-
-/**
- * stage2_wp_p4ds - write protect PGD range
- * @pgd: pointer to pgd entry
- * @addr: range start address
- * @end: range end address
- */
-static void stage2_wp_p4ds(struct kvm_s2_mmu *mmu, pgd_t *pgd,
- phys_addr_t addr, phys_addr_t end)
-{
- struct kvm *kvm = mmu->kvm;
- p4d_t *p4d;
- phys_addr_t next;
-
- p4d = stage2_p4d_offset(kvm, pgd, addr);
- do {
- next = stage2_p4d_addr_end(kvm, addr, end);
- if (!stage2_p4d_none(kvm, *p4d))
- stage2_wp_puds(mmu, p4d, addr, next);
- } while (p4d++, addr = next, addr != end);
-}
-
/**
* stage2_wp_range() - write protect stage2 memory region range
* @kvm: The KVM pointer
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 19/21] KVM: arm64: Remove unused page-table code
2020-09-07 15:23 ` [PATCH v4 19/21] KVM: arm64: Remove unused page-table code Will Deacon
@ 2020-09-08 10:33 ` Marc Zyngier
2020-09-10 10:54 ` Will Deacon
0 siblings, 1 reply; 52+ messages in thread
From: Marc Zyngier @ 2020-09-08 10:33 UTC (permalink / raw)
To: Will Deacon; +Cc: Catalin Marinas, kernel-team, kvmarm, linux-arm-kernel
Hi Will,
On 2020-09-07 16:23, Will Deacon wrote:
> Now that KVM is using the generic page-table code to manage the guest
> stage-2 page-tables, we can remove a bunch of unused macros, #defines
> and static inline functions from the old implementation.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/include/asm/kvm_mmu.h | 171 ------
> arch/arm64/include/asm/pgtable-hwdef.h | 17 -
> arch/arm64/include/asm/pgtable-prot.h | 13 -
> arch/arm64/include/asm/stage2_pgtable.h | 215 -------
> arch/arm64/kvm/mmu.c | 755 ------------------------
> 5 files changed, 1171 deletions(-)
>
[...]
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h
> b/arch/arm64/include/asm/pgtable-hwdef.h
> index 1a989353144e..bb97d464f42b 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -172,23 +172,6 @@
> #define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2)
> #define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2)
>
> -/*
> - * 2nd stage PTE definitions
> - */
> -#define PTE_S2_RDONLY (_AT(pteval_t, 1) << 6) /* HAP[2:1] */
> -#define PTE_S2_RDWR (_AT(pteval_t, 3) << 6) /* HAP[2:1] */
> -#define PTE_S2_XN (_AT(pteval_t, 2) << 53) /* XN[1:0] */
> -#define PTE_S2_SW_RESVD (_AT(pteval_t, 15) << 55) /* Reserved for SW
> */
> -
> -#define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6) /* HAP[2:1] */
> -#define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6) /* HAP[2:1] */
> -#define PMD_S2_XN (_AT(pmdval_t, 2) << 53) /* XN[1:0] */
> -#define PMD_S2_SW_RESVD (_AT(pmdval_t, 15) << 55) /* Reserved for SW
> */
> -
> -#define PUD_S2_RDONLY (_AT(pudval_t, 1) << 6) /* HAP[2:1] */
> -#define PUD_S2_RDWR (_AT(pudval_t, 3) << 6) /* HAP[2:1] */
> -#define PUD_S2_XN (_AT(pudval_t, 2) << 53) /* XN[1:0] */
> -
> /*
> * Memory Attribute override for Stage-2 (MemAttr[3:0])
> */
This breaks the (still out of tree) NV patches, which use some of
these definitions as part of the architectural S2 PTW.
I can either keep a KVM-private copy, or revert this hunk. What do
you think?
M.
--
Jazz is not dead. It just smells funny...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 19/21] KVM: arm64: Remove unused page-table code
2020-09-08 10:33 ` Marc Zyngier
@ 2020-09-10 10:54 ` Will Deacon
0 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-10 10:54 UTC (permalink / raw)
To: Marc Zyngier; +Cc: Catalin Marinas, kernel-team, kvmarm, linux-arm-kernel
On Tue, Sep 08, 2020 at 11:33:35AM +0100, Marc Zyngier wrote:
> On 2020-09-07 16:23, Will Deacon wrote:
> > diff --git a/arch/arm64/include/asm/pgtable-hwdef.h
> > b/arch/arm64/include/asm/pgtable-hwdef.h
> > index 1a989353144e..bb97d464f42b 100644
> > --- a/arch/arm64/include/asm/pgtable-hwdef.h
> > +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> > @@ -172,23 +172,6 @@
> > #define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2)
> > #define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2)
> >
> > -/*
> > - * 2nd stage PTE definitions
> > - */
> > -#define PTE_S2_RDONLY (_AT(pteval_t, 1) << 6) /* HAP[2:1] */
> > -#define PTE_S2_RDWR (_AT(pteval_t, 3) << 6) /* HAP[2:1] */
> > -#define PTE_S2_XN (_AT(pteval_t, 2) << 53) /* XN[1:0] */
> > -#define PTE_S2_SW_RESVD (_AT(pteval_t, 15) << 55) /* Reserved for SW
> > */
> > -
> > -#define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6) /* HAP[2:1] */
> > -#define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6) /* HAP[2:1] */
> > -#define PMD_S2_XN (_AT(pmdval_t, 2) << 53) /* XN[1:0] */
> > -#define PMD_S2_SW_RESVD (_AT(pmdval_t, 15) << 55) /* Reserved for SW
> > */
> > -
> > -#define PUD_S2_RDONLY (_AT(pudval_t, 1) << 6) /* HAP[2:1] */
> > -#define PUD_S2_RDWR (_AT(pudval_t, 3) << 6) /* HAP[2:1] */
> > -#define PUD_S2_XN (_AT(pudval_t, 2) << 53) /* XN[1:0] */
> > -
> > /*
> > * Memory Attribute override for Stage-2 (MemAttr[3:0])
> > */
>
> This breaks the (still out of tree) NV patches, which use some of
> these definitions as part of the architectural S2 PTW.
>
> I can either keep a KVM-private copy, or revert this hunk. What do
> you think?
I'd be inclined to stick a private copy in your .c file, and perhaps we can
try to move NV over to the new page-table code in the future (which will
definitely need some extensions to handle the walk).
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v4 20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu'
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (18 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 19/21] KVM: arm64: Remove unused page-table code Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-07 15:23 ` [PATCH v4 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration Will Deacon
` (2 subsequent siblings)
22 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
The stage-2 page-tables are entirely encapsulated by the 'pgt' field of
'struct kvm_s2_mmu', so remove the unused 'pgd' field.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/kvm_host.h | 1 -
arch/arm64/kvm/mmu.c | 2 --
2 files changed, 3 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 0b7c702b2151..41caf29bd93c 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -79,7 +79,6 @@ struct kvm_s2_mmu {
* for vEL1/EL0 with vHCR_EL2.VM == 0. In that case, we use the
* canonical stage-2 page tables.
*/
- pgd_t *pgd;
phys_addr_t pgd_phys;
struct kvm_pgtable *pgt;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index ba27eb537e9d..1041be1fafe4 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -389,7 +389,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
mmu->kvm = kvm;
mmu->pgt = pgt;
mmu->pgd_phys = __pa(pgt->pgd);
- mmu->pgd = (void *)pgt->pgd;
mmu->vmid.vmid_gen = 0;
return 0;
@@ -475,7 +474,6 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
spin_lock(&kvm->mmu_lock);
pgt = mmu->pgt;
if (pgt) {
- mmu->pgd = NULL;
mmu->pgd_phys = 0;
mmu->pgt = NULL;
free_percpu(mmu->last_vcpu_ran);
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* [PATCH v4 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (19 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu' Will Deacon
@ 2020-09-07 15:23 ` Will Deacon
2020-09-09 14:53 ` Alexandru Elisei
2020-09-07 17:16 ` [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Marc Zyngier
2020-09-10 4:06 ` Gavin Shan
22 siblings, 1 reply; 52+ messages in thread
From: Will Deacon @ 2020-09-07 15:23 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Gavin Shan, Suzuki Poulose, Marc Zyngier,
Quentin Perret, Alexandru Elisei, James Morse, Catalin Marinas,
Will Deacon, linux-arm-kernel
Now that the guest stage-2 page-tables are managed independently from
the host stage-1 page-tables, we can avoid constraining the IPA size
based on the host and instead limit it only based on the PARange field
of the ID_AA64MMFR0 register.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/kvm/reset.c | 38 +++++---------------------------------
1 file changed, 5 insertions(+), 33 deletions(-)
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index ee33875c5c2a..471ee9234e40 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -339,7 +339,7 @@ u32 get_kvm_ipa_limit(void)
int kvm_set_ipa_limit(void)
{
- unsigned int ipa_max, pa_max, va_max, parange, tgran_2;
+ unsigned int parange, tgran_2;
u64 mmfr0;
mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
@@ -376,38 +376,10 @@ int kvm_set_ipa_limit(void)
break;
}
- pa_max = id_aa64mmfr0_parange_to_phys_shift(parange);
-
- /* Clamp the IPA limit to the PA size supported by the kernel */
- ipa_max = (pa_max > PHYS_MASK_SHIFT) ? PHYS_MASK_SHIFT : pa_max;
- /*
- * Since our stage2 table is dependent on the stage1 page table code,
- * we must always honor the following condition:
- *
- * Number of levels in Stage1 >= Number of levels in Stage2.
- *
- * So clamp the ipa limit further down to limit the number of levels.
- * Since we can concatenate upto 16 tables at entry level, we could
- * go upto 4bits above the maximum VA addressable with the current
- * number of levels.
- */
- va_max = PGDIR_SHIFT + PAGE_SHIFT - 3;
- va_max += 4;
-
- if (va_max < ipa_max)
- ipa_max = va_max;
-
- /*
- * If the final limit is lower than the real physical address
- * limit of the CPUs, report the reason.
- */
- if (ipa_max < pa_max)
- pr_info("kvm: Limiting the IPA size due to kernel %s Address limit\n",
- (va_max < pa_max) ? "Virtual" : "Physical");
-
- WARN(ipa_max < KVM_PHYS_SHIFT,
- "KVM IPA limit (%d bit) is smaller than default size\n", ipa_max);
- kvm_ipa_limit = ipa_max;
+ kvm_ipa_limit = id_aa64mmfr0_parange_to_phys_shift(parange);
+ WARN(kvm_ipa_limit < KVM_PHYS_SHIFT,
+ "KVM IPA limit (%d bit) is smaller than default size\n",
+ kvm_ipa_limit);
kvm_info("IPA Size Limit: %dbits\n", kvm_ipa_limit);
return 0;
--
2.28.0.526.ge36021eeef-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 52+ messages in thread* Re: [PATCH v4 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration
2020-09-07 15:23 ` [PATCH v4 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration Will Deacon
@ 2020-09-09 14:53 ` Alexandru Elisei
0 siblings, 0 replies; 52+ messages in thread
From: Alexandru Elisei @ 2020-09-09 14:53 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: Gavin Shan, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, kernel-team, linux-arm-kernel
Hi Will,
On 9/7/20 4:23 PM, Will Deacon wrote:
> Now that the guest stage-2 page-tables are managed independently from
> the host stage-1 page-tables, we can avoid constraining the IPA size
> based on the host and instead limit it only based on the PARange field
> of the ID_AA64MMFR0 register.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Quentin Perret <qperret@google.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
> arch/arm64/kvm/reset.c | 38 +++++---------------------------------
> 1 file changed, 5 insertions(+), 33 deletions(-)
>
> diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
> index ee33875c5c2a..471ee9234e40 100644
> --- a/arch/arm64/kvm/reset.c
> +++ b/arch/arm64/kvm/reset.c
> @@ -339,7 +339,7 @@ u32 get_kvm_ipa_limit(void)
>
> int kvm_set_ipa_limit(void)
> {
> - unsigned int ipa_max, pa_max, va_max, parange, tgran_2;
> + unsigned int parange, tgran_2;
> u64 mmfr0;
>
> mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
> @@ -376,38 +376,10 @@ int kvm_set_ipa_limit(void)
> break;
> }
>
> - pa_max = id_aa64mmfr0_parange_to_phys_shift(parange);
> -
> - /* Clamp the IPA limit to the PA size supported by the kernel */
> - ipa_max = (pa_max > PHYS_MASK_SHIFT) ? PHYS_MASK_SHIFT : pa_max;
> - /*
> - * Since our stage2 table is dependent on the stage1 page table code,
> - * we must always honor the following condition:
> - *
> - * Number of levels in Stage1 >= Number of levels in Stage2.
> - *
> - * So clamp the ipa limit further down to limit the number of levels.
> - * Since we can concatenate upto 16 tables at entry level, we could
> - * go upto 4bits above the maximum VA addressable with the current
> - * number of levels.
> - */
> - va_max = PGDIR_SHIFT + PAGE_SHIFT - 3;
> - va_max += 4;
> -
> - if (va_max < ipa_max)
> - ipa_max = va_max;
> -
> - /*
> - * If the final limit is lower than the real physical address
> - * limit of the CPUs, report the reason.
> - */
> - if (ipa_max < pa_max)
> - pr_info("kvm: Limiting the IPA size due to kernel %s Address limit\n",
> - (va_max < pa_max) ? "Virtual" : "Physical");
> -
> - WARN(ipa_max < KVM_PHYS_SHIFT,
> - "KVM IPA limit (%d bit) is smaller than default size\n", ipa_max);
> - kvm_ipa_limit = ipa_max;
> + kvm_ipa_limit = id_aa64mmfr0_parange_to_phys_shift(parange);
> + WARN(kvm_ipa_limit < KVM_PHYS_SHIFT,
> + "KVM IPA limit (%d bit) is smaller than default size\n",
> + kvm_ipa_limit);
> kvm_info("IPA Size Limit: %dbits\n", kvm_ipa_limit);
Bikeshedding: the two messages are slightly inconsistent: "IPA limit" vs "IPA Size
Limit" and "%d bit" vs "%dbits. Might be worth using the exact wording to make it
painfully obvious that the messages refer to the same variable, kvm_ipa_limit.
Thanks,
Alex
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (20 preceding siblings ...)
2020-09-07 15:23 ` [PATCH v4 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration Will Deacon
@ 2020-09-07 17:16 ` Marc Zyngier
2020-09-07 17:31 ` Will Deacon
2020-09-10 4:06 ` Gavin Shan
22 siblings, 1 reply; 52+ messages in thread
From: Marc Zyngier @ 2020-09-07 17:16 UTC (permalink / raw)
To: kvmarm, Will Deacon; +Cc: Catalin Marinas, kernel-team, linux-arm-kernel
On Mon, 7 Sep 2020 16:23:23 +0100, Will Deacon wrote:
> Here's version four of the KVM page-table rework I previously posted here:
>
> v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
> v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
> v3: https://lore.kernel.org/r/20200825093953.26493-1-will@kernel.org
>
> Changes since v3 include:
>
> [...]
I have provisionally applied this to kvmarm/next in order to give it
some exposure. Please have a look at the conflict resolution with
the latest fixes though.
[01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches()
commit: 4ae8bc9bde11b8d33cba71097d0ad078310f5696
[02/21] KVM: arm64: Add stand-alone page-table walker infrastructure
commit: 872b7ac07aca4d475b5580611d2a901c3cf81734
[03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables
commit: b12d0b7c5f340dcfb40e02a323e0c1304ac739c9
[04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables
commit: d0568315c2c777e38a2d6643e8ddd4f1a1135c88
[05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables
commit: b0742113046c4958d7d86ee5d5c2802aeb0b0b98
[06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table
commit: 21c40390c7f75846f086451336245c10ae8da576
[07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API
commit: 63cb38a43c88f679e58821b80c886a7bdb8e57fa
[08/21] KVM: arm64: Convert kvm_set_spte_hva() to generic page-table API
commit: da1a8a88d33405556dfc19d90515ff548c413642
[09/21] KVM: arm64: Convert unmap_stage2_range() to generic page-table API
commit: 7715a065a804933c050f4a2aaffd3eb1e1a8aec6
[10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table
commit: 6b7311f0e727cb8f06a62e694a0a9a11b16475f2
[11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API
commit: 7769a437f471da1ce6482613d83e4ba2815d8a3f
[12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table
commit: b4efa09299c0d23fde593d670e4de92ac292f72f
[13/21] KVM: arm64: Convert write-protect operation to generic page-table API
commit: 98258e3a1f3078e8458f3216d446e8306b6d9c9a
[14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table
commit: 68e974981a7526551cd0a0c975578a5c65881017
[15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API
commit: 45fef5ebfad012b546786886dda47112960c6cdc
[16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code
commit: ecec724683c6359a66786cd4f10cb2b6b5b2eec4
[17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API
commit: 62a763bccca7ca55666650c2340c1eeb0cabbf16
[18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table
commit: d8267676b4bdd18fe9799f3ef4073c5719d4f596
[19/21] KVM: arm64: Remove unused page-table code
commit: 480fc5a53118335767c56cb9836bb5fcb87e1197
[20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu'
commit: 593411fee44965e24b4f9cd560b6e40498917237
[21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration
commit: f8883302774bf876a76583ca1a85b15bf3e180f9
Cheers,
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling
2020-09-07 17:16 ` [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Marc Zyngier
@ 2020-09-07 17:31 ` Will Deacon
0 siblings, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-07 17:31 UTC (permalink / raw)
To: Marc Zyngier; +Cc: Catalin Marinas, kernel-team, kvmarm, linux-arm-kernel
On Mon, Sep 07, 2020 at 06:16:14PM +0100, Marc Zyngier wrote:
> On Mon, 7 Sep 2020 16:23:23 +0100, Will Deacon wrote:
> > Here's version four of the KVM page-table rework I previously posted here:
> >
> > v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
> > v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
> > v3: https://lore.kernel.org/r/20200825093953.26493-1-will@kernel.org
> >
> > Changes since v3 include:
> >
> > [...]
>
> I have provisionally applied this to kvmarm/next in order to give it
> some exposure. Please have a look at the conflict resolution with
> the latest fixes though.
Cheers, looks good to me. Holler if anything explodes.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling
2020-09-07 15:23 [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
` (21 preceding siblings ...)
2020-09-07 17:16 ` [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling Marc Zyngier
@ 2020-09-10 4:06 ` Gavin Shan
2020-09-10 4:11 ` Gavin Shan
2020-09-10 10:58 ` Will Deacon
22 siblings, 2 replies; 52+ messages in thread
From: Gavin Shan @ 2020-09-10 4:06 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, linux-arm-kernel
Hi Will and Marc,
On 9/8/20 1:23 AM, Will Deacon wrote:
> Hi all,
>
> Here's version four of the KVM page-table rework I previously posted here:
>
> v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
> v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
> v3: https://lore.kernel.org/r/20200825093953.26493-1-will@kernel.org
>
> Changes since v3 include:
>
> * Removed some more unused code from the old implementation
> * Renumbered KVM_PGTABLE_PROT_* to match file permissions in hex dumps
> * Simplified kvm_granule_shift() and removed kvm_start_level()
> * Only clear valid bit when invalidating a PTE, leaving phys addr intact
> * Fail gracefully when attempting to allocate without a memory cache
> * Rewrote stage2_apply_range() as a function instead of a macro
> * Fixed fault address rounding with hugetlbfs
> * Fixed permission fault handling when dirty logging is enabled
> * Added comments
> * Added reviewer tags
>
> Many thanks to Alex and Gavin for their comments.
>
> Will
>
[...]
Eventually, I'm lucky finding a machine from our pool where 16KB page
size is supported. I tried this series (v4) on this machine, everything
looks good as we expected:
Host: 5.9.0-rc3 + patchset (v4)
CONFIG_ARM64_VA_BITS=47
CONFIG_ARM64_PAGE_SHIFT=14
Hugepagesize: 32768 kB (from /proc/meminfo)
Config: -HugeTLBfs/-THP
+HugeTLBfs/-THP
-HugeTLBfs/THP
Guest: CONFIG_ARM64_PAGE_SHIFT=12/14/16
Let me know in case there are more tests needed from my side.
Thanks,
Gavin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread* Re: [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling
2020-09-10 4:06 ` Gavin Shan
@ 2020-09-10 4:11 ` Gavin Shan
2020-09-10 10:58 ` Will Deacon
1 sibling, 0 replies; 52+ messages in thread
From: Gavin Shan @ 2020-09-10 4:11 UTC (permalink / raw)
To: Will Deacon, kvmarm
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, linux-arm-kernel
On 9/10/20 2:06 PM, Gavin Shan wrote:
> Hi Will and Marc,
>
> On 9/8/20 1:23 AM, Will Deacon wrote:
>> Hi all,
>>
>> Here's version four of the KVM page-table rework I previously posted here:
>>
>> v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
>> v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
>> v3: https://lore.kernel.org/r/20200825093953.26493-1-will@kernel.org
>>
>> Changes since v3 include:
>>
>> * Removed some more unused code from the old implementation
>> * Renumbered KVM_PGTABLE_PROT_* to match file permissions in hex dumps
>> * Simplified kvm_granule_shift() and removed kvm_start_level()
>> * Only clear valid bit when invalidating a PTE, leaving phys addr intact
>> * Fail gracefully when attempting to allocate without a memory cache
>> * Rewrote stage2_apply_range() as a function instead of a macro
>> * Fixed fault address rounding with hugetlbfs
>> * Fixed permission fault handling when dirty logging is enabled
>> * Added comments
>> * Added reviewer tags
>>
>> Many thanks to Alex and Gavin for their comments.
>>
>> Will
>>
>
> [...]
>
> Eventually, I'm lucky finding a machine from our pool where 16KB page
> size is supported. I tried this series (v4) on this machine, everything
> looks good as we expected:
>
> Host: 5.9.0-rc3 + patchset (v4)
> CONFIG_ARM64_VA_BITS=47
> CONFIG_ARM64_PAGE_SHIFT=14
> Hugepagesize: 32768 kB (from /proc/meminfo)
> Config: -HugeTLBfs/-THP
> +HugeTLBfs/-THP
> -HugeTLBfs/THP
> Guest: CONFIG_ARM64_PAGE_SHIFT=12/14/16
>
> Let me know in case there are more tests needed from my side.
>
Here is the CPU info retrieved from dmidecode:
Handle 0x0004, DMI type 4, 48 bytes
Processor Information
Socket Designation: CPU 1
Type: Central Processor
Family: ARMv8
Manufacturer: Ampere(TM)
ID: 02 00 3F 50 00 00 00 00
Signature: Implementor 0x50, Variant 0x3, Architecture 15, Part 0x000, Revision 2
Version: eMAG
Voltage: 0.9 V
External Clock: 3000 MHz
Max Speed: 3300 MHz
Current Speed: 3000 MHz
Status: Populated, Enabled
Upgrade: None
L1 Cache Handle: 0x0005
L2 Cache Handle: 0x0006
L3 Cache Handle: 0x0007
Serial Number: 000000000000000000D7080502F180DC
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 32
Core Enabled: 32
Thread Count: 32
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization
Thanks,
Gavin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v4 00/21] KVM: arm64: Rewrite page-table code and fault handling
2020-09-10 4:06 ` Gavin Shan
2020-09-10 4:11 ` Gavin Shan
@ 2020-09-10 10:58 ` Will Deacon
1 sibling, 0 replies; 52+ messages in thread
From: Will Deacon @ 2020-09-10 10:58 UTC (permalink / raw)
To: Gavin Shan
Cc: kernel-team, Suzuki Poulose, Marc Zyngier, Quentin Perret,
James Morse, Catalin Marinas, Alexandru Elisei, kvmarm,
linux-arm-kernel
On Thu, Sep 10, 2020 at 02:06:04PM +1000, Gavin Shan wrote:
> On 9/8/20 1:23 AM, Will Deacon wrote:
> > Here's version four of the KVM page-table rework I previously posted here:
> >
> > v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
> > v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
> > v3: https://lore.kernel.org/r/20200825093953.26493-1-will@kernel.org
> >
> > Changes since v3 include:
> >
> > * Removed some more unused code from the old implementation
> > * Renumbered KVM_PGTABLE_PROT_* to match file permissions in hex dumps
> > * Simplified kvm_granule_shift() and removed kvm_start_level()
> > * Only clear valid bit when invalidating a PTE, leaving phys addr intact
> > * Fail gracefully when attempting to allocate without a memory cache
> > * Rewrote stage2_apply_range() as a function instead of a macro
> > * Fixed fault address rounding with hugetlbfs
> > * Fixed permission fault handling when dirty logging is enabled
> > * Added comments
> > * Added reviewer tags
> >
> > Many thanks to Alex and Gavin for their comments.
> >
> > Will
> >
>
> [...]
>
> Eventually, I'm lucky finding a machine from our pool where 16KB page
> size is supported. I tried this series (v4) on this machine, everything
> looks good as we expected:
Thanks, Gavin! I'll probably spin another version this week, just collecting
the last few comments I've had.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 52+ messages in thread