[PATCH 00/10] arm64: 52-bit physical address support

[PATCH 00/10] arm64: 52-bit physical address support

[Kristina Martsenko]

Hi,

This series adds 52-bit physical address space support to arm64, up from
the current 48 bits. This is an ARMv8.2 feature (ARMv8.2-LPA).

The series is based on 4.15-rc3. It has been lightly tested on an ARM
Fast Model. There's still some cases and areas to think through, as well
as more testing to do.

Patches for SMMU 52-bit PA support have been sent separately [1]. A GIC
ITS patch has already been merged [2]. ARMv8.2 also allows 52-bit IPA,
but support for that is not part of this series.

This version mostly addresses various review comments received.

Changes from RFC:
 - Split kconfig symbol into two patches, to enable 52-bit PA at the end
 - Patch #3: Changed phys_to_ttbr to use a macro, added an #include
 - Patch #4: Changed phys_to_pte to use a macro
 - Patch #6: Replaced __phys_to_pte with __phys_to_pte_val (same for
   pmd/pud/pgd)
 - Patch #6: Changed __phys_to_pte_val, __pte_to_phys, and
   pgtable-hwdef.h macros
 - Patches #5, #6: Removed kvm_extended_idmap_pgd, inlined its code,
   moved the comment
 - Patch #5: Added pfn_pud definition (to make the kernel build on that
   commit)

Thanks,
Kristina

[1] https://www.spinics.net/lists/arm-kernel/msg619040.html
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=30ae9610d275f8f03f5bf7612ce71d8af6fc400b


Kristina Martsenko (10):
  arm64: add kconfig symbol to configure physical address size
  arm64: limit PA size to supported range
  arm64: handle 52-bit addresses in TTBR
  arm64: head.S: handle 52-bit PAs in PTEs in early page table setup
  arm64: don't open code page table entry creation
  arm64: handle 52-bit physical addresses in page table entries
  arm64: increase PHYS_MASK to 52 bits
  arm64: increase sparsemem MAX_PHYSMEM_BITS to 52
  arm64: allow ID map to be extended to 52 bits
  arm64: enable 52-bit physical address support

 arch/arm/include/asm/kvm_mmu.h         |   7 ++
 arch/arm64/Kconfig                     |  29 ++++++++
 arch/arm64/include/asm/assembler.h     |  31 ++++++++-
 arch/arm64/include/asm/kvm_mmu.h       |  21 +++++-
 arch/arm64/include/asm/mmu_context.h   |  16 ++++-
 arch/arm64/include/asm/pgalloc.h       |   6 +-
 arch/arm64/include/asm/pgtable-hwdef.h |  19 +++++-
 arch/arm64/include/asm/pgtable.h       |  53 ++++++++++++---
 arch/arm64/include/asm/sparsemem.h     |   2 +-
 arch/arm64/include/asm/sysreg.h        |   8 +++
 arch/arm64/kernel/head.S               | 118 +++++++++++++++++++++------------
 arch/arm64/kernel/hibernate-asm.S      |  12 ++--
 arch/arm64/kernel/hibernate.c          |   5 +-
 arch/arm64/kvm/hyp-init.S              |  26 ++++----
 arch/arm64/kvm/hyp/s2-setup.c          |   2 +
 arch/arm64/mm/mmu.c                    |  15 +++--
 arch/arm64/mm/pgd.c                    |   8 +++
 arch/arm64/mm/proc.S                   |  19 +++---
 virt/kvm/arm/arm.c                     |   2 +-
 virt/kvm/arm/mmu.c                     |  12 ++--
 20 files changed, 302 insertions(+), 109 deletions(-)

-- 
2.1.4

[PATCH 01/10] arm64: add kconfig symbol to configure physical address size

[Kristina Martsenko]

ARMv8.2 introduces support for 52-bit physical addresses. To prepare for
supporting this, add a new kconfig symbol to configure the physical
address space size. The symbols will be used in subsequent patches.
Currently the only choice is 48, a later patch will add the option of 52
once the required code is in place.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/Kconfig | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index a93339f5178f..8dc937823eeb 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -646,6 +646,22 @@ config ARM64_VA_BITS
 	default 47 if ARM64_VA_BITS_47
 	default 48 if ARM64_VA_BITS_48
 
+choice
+	prompt "Physical address space size"
+	default ARM64_PA_BITS_48
+	help
+	  Choose the maximum physical address range that the kernel will
+	  support.
+
+config ARM64_PA_BITS_48
+	bool "48-bit"
+
+endchoice
+
+config ARM64_PA_BITS
+	int
+	default 48 if ARM64_PA_BITS_48
+
 config CPU_BIG_ENDIAN
        bool "Build big-endian kernel"
        help
-- 
2.1.4

[PATCH 02/10] arm64: limit PA size to supported range

[Kristina Martsenko]

We currently copy the physical address size from
ID_AA64MMFR0_EL1.PARange directly into TCR.(I)PS. This will not work for
4k and 16k granule kernels on systems that support 52-bit physical
addresses, since 52-bit addresses are only permitted with the 64k
granule.

To fix this, fall back to 48 bits when configuring the PA size when the
kernel does not support 52-bit PAs. When it does, fall back to 52, to
avoid similar problems in the future if the PA size is ever increased
above 52.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/assembler.h | 13 +++++++++++++
 arch/arm64/include/asm/sysreg.h    |  8 ++++++++
 arch/arm64/kvm/hyp-init.S          |  6 ++----
 arch/arm64/kvm/hyp/s2-setup.c      |  2 ++
 arch/arm64/mm/proc.S               |  6 ++----
 5 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index aef72d886677..6cddf12a0250 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -351,6 +351,19 @@ alternative_endif
 	.endm
 
 /*
+ * tcr_set_pa_size - set TCR.(I)PS to the highest supported
+ * ID_AA64MMFR0_EL1.PARange value
+ */
+	.macro	tcr_set_pa_size, tcr, pos, tmp0, tmp1
+	mrs	\tmp0, ID_AA64MMFR0_EL1
+	ubfx	\tmp0, \tmp0, #ID_AA64MMFR0_PARANGE_SHIFT, #3
+	mov	\tmp1, #ID_AA64MMFR0_PARANGE_MAX
+	cmp	\tmp0, \tmp1
+	csel	\tmp0, \tmp1, \tmp0, hi
+	bfi	\tcr, \tmp0, \pos, #3
+	.endm
+
+/*
  * Macro to perform a data cache maintenance for the interval
  * [kaddr, kaddr + size)
  *
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 08cc88574659..ec144f480b39 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -471,6 +471,14 @@
 #define ID_AA64MMFR0_TGRAN64_SUPPORTED	0x0
 #define ID_AA64MMFR0_TGRAN16_NI		0x0
 #define ID_AA64MMFR0_TGRAN16_SUPPORTED	0x1
+#define ID_AA64MMFR0_PARANGE_48		0x5
+#define ID_AA64MMFR0_PARANGE_52		0x6
+
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_52
+#else
+#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_48
+#endif
 
 /* id_aa64mmfr1 */
 #define ID_AA64MMFR1_PAN_SHIFT		20
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 3f9615582377..f731a48bd9f1 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -90,11 +90,9 @@ __do_hyp_init:
 	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
 #endif
 	/*
-	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
-	 * TCR_EL2.
+	 * Set the PS bits in TCR_EL2.
 	 */
-	mrs	x5, ID_AA64MMFR0_EL1
-	bfi	x4, x5, #16, #3
+	tcr_set_pa_size x4, #16, x5, x6
 
 	msr	tcr_el2, x4
 
diff --git a/arch/arm64/kvm/hyp/s2-setup.c b/arch/arm64/kvm/hyp/s2-setup.c
index a81f5e10fc8c..603e1ee83e89 100644
--- a/arch/arm64/kvm/hyp/s2-setup.c
+++ b/arch/arm64/kvm/hyp/s2-setup.c
@@ -32,6 +32,8 @@ u32 __hyp_text __init_stage2_translation(void)
 	 * PS is only 3. Fortunately, bit 19 is RES0 in VTCR_EL2...
 	 */
 	parange = read_sysreg(id_aa64mmfr0_el1) & 7;
+	if (parange > ID_AA64MMFR0_PARANGE_MAX)
+		parange = ID_AA64MMFR0_PARANGE_MAX;
 	val |= parange << 16;
 
 	/* Compute the actual PARange... */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 95233dfc4c39..c10c6c180961 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -228,11 +228,9 @@ ENTRY(__cpu_setup)
 	tcr_set_idmap_t0sz	x10, x9
 
 	/*
-	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the IPS bits in
-	 * TCR_EL1.
+	 * Set the IPS bits in TCR_EL1.
 	 */
-	mrs	x9, ID_AA64MMFR0_EL1
-	bfi	x10, x9, #32, #3
+	tcr_set_pa_size x10, #32, x5, x6
 #ifdef CONFIG_ARM64_HW_AFDBM
 	/*
 	 * Hardware update of the Access and Dirty bits.
-- 
2.1.4

[PATCH 03/10] arm64: handle 52-bit addresses in TTBR

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits 2..5
in the TTBR registers. Introduce a couple of macros to move the bits
there, and change all TTBR writers to use them.

Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
system without 52-bit PA can only use up to 48-bit PAs. A later patch in
this series will add a kconfig dependency to ensure PAN is configured.

In addition, when using 52-bit PA there is a special alignment
requirement on the top-level table. We don't currently have any VA_BITS
configuration that would violate the requirement, but one could be added
in the future, so add a compile-time BUG_ON to check for it.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h         |  2 ++
 arch/arm64/include/asm/assembler.h     | 16 ++++++++++++++++
 arch/arm64/include/asm/kvm_mmu.h       |  2 ++
 arch/arm64/include/asm/mmu_context.h   |  2 +-
 arch/arm64/include/asm/pgtable-hwdef.h |  9 +++++++++
 arch/arm64/include/asm/pgtable.h       |  6 ++++++
 arch/arm64/kernel/head.S               |  6 ++++--
 arch/arm64/kernel/hibernate-asm.S      | 12 +++++++-----
 arch/arm64/kernel/hibernate.c          |  2 +-
 arch/arm64/kvm/hyp-init.S              |  3 ++-
 arch/arm64/mm/pgd.c                    |  8 ++++++++
 arch/arm64/mm/proc.S                   | 13 ++++++++-----
 virt/kvm/arm/arm.c                     |  2 +-
 13 files changed, 67 insertions(+), 16 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index fa6f2174276b..8dbec683638b 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -221,6 +221,8 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return 8;
 }
 
+#define kvm_phys_to_vttbr(addr)		(addr)
+
 #endif	/* !__ASSEMBLY__ */
 
 #endif /* __ARM_KVM_MMU_H__ */
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 6cddf12a0250..2058fd864bfb 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -525,4 +525,20 @@ alternative_else_nop_endif
 #endif
 	.endm
 
+/*
+ * Arrange a physical address in a TTBR register, taking care of 52-bit
+ * addresses.
+ *
+ * 	phys:	physical address, preserved
+ * 	ttbr:	returns the TTBR value
+ */
+	.macro	phys_to_ttbr, phys, ttbr
+#ifdef CONFIG_ARM64_PA_BITS_52
+	orr	\ttbr, \phys, \phys, lsr #46
+	and	\ttbr, \ttbr, #TTBR_BADDR_MASK_52
+#else
+	mov	\ttbr, \phys
+#endif
+	.endm
+
 #endif	/* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 672c8684d5c2..747bfff92948 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -309,5 +309,7 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
 }
 
+#define kvm_phys_to_vttbr(addr)		phys_to_ttbr(addr)
+
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 9d155fa9a507..accc2ff32a0e 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -51,7 +51,7 @@ static inline void contextidr_thread_switch(struct task_struct *next)
  */
 static inline void cpu_set_reserved_ttbr0(void)
 {
-	unsigned long ttbr = __pa_symbol(empty_zero_page);
+	unsigned long ttbr = phys_to_ttbr(__pa_symbol(empty_zero_page));
 
 	write_sysreg(ttbr, ttbr0_el1);
 	isb();
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index eb0c2bd90de9..2b3104af79d0 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -16,6 +16,8 @@
 #ifndef __ASM_PGTABLE_HWDEF_H
 #define __ASM_PGTABLE_HWDEF_H
 
+#include <asm/memory.h>
+
 /*
  * Number of page-table levels required to address 'va_bits' wide
  * address, without section mapping. We resolve the top (va_bits - PAGE_SHIFT)
@@ -277,4 +279,11 @@
 #define TCR_HA			(UL(1) << 39)
 #define TCR_HD			(UL(1) << 40)
 
+/*
+ * TTBR
+ */
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define TTBR_BADDR_MASK_52	(((UL(1) << 46) - 1) << 2)
+#endif
+
 #endif
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 149d05fb9421..93677b9db947 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -733,6 +733,12 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
 #define kc_vaddr_to_offset(v)	((v) & ~VA_START)
 #define kc_offset_to_vaddr(o)	((o) | VA_START)
 
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define phys_to_ttbr(addr)	(((addr) | ((addr) >> 46)) & TTBR_BADDR_MASK_52)
+#else
+#define phys_to_ttbr(addr)	(addr)
+#endif
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __ASM_PGTABLE_H */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 67e86a0f57ac..0addea3760a6 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -679,8 +679,10 @@ ENTRY(__enable_mmu)
 	update_early_cpu_boot_status 0, x1, x2
 	adrp	x1, idmap_pg_dir
 	adrp	x2, swapper_pg_dir
-	msr	ttbr0_el1, x1			// load TTBR0
-	msr	ttbr1_el1, x2			// load TTBR1
+	phys_to_ttbr x1, x3
+	phys_to_ttbr x2, x4
+	msr	ttbr0_el1, x3			// load TTBR0
+	msr	ttbr1_el1, x4			// load TTBR1
 	isb
 	msr	sctlr_el1, x0
 	isb
diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
index e56d848b6466..84f5d52fddda 100644
--- a/arch/arm64/kernel/hibernate-asm.S
+++ b/arch/arm64/kernel/hibernate-asm.S
@@ -33,12 +33,14 @@
  * Even switching to our copied tables will cause a changed output address at
  * each stage of the walk.
  */
-.macro break_before_make_ttbr_switch zero_page, page_table
-	msr	ttbr1_el1, \zero_page
+.macro break_before_make_ttbr_switch zero_page, page_table, tmp
+	phys_to_ttbr \zero_page, \tmp
+	msr	ttbr1_el1, \tmp
 	isb
 	tlbi	vmalle1
 	dsb	nsh
-	msr	ttbr1_el1, \page_table
+	phys_to_ttbr \page_table, \tmp
+	msr	ttbr1_el1, \tmp
 	isb
 .endm
 
@@ -78,7 +80,7 @@ ENTRY(swsusp_arch_suspend_exit)
 	 * We execute from ttbr0, change ttbr1 to our copied linear map tables
 	 * with a break-before-make via the zero page
 	 */
-	break_before_make_ttbr_switch	x5, x0
+	break_before_make_ttbr_switch	x5, x0, x6
 
 	mov	x21, x1
 	mov	x30, x2
@@ -109,7 +111,7 @@ ENTRY(swsusp_arch_suspend_exit)
 	dsb	ish		/* wait for PoU cleaning to finish */
 
 	/* switch to the restored kernels page tables */
-	break_before_make_ttbr_switch	x25, x21
+	break_before_make_ttbr_switch	x25, x21, x6
 
 	ic	ialluis
 	dsb	ish
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 3009b8b80f08..efbf6dbd93c8 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -264,7 +264,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
 	 */
 	cpu_set_reserved_ttbr0();
 	local_flush_tlb_all();
-	write_sysreg(virt_to_phys(pgd), ttbr0_el1);
+	write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1);
 	isb();
 
 	*phys_dst_addr = virt_to_phys((void *)dst);
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index f731a48bd9f1..a99718f32af9 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -63,7 +63,8 @@ __do_hyp_init:
 	cmp	x0, #HVC_STUB_HCALL_NR
 	b.lo	__kvm_handle_stub_hvc
 
-	msr	ttbr0_el2, x0
+	phys_to_ttbr x0, x4
+	msr	ttbr0_el2, x4
 
 	mrs	x4, tcr_el1
 	ldr	x5, =TCR_EL2_MASK
diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c
index 051e71ec3335..289f9113a27a 100644
--- a/arch/arm64/mm/pgd.c
+++ b/arch/arm64/mm/pgd.c
@@ -49,6 +49,14 @@ void __init pgd_cache_init(void)
 	if (PGD_SIZE == PAGE_SIZE)
 		return;
 
+#ifdef CONFIG_ARM64_PA_BITS_52
+	/*
+	 * With 52-bit physical addresses, the architecture requires the
+	 * top-level table to be aligned to at least 64 bytes.
+	 */
+	BUILD_BUG_ON(PGD_SIZE < 64);
+#endif
+
 	/*
 	 * Naturally aligned pgds required by the architecture.
 	 */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index c10c6c180961..820afe2d0d91 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -138,10 +138,11 @@ ENDPROC(cpu_do_resume)
  *	- pgd_phys - physical address of new TTB
  */
 ENTRY(cpu_do_switch_mm)
-	pre_ttbr0_update_workaround x0, x2, x3
+	phys_to_ttbr x0, x2
+	pre_ttbr0_update_workaround x2, x3, x4
 	mmid	x1, x1				// get mm->context.id
-	bfi	x0, x1, #48, #16		// set the ASID
-	msr	ttbr0_el1, x0			// set TTBR0
+	bfi	x2, x1, #48, #16		// set the ASID
+	msr	ttbr0_el1, x2			// set TTBR0
 	isb
 	post_ttbr0_update_workaround
 	ret
@@ -158,14 +159,16 @@ ENTRY(idmap_cpu_replace_ttbr1)
 	save_and_disable_daif flags=x2
 
 	adrp	x1, empty_zero_page
-	msr	ttbr1_el1, x1
+	phys_to_ttbr x1, x3
+	msr	ttbr1_el1, x3
 	isb
 
 	tlbi	vmalle1
 	dsb	nsh
 	isb
 
-	msr	ttbr1_el1, x0
+	phys_to_ttbr x0, x3
+	msr	ttbr1_el1, x3
 	isb
 
 	restore_daif x2
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 6b60c98a6e22..c8d49879307f 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -509,7 +509,7 @@ static void update_vttbr(struct kvm *kvm)
 	pgd_phys = virt_to_phys(kvm->arch.pgd);
 	BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK);
 	vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits);
-	kvm->arch.vttbr = pgd_phys | vmid;
+	kvm->arch.vttbr = kvm_phys_to_vttbr(pgd_phys) | vmid;
 
 	spin_unlock(&kvm_vmid_lock);
 }
-- 
2.1.4

[PATCH 04/10] arm64: head.S: handle 52-bit PAs in PTEs in early page table setup

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits
12..15 in page table entries. Introduce a macro to move the bits there,
and change the early ID map and swapper table setup code to use it.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/pgtable-hwdef.h |  6 ++++++
 arch/arm64/kernel/head.S               | 36 +++++++++++++++++++++++++---------
 2 files changed, 33 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 2b3104af79d0..c59c69e02036 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -168,6 +168,12 @@
 #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
 #define PTE_HYP_XN		(_AT(pteval_t, 1) << 54)	/* HYP XN */
 
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define PTE_ADDR_LOW		(((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
+#define PTE_ADDR_HIGH		(_AT(pteval_t, 0xf) << 12)
+#define PTE_ADDR_MASK_52	(PTE_ADDR_LOW | PTE_ADDR_HIGH)
+#endif
+
 /*
  * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).
  */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 0addea3760a6..ddee8b347f60 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -148,6 +148,22 @@ preserve_boot_args:
 ENDPROC(preserve_boot_args)
 
 /*
+ * Macro to arrange a physical address in a page table entry, taking care of
+ * 52-bit addresses.
+ *
+ * Preserves:	phys
+ * Returns:	pte
+ */
+	.macro	phys_to_pte, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+	orr	\pte, \phys, \phys, lsr #36
+	and	\pte, \pte, #PTE_ADDR_MASK_52
+#else
+	mov	\pte, \phys
+#endif
+	.endm
+
+/*
  * Macro to create a table entry to the next page.
  *
  *	tbl:	page table address
@@ -160,10 +176,11 @@ ENDPROC(preserve_boot_args)
  * Returns:	tbl -> next level table page address
  */
 	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
+	add	\tmp1, \tbl, #PAGE_SIZE
+	phys_to_pte \tmp1, \tmp2
+	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	lsr	\tmp1, \virt, #\shift
 	and	\tmp1, \tmp1, #\ptrs - 1	// table index
-	add	\tmp2, \tbl, #PAGE_SIZE
-	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	str	\tmp2, [\tbl, \tmp1, lsl #3]
 	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
 	.endm
@@ -190,16 +207,17 @@ ENDPROC(preserve_boot_args)
  * virtual range (inclusive).
  *
  * Preserves:	tbl, flags
- * Corrupts:	phys, start, end, pstate
+ * Corrupts:	phys, start, end, tmp
  */
-	.macro	create_block_map, tbl, flags, phys, start, end
-	lsr	\phys, \phys, #SWAPPER_BLOCK_SHIFT
+	.macro	create_block_map, tbl, flags, phys, start, end, tmp
 	lsr	\start, \start, #SWAPPER_BLOCK_SHIFT
 	and	\start, \start, #PTRS_PER_PTE - 1	// table index
-	orr	\phys, \flags, \phys, lsl #SWAPPER_BLOCK_SHIFT	// table entry
+	bic	\phys, \phys, #SWAPPER_BLOCK_SIZE - 1
 	lsr	\end, \end, #SWAPPER_BLOCK_SHIFT
 	and	\end, \end, #PTRS_PER_PTE - 1		// table end index
-9999:	str	\phys, [\tbl, \start, lsl #3]		// store the entry
+9999:	phys_to_pte \phys, \tmp
+	orr	\tmp, \tmp, \flags			// table entry
+	str	\tmp, [\tbl, \start, lsl #3]		// store the entry
 	add	\start, \start, #1			// next entry
 	add	\phys, \phys, #SWAPPER_BLOCK_SIZE		// next block
 	cmp	\start, \end
@@ -286,7 +304,7 @@ __create_page_tables:
 	create_pgd_entry x0, x3, x5, x6
 	mov	x5, x3				// __pa(__idmap_text_start)
 	adr_l	x6, __idmap_text_end		// __pa(__idmap_text_end)
-	create_block_map x0, x7, x3, x5, x6
+	create_block_map x0, x7, x3, x5, x6, x4
 
 	/*
 	 * Map the kernel image (starting with PHYS_OFFSET).
@@ -299,7 +317,7 @@ __create_page_tables:
 	adrp	x3, _text			// runtime __pa(_text)
 	sub	x6, x6, x3			// _end - _text
 	add	x6, x6, x5			// runtime __va(_end)
-	create_block_map x0, x7, x3, x5, x6
+	create_block_map x0, x7, x3, x5, x6, x4
 
 	/*
 	 * Since the page tables have been populated with non-cacheable
-- 
2.1.4

[PATCH 05/10] arm64: don't open code page table entry creation

[Kristina Martsenko]

Instead of open coding the generation of page table entries, use the
macros/functions that exist for this - pfn_p*d and p*d_populate. Most
code in the kernel already uses these macros, this patch tries to fix
up the few places that don't. This is useful for the next patch in this
series, which needs to change the page table entry logic, and it's
better to have that logic in one place.

The KVM extended ID map is special, since we're creating a level above
CONFIG_PGTABLE_LEVELS and the required function isn't available. Leave
it as is and add a comment to explain it. (The normal kernel ID map code
doesn't need this change because its page tables are created in assembly
(__create_page_tables)).

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h |  5 +++++
 arch/arm64/include/asm/pgtable.h |  1 +
 arch/arm64/kernel/hibernate.c    |  3 +--
 arch/arm64/mm/mmu.c              | 14 +++++++++-----
 4 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 747bfff92948..9810ebf949b3 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -276,6 +276,11 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
 	return __cpu_uses_extended_idmap();
 }
 
+/*
+ * Can't use pgd_populate here, because the extended idmap adds an extra level
+ * above CONFIG_PGTABLE_LEVELS (which is 2 or 3 if we're using the extended
+ * idmap), and pgd_populate is only available if CONFIG_PGTABLE_LEVELS = 4.
+ */
 static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       pgd_t *hyp_pgd,
 				       pgd_t *merged_hyp_pgd,
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 93677b9db947..5d9554fb2692 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -355,6 +355,7 @@ static inline int pmd_protnone(pmd_t pmd)
 
 #define pud_write(pud)		pte_write(pud_pte(pud))
 #define pud_pfn(pud)		(((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
+#define pfn_pud(pfn,prot)	(__pud(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
 
 #define set_pmd_at(mm, addr, pmdp, pmd)	set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
 
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index efbf6dbd93c8..f20cf7e99249 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -247,8 +247,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
 	}
 
 	pte = pte_offset_kernel(pmd, dst_addr);
-	set_pte(pte, __pte(virt_to_phys((void *)dst) |
-			 pgprot_val(PAGE_KERNEL_EXEC)));
+	set_pte(pte, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC));
 
 	/*
 	 * Load our new page tables. A strict BBM approach requires that we
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 267d2b79d52d..0c631a17ae1d 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -570,8 +570,8 @@ static void __init map_kernel(pgd_t *pgd)
 		 * entry instead.
 		 */
 		BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES));
-		set_pud(pud_set_fixmap_offset(pgd, FIXADDR_START),
-			__pud(__pa_symbol(bm_pmd) | PUD_TYPE_TABLE));
+		pud_populate(&init_mm, pud_set_fixmap_offset(pgd, FIXADDR_START),
+			     lm_alias(bm_pmd));
 		pud_clear_fixmap();
 	} else {
 		BUG();
@@ -686,7 +686,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node)
 			if (!p)
 				return -ENOMEM;
 
-			set_pmd(pmd, __pmd(__pa(p) | PROT_SECT_NORMAL));
+			pmd_set_huge(pmd, __pa(p), __pgprot(PROT_SECT_NORMAL));
 		} else
 			vmemmap_verify((pte_t *)pmd, node, addr, next);
 	} while (addr = next, addr != end);
@@ -879,15 +879,19 @@ int __init arch_ioremap_pmd_supported(void)
 
 int pud_set_huge(pud_t *pud, phys_addr_t phys, pgprot_t prot)
 {
+	pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
+					pgprot_val(mk_sect_prot(prot)));
 	BUG_ON(phys & ~PUD_MASK);
-	set_pud(pud, __pud(phys | PUD_TYPE_SECT | pgprot_val(mk_sect_prot(prot))));
+	set_pud(pud, pfn_pud(__phys_to_pfn(phys), sect_prot));
 	return 1;
 }
 
 int pmd_set_huge(pmd_t *pmd, phys_addr_t phys, pgprot_t prot)
 {
+	pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
+					pgprot_val(mk_sect_prot(prot)));
 	BUG_ON(phys & ~PMD_MASK);
-	set_pmd(pmd, __pmd(phys | PMD_TYPE_SECT | pgprot_val(mk_sect_prot(prot))));
+	set_pmd(pmd, pfn_pmd(__phys_to_pfn(phys), sect_prot));
 	return 1;
 }
 
-- 
2.1.4

[PATCH 06/10] arm64: handle 52-bit physical addresses in page table entries

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits
12..15 of a page table entry. Introduce macros to convert between a
physical address and its placement in a table entry, and change all
macros/functions that access PTEs to use them.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h       |  7 +++--
 arch/arm64/include/asm/pgalloc.h       |  6 ++---
 arch/arm64/include/asm/pgtable-hwdef.h |  6 +++--
 arch/arm64/include/asm/pgtable.h       | 48 +++++++++++++++++++++++++---------
 arch/arm64/kernel/head.S               |  2 +-
 5 files changed, 49 insertions(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 9810ebf949b3..b3f7b68b042d 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -287,6 +287,7 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       unsigned long hyp_idmap_start)
 {
 	int idmap_idx;
+	u64 pgd_addr;
 
 	/*
 	 * Use the first entry to access the HYP mappings. It is
@@ -294,7 +295,8 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 	 * extended idmap.
 	 */
 	VM_BUG_ON(pgd_val(merged_hyp_pgd[0]));
-	merged_hyp_pgd[0] = __pgd(__pa(hyp_pgd) | PMD_TYPE_TABLE);
+	pgd_addr = __phys_to_pgd_val(__pa(hyp_pgd));
+	merged_hyp_pgd[0] = __pgd(pgd_addr | PMD_TYPE_TABLE);
 
 	/*
 	 * Create another extended level entry that points to the boot HYP map,
@@ -304,7 +306,8 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 	 */
 	idmap_idx = hyp_idmap_start >> VA_BITS;
 	VM_BUG_ON(pgd_val(merged_hyp_pgd[idmap_idx]));
-	merged_hyp_pgd[idmap_idx] = __pgd(__pa(boot_hyp_pgd) | PMD_TYPE_TABLE);
+	pgd_addr = __phys_to_pgd_val(__pa(boot_hyp_pgd));
+	merged_hyp_pgd[idmap_idx] = __pgd(pgd_addr | PMD_TYPE_TABLE);
 }
 
 static inline unsigned int kvm_get_vmid_bits(void)
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 5ca6a573a701..e9d9f1b006ef 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -44,7 +44,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
 
 static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot)
 {
-	set_pud(pud, __pud(pmd | prot));
+	set_pud(pud, __pud(__phys_to_pud_val(pmd) | prot));
 }
 
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
@@ -73,7 +73,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
 
 static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot)
 {
-	set_pgd(pgdp, __pgd(pud | prot));
+	set_pgd(pgdp, __pgd(__phys_to_pgd_val(pud) | prot));
 }
 
 static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
@@ -129,7 +129,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pte)
 static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t pte,
 				  pmdval_t prot)
 {
-	set_pmd(pmdp, __pmd(pte | prot));
+	set_pmd(pmdp, __pmd(__phys_to_pmd_val(pte) | prot));
 }
 
 /*
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index c59c69e02036..60e0432df559 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -168,10 +168,12 @@
 #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
 #define PTE_HYP_XN		(_AT(pteval_t, 1) << 54)	/* HYP XN */
 
-#ifdef CONFIG_ARM64_PA_BITS_52
 #define PTE_ADDR_LOW		(((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
+#ifdef CONFIG_ARM64_PA_BITS_52
 #define PTE_ADDR_HIGH		(_AT(pteval_t, 0xf) << 12)
-#define PTE_ADDR_MASK_52	(PTE_ADDR_LOW | PTE_ADDR_HIGH)
+#define PTE_ADDR_MASK		(PTE_ADDR_LOW | PTE_ADDR_HIGH)
+#else
+#define PTE_ADDR_MASK		PTE_ADDR_LOW
 #endif
 
 /*
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 5d9554fb2692..65856a81e692 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -57,9 +57,20 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
 
 #define pte_ERROR(pte)		__pte_error(__FILE__, __LINE__, pte_val(pte))
 
-#define pte_pfn(pte)		((pte_val(pte) & PHYS_MASK) >> PAGE_SHIFT)
+/*
+ * Macros to convert between a physical address and its placement in a
+ * page table entry, taking care of 52-bit addresses.
+ */
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define __pte_to_phys(pte)	((pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << 36))
+#define __phys_to_pte_val(phys)	(((phys) | ((phys) >> 36)) & PTE_ADDR_MASK)
+#else
+#define __pte_to_phys(pte)	(pte_val(pte) & PTE_ADDR_MASK)
+#define __phys_to_pte_val(phys)	(phys)
+#endif
 
-#define pfn_pte(pfn,prot)	(__pte(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+#define pte_pfn(pte)		(__pte_to_phys(pte) >> PAGE_SHIFT)
+#define pfn_pte(pfn,prot)	__pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
 
 #define pte_none(pte)		(!pte_val(pte))
 #define pte_clear(mm,addr,ptep)	set_pte(ptep, __pte(0))
@@ -284,6 +295,11 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
 
 #define __HAVE_ARCH_PTE_SPECIAL
 
+static inline pte_t pgd_pte(pgd_t pgd)
+{
+	return __pte(pgd_val(pgd));
+}
+
 static inline pte_t pud_pte(pud_t pud)
 {
 	return __pte(pud_val(pud));
@@ -349,16 +365,24 @@ static inline int pmd_protnone(pmd_t pmd)
 
 #define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
 
-#define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
-#define pfn_pmd(pfn,prot)	(__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+#define __pmd_to_phys(pmd)	__pte_to_phys(pmd_pte(pmd))
+#define __phys_to_pmd_val(phys)	__phys_to_pte_val(phys)
+#define pmd_pfn(pmd)		((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT)
+#define pfn_pmd(pfn,prot)	__pmd(__phys_to_pmd_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
 #define mk_pmd(page,prot)	pfn_pmd(page_to_pfn(page),prot)
 
 #define pud_write(pud)		pte_write(pud_pte(pud))
-#define pud_pfn(pud)		(((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
-#define pfn_pud(pfn,prot)	(__pud(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+
+#define __pud_to_phys(pud)	__pte_to_phys(pud_pte(pud))
+#define __phys_to_pud_val(phys)	__phys_to_pte_val(phys)
+#define pud_pfn(pud)		((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT)
+#define pfn_pud(pfn,prot)	__pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
 
 #define set_pmd_at(mm, addr, pmdp, pmd)	set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
 
+#define __pgd_to_phys(pgd)	__pte_to_phys(pgd_pte(pgd))
+#define __phys_to_pgd_val(phys)	__phys_to_pte_val(phys)
+
 #define __pgprot_modify(prot,mask,bits) \
 	__pgprot((pgprot_val(prot) & ~(mask)) | (bits))
 
@@ -409,7 +433,7 @@ static inline void pmd_clear(pmd_t *pmdp)
 
 static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
 {
-	return pmd_val(pmd) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pmd_to_phys(pmd);
 }
 
 /* Find an entry in the third-level page table. */
@@ -427,7 +451,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
 #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
 #define pte_clear_fixmap()		clear_fixmap(FIX_PTE)
 
-#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK))
+#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(__pmd_to_phys(pmd)))
 
 /* use ONLY for statically allocated translation tables */
 #define pte_offset_kimg(dir,addr)	((pte_t *)__phys_to_kimg(pte_offset_phys((dir), (addr))))
@@ -460,7 +484,7 @@ static inline void pud_clear(pud_t *pudp)
 
 static inline phys_addr_t pud_page_paddr(pud_t pud)
 {
-	return pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pud_to_phys(pud);
 }
 
 /* Find an entry in the second-level page table. */
@@ -473,7 +497,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud)
 #define pmd_set_fixmap_offset(pud, addr)	pmd_set_fixmap(pmd_offset_phys(pud, addr))
 #define pmd_clear_fixmap()		clear_fixmap(FIX_PMD)
 
-#define pud_page(pud)		pfn_to_page(__phys_to_pfn(pud_val(pud) & PHYS_MASK))
+#define pud_page(pud)		pfn_to_page(__phys_to_pfn(__pud_to_phys(pud)))
 
 /* use ONLY for statically allocated translation tables */
 #define pmd_offset_kimg(dir,addr)	((pmd_t *)__phys_to_kimg(pmd_offset_phys((dir), (addr))))
@@ -512,7 +536,7 @@ static inline void pgd_clear(pgd_t *pgdp)
 
 static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 {
-	return pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pgd_to_phys(pgd);
 }
 
 /* Find an entry in the frst-level page table. */
@@ -525,7 +549,7 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 #define pud_set_fixmap_offset(pgd, addr)	pud_set_fixmap(pud_offset_phys(pgd, addr))
 #define pud_clear_fixmap()		clear_fixmap(FIX_PUD)
 
-#define pgd_page(pgd)		pfn_to_page(__phys_to_pfn(pgd_val(pgd) & PHYS_MASK))
+#define pgd_page(pgd)		pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd)))
 
 /* use ONLY for statically allocated translation tables */
 #define pud_offset_kimg(dir,addr)	((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr))))
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index ddee8b347f60..64e09f207d1d 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -157,7 +157,7 @@ ENDPROC(preserve_boot_args)
 	.macro	phys_to_pte, phys, pte
 #ifdef CONFIG_ARM64_PA_BITS_52
 	orr	\pte, \phys, \phys, lsr #36
-	and	\pte, \pte, #PTE_ADDR_MASK_52
+	and	\pte, \pte, #PTE_ADDR_MASK
 #else
 	mov	\pte, \phys
 #endif
-- 
2.1.4

[PATCH 07/10] arm64: increase PHYS_MASK to 52 bits

[Kristina Martsenko]

PHYS_MASK_SHIFT represents the highest possible physical address
supported by the kernel, and is used in a number of places. In order to
support 52-bit physical memory, increase PHYS_MASK_SHIFT to 52 when
52-bit physical memory is configured, and retain 48 if it is not, to
e.g. keep IDMAP_PGTABLE_LEVELS accurate. (The kconfig option to set
ARM64_PA_BITS to 52 will be added in a later patch.)

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/pgtable-hwdef.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 60e0432df559..03fe82057bd1 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -206,7 +206,7 @@
 /*
  * Highest possible physical address supported.
  */
-#define PHYS_MASK_SHIFT		(48)
+#define PHYS_MASK_SHIFT		(CONFIG_ARM64_PA_BITS)
 #define PHYS_MASK		((UL(1) << PHYS_MASK_SHIFT) - 1)
 
 /*
-- 
2.1.4

[PATCH 08/10] arm64: increase sparsemem MAX_PHYSMEM_BITS to 52

[Kristina Martsenko]

Sparsemem is used to store page structs and other information for each
physical page in the system. In order to support 52-bit physical memory,
increase MAX_PHYSMEM_BITS to 52 when 52-bit physical memory is
configured. If it is not configured, then the kernel can't use 52-bit
memory, so leave MAX_PHYSMEM_BITS at 48 in that case. (The kconfig
option to set ARM64_PA_BITS to 52 will be added in a later patch.)

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
index 74a9d301819f..b299929fe56c 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -17,7 +17,7 @@
 #define __ASM_SPARSEMEM_H
 
 #ifdef CONFIG_SPARSEMEM
-#define MAX_PHYSMEM_BITS	48
+#define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
 #define SECTION_SIZE_BITS	30
 #endif
 
-- 
2.1.4

[PATCH 09/10] arm64: allow ID map to be extended to 52 bits

[Kristina Martsenko]

Currently, when using VA_BITS < 48, if the ID map text happens to be
placed in physical memory above VA_BITS, we increase the VA size (up to
48) and create a new table level, in order to map in the ID map text.
This is okay because the system always supports 48 bits of VA.

This patch extends the code such that if the system supports 52 bits of
VA, and the ID map text is placed that high up, then we increase the VA
size accordingly, up to 52.

One difference from the current implementation is that so far the
condition of VA_BITS < 48 has meant that the top level table is always
"full", with the maximum number of entries, and an extra table level is
always needed. Now, when VA_BITS = 48 (and using 64k pages), the top
level table is not full, and we simply need to increase the number of
entries in it, instead of creating a new table level.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h       |  5 +++
 arch/arm64/include/asm/assembler.h   |  2 -
 arch/arm64/include/asm/kvm_mmu.h     |  7 +++-
 arch/arm64/include/asm/mmu_context.h | 14 ++++++-
 arch/arm64/kernel/head.S             | 76 +++++++++++++++++++++---------------
 arch/arm64/kvm/hyp-init.S            | 17 ++++----
 arch/arm64/mm/mmu.c                  |  1 +
 virt/kvm/arm/mmu.c                   | 12 +++---
 8 files changed, 83 insertions(+), 51 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 8dbec683638b..8c5643e2eea4 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -211,6 +211,11 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
 	return false;
 }
 
+static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
+{
+	return PTRS_PER_PGD;
+}
+
 static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       pgd_t *hyp_pgd,
 				       pgd_t *merged_hyp_pgd,
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 2058fd864bfb..11719b11f4a7 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -344,10 +344,8 @@ alternative_endif
  * tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map
  */
 	.macro	tcr_set_idmap_t0sz, valreg, tmpreg
-#ifndef CONFIG_ARM64_VA_BITS_48
 	ldr_l	\tmpreg, idmap_t0sz
 	bfi	\valreg, \tmpreg, #TCR_T0SZ_OFFSET, #TCR_TxSZ_WIDTH
-#endif
 	.endm
 
 /*
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index b3f7b68b042d..8d663ca0d50c 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -273,7 +273,12 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
 
 static inline bool __kvm_cpu_uses_extended_idmap(void)
 {
-	return __cpu_uses_extended_idmap();
+	return __cpu_uses_extended_idmap_table();
+}
+
+static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
+{
+	return idmap_ptrs_per_pgd;
 }
 
 /*
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index accc2ff32a0e..043eed856b55 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -63,11 +63,21 @@ static inline void cpu_set_reserved_ttbr0(void)
  * physical memory, in which case it will be smaller.
  */
 extern u64 idmap_t0sz;
+extern u64 idmap_ptrs_per_pgd;
 
 static inline bool __cpu_uses_extended_idmap(void)
 {
-	return (!IS_ENABLED(CONFIG_ARM64_VA_BITS_48) &&
-		unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS)));
+	return unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS));
+}
+
+/*
+ * True if the extended ID map requires an extra level of translation table
+ * to be configured.
+ */
+static inline bool __cpu_uses_extended_idmap_table(void)
+{
+	return __cpu_uses_extended_idmap() &&
+		(idmap_ptrs_per_pgd == PTRS_PER_PGD);
 }
 
 /*
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 64e09f207d1d..465f70328ba0 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -172,7 +172,7 @@ ENDPROC(preserve_boot_args)
  *	ptrs:	#imm pointers per table page
  *
  * Preserves:	virt
- * Corrupts:	tmp1, tmp2
+ * Corrupts:	ptrs, tmp1, tmp2
  * Returns:	tbl -> next level table page address
  */
 	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
@@ -180,7 +180,8 @@ ENDPROC(preserve_boot_args)
 	phys_to_pte \tmp1, \tmp2
 	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	lsr	\tmp1, \virt, #\shift
-	and	\tmp1, \tmp1, #\ptrs - 1	// table index
+	sub	\ptrs, \ptrs, #1
+	and	\tmp1, \tmp1, \ptrs		// table index
 	str	\tmp2, [\tbl, \tmp1, lsl #3]
 	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
 	.endm
@@ -190,15 +191,17 @@ ENDPROC(preserve_boot_args)
  * block entry in the next level (tbl) for the given virtual address.
  *
  * Preserves:	tbl, next, virt
- * Corrupts:	tmp1, tmp2
+ * Corrupts:	ptrs_per_pgd, tmp1, tmp2
  */
-	.macro	create_pgd_entry, tbl, virt, tmp1, tmp2
-	create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
+	.macro	create_pgd_entry, tbl, virt, ptrs_per_pgd, tmp1, tmp2
+	create_table_entry \tbl, \virt, PGDIR_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #if SWAPPER_PGTABLE_LEVELS > 3
-	create_table_entry \tbl, \virt, PUD_SHIFT, PTRS_PER_PUD, \tmp1, \tmp2
+	mov	\ptrs_per_pgd, PTRS_PER_PUD
+	create_table_entry \tbl, \virt, PUD_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #endif
 #if SWAPPER_PGTABLE_LEVELS > 2
-	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, PTRS_PER_PTE, \tmp1, \tmp2
+	mov	\ptrs_per_pgd, PTRS_PER_PTE
+	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #endif
 	.endm
 
@@ -262,26 +265,13 @@ __create_page_tables:
 	adrp	x0, idmap_pg_dir
 	adrp	x3, __idmap_text_start		// __pa(__idmap_text_start)
 
-#ifndef CONFIG_ARM64_VA_BITS_48
-#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
-#define EXTRA_PTRS	(1 << (48 - EXTRA_SHIFT))
-
-	/*
-	 * If VA_BITS < 48, it may be too small to allow for an ID mapping to be
-	 * created that covers system RAM if that is located sufficiently high
-	 * in the physical address space. So for the ID map, use an extended
-	 * virtual range in that case, by configuring an additional translation
-	 * level.
-	 * First, we have to verify our assumption that the current value of
-	 * VA_BITS was chosen such that all translation levels are fully
-	 * utilised, and that lowering T0SZ will always result in an additional
-	 * translation level to be configured.
-	 */
-#if VA_BITS != EXTRA_SHIFT
-#error "Mismatch between VA_BITS and page size/number of translation levels"
-#endif
-
 	/*
+	 * VA_BITS may be too small to allow for an ID mapping to be created
+	 * that covers system RAM if that is located sufficiently high in the
+	 * physical address space. So for the ID map, use an extended virtual
+	 * range in that case, and configure an additional translation level
+	 * if needed.
+	 *
 	 * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the
 	 * entire ID map region can be mapped. As T0SZ == (64 - #bits used),
 	 * this number conveniently equals the number of leading zeroes in
@@ -290,18 +280,41 @@ __create_page_tables:
 	adrp	x5, __idmap_text_end
 	clz	x5, x5
 	cmp	x5, TCR_T0SZ(VA_BITS)	// default T0SZ small enough?
-	b.ge	1f			// .. then skip additional level
+	b.ge	1f			// .. then skip VA range extension
 
 	adr_l	x6, idmap_t0sz
 	str	x5, [x6]
 	dmb	sy
 	dc	ivac, x6		// Invalidate potentially stale cache line
 
-	create_table_entry x0, x3, EXTRA_SHIFT, EXTRA_PTRS, x5, x6
-1:
+#if (VA_BITS < 48)
+#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
+#define EXTRA_PTRS	(1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT))
+
+	/*
+	 * If VA_BITS < 48, we have to configure an additional table level.
+	 * First, we have to verify our assumption that the current value of
+	 * VA_BITS was chosen such that all translation levels are fully
+	 * utilised, and that lowering T0SZ will always result in an additional
+	 * translation level to be configured.
+	 */
+#if VA_BITS != EXTRA_SHIFT
+#error "Mismatch between VA_BITS and page size/number of translation levels"
 #endif
 
-	create_pgd_entry x0, x3, x5, x6
+	mov	x4, EXTRA_PTRS
+	create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6
+#else
+	/*
+	 * If VA_BITS == 48, we don't have to configure an additional
+	 * translation level, but the top-level table has more entries.
+	 */
+	mov	x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT)
+	str_l	x4, idmap_ptrs_per_pgd, x5
+#endif
+1:
+	ldr_l	x4, idmap_ptrs_per_pgd
+	create_pgd_entry x0, x3, x4, x5, x6
 	mov	x5, x3				// __pa(__idmap_text_start)
 	adr_l	x6, __idmap_text_end		// __pa(__idmap_text_end)
 	create_block_map x0, x7, x3, x5, x6, x4
@@ -312,7 +325,8 @@ __create_page_tables:
 	adrp	x0, swapper_pg_dir
 	mov_q	x5, KIMAGE_VADDR + TEXT_OFFSET	// compile time __va(_text)
 	add	x5, x5, x23			// add KASLR displacement
-	create_pgd_entry x0, x5, x3, x6
+	mov	x4, PTRS_PER_PGD
+	create_pgd_entry x0, x5, x4, x3, x6
 	adrp	x6, _end			// runtime __pa(_end)
 	adrp	x3, _text			// runtime __pa(_text)
 	sub	x6, x6, x3			// _end - _text
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index a99718f32af9..c2ebe4e992df 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -72,24 +72,23 @@ __do_hyp_init:
 	mov	x5, #TCR_EL2_RES1
 	orr	x4, x4, x5
 
-#ifndef CONFIG_ARM64_VA_BITS_48
 	/*
-	 * If we are running with VA_BITS < 48, we may be running with an extra
-	 * level of translation in the ID map. This is only the case if system
-	 * RAM is out of range for the currently configured page size and number
-	 * of translation levels, in which case we will also need the extra
-	 * level for the HYP ID map, or we won't be able to enable the EL2 MMU.
+	 * The ID map may be configured to use an extended virtual address
+	 * range. This is only the case if system RAM is out of range for the
+	 * currently configured page size and VA_BITS, in which case we will
+	 * also need the extended virtual range for the HYP ID map, or we won't
+	 * be able to enable the EL2 MMU.
 	 *
 	 * However, at EL2, there is only one TTBR register, and we can't switch
 	 * between translation tables *and* update TCR_EL2.T0SZ@the same
-	 * time. Bottom line: we need the extra level in *both* our translation
-	 * tables.
+	 * time. Bottom line: we need to use the extended range with *both* our
+	 * translation tables.
 	 *
 	 * So use the same T0SZ value we use for the ID map.
 	 */
 	ldr_l	x5, idmap_t0sz
 	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
-#endif
+
 	/*
 	 * Set the PS bits in TCR_EL2.
 	 */
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 0c631a17ae1d..baa34418c3bf 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -50,6 +50,7 @@
 #define NO_CONT_MAPPINGS	BIT(1)
 
 u64 idmap_t0sz = TCR_T0SZ(VA_BITS);
+u64 idmap_ptrs_per_pgd = PTRS_PER_PGD;
 
 u64 kimage_voffset __ro_after_init;
 EXPORT_SYMBOL(kimage_voffset);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b36945d49986..876caf531d32 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -623,7 +623,7 @@ static int create_hyp_pud_mappings(pgd_t *pgd, unsigned long start,
 	return 0;
 }
 
-static int __create_hyp_mappings(pgd_t *pgdp,
+static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
 				 unsigned long start, unsigned long end,
 				 unsigned long pfn, pgprot_t prot)
 {
@@ -636,7 +636,7 @@ static int __create_hyp_mappings(pgd_t *pgdp,
 	addr = start & PAGE_MASK;
 	end = PAGE_ALIGN(end);
 	do {
-		pgd = pgdp + pgd_index(addr);
+		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
 
 		if (pgd_none(*pgd)) {
 			pud = pud_alloc_one(NULL, addr);
@@ -699,8 +699,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 		int err;
 
 		phys_addr = kvm_kaddr_to_phys(from + virt_addr - start);
-		err = __create_hyp_mappings(hyp_pgd, virt_addr,
-					    virt_addr + PAGE_SIZE,
+		err = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD,
+					    virt_addr, virt_addr + PAGE_SIZE,
 					    __phys_to_pfn(phys_addr),
 					    prot);
 		if (err)
@@ -731,7 +731,7 @@ int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
 	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
 		return -EINVAL;
 
-	return __create_hyp_mappings(hyp_pgd, start, end,
+	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
 }
 
@@ -1737,7 +1737,7 @@ static int kvm_map_idmap_text(pgd_t *pgd)
 	int err;
 
 	/* Create the idmap in the boot page tables */
-	err = 	__create_hyp_mappings(pgd,
+	err = 	__create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
 				      hyp_idmap_start, hyp_idmap_end,
 				      __phys_to_pfn(hyp_idmap_start),
 				      PAGE_HYP_EXEC);
-- 
2.1.4

[PATCH 10/10] arm64: enable 52-bit physical address support

[Kristina Martsenko]

Now that 52-bit physical address support is in place, add the kconfig
symbol to enable it. As described in ARMv8.2, the larger addresses are
only supported with the 64k granule. Also ensure that PAN is configured
(or TTBR0 PAN is not), as explained in an earlier patch in this series.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/Kconfig | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 8dc937823eeb..337a696c9b02 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -656,11 +656,24 @@ choice
 config ARM64_PA_BITS_48
 	bool "48-bit"
 
+config ARM64_PA_BITS_52
+	bool "52-bit (ARMv8.2)"
+	depends on ARM64_64K_PAGES
+	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
+	help
+	  Enable support for a 52-bit physical address space, introduced as
+	  part of the ARMv8.2-LPA extension.
+
+	  With this enabled, the kernel will also continue to work on CPUs that
+	  do not support ARMv8.2-LPA, but with some added memory overhead (and
+	  minor performance overhead).
+
 endchoice
 
 config ARM64_PA_BITS
 	int
 	default 48 if ARM64_PA_BITS_48
+	default 52 if ARM64_PA_BITS_52
 
 config CPU_BIG_ENDIAN
        bool "Build big-endian kernel"
-- 
2.1.4

[PATCH 07/10] arm64: increase PHYS_MASK to 52 bits

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> PHYS_MASK_SHIFT represents the highest possible physical address
> supported by the kernel, and is used in a number of places. In order to
> support 52-bit physical memory, increase PHYS_MASK_SHIFT to 52 when
> 52-bit physical memory is configured, and retain 48 if it is not, to
> e.g. keep IDMAP_PGTABLE_LEVELS accurate. (The kconfig option to set
> ARM64_PA_BITS to 52 will be added in a later patch.)
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm64/include/asm/pgtable-hwdef.h | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index 60e0432df559..03fe82057bd1 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -206,7 +206,7 @@
>   /*
>    * Highest possible physical address supported.
>    */
> -#define PHYS_MASK_SHIFT		(48)
> +#define PHYS_MASK_SHIFT		(CONFIG_ARM64_PA_BITS)
>   #define PHYS_MASK		((UL(1) << PHYS_MASK_SHIFT) - 1)
>   
>   /*
> 

Kristina,

The changes look good to me. I believe this patch and the next one could be merged
to patch 1 in the series.

Eitherway,

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Cheers
Suzuki

[PATCH 01/10] arm64: add kconfig symbol to configure physical address size

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> ARMv8.2 introduces support for 52-bit physical addresses. To prepare for
> supporting this, add a new kconfig symbol to configure the physical
> address space size. The symbols will be used in subsequent patches.
> Currently the only choice is 48, a later patch will add the option of 52
> once the required code is in place.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm64/Kconfig | 16 ++++++++++++++++
>   1 file changed, 16 insertions(+)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index a93339f5178f..8dc937823eeb 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -646,6 +646,22 @@ config ARM64_VA_BITS
>   	default 47 if ARM64_VA_BITS_47
>   	default 48 if ARM64_VA_BITS_48
>   
> +choice
> +	prompt "Physical address space size"
> +	default ARM64_PA_BITS_48
> +	help
> +	  Choose the maximum physical address range that the kernel will
> +	  support.
> +
> +config ARM64_PA_BITS_48
> +	bool "48-bit"
> +
> +endchoice
> +
> +config ARM64_PA_BITS
> +	int
> +	default 48 if ARM64_PA_BITS_48
> +
>   config CPU_BIG_ENDIAN
>          bool "Build big-endian kernel"
>          help
> 

We could replace most of the hard coded "48" PA limit values to ARM64_PA_BITS,
now that we have a configurable entity. i.e, you could fold  patch 7 & 8 into
this one.

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

[PATCH 02/10] arm64: limit PA size to supported range

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> We currently copy the physical address size from
> ID_AA64MMFR0_EL1.PARange directly into TCR.(I)PS. This will not work for
> 4k and 16k granule kernels on systems that support 52-bit physical
> addresses, since 52-bit addresses are only permitted with the 64k
> granule.
> 
> To fix this, fall back to 48 bits when configuring the PA size when the
> kernel does not support 52-bit PAs. When it does, fall back to 52, to
> avoid similar problems in the future if the PA size is ever increased
> above 52.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm64/include/asm/assembler.h | 13 +++++++++++++
>   arch/arm64/include/asm/sysreg.h    |  8 ++++++++
>   arch/arm64/kvm/hyp-init.S          |  6 ++----
>   arch/arm64/kvm/hyp/s2-setup.c      |  2 ++
>   arch/arm64/mm/proc.S               |  6 ++----
>   5 files changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index aef72d886677..6cddf12a0250 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -351,6 +351,19 @@ alternative_endif
>   	.endm
>   
>   /*
> + * tcr_set_pa_size - set TCR.(I)PS to the highest supported
> + * ID_AA64MMFR0_EL1.PARange value
> + */
> +	.macro	tcr_set_pa_size, tcr, pos, tmp0, tmp1
> +	mrs	\tmp0, ID_AA64MMFR0_EL1
> +	ubfx	\tmp0, \tmp0, #ID_AA64MMFR0_PARANGE_SHIFT, #3
> +	mov	\tmp1, #ID_AA64MMFR0_PARANGE_MAX
> +	cmp	\tmp0, \tmp1
> +	csel	\tmp0, \tmp1, \tmp0, hi
> +	bfi	\tcr, \tmp0, \pos, #3
> +	.endm
> +
> +/*
>    * Macro to perform a data cache maintenance for the interval
>    * [kaddr, kaddr + size)
>    *
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 08cc88574659..ec144f480b39 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -471,6 +471,14 @@
>   #define ID_AA64MMFR0_TGRAN64_SUPPORTED	0x0
>   #define ID_AA64MMFR0_TGRAN16_NI		0x0
>   #define ID_AA64MMFR0_TGRAN16_SUPPORTED	0x1
> +#define ID_AA64MMFR0_PARANGE_48		0x5
> +#define ID_AA64MMFR0_PARANGE_52		0x6
> +
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_52
> +#else
> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_48
> +#endif
>   
>   /* id_aa64mmfr1 */
>   #define ID_AA64MMFR1_PAN_SHIFT		20
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index 3f9615582377..f731a48bd9f1 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -90,11 +90,9 @@ __do_hyp_init:
>   	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
>   #endif
>   	/*
> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
> -	 * TCR_EL2.
> +	 * Set the PS bits in TCR_EL2.
>   	 */
> -	mrs	x5, ID_AA64MMFR0_EL1
> -	bfi	x4, x5, #16, #3
> +	tcr_set_pa_size x4, #16, x5, x6
>   
>   	msr	tcr_el2, x4
>   
> diff --git a/arch/arm64/kvm/hyp/s2-setup.c b/arch/arm64/kvm/hyp/s2-setup.c
> index a81f5e10fc8c..603e1ee83e89 100644
> --- a/arch/arm64/kvm/hyp/s2-setup.c
> +++ b/arch/arm64/kvm/hyp/s2-setup.c
> @@ -32,6 +32,8 @@ u32 __hyp_text __init_stage2_translation(void)
>   	 * PS is only 3. Fortunately, bit 19 is RES0 in VTCR_EL2...
>   	 */
>   	parange = read_sysreg(id_aa64mmfr0_el1) & 7;
> +	if (parange > ID_AA64MMFR0_PARANGE_MAX)
> +		parange = ID_AA64MMFR0_PARANGE_MAX;
>   	val |= parange << 16;
>   
>   	/* Compute the actual PARange... */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 95233dfc4c39..c10c6c180961 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -228,11 +228,9 @@ ENTRY(__cpu_setup)
>   	tcr_set_idmap_t0sz	x10, x9
>   
>   	/*
> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the IPS bits in
> -	 * TCR_EL1.
> +	 * Set the IPS bits in TCR_EL1.
>   	 */
> -	mrs	x9, ID_AA64MMFR0_EL1
> -	bfi	x10, x9, #32, #3
> +	tcr_set_pa_size x10, #32, x5, x6

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

[PATCH 00/10] arm64: 52-bit physical address support

[Bob Picco]

Kristina Martsenko wrote:	[Wed Dec 13 2017, 12:07:15PM EST]
> Hi,
> 
> This series adds 52-bit physical address space support to arm64, up from
> the current 48 bits. This is an ARMv8.2 feature (ARMv8.2-LPA).
> 
> The series is based on 4.15-rc3. It has been lightly tested on an ARM
> Fast Model. There's still some cases and areas to think through, as well
> as more testing to do.
> 
> Patches for SMMU 52-bit PA support have been sent separately [1]. A GIC
> ITS patch has already been merged [2]. ARMv8.2 also allows 52-bit IPA,
> but support for that is not part of this series.
> 
> This version mostly addresses various review comments received.
> 
> Changes from RFC:
>  - Split kconfig symbol into two patches, to enable 52-bit PA at the end
>  - Patch #3: Changed phys_to_ttbr to use a macro, added an #include
>  - Patch #4: Changed phys_to_pte to use a macro
>  - Patch #6: Replaced __phys_to_pte with __phys_to_pte_val (same for
>    pmd/pud/pgd)
>  - Patch #6: Changed __phys_to_pte_val, __pte_to_phys, and
>    pgtable-hwdef.h macros
>  - Patches #5, #6: Removed kvm_extended_idmap_pgd, inlined its code,
>    moved the comment
>  - Patch #5: Added pfn_pud definition (to make the kernel build on that
>    commit)
> 
> Thanks,
> Kristina
> 
> [1] https://www.spinics.net/lists/arm-kernel/msg619040.html
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=30ae9610d275f8f03f5bf7612ce71d8af6fc400b
> 
> 
> Kristina Martsenko (10):
>   arm64: add kconfig symbol to configure physical address size
>   arm64: limit PA size to supported range
>   arm64: handle 52-bit addresses in TTBR
>   arm64: head.S: handle 52-bit PAs in PTEs in early page table setup
>   arm64: don't open code page table entry creation
>   arm64: handle 52-bit physical addresses in page table entries
>   arm64: increase PHYS_MASK to 52 bits
>   arm64: increase sparsemem MAX_PHYSMEM_BITS to 52
>   arm64: allow ID map to be extended to 52 bits
>   arm64: enable 52-bit physical address support
> 
>  arch/arm/include/asm/kvm_mmu.h         |   7 ++
>  arch/arm64/Kconfig                     |  29 ++++++++
>  arch/arm64/include/asm/assembler.h     |  31 ++++++++-
>  arch/arm64/include/asm/kvm_mmu.h       |  21 +++++-
>  arch/arm64/include/asm/mmu_context.h   |  16 ++++-
>  arch/arm64/include/asm/pgalloc.h       |   6 +-
>  arch/arm64/include/asm/pgtable-hwdef.h |  19 +++++-
>  arch/arm64/include/asm/pgtable.h       |  53 ++++++++++++---
>  arch/arm64/include/asm/sparsemem.h     |   2 +-
>  arch/arm64/include/asm/sysreg.h        |   8 +++
>  arch/arm64/kernel/head.S               | 118 +++++++++++++++++++++------------
>  arch/arm64/kernel/hibernate-asm.S      |  12 ++--
>  arch/arm64/kernel/hibernate.c          |   5 +-
>  arch/arm64/kvm/hyp-init.S              |  26 ++++----
>  arch/arm64/kvm/hyp/s2-setup.c          |   2 +
>  arch/arm64/mm/mmu.c                    |  15 +++--
>  arch/arm64/mm/pgd.c                    |   8 +++
>  arch/arm64/mm/proc.S                   |  19 +++---
>  virt/kvm/arm/arm.c                     |   2 +-
>  virt/kvm/arm/mmu.c                     |  12 ++--
>  20 files changed, 302 insertions(+), 109 deletions(-)
> 
> -- 
> 2.1.4
Hi Kristina,

I boot tested but on VM and had a couple issues. I will examine and share
should the issues be of value.
Tested-by: Bob Picco <bob.picco@oracle.com>

I reviewed and thank you. Your effort caused me to return to some code
examined/learned during the last few months.
Reviewed-by: Bob Picco <bob.picco@oracle.com>

bob
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 02/10] arm64: limit PA size to supported range

[Marc Zyngier]

On 13/12/17 17:07, Kristina Martsenko wrote:
> We currently copy the physical address size from
> ID_AA64MMFR0_EL1.PARange directly into TCR.(I)PS. This will not work for
> 4k and 16k granule kernels on systems that support 52-bit physical
> addresses, since 52-bit addresses are only permitted with the 64k
> granule.
> 
> To fix this, fall back to 48 bits when configuring the PA size when the
> kernel does not support 52-bit PAs. When it does, fall back to 52, to
> avoid similar problems in the future if the PA size is ever increased
> above 52.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>  arch/arm64/include/asm/assembler.h | 13 +++++++++++++
>  arch/arm64/include/asm/sysreg.h    |  8 ++++++++
>  arch/arm64/kvm/hyp-init.S          |  6 ++----
>  arch/arm64/kvm/hyp/s2-setup.c      |  2 ++
>  arch/arm64/mm/proc.S               |  6 ++----
>  5 files changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index aef72d886677..6cddf12a0250 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -351,6 +351,19 @@ alternative_endif
>  	.endm
>  
>  /*
> + * tcr_set_pa_size - set TCR.(I)PS to the highest supported
> + * ID_AA64MMFR0_EL1.PARange value

It'd be good to document what are the expected parameters here.

> + */
> +	.macro	tcr_set_pa_size, tcr, pos, tmp0, tmp1

Small nit: "tcr_compute_pa_size" would better describe what this does.

> +	mrs	\tmp0, ID_AA64MMFR0_EL1
> +	ubfx	\tmp0, \tmp0, #ID_AA64MMFR0_PARANGE_SHIFT, #3

It'd be good to have a comment explaining that we narrow the PARange to
fit the PS firld in TCR. Who knows what will happen once (if ever) we
get two other PARange extentions... ;-)

> +	mov	\tmp1, #ID_AA64MMFR0_PARANGE_MAX
> +	cmp	\tmp0, \tmp1
> +	csel	\tmp0, \tmp1, \tmp0, hi
> +	bfi	\tcr, \tmp0, \pos, #3> +	.endm
> +
> +/*
>   * Macro to perform a data cache maintenance for the interval
>   * [kaddr, kaddr + size)
>   *
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 08cc88574659..ec144f480b39 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -471,6 +471,14 @@
>  #define ID_AA64MMFR0_TGRAN64_SUPPORTED	0x0
>  #define ID_AA64MMFR0_TGRAN16_NI		0x0
>  #define ID_AA64MMFR0_TGRAN16_SUPPORTED	0x1
> +#define ID_AA64MMFR0_PARANGE_48		0x5
> +#define ID_AA64MMFR0_PARANGE_52		0x6
> +
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_52
> +#else
> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_48
> +#endif
>  
>  /* id_aa64mmfr1 */
>  #define ID_AA64MMFR1_PAN_SHIFT		20
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index 3f9615582377..f731a48bd9f1 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -90,11 +90,9 @@ __do_hyp_init:
>  	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
>  #endif
>  	/*
> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
> -	 * TCR_EL2.
> +	 * Set the PS bits in TCR_EL2.
>  	 */
> -	mrs	x5, ID_AA64MMFR0_EL1
> -	bfi	x4, x5, #16, #3
> +	tcr_set_pa_size x4, #16, x5, x6
>  
>  	msr	tcr_el2, x4
>  
> diff --git a/arch/arm64/kvm/hyp/s2-setup.c b/arch/arm64/kvm/hyp/s2-setup.c
> index a81f5e10fc8c..603e1ee83e89 100644
> --- a/arch/arm64/kvm/hyp/s2-setup.c
> +++ b/arch/arm64/kvm/hyp/s2-setup.c
> @@ -32,6 +32,8 @@ u32 __hyp_text __init_stage2_translation(void)
>  	 * PS is only 3. Fortunately, bit 19 is RES0 in VTCR_EL2...
>  	 */
>  	parange = read_sysreg(id_aa64mmfr0_el1) & 7;
> +	if (parange > ID_AA64MMFR0_PARANGE_MAX)
> +		parange = ID_AA64MMFR0_PARANGE_MAX;
>  	val |= parange << 16;
>  
>  	/* Compute the actual PARange... */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 95233dfc4c39..c10c6c180961 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -228,11 +228,9 @@ ENTRY(__cpu_setup)
>  	tcr_set_idmap_t0sz	x10, x9
>  
>  	/*
> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the IPS bits in
> -	 * TCR_EL1.
> +	 * Set the IPS bits in TCR_EL1.
>  	 */
> -	mrs	x9, ID_AA64MMFR0_EL1
> -	bfi	x10, x9, #32, #3
> +	tcr_set_pa_size x10, #32, x5, x6
>  #ifdef CONFIG_ARM64_HW_AFDBM
>  	/*
>  	 * Hardware update of the Access and Dirty bits.
> 

Other than the nits above:

Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 03/10] arm64: handle 52-bit addresses in TTBR

[Marc Zyngier]

On 13/12/17 17:07, Kristina Martsenko wrote:
> The top 4 bits of a 52-bit physical address are positioned at bits 2..5
> in the TTBR registers. Introduce a couple of macros to move the bits
> there, and change all TTBR writers to use them.
> 
> Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
> 52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
> system without 52-bit PA can only use up to 48-bit PAs. A later patch in
> this series will add a kconfig dependency to ensure PAN is configured.
> 
> In addition, when using 52-bit PA there is a special alignment
> requirement on the top-level table. We don't currently have any VA_BITS
> configuration that would violate the requirement, but one could be added
> in the future, so add a compile-time BUG_ON to check for it.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>  arch/arm/include/asm/kvm_mmu.h         |  2 ++
>  arch/arm64/include/asm/assembler.h     | 16 ++++++++++++++++
>  arch/arm64/include/asm/kvm_mmu.h       |  2 ++
>  arch/arm64/include/asm/mmu_context.h   |  2 +-
>  arch/arm64/include/asm/pgtable-hwdef.h |  9 +++++++++
>  arch/arm64/include/asm/pgtable.h       |  6 ++++++
>  arch/arm64/kernel/head.S               |  6 ++++--
>  arch/arm64/kernel/hibernate-asm.S      | 12 +++++++-----
>  arch/arm64/kernel/hibernate.c          |  2 +-
>  arch/arm64/kvm/hyp-init.S              |  3 ++-
>  arch/arm64/mm/pgd.c                    |  8 ++++++++
>  arch/arm64/mm/proc.S                   | 13 ++++++++-----
>  virt/kvm/arm/arm.c                     |  2 +-
>  13 files changed, 67 insertions(+), 16 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index fa6f2174276b..8dbec683638b 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -221,6 +221,8 @@ static inline unsigned int kvm_get_vmid_bits(void)
>  	return 8;
>  }
>  
> +#define kvm_phys_to_vttbr(addr)		(addr)
> +
>  #endif	/* !__ASSEMBLY__ */
>  
>  #endif /* __ARM_KVM_MMU_H__ */
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 6cddf12a0250..2058fd864bfb 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -525,4 +525,20 @@ alternative_else_nop_endif
>  #endif
>  	.endm
>  
> +/*
> + * Arrange a physical address in a TTBR register, taking care of 52-bit
> + * addresses.
> + *
> + * 	phys:	physical address, preserved
> + * 	ttbr:	returns the TTBR value
> + */
> +	.macro	phys_to_ttbr, phys, ttbr
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +	orr	\ttbr, \phys, \phys, lsr #46
> +	and	\ttbr, \ttbr, #TTBR_BADDR_MASK_52
> +#else
> +	mov	\ttbr, \phys
> +#endif
> +	.endm
> +
>  #endif	/* __ASM_ASSEMBLER_H */
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 672c8684d5c2..747bfff92948 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -309,5 +309,7 @@ static inline unsigned int kvm_get_vmid_bits(void)
>  	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>  }
>  
> +#define kvm_phys_to_vttbr(addr)		phys_to_ttbr(addr)
> +
>  #endif /* __ASSEMBLY__ */
>  #endif /* __ARM64_KVM_MMU_H__ */
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index 9d155fa9a507..accc2ff32a0e 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -51,7 +51,7 @@ static inline void contextidr_thread_switch(struct task_struct *next)
>   */
>  static inline void cpu_set_reserved_ttbr0(void)
>  {
> -	unsigned long ttbr = __pa_symbol(empty_zero_page);
> +	unsigned long ttbr = phys_to_ttbr(__pa_symbol(empty_zero_page));
>  
>  	write_sysreg(ttbr, ttbr0_el1);
>  	isb();
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index eb0c2bd90de9..2b3104af79d0 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -16,6 +16,8 @@
>  #ifndef __ASM_PGTABLE_HWDEF_H
>  #define __ASM_PGTABLE_HWDEF_H
>  
> +#include <asm/memory.h>
> +
>  /*
>   * Number of page-table levels required to address 'va_bits' wide
>   * address, without section mapping. We resolve the top (va_bits - PAGE_SHIFT)
> @@ -277,4 +279,11 @@
>  #define TCR_HA			(UL(1) << 39)
>  #define TCR_HD			(UL(1) << 40)
>  
> +/*
> + * TTBR
> + */
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define TTBR_BADDR_MASK_52	(((UL(1) << 46) - 1) << 2)

This really hurts by brain. How about

#define TTBR_BADDR_MASK_52	GENMASK_UL(47, 2)

instead, together with a comment saying that TTBR[1] is RES0.

> +#endif
> +
>  #endif
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index 149d05fb9421..93677b9db947 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -733,6 +733,12 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
>  #define kc_vaddr_to_offset(v)	((v) & ~VA_START)
>  #define kc_offset_to_vaddr(o)	((o) | VA_START)
>  
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define phys_to_ttbr(addr)	(((addr) | ((addr) >> 46)) & TTBR_BADDR_MASK_52)
> +#else
> +#define phys_to_ttbr(addr)	(addr)
> +#endif
> +
>  #endif /* !__ASSEMBLY__ */
>  
>  #endif /* __ASM_PGTABLE_H */
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 67e86a0f57ac..0addea3760a6 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -679,8 +679,10 @@ ENTRY(__enable_mmu)
>  	update_early_cpu_boot_status 0, x1, x2
>  	adrp	x1, idmap_pg_dir
>  	adrp	x2, swapper_pg_dir
> -	msr	ttbr0_el1, x1			// load TTBR0
> -	msr	ttbr1_el1, x2			// load TTBR1
> +	phys_to_ttbr x1, x3
> +	phys_to_ttbr x2, x4
> +	msr	ttbr0_el1, x3			// load TTBR0
> +	msr	ttbr1_el1, x4			// load TTBR1
>  	isb
>  	msr	sctlr_el1, x0
>  	isb
> diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
> index e56d848b6466..84f5d52fddda 100644
> --- a/arch/arm64/kernel/hibernate-asm.S
> +++ b/arch/arm64/kernel/hibernate-asm.S
> @@ -33,12 +33,14 @@
>   * Even switching to our copied tables will cause a changed output address at
>   * each stage of the walk.
>   */
> -.macro break_before_make_ttbr_switch zero_page, page_table
> -	msr	ttbr1_el1, \zero_page
> +.macro break_before_make_ttbr_switch zero_page, page_table, tmp
> +	phys_to_ttbr \zero_page, \tmp
> +	msr	ttbr1_el1, \tmp
>  	isb
>  	tlbi	vmalle1
>  	dsb	nsh
> -	msr	ttbr1_el1, \page_table
> +	phys_to_ttbr \page_table, \tmp
> +	msr	ttbr1_el1, \tmp
>  	isb
>  .endm
>  
> @@ -78,7 +80,7 @@ ENTRY(swsusp_arch_suspend_exit)
>  	 * We execute from ttbr0, change ttbr1 to our copied linear map tables
>  	 * with a break-before-make via the zero page
>  	 */
> -	break_before_make_ttbr_switch	x5, x0
> +	break_before_make_ttbr_switch	x5, x0, x6
>  
>  	mov	x21, x1
>  	mov	x30, x2
> @@ -109,7 +111,7 @@ ENTRY(swsusp_arch_suspend_exit)
>  	dsb	ish		/* wait for PoU cleaning to finish */
>  
>  	/* switch to the restored kernels page tables */
> -	break_before_make_ttbr_switch	x25, x21
> +	break_before_make_ttbr_switch	x25, x21, x6
>  
>  	ic	ialluis
>  	dsb	ish
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 3009b8b80f08..efbf6dbd93c8 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -264,7 +264,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
>  	 */
>  	cpu_set_reserved_ttbr0();
>  	local_flush_tlb_all();
> -	write_sysreg(virt_to_phys(pgd), ttbr0_el1);
> +	write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1);
>  	isb();
>  
>  	*phys_dst_addr = virt_to_phys((void *)dst);
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index f731a48bd9f1..a99718f32af9 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -63,7 +63,8 @@ __do_hyp_init:
>  	cmp	x0, #HVC_STUB_HCALL_NR
>  	b.lo	__kvm_handle_stub_hvc
>  
> -	msr	ttbr0_el2, x0
> +	phys_to_ttbr x0, x4
> +	msr	ttbr0_el2, x4
>  
>  	mrs	x4, tcr_el1
>  	ldr	x5, =TCR_EL2_MASK
> diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c
> index 051e71ec3335..289f9113a27a 100644
> --- a/arch/arm64/mm/pgd.c
> +++ b/arch/arm64/mm/pgd.c
> @@ -49,6 +49,14 @@ void __init pgd_cache_init(void)
>  	if (PGD_SIZE == PAGE_SIZE)
>  		return;
>  
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +	/*
> +	 * With 52-bit physical addresses, the architecture requires the
> +	 * top-level table to be aligned to at least 64 bytes.
> +	 */
> +	BUILD_BUG_ON(PGD_SIZE < 64);
> +#endif
> +
>  	/*
>  	 * Naturally aligned pgds required by the architecture.
>  	 */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index c10c6c180961..820afe2d0d91 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -138,10 +138,11 @@ ENDPROC(cpu_do_resume)
>   *	- pgd_phys - physical address of new TTB
>   */
>  ENTRY(cpu_do_switch_mm)
> -	pre_ttbr0_update_workaround x0, x2, x3
> +	phys_to_ttbr x0, x2
> +	pre_ttbr0_update_workaround x2, x3, x4
>  	mmid	x1, x1				// get mm->context.id
> -	bfi	x0, x1, #48, #16		// set the ASID
> -	msr	ttbr0_el1, x0			// set TTBR0
> +	bfi	x2, x1, #48, #16		// set the ASID
> +	msr	ttbr0_el1, x2			// set TTBR0
>  	isb
>  	post_ttbr0_update_workaround
>  	ret
> @@ -158,14 +159,16 @@ ENTRY(idmap_cpu_replace_ttbr1)
>  	save_and_disable_daif flags=x2
>  
>  	adrp	x1, empty_zero_page
> -	msr	ttbr1_el1, x1
> +	phys_to_ttbr x1, x3
> +	msr	ttbr1_el1, x3
>  	isb
>  
>  	tlbi	vmalle1
>  	dsb	nsh
>  	isb
>  
> -	msr	ttbr1_el1, x0
> +	phys_to_ttbr x0, x3
> +	msr	ttbr1_el1, x3
>  	isb
>  
>  	restore_daif x2
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 6b60c98a6e22..c8d49879307f 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -509,7 +509,7 @@ static void update_vttbr(struct kvm *kvm)
>  	pgd_phys = virt_to_phys(kvm->arch.pgd);
>  	BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK);
>  	vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits);
> -	kvm->arch.vttbr = pgd_phys | vmid;
> +	kvm->arch.vttbr = kvm_phys_to_vttbr(pgd_phys) | vmid;
>  
>  	spin_unlock(&kvm_vmid_lock);
>  }
> 

Otherwise:

Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 02/10] arm64: limit PA size to supported range

[Suzuki K Poulose]

On 14/12/17 18:34, Marc Zyngier wrote:
> On 13/12/17 17:07, Kristina Martsenko wrote:
>> We currently copy the physical address size from
>> ID_AA64MMFR0_EL1.PARange directly into TCR.(I)PS. This will not work for
>> 4k and 16k granule kernels on systems that support 52-bit physical
>> addresses, since 52-bit addresses are only permitted with the 64k
>> granule.
>>
>> To fix this, fall back to 48 bits when configuring the PA size when the
>> kernel does not support 52-bit PAs. When it does, fall back to 52, to
>> avoid similar problems in the future if the PA size is ever increased
>> above 52.
>>
>> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
>> ---
>>   arch/arm64/include/asm/assembler.h | 13 +++++++++++++
>>   arch/arm64/include/asm/sysreg.h    |  8 ++++++++
>>   arch/arm64/kvm/hyp-init.S          |  6 ++----
>>   arch/arm64/kvm/hyp/s2-setup.c      |  2 ++
>>   arch/arm64/mm/proc.S               |  6 ++----
>>   5 files changed, 27 insertions(+), 8 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
>> index aef72d886677..6cddf12a0250 100644
>> --- a/arch/arm64/include/asm/assembler.h
>> +++ b/arch/arm64/include/asm/assembler.h
>> @@ -351,6 +351,19 @@ alternative_endif
>>   	.endm
>>   
>>   /*
>> + * tcr_set_pa_size - set TCR.(I)PS to the highest supported
>> + * ID_AA64MMFR0_EL1.PARange value
> 
> It'd be good to document what are the expected parameters here.
> 
>> + */
>> +	.macro	tcr_set_pa_size, tcr, pos, tmp0, tmp1
> 
> Small nit: "tcr_compute_pa_size" would better describe what this does.
> 
>> +	mrs	\tmp0, ID_AA64MMFR0_EL1
>> +	ubfx	\tmp0, \tmp0, #ID_AA64MMFR0_PARANGE_SHIFT, #3
> 
> It'd be good to have a comment explaining that we narrow the PARange to
> fit the PS firld in TCR. Who knows what will happen once (if ever) we
> get two other PARange extentions... ;-)
> 
>> +	mov	\tmp1, #ID_AA64MMFR0_PARANGE_MAX
>> +	cmp	\tmp0, \tmp1
>> +	csel	\tmp0, \tmp1, \tmp0, hi
>> +	bfi	\tcr, \tmp0, \pos, #3> +	.endm
>> +
>> +/*
>>    * Macro to perform a data cache maintenance for the interval
>>    * [kaddr, kaddr + size)
>>    *
>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>> index 08cc88574659..ec144f480b39 100644
>> --- a/arch/arm64/include/asm/sysreg.h
>> +++ b/arch/arm64/include/asm/sysreg.h
>> @@ -471,6 +471,14 @@
>>   #define ID_AA64MMFR0_TGRAN64_SUPPORTED	0x0
>>   #define ID_AA64MMFR0_TGRAN16_NI		0x0
>>   #define ID_AA64MMFR0_TGRAN16_SUPPORTED	0x1
>> +#define ID_AA64MMFR0_PARANGE_48		0x5
>> +#define ID_AA64MMFR0_PARANGE_52		0x6
>> +
>> +#ifdef CONFIG_ARM64_PA_BITS_52
>> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_52
>> +#else
>> +#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_48
>> +#endif
>>   
>>   /* id_aa64mmfr1 */
>>   #define ID_AA64MMFR1_PAN_SHIFT		20
>> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
>> index 3f9615582377..f731a48bd9f1 100644
>> --- a/arch/arm64/kvm/hyp-init.S
>> +++ b/arch/arm64/kvm/hyp-init.S
>> @@ -90,11 +90,9 @@ __do_hyp_init:
>>   	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
>>   #endif
>>   	/*
>> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
>> -	 * TCR_EL2.
>> +	 * Set the PS bits in TCR_EL2.
>>   	 */
>> -	mrs	x5, ID_AA64MMFR0_EL1
>> -	bfi	x4, x5, #16, #3
>> +	tcr_set_pa_size x4, #16, x5, x6
>>   
>>   	msr	tcr_el2, x4
>>   
>> diff --git a/arch/arm64/kvm/hyp/s2-setup.c b/arch/arm64/kvm/hyp/s2-setup.c
>> index a81f5e10fc8c..603e1ee83e89 100644
>> --- a/arch/arm64/kvm/hyp/s2-setup.c
>> +++ b/arch/arm64/kvm/hyp/s2-setup.c
>> @@ -32,6 +32,8 @@ u32 __hyp_text __init_stage2_translation(void)
>>   	 * PS is only 3. Fortunately, bit 19 is RES0 in VTCR_EL2...
>>   	 */
>>   	parange = read_sysreg(id_aa64mmfr0_el1) & 7;
>> +	if (parange > ID_AA64MMFR0_PARANGE_MAX)
>> +		parange = ID_AA64MMFR0_PARANGE_MAX;
>>   	val |= parange << 16;
>>   
>>   	/* Compute the actual PARange... */
>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
>> index 95233dfc4c39..c10c6c180961 100644
>> --- a/arch/arm64/mm/proc.S
>> +++ b/arch/arm64/mm/proc.S
>> @@ -228,11 +228,9 @@ ENTRY(__cpu_setup)
>>   	tcr_set_idmap_t0sz	x10, x9
>>   
>>   	/*
>> -	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the IPS bits in
>> -	 * TCR_EL1.
>> +	 * Set the IPS bits in TCR_EL1.
>>   	 */
>> -	mrs	x9, ID_AA64MMFR0_EL1
>> -	bfi	x10, x9, #32, #3
>> +	tcr_set_pa_size x10, #32, x5, x6
>>   #ifdef CONFIG_ARM64_HW_AFDBM
>>   	/*
>>   	 * Hardware update of the Access and Dirty bits.
>>
> 
> Other than the nits above:
> 

Kristina,

If you are spinning another version correcting those, please could you
also add the bit definitions for TCR_EL2_PS and TCR_EL1_IPS and use them
here instead of the constants above ?

Cheers
Suzuki

[PATCH 09/10] arm64: allow ID map to be extended to 52 bits

[Marc Zyngier]

On 13/12/17 17:07, Kristina Martsenko wrote:
> Currently, when using VA_BITS < 48, if the ID map text happens to be
> placed in physical memory above VA_BITS, we increase the VA size (up to
> 48) and create a new table level, in order to map in the ID map text.
> This is okay because the system always supports 48 bits of VA.
> 
> This patch extends the code such that if the system supports 52 bits of
> VA, and the ID map text is placed that high up, then we increase the VA
> size accordingly, up to 52.
> 
> One difference from the current implementation is that so far the
> condition of VA_BITS < 48 has meant that the top level table is always
> "full", with the maximum number of entries, and an extra table level is
> always needed. Now, when VA_BITS = 48 (and using 64k pages), the top
> level table is not full, and we simply need to increase the number of
> entries in it, instead of creating a new table level.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>  arch/arm/include/asm/kvm_mmu.h       |  5 +++
>  arch/arm64/include/asm/assembler.h   |  2 -
>  arch/arm64/include/asm/kvm_mmu.h     |  7 +++-
>  arch/arm64/include/asm/mmu_context.h | 14 ++++++-
>  arch/arm64/kernel/head.S             | 76 +++++++++++++++++++++---------------
>  arch/arm64/kvm/hyp-init.S            | 17 ++++----
>  arch/arm64/mm/mmu.c                  |  1 +
>  virt/kvm/arm/mmu.c                   | 12 +++---
>  8 files changed, 83 insertions(+), 51 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index 8dbec683638b..8c5643e2eea4 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -211,6 +211,11 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
>  	return false;
>  }
>  
> +static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
> +{
> +	return PTRS_PER_PGD;
> +}
> +
>  static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
>  				       pgd_t *hyp_pgd,
>  				       pgd_t *merged_hyp_pgd,
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 2058fd864bfb..11719b11f4a7 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -344,10 +344,8 @@ alternative_endif
>   * tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map
>   */
>  	.macro	tcr_set_idmap_t0sz, valreg, tmpreg
> -#ifndef CONFIG_ARM64_VA_BITS_48
>  	ldr_l	\tmpreg, idmap_t0sz
>  	bfi	\valreg, \tmpreg, #TCR_T0SZ_OFFSET, #TCR_TxSZ_WIDTH
> -#endif
>  	.endm
>  
>  /*
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index b3f7b68b042d..8d663ca0d50c 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -273,7 +273,12 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
>  
>  static inline bool __kvm_cpu_uses_extended_idmap(void)
>  {
> -	return __cpu_uses_extended_idmap();
> +	return __cpu_uses_extended_idmap_table();
> +}
> +
> +static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
> +{
> +	return idmap_ptrs_per_pgd;
>  }
>  
>  /*
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index accc2ff32a0e..043eed856b55 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -63,11 +63,21 @@ static inline void cpu_set_reserved_ttbr0(void)
>   * physical memory, in which case it will be smaller.
>   */
>  extern u64 idmap_t0sz;
> +extern u64 idmap_ptrs_per_pgd;
>  
>  static inline bool __cpu_uses_extended_idmap(void)
>  {
> -	return (!IS_ENABLED(CONFIG_ARM64_VA_BITS_48) &&
> -		unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS)));
> +	return unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS));
> +}
> +
> +/*
> + * True if the extended ID map requires an extra level of translation table
> + * to be configured.
> + */
> +static inline bool __cpu_uses_extended_idmap_table(void)
> +{
> +	return __cpu_uses_extended_idmap() &&
> +		(idmap_ptrs_per_pgd == PTRS_PER_PGD);
>  }
>  
>  /*
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 64e09f207d1d..465f70328ba0 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -172,7 +172,7 @@ ENDPROC(preserve_boot_args)
>   *	ptrs:	#imm pointers per table page
>   *
>   * Preserves:	virt
> - * Corrupts:	tmp1, tmp2
> + * Corrupts:	ptrs, tmp1, tmp2
>   * Returns:	tbl -> next level table page address
>   */
>  	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
> @@ -180,7 +180,8 @@ ENDPROC(preserve_boot_args)
>  	phys_to_pte \tmp1, \tmp2
>  	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
>  	lsr	\tmp1, \virt, #\shift
> -	and	\tmp1, \tmp1, #\ptrs - 1	// table index
> +	sub	\ptrs, \ptrs, #1
> +	and	\tmp1, \tmp1, \ptrs		// table index
>  	str	\tmp2, [\tbl, \tmp1, lsl #3]
>  	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
>  	.endm
> @@ -190,15 +191,17 @@ ENDPROC(preserve_boot_args)
>   * block entry in the next level (tbl) for the given virtual address.
>   *
>   * Preserves:	tbl, next, virt
> - * Corrupts:	tmp1, tmp2
> + * Corrupts:	ptrs_per_pgd, tmp1, tmp2
>   */
> -	.macro	create_pgd_entry, tbl, virt, tmp1, tmp2
> -	create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
> +	.macro	create_pgd_entry, tbl, virt, ptrs_per_pgd, tmp1, tmp2
> +	create_table_entry \tbl, \virt, PGDIR_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
>  #if SWAPPER_PGTABLE_LEVELS > 3
> -	create_table_entry \tbl, \virt, PUD_SHIFT, PTRS_PER_PUD, \tmp1, \tmp2
> +	mov	\ptrs_per_pgd, PTRS_PER_PUD
> +	create_table_entry \tbl, \virt, PUD_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
>  #endif
>  #if SWAPPER_PGTABLE_LEVELS > 2
> -	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, PTRS_PER_PTE, \tmp1, \tmp2
> +	mov	\ptrs_per_pgd, PTRS_PER_PTE
> +	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
>  #endif
>  	.endm
>  
> @@ -262,26 +265,13 @@ __create_page_tables:
>  	adrp	x0, idmap_pg_dir
>  	adrp	x3, __idmap_text_start		// __pa(__idmap_text_start)
>  
> -#ifndef CONFIG_ARM64_VA_BITS_48
> -#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
> -#define EXTRA_PTRS	(1 << (48 - EXTRA_SHIFT))
> -
> -	/*
> -	 * If VA_BITS < 48, it may be too small to allow for an ID mapping to be
> -	 * created that covers system RAM if that is located sufficiently high
> -	 * in the physical address space. So for the ID map, use an extended
> -	 * virtual range in that case, by configuring an additional translation
> -	 * level.
> -	 * First, we have to verify our assumption that the current value of
> -	 * VA_BITS was chosen such that all translation levels are fully
> -	 * utilised, and that lowering T0SZ will always result in an additional
> -	 * translation level to be configured.
> -	 */
> -#if VA_BITS != EXTRA_SHIFT
> -#error "Mismatch between VA_BITS and page size/number of translation levels"
> -#endif
> -
>  	/*
> +	 * VA_BITS may be too small to allow for an ID mapping to be created
> +	 * that covers system RAM if that is located sufficiently high in the
> +	 * physical address space. So for the ID map, use an extended virtual
> +	 * range in that case, and configure an additional translation level
> +	 * if needed.
> +	 *
>  	 * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the
>  	 * entire ID map region can be mapped. As T0SZ == (64 - #bits used),
>  	 * this number conveniently equals the number of leading zeroes in
> @@ -290,18 +280,41 @@ __create_page_tables:
>  	adrp	x5, __idmap_text_end
>  	clz	x5, x5
>  	cmp	x5, TCR_T0SZ(VA_BITS)	// default T0SZ small enough?
> -	b.ge	1f			// .. then skip additional level
> +	b.ge	1f			// .. then skip VA range extension
>  
>  	adr_l	x6, idmap_t0sz
>  	str	x5, [x6]
>  	dmb	sy
>  	dc	ivac, x6		// Invalidate potentially stale cache line
>  
> -	create_table_entry x0, x3, EXTRA_SHIFT, EXTRA_PTRS, x5, x6
> -1:
> +#if (VA_BITS < 48)
> +#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
> +#define EXTRA_PTRS	(1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT))
> +
> +	/*
> +	 * If VA_BITS < 48, we have to configure an additional table level.
> +	 * First, we have to verify our assumption that the current value of
> +	 * VA_BITS was chosen such that all translation levels are fully
> +	 * utilised, and that lowering T0SZ will always result in an additional
> +	 * translation level to be configured.
> +	 */
> +#if VA_BITS != EXTRA_SHIFT
> +#error "Mismatch between VA_BITS and page size/number of translation levels"
>  #endif
>  
> -	create_pgd_entry x0, x3, x5, x6
> +	mov	x4, EXTRA_PTRS
> +	create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6
> +#else
> +	/*
> +	 * If VA_BITS == 48, we don't have to configure an additional
> +	 * translation level, but the top-level table has more entries.
> +	 */
> +	mov	x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT)
> +	str_l	x4, idmap_ptrs_per_pgd, x5
> +#endif
> +1:
> +	ldr_l	x4, idmap_ptrs_per_pgd
> +	create_pgd_entry x0, x3, x4, x5, x6
>  	mov	x5, x3				// __pa(__idmap_text_start)
>  	adr_l	x6, __idmap_text_end		// __pa(__idmap_text_end)
>  	create_block_map x0, x7, x3, x5, x6, x4
> @@ -312,7 +325,8 @@ __create_page_tables:
>  	adrp	x0, swapper_pg_dir
>  	mov_q	x5, KIMAGE_VADDR + TEXT_OFFSET	// compile time __va(_text)
>  	add	x5, x5, x23			// add KASLR displacement
> -	create_pgd_entry x0, x5, x3, x6
> +	mov	x4, PTRS_PER_PGD
> +	create_pgd_entry x0, x5, x4, x3, x6
>  	adrp	x6, _end			// runtime __pa(_end)
>  	adrp	x3, _text			// runtime __pa(_text)
>  	sub	x6, x6, x3			// _end - _text
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index a99718f32af9..c2ebe4e992df 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -72,24 +72,23 @@ __do_hyp_init:
>  	mov	x5, #TCR_EL2_RES1
>  	orr	x4, x4, x5
>  
> -#ifndef CONFIG_ARM64_VA_BITS_48
>  	/*
> -	 * If we are running with VA_BITS < 48, we may be running with an extra
> -	 * level of translation in the ID map. This is only the case if system
> -	 * RAM is out of range for the currently configured page size and number
> -	 * of translation levels, in which case we will also need the extra
> -	 * level for the HYP ID map, or we won't be able to enable the EL2 MMU.
> +	 * The ID map may be configured to use an extended virtual address
> +	 * range. This is only the case if system RAM is out of range for the
> +	 * currently configured page size and VA_BITS, in which case we will
> +	 * also need the extended virtual range for the HYP ID map, or we won't
> +	 * be able to enable the EL2 MMU.
>  	 *
>  	 * However, at EL2, there is only one TTBR register, and we can't switch
>  	 * between translation tables *and* update TCR_EL2.T0SZ at the same
> -	 * time. Bottom line: we need the extra level in *both* our translation
> -	 * tables.
> +	 * time. Bottom line: we need to use the extended range with *both* our
> +	 * translation tables.
>  	 *
>  	 * So use the same T0SZ value we use for the ID map.
>  	 */
>  	ldr_l	x5, idmap_t0sz
>  	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
> -#endif
> +
>  	/*
>  	 * Set the PS bits in TCR_EL2.
>  	 */
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 0c631a17ae1d..baa34418c3bf 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -50,6 +50,7 @@
>  #define NO_CONT_MAPPINGS	BIT(1)
>  
>  u64 idmap_t0sz = TCR_T0SZ(VA_BITS);
> +u64 idmap_ptrs_per_pgd = PTRS_PER_PGD;
>  
>  u64 kimage_voffset __ro_after_init;
>  EXPORT_SYMBOL(kimage_voffset);
> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index b36945d49986..876caf531d32 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -623,7 +623,7 @@ static int create_hyp_pud_mappings(pgd_t *pgd, unsigned long start,
>  	return 0;
>  }
>  
> -static int __create_hyp_mappings(pgd_t *pgdp,
> +static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
>  				 unsigned long start, unsigned long end,
>  				 unsigned long pfn, pgprot_t prot)
>  {
> @@ -636,7 +636,7 @@ static int __create_hyp_mappings(pgd_t *pgdp,
>  	addr = start & PAGE_MASK;
>  	end = PAGE_ALIGN(end);
>  	do {
> -		pgd = pgdp + pgd_index(addr);
> +		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
>  
>  		if (pgd_none(*pgd)) {
>  			pud = pud_alloc_one(NULL, addr);
> @@ -699,8 +699,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
>  		int err;
>  
>  		phys_addr = kvm_kaddr_to_phys(from + virt_addr - start);
> -		err = __create_hyp_mappings(hyp_pgd, virt_addr,
> -					    virt_addr + PAGE_SIZE,
> +		err = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD,
> +					    virt_addr, virt_addr + PAGE_SIZE,
>  					    __phys_to_pfn(phys_addr),
>  					    prot);
>  		if (err)
> @@ -731,7 +731,7 @@ int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
>  	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
>  		return -EINVAL;
>  
> -	return __create_hyp_mappings(hyp_pgd, start, end,
> +	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
>  				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
>  }
>  
> @@ -1737,7 +1737,7 @@ static int kvm_map_idmap_text(pgd_t *pgd)
>  	int err;
>  
>  	/* Create the idmap in the boot page tables */
> -	err = 	__create_hyp_mappings(pgd,
> +	err = 	__create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>  				      hyp_idmap_start, hyp_idmap_end,
>  				      __phys_to_pfn(hyp_idmap_start),
>  				      PAGE_HYP_EXEC);
> 

I wonder if you could push these changes into __create_hyp_mappings()
instead of changing all the call sites:

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b36945d49986..0d44fb2bd9c6 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -629,14 +629,17 @@ static int __create_hyp_mappings(pgd_t *pgdp,
 {
 	pgd_t *pgd;
 	pud_t *pud;
-	unsigned long addr, next;
+	unsigned long addr, next, ptrs_per_pgd = PTRS_PER_PGD;
 	int err = 0;
 
+	if (pgdp != hyp_pgd)
+		ptrs_per_pgd = __kvm_idmap_ptrs_per_pgd();
+
 	mutex_lock(&kvm_hyp_pgd_mutex);
 	addr = start & PAGE_MASK;
 	end = PAGE_ALIGN(end);
 	do {
-		pgd = pgdp + pgd_index(addr);
+		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
 
 		if (pgd_none(*pgd)) {
 			pud = pud_alloc_one(NULL, addr);

with a suitable comment explaining why this is needed?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 06/10] arm64: handle 52-bit physical addresses in page table entries

[Marc Zyngier]

On 13/12/17 17:07, Kristina Martsenko wrote:
> The top 4 bits of a 52-bit physical address are positioned at bits
> 12..15 of a page table entry. Introduce macros to convert between a
> physical address and its placement in a table entry, and change all
> macros/functions that access PTEs to use them.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>

Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 04/10] arm64: head.S: handle 52-bit PAs in PTEs in early page table setup

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> The top 4 bits of a 52-bit physical address are positioned at bits
> 12..15 in page table entries. Introduce a macro to move the bits there,
> and change the early ID map and swapper table setup code to use it.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm64/include/asm/pgtable-hwdef.h |  6 ++++++
>   arch/arm64/kernel/head.S               | 36 +++++++++++++++++++++++++---------
>   2 files changed, 33 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index 2b3104af79d0..c59c69e02036 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -168,6 +168,12 @@
>   #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
>   #define PTE_HYP_XN		(_AT(pteval_t, 1) << 54)	/* HYP XN */
>   
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define PTE_ADDR_LOW		(((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
> +#define PTE_ADDR_HIGH		(_AT(pteval_t, 0xf) << 12)
> +#define PTE_ADDR_MASK_52	(PTE_ADDR_LOW | PTE_ADDR_HIGH)
> +#endif
> +
>   /*
>    * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).
>    */
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 0addea3760a6..ddee8b347f60 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -148,6 +148,22 @@ preserve_boot_args:
>   ENDPROC(preserve_boot_args)
>   
>   /*
> + * Macro to arrange a physical address in a page table entry, taking care of
> + * 52-bit addresses.
> + *
> + * Preserves:	phys
> + * Returns:	pte
> + */
> +	.macro	phys_to_pte, phys, pte
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +	orr	\pte, \phys, \phys, lsr #36
> +	and	\pte, \pte, #PTE_ADDR_MASK_52

We could have a corrupt "pte" if the "phys" is not aligned
to page size (i.e, 64K here). However, given that the only callers
of this (create_table_entry and create_block_map) passes page
aligned addresses, we are fine. It would be good to add a comment
mentioning that, to prevent future misuses.

> +#else
> +	mov	\pte, \phys
> +#endif
> +	.endm
> +
> +/*
>    * Macro to create a table entry to the next page.
>    *
>    *	tbl:	page table address
> @@ -160,10 +176,11 @@ ENDPROC(preserve_boot_args)
>    * Returns:	tbl -> next level table page address
>    */
>   	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
> +	add	\tmp1, \tbl, #PAGE_SIZE
> +	phys_to_pte \tmp1, \tmp2
> +	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
>   	lsr	\tmp1, \virt, #\shift
>   	and	\tmp1, \tmp1, #\ptrs - 1	// table index
> -	add	\tmp2, \tbl, #PAGE_SIZE
> -	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
>   	str	\tmp2, [\tbl, \tmp1, lsl #3]
>   	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
>   	.endm
> @@ -190,16 +207,17 @@ ENDPROC(preserve_boot_args)
>    * virtual range (inclusive).
>    *
>    * Preserves:	tbl, flags
> - * Corrupts:	phys, start, end, pstate
> + * Corrupts:	phys, start, end, tmp
>    */

nit: We still corrupt pstate. So, it would be good retain that here.

Other than those nits, looks good to me.

Reviewed-by : Suzuki K Poulose <suzuki.poulose@arm.com>

[PATCH 06/10] arm64: handle 52-bit physical addresses in page table entries

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> The top 4 bits of a 52-bit physical address are positioned at bits
> 12..15 of a page table entry. Introduce macros to convert between a
> physical address and its placement in a table entry, and change all
> macros/functions that access PTEs to use them.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>

Kristina

Patch looks good, except for a couple of nits below


> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index c59c69e02036..60e0432df559 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -168,10 +168,12 @@
>   #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
>   #define PTE_HYP_XN		(_AT(pteval_t, 1) << 54)	/* HYP XN */
>   
> -#ifdef CONFIG_ARM64_PA_BITS_52
>   #define PTE_ADDR_LOW		(((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
> +#ifdef CONFIG_ARM64_PA_BITS_52
>   #define PTE_ADDR_HIGH		(_AT(pteval_t, 0xf) << 12)
> -#define PTE_ADDR_MASK_52	(PTE_ADDR_LOW | PTE_ADDR_HIGH)
> +#define PTE_ADDR_MASK		(PTE_ADDR_LOW | PTE_ADDR_HIGH)
> +#else
> +#define PTE_ADDR_MASK		PTE_ADDR_LOW
>   #endif
>   
>   /*
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index 5d9554fb2692..65856a81e692 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -57,9 +57,20 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
>   
>   #define pte_ERROR(pte)		__pte_error(__FILE__, __LINE__, pte_val(pte))
>   
> -#define pte_pfn(pte)		((pte_val(pte) & PHYS_MASK) >> PAGE_SHIFT)
> +/*
> + * Macros to convert between a physical address and its placement in a
> + * page table entry, taking care of 52-bit addresses.
> + */
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define __pte_to_phys(pte)	((pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << 36))

nit: The line is too long, 100+, it may be worth putting it in a separate line :

i.e,

+#define __pte_to_phys(pte)	\
   ((pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << 36))

similarly below.

> +#define __phys_to_pte_val(phys)	(((phys) | ((phys) >> 36)) & PTE_ADDR_MASK)
> +#else
> +#define __pte_to_phys(pte)	(pte_val(pte) & PTE_ADDR_MASK)
> +#define __phys_to_pte_val(phys)	(phys)
> +#endif
>   
> -#define pfn_pte(pfn,prot)	(__pte(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
> +#define pte_pfn(pte)		(__pte_to_phys(pte) >> PAGE_SHIFT)
> +#define pfn_pte(pfn,prot)	__pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
>   
>   #define pte_none(pte)		(!pte_val(pte))
>   #define pte_clear(mm,addr,ptep)	set_pte(ptep, __pte(0))
> @@ -284,6 +295,11 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
>   
>   #define __HAVE_ARCH_PTE_SPECIAL
>   
> +static inline pte_t pgd_pte(pgd_t pgd)
> +{
> +	return __pte(pgd_val(pgd));
> +}
> +
>   static inline pte_t pud_pte(pud_t pud)
>   {
>   	return __pte(pud_val(pud));
> @@ -349,16 +365,24 @@ static inline int pmd_protnone(pmd_t pmd)
>   
>   #define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
>   
> -#define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
> -#define pfn_pmd(pfn,prot)	(__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
> +#define __pmd_to_phys(pmd)	__pte_to_phys(pmd_pte(pmd))
> +#define __phys_to_pmd_val(phys)	__phys_to_pte_val(phys)
> +#define pmd_pfn(pmd)		((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT)
> +#define pfn_pmd(pfn,prot)	__pmd(__phys_to_pmd_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
>   #define mk_pmd(page,prot)	pfn_pmd(page_to_pfn(page),prot)
>   
>   #define pud_write(pud)		pte_write(pud_pte(pud))
> -#define pud_pfn(pud)		(((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
> -#define pfn_pud(pfn,prot)	(__pud(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
> +
> +#define __pud_to_phys(pud)	__pte_to_phys(pud_pte(pud))
> +#define __phys_to_pud_val(phys)	__phys_to_pte_val(phys)
> +#define pud_pfn(pud)		((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT)
> +#define pfn_pud(pfn,prot)	__pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
>   
>   #define set_pmd_at(mm, addr, pmdp, pmd)	set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
>   
> +#define __pgd_to_phys(pgd)	__pte_to_phys(pgd_pte(pgd))
> +#define __phys_to_pgd_val(phys)	__phys_to_pte_val(phys)
> +
>   #define __pgprot_modify(prot,mask,bits) \
>   	__pgprot((pgprot_val(prot) & ~(mask)) | (bits))
>   
> @@ -409,7 +433,7 @@ static inline void pmd_clear(pmd_t *pmdp)
>   
>   static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
>   {
> -	return pmd_val(pmd) & PHYS_MASK & (s32)PAGE_MASK;
> +	return __pmd_to_phys(pmd);
>   }
>   
>   /* Find an entry in the third-level page table. */
> @@ -427,7 +451,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
>   #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
>   #define pte_clear_fixmap()		clear_fixmap(FIX_PTE)
>   
> -#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK))
> +#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(__pmd_to_phys(pmd)))

You could simplify the above to :

#define pmd_page(pmd)			pfn_to_page(pmd_pfn(pmd))

? similarly for pud_page.


Other than the above nits:

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

[PATCH 01/10] arm64: add kconfig symbol to configure physical address size

[Kristina Martsenko]

On 14/12/17 10:22, Suzuki K Poulose wrote:
> On 13/12/17 17:07, Kristina Martsenko wrote:
>> ARMv8.2 introduces support for 52-bit physical addresses. To prepare for
>> supporting this, add a new kconfig symbol to configure the physical
>> address space size. The symbols will be used in subsequent patches.
>> Currently the only choice is 48, a later patch will add the option of 52
>> once the required code is in place.
>>
>> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
>> ---
>> ? arch/arm64/Kconfig | 16 ++++++++++++++++
>> ? 1 file changed, 16 insertions(+)
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index a93339f5178f..8dc937823eeb 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -646,6 +646,22 @@ config ARM64_VA_BITS
>> ????? default 47 if ARM64_VA_BITS_47
>> ????? default 48 if ARM64_VA_BITS_48
>> ? +choice
>> +??? prompt "Physical address space size"
>> +??? default ARM64_PA_BITS_48
>> +??? help
>> +????? Choose the maximum physical address range that the kernel will
>> +????? support.
>> +
>> +config ARM64_PA_BITS_48
>> +??? bool "48-bit"
>> +
>> +endchoice
>> +
>> +config ARM64_PA_BITS
>> +??? int
>> +??? default 48 if ARM64_PA_BITS_48
>> +
>> ? config CPU_BIG_ENDIAN
>> ???????? bool "Build big-endian kernel"
>> ???????? help
>>
> 
> We could replace most of the hard coded "48" PA limit values to ARM64_PA_BITS,
> now that we have a configurable entity. i.e, you could fold? patch 7 & 8 into
> this one.

Makes sense, I'll merge patches 7 and 8 into this one in v2.

> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks!

Kristina

[PATCH 00/10] arm64: 52-bit physical address support

[Kristina Martsenko]

On 14/12/17 17:52, Bob Picco wrote:
> Kristina Martsenko wrote:	[Wed Dec 13 2017, 12:07:15PM EST]
>> Hi,
>>
>> This series adds 52-bit physical address space support to arm64, up from
>> the current 48 bits. This is an ARMv8.2 feature (ARMv8.2-LPA).
>>
>> The series is based on 4.15-rc3. It has been lightly tested on an ARM
>> Fast Model. There's still some cases and areas to think through, as well
>> as more testing to do.
>>
>> Patches for SMMU 52-bit PA support have been sent separately [1]. A GIC
>> ITS patch has already been merged [2]. ARMv8.2 also allows 52-bit IPA,
>> but support for that is not part of this series.
>>
>> This version mostly addresses various review comments received.
>>
>> Changes from RFC:
>>  - Split kconfig symbol into two patches, to enable 52-bit PA at the end
>>  - Patch #3: Changed phys_to_ttbr to use a macro, added an #include
>>  - Patch #4: Changed phys_to_pte to use a macro
>>  - Patch #6: Replaced __phys_to_pte with __phys_to_pte_val (same for
>>    pmd/pud/pgd)
>>  - Patch #6: Changed __phys_to_pte_val, __pte_to_phys, and
>>    pgtable-hwdef.h macros
>>  - Patches #5, #6: Removed kvm_extended_idmap_pgd, inlined its code,
>>    moved the comment
>>  - Patch #5: Added pfn_pud definition (to make the kernel build on that
>>    commit)
>>
>> Thanks,
>> Kristina
>>
>> [1] https://www.spinics.net/lists/arm-kernel/msg619040.html
>> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=30ae9610d275f8f03f5bf7612ce71d8af6fc400b
>>
>>
>> Kristina Martsenko (10):
>>   arm64: add kconfig symbol to configure physical address size
>>   arm64: limit PA size to supported range
>>   arm64: handle 52-bit addresses in TTBR
>>   arm64: head.S: handle 52-bit PAs in PTEs in early page table setup
>>   arm64: don't open code page table entry creation
>>   arm64: handle 52-bit physical addresses in page table entries
>>   arm64: increase PHYS_MASK to 52 bits
>>   arm64: increase sparsemem MAX_PHYSMEM_BITS to 52
>>   arm64: allow ID map to be extended to 52 bits
>>   arm64: enable 52-bit physical address support
>>
>>  arch/arm/include/asm/kvm_mmu.h         |   7 ++
>>  arch/arm64/Kconfig                     |  29 ++++++++
>>  arch/arm64/include/asm/assembler.h     |  31 ++++++++-
>>  arch/arm64/include/asm/kvm_mmu.h       |  21 +++++-
>>  arch/arm64/include/asm/mmu_context.h   |  16 ++++-
>>  arch/arm64/include/asm/pgalloc.h       |   6 +-
>>  arch/arm64/include/asm/pgtable-hwdef.h |  19 +++++-
>>  arch/arm64/include/asm/pgtable.h       |  53 ++++++++++++---
>>  arch/arm64/include/asm/sparsemem.h     |   2 +-
>>  arch/arm64/include/asm/sysreg.h        |   8 +++
>>  arch/arm64/kernel/head.S               | 118 +++++++++++++++++++++------------
>>  arch/arm64/kernel/hibernate-asm.S      |  12 ++--
>>  arch/arm64/kernel/hibernate.c          |   5 +-
>>  arch/arm64/kvm/hyp-init.S              |  26 ++++----
>>  arch/arm64/kvm/hyp/s2-setup.c          |   2 +
>>  arch/arm64/mm/mmu.c                    |  15 +++--
>>  arch/arm64/mm/pgd.c                    |   8 +++
>>  arch/arm64/mm/proc.S                   |  19 +++---
>>  virt/kvm/arm/arm.c                     |   2 +-
>>  virt/kvm/arm/mmu.c                     |  12 ++--
>>  20 files changed, 302 insertions(+), 109 deletions(-)
>>
>> -- 
>> 2.1.4
> Hi Kristina,
> 
> I boot tested but on VM and had a couple issues. I will examine and share
> should the issues be of value.
> Tested-by: Bob Picco <bob.picco@oracle.com>

Thanks for testing! I assume it was on a VM with less than 52 bits of
PA? In any case, let me know if you find any issues.

> I reviewed and thank you. Your effort caused me to return to some code
> examined/learned during the last few months.
> Reviewed-by: Bob Picco <bob.picco@oracle.com>

Thanks for reviewing! Hope it was useful.

Kristina

[PATCH 00/10] arm64: 52-bit physical address support

[Bob Picco]

Kristina Martsenko wrote:	[Tue Dec 19 2017, 12:15:46PM EST]
> On 14/12/17 17:52, Bob Picco wrote:
> > Kristina Martsenko wrote:	[Wed Dec 13 2017, 12:07:15PM EST]
> >> Hi,
> >>
> >> This series adds 52-bit physical address space support to arm64, up from
> >> the current 48 bits. This is an ARMv8.2 feature (ARMv8.2-LPA).
> >>
> >> The series is based on 4.15-rc3. It has been lightly tested on an ARM
> >> Fast Model. There's still some cases and areas to think through, as well
> >> as more testing to do.
> >>
> >> Patches for SMMU 52-bit PA support have been sent separately [1]. A GIC
> >> ITS patch has already been merged [2]. ARMv8.2 also allows 52-bit IPA,
> >> but support for that is not part of this series.
> >>
> >> This version mostly addresses various review comments received.
> >>
> >> Changes from RFC:
> >>  - Split kconfig symbol into two patches, to enable 52-bit PA at the end
> >>  - Patch #3: Changed phys_to_ttbr to use a macro, added an #include
> >>  - Patch #4: Changed phys_to_pte to use a macro
> >>  - Patch #6: Replaced __phys_to_pte with __phys_to_pte_val (same for
> >>    pmd/pud/pgd)
> >>  - Patch #6: Changed __phys_to_pte_val, __pte_to_phys, and
> >>    pgtable-hwdef.h macros
> >>  - Patches #5, #6: Removed kvm_extended_idmap_pgd, inlined its code,
> >>    moved the comment
> >>  - Patch #5: Added pfn_pud definition (to make the kernel build on that
> >>    commit)
> >>
> >> Thanks,
> >> Kristina
> >>
> >> [1] https://www.spinics.net/lists/arm-kernel/msg619040.html
> >> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=30ae9610d275f8f03f5bf7612ce71d8af6fc400b
> >>
> >>
> >> Kristina Martsenko (10):
> >>   arm64: add kconfig symbol to configure physical address size
> >>   arm64: limit PA size to supported range
> >>   arm64: handle 52-bit addresses in TTBR
> >>   arm64: head.S: handle 52-bit PAs in PTEs in early page table setup
> >>   arm64: don't open code page table entry creation
> >>   arm64: handle 52-bit physical addresses in page table entries
> >>   arm64: increase PHYS_MASK to 52 bits
> >>   arm64: increase sparsemem MAX_PHYSMEM_BITS to 52
> >>   arm64: allow ID map to be extended to 52 bits
> >>   arm64: enable 52-bit physical address support
> >>
> >>  arch/arm/include/asm/kvm_mmu.h         |   7 ++
> >>  arch/arm64/Kconfig                     |  29 ++++++++
> >>  arch/arm64/include/asm/assembler.h     |  31 ++++++++-
> >>  arch/arm64/include/asm/kvm_mmu.h       |  21 +++++-
> >>  arch/arm64/include/asm/mmu_context.h   |  16 ++++-
> >>  arch/arm64/include/asm/pgalloc.h       |   6 +-
> >>  arch/arm64/include/asm/pgtable-hwdef.h |  19 +++++-
> >>  arch/arm64/include/asm/pgtable.h       |  53 ++++++++++++---
> >>  arch/arm64/include/asm/sparsemem.h     |   2 +-
> >>  arch/arm64/include/asm/sysreg.h        |   8 +++
> >>  arch/arm64/kernel/head.S               | 118 +++++++++++++++++++++------------
> >>  arch/arm64/kernel/hibernate-asm.S      |  12 ++--
> >>  arch/arm64/kernel/hibernate.c          |   5 +-
> >>  arch/arm64/kvm/hyp-init.S              |  26 ++++----
> >>  arch/arm64/kvm/hyp/s2-setup.c          |   2 +
> >>  arch/arm64/mm/mmu.c                    |  15 +++--
> >>  arch/arm64/mm/pgd.c                    |   8 +++
> >>  arch/arm64/mm/proc.S                   |  19 +++---
> >>  virt/kvm/arm/arm.c                     |   2 +-
> >>  virt/kvm/arm/mmu.c                     |  12 ++--
> >>  20 files changed, 302 insertions(+), 109 deletions(-)
> >>
> >> -- 
> >> 2.1.4
> > Hi Kristina,
> > 
> > I boot tested but on VM and had a couple issues. I will examine and share
> > should the issues be of value.
> > Tested-by: Bob Picco <bob.picco@oracle.com>
> 
> Thanks for testing! I assume it was on a VM with less than 52 bits of
> PA? In any case, let me know if you find any issues.
Hi Kristina,

Yes it was substantially less than 52b PA. I was surprised we booted.
> 
> > I reviewed and thank you. Your effort caused me to return to some code
> > examined/learned during the last few months.
> > Reviewed-by: Bob Picco <bob.picco@oracle.com>
> 
> Thanks for reviewing! Hope it was useful.
You are welcome. Well I had to commence at some point. I happened to review
most of boot up two months ago. Your effort caused me to return and review
more... A few more iterations before being reasonably solid.

bob
> 
> Kristina
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 09/10] arm64: allow ID map to be extended to 52 bits

[Suzuki K Poulose]

On 13/12/17 17:07, Kristina Martsenko wrote:
> Currently, when using VA_BITS < 48, if the ID map text happens to be
> placed in physical memory above VA_BITS, we increase the VA size (up to
> 48) and create a new table level, in order to map in the ID map text.
> This is okay because the system always supports 48 bits of VA.
> 
> This patch extends the code such that if the system supports 52 bits of
> VA, and the ID map text is placed that high up, then we increase the VA
> size accordingly, up to 52.
> 
> One difference from the current implementation is that so far the
> condition of VA_BITS < 48 has meant that the top level table is always
> "full", with the maximum number of entries, and an extra table level is
> always needed. Now, when VA_BITS = 48 (and using 64k pages), the top
> level table is not full, and we simply need to increase the number of
> entries in it, instead of creating a new table level.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm/include/asm/kvm_mmu.h       |  5 +++
>   arch/arm64/include/asm/assembler.h   |  2 -
>   arch/arm64/include/asm/kvm_mmu.h     |  7 +++-
>   arch/arm64/include/asm/mmu_context.h | 14 ++++++-
>   arch/arm64/kernel/head.S             | 76 +++++++++++++++++++++---------------
>   arch/arm64/kvm/hyp-init.S            | 17 ++++----
>   arch/arm64/mm/mmu.c                  |  1 +
>   virt/kvm/arm/mmu.c                   | 12 +++---
>   8 files changed, 83 insertions(+), 51 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index 8dbec683638b..8c5643e2eea4 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -211,6 +211,11 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
>   	return false;
>   }
>   
> +static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
> +{
> +	return PTRS_PER_PGD;
> +}
> +
>   static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
>   				       pgd_t *hyp_pgd,
>   				       pgd_t *merged_hyp_pgd,
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 2058fd864bfb..11719b11f4a7 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -344,10 +344,8 @@ alternative_endif
>    * tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map
>    */
>   	.macro	tcr_set_idmap_t0sz, valreg, tmpreg
> -#ifndef CONFIG_ARM64_VA_BITS_48
>   	ldr_l	\tmpreg, idmap_t0sz
>   	bfi	\valreg, \tmpreg, #TCR_T0SZ_OFFSET, #TCR_TxSZ_WIDTH
> -#endif
>   	.endm
>   
>   /*
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index b3f7b68b042d..8d663ca0d50c 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -273,7 +273,12 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
>   
>   static inline bool __kvm_cpu_uses_extended_idmap(void)
>   {
> -	return __cpu_uses_extended_idmap();
> +	return __cpu_uses_extended_idmap_table();
> +}
> +
> +static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
> +{
> +	return idmap_ptrs_per_pgd;
>   }
>   
>   /*
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index accc2ff32a0e..043eed856b55 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -63,11 +63,21 @@ static inline void cpu_set_reserved_ttbr0(void)
>    * physical memory, in which case it will be smaller.
>    */
>   extern u64 idmap_t0sz;
> +extern u64 idmap_ptrs_per_pgd;
>   
>   static inline bool __cpu_uses_extended_idmap(void)
>   {
> -	return (!IS_ENABLED(CONFIG_ARM64_VA_BITS_48) &&
> -		unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS)));
> +	return unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS));
> +}
> +
> +/*
> + * True if the extended ID map requires an extra level of translation table
> + * to be configured.
> + */
> +static inline bool __cpu_uses_extended_idmap_table(void)
> +{
> +	return __cpu_uses_extended_idmap() &&
> +		(idmap_ptrs_per_pgd == PTRS_PER_PGD);
>   }
>   

Kristina,

I feel this is a bit confusing, as we use idmap_ptrs_per_pgd value to convey two
different information.

1) Whether we use an extended level of table
2) Number of ptrs in the host VA PGD level for idmap

I think we are able to figure out if the idmap uses an extended table using the idmap_t0sz alone :

static inline bool __cpu_uses_extended_idmap_level(void)
{
	return ARM64_HW_PGTABLE_LEVELS((64 - idmap_t0sz)) > CONFIG_PGTABLE_LEVELS;
}

Also, the name could be changed to __cpu_uses_extended_idmap_level() to imply we are talking
about an additional level of table, rather than extending the table, which could either mean
extending the table to include more pgd ptrs or an additional level.


Rest looks good to me.

Suzuki

[PATCH 03/10] arm64: handle 52-bit addresses in TTBR

[Catalin Marinas]

On Thu, Dec 14, 2017 at 06:50:05PM +0000, Marc Zyngier wrote:
> On 13/12/17 17:07, Kristina Martsenko wrote:
> > diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> > index eb0c2bd90de9..2b3104af79d0 100644
> > --- a/arch/arm64/include/asm/pgtable-hwdef.h
> > +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> > @@ -16,6 +16,8 @@
> >  #ifndef __ASM_PGTABLE_HWDEF_H
> >  #define __ASM_PGTABLE_HWDEF_H
> >  
> > +#include <asm/memory.h>
> > +
> >  /*
> >   * Number of page-table levels required to address 'va_bits' wide
> >   * address, without section mapping. We resolve the top (va_bits - PAGE_SHIFT)
> > @@ -277,4 +279,11 @@
> >  #define TCR_HA			(UL(1) << 39)
> >  #define TCR_HD			(UL(1) << 40)
> >  
> > +/*
> > + * TTBR
> > + */
> > +#ifdef CONFIG_ARM64_PA_BITS_52
> > +#define TTBR_BADDR_MASK_52	(((UL(1) << 46) - 1) << 2)
> 
> This really hurts by brain. How about
> 
> #define TTBR_BADDR_MASK_52	GENMASK_UL(47, 2)

This file is included in assembly code and GENMASK_ULL has a C-only
version (include/linux/bitops.h). I'll leave Kristina's original code in
place.

> instead, together with a comment saying that TTBR[1] is RES0.

I can add the comment.

-- 
Catalin

[PATCH 03/10] arm64: handle 52-bit addresses in TTBR

[Marc Zyngier]

On 21/12/17 16:48, Catalin Marinas wrote:
> On Thu, Dec 14, 2017 at 06:50:05PM +0000, Marc Zyngier wrote:
>> On 13/12/17 17:07, Kristina Martsenko wrote:
>>> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
>>> index eb0c2bd90de9..2b3104af79d0 100644
>>> --- a/arch/arm64/include/asm/pgtable-hwdef.h
>>> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
>>> @@ -16,6 +16,8 @@
>>>  #ifndef __ASM_PGTABLE_HWDEF_H
>>>  #define __ASM_PGTABLE_HWDEF_H
>>>  
>>> +#include <asm/memory.h>
>>> +
>>>  /*
>>>   * Number of page-table levels required to address 'va_bits' wide
>>>   * address, without section mapping. We resolve the top (va_bits - PAGE_SHIFT)
>>> @@ -277,4 +279,11 @@
>>>  #define TCR_HA			(UL(1) << 39)
>>>  #define TCR_HD			(UL(1) << 40)
>>>  
>>> +/*
>>> + * TTBR
>>> + */
>>> +#ifdef CONFIG_ARM64_PA_BITS_52
>>> +#define TTBR_BADDR_MASK_52	(((UL(1) << 46) - 1) << 2)
>>
>> This really hurts by brain. How about
>>
>> #define TTBR_BADDR_MASK_52	GENMASK_UL(47, 2)
> 
> This file is included in assembly code and GENMASK_ULL has a C-only
> version (include/linux/bitops.h). I'll leave Kristina's original code in
> place.

Ah, that's a shame. I really liked it! ;-)

> 
>> instead, together with a comment saying that TTBR[1] is RES0.
> 
> I can add the comment.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH 06/10] arm64: handle 52-bit physical addresses in page table entries

[Catalin Marinas]

On Mon, Dec 18, 2017 at 04:36:46PM +0000, Suzuki K. Poulose wrote:
> On 13/12/17 17:07, Kristina Martsenko wrote:
> > @@ -427,7 +451,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
> >   #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
> >   #define pte_clear_fixmap()		clear_fixmap(FIX_PTE)
> > -#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK))
> > +#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(__pmd_to_phys(pmd)))
> 
> You could simplify the above to :
> 
> #define pmd_page(pmd)			pfn_to_page(pmd_pfn(pmd))
> 
> ? similarly for pud_page.

Just for the record as we already discussed: pmd_pfn(pmd) gives a
different result from __phys_to_pfn(__pmd_to_phys(pmd)). Weird but
pmd_pfn() is used with huge page stuff and it ensures that the pfn
corresponds to a PMD_SIZE-aligned page while pmd_page() is used to
retrieve the page of the pte pointed at by the pmd, so only PAGE_SIZE
aligned.

-- 
Catalin