[RFC 0/9] arm64: 52-bit physical address support

[RFC 0/9] arm64: 52-bit physical address support

[Kristina Martsenko]

Hi,

This series adds 52-bit physical address space support to arm64, up from
the current 48 bits. This is an ARMv8.2 feature (ARMv8.2-LPA).

The series is based on 4.14. It has been lightly tested on an ARM Fast
Model. There's still some cases and areas to think through, as well as
more testing to do, which is why this is an RFC for now.

Patches for SMMU 52-bit PA support will be sent separately. A GIC ITS
patch has already been merged [1]. ARMv8.2 also adds 52-bit IPA and VA
features, neither of those are part of this series.

The series depends on the patch "arm64: KVM: fix VTTBR_BADDR_MASK BUG_ON
off-by-one", which has been sent separately.

Thanks,
Kristina

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=30ae9610d275f8f03f5bf7612ce71d8af6fc400b

Kristina Martsenko (9):
  arm64: add kconfig symbol to enable 52-bit PA
  arm64: limit PA size to supported range
  arm64: handle 52-bit addresses in TTBR
  arm64: head.S: handle 52-bit PAs in PTEs in early page table setup
  arm64: don't open code page table entry creation
  arm64: handle 52-bit physical addresses in page table entries
  arm64: increase PHYS_MASK to 52 bits
  arm64: increase sparsemem MAX_PHYSMEM_BITS to 52
  arm64: allow ID map to be extended to 52 bits

 arch/arm/include/asm/kvm_mmu.h         |   7 ++
 arch/arm64/Kconfig                     |  29 ++++++++
 arch/arm64/include/asm/assembler.h     |  32 ++++++++-
 arch/arm64/include/asm/kvm_mmu.h       |  21 +++++-
 arch/arm64/include/asm/mmu_context.h   |  16 ++++-
 arch/arm64/include/asm/pgalloc.h       |   6 +-
 arch/arm64/include/asm/pgtable-hwdef.h |   5 +-
 arch/arm64/include/asm/pgtable.h       |  63 ++++++++++++++---
 arch/arm64/include/asm/sparsemem.h     |   2 +-
 arch/arm64/include/asm/sysreg.h        |   8 +++
 arch/arm64/kernel/head.S               | 119 +++++++++++++++++++++------------
 arch/arm64/kernel/hibernate-asm.S      |  12 ++--
 arch/arm64/kernel/hibernate.c          |   5 +-
 arch/arm64/kvm/hyp-init.S              |  26 ++++---
 arch/arm64/kvm/hyp/s2-setup.c          |   2 +
 arch/arm64/mm/mmu.c                    |  15 +++--
 arch/arm64/mm/pgd.c                    |   8 +++
 arch/arm64/mm/proc.S                   |  19 +++---
 virt/kvm/arm/arm.c                     |   2 +-
 virt/kvm/arm/mmu.c                     |  12 ++--
 20 files changed, 300 insertions(+), 109 deletions(-)

-- 
2.1.4

[RFC 1/9] arm64: add kconfig symbol to enable 52-bit PA

[Kristina Martsenko]

ARMv8.2 introduces support for 52-bit physical addresses if the 64k
granule is used. To enable this, add a new kconfig symbol to configure
the physical address space size, with 52-bit as one option. The symbols
will be used to enable 52-bit PA functionality in subsequent patches.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/Kconfig | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 0df64a6a56d4..30d0cc272903 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -609,6 +609,34 @@ config ARM64_VA_BITS
 	default 47 if ARM64_VA_BITS_47
 	default 48 if ARM64_VA_BITS_48
 
+choice
+	prompt "Physical address space size"
+	default ARM64_PA_BITS_48
+	help
+	  Choose the maximum physical address range that the kernel will
+	  support.
+
+config ARM64_PA_BITS_48
+	bool "48-bit"
+
+config ARM64_PA_BITS_52
+	bool "52-bit (ARMv8.2)"
+	depends on ARM64_64K_PAGES
+	help
+	  Enable support for a 52-bit physical address space, introduced as
+	  part of the ARMv8.2-LPA extension.
+
+	  With this enabled, the kernel will also continue to work on CPUs that
+	  do not support ARMv8.2-LPA, but with some added memory overhead (and
+	  minor performance overhead).
+
+endchoice
+
+config ARM64_PA_BITS
+	int
+	default 48 if ARM64_PA_BITS_48
+	default 52 if ARM64_PA_BITS_52
+
 config CPU_BIG_ENDIAN
        bool "Build big-endian kernel"
        help
-- 
2.1.4

[RFC 2/9] arm64: limit PA size to supported range

[Kristina Martsenko]

We currently copy the physical address size from
ID_AA64MMFR0_EL1.PARange directly into TCR.(I)PS. This will not work for
4k and 16k granule kernels on systems that support 52-bit physical
addresses, since 52-bit addresses are only permitted with the 64k
granule.

To fix this, fall back to 48 bits when configuring the PA size when the
kernel does not support 52-bit PAs. When it does, fall back to 52, to
avoid similar problems in the future if the PA size is ever increased
above 52.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/assembler.h | 13 +++++++++++++
 arch/arm64/include/asm/sysreg.h    |  8 ++++++++
 arch/arm64/kvm/hyp-init.S          |  6 ++----
 arch/arm64/kvm/hyp/s2-setup.c      |  2 ++
 arch/arm64/mm/proc.S               |  6 ++----
 5 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index d58a6253c6ab..04cf94766b78 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -338,6 +338,19 @@ alternative_endif
 	.endm
 
 /*
+ * tcr_set_pa_size - set TCR.(I)PS to the highest supported
+ * ID_AA64MMFR0_EL1.PARange value
+ */
+	.macro	tcr_set_pa_size, tcr, pos, tmp0, tmp1
+	mrs	\tmp0, ID_AA64MMFR0_EL1
+	ubfx	\tmp0, \tmp0, #ID_AA64MMFR0_PARANGE_SHIFT, #3
+	mov	\tmp1, #ID_AA64MMFR0_PARANGE_MAX
+	cmp	\tmp0, \tmp1
+	csel	\tmp0, \tmp1, \tmp0, hi
+	bfi	\tcr, \tmp0, \pos, #3
+	.endm
+
+/*
  * Macro to perform a data cache maintenance for the interval
  * [kaddr, kaddr + size)
  *
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index f707fed5886f..52225165df1a 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -364,6 +364,14 @@
 #define ID_AA64MMFR0_TGRAN64_SUPPORTED	0x0
 #define ID_AA64MMFR0_TGRAN16_NI		0x0
 #define ID_AA64MMFR0_TGRAN16_SUPPORTED	0x1
+#define ID_AA64MMFR0_PARANGE_48		0x5
+#define ID_AA64MMFR0_PARANGE_52		0x6
+
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_52
+#else
+#define ID_AA64MMFR0_PARANGE_MAX	ID_AA64MMFR0_PARANGE_48
+#endif
 
 /* id_aa64mmfr1 */
 #define ID_AA64MMFR1_PAN_SHIFT		20
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 3f9615582377..f731a48bd9f1 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -90,11 +90,9 @@ __do_hyp_init:
 	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
 #endif
 	/*
-	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the PS bits in
-	 * TCR_EL2.
+	 * Set the PS bits in TCR_EL2.
 	 */
-	mrs	x5, ID_AA64MMFR0_EL1
-	bfi	x4, x5, #16, #3
+	tcr_set_pa_size x4, #16, x5, x6
 
 	msr	tcr_el2, x4
 
diff --git a/arch/arm64/kvm/hyp/s2-setup.c b/arch/arm64/kvm/hyp/s2-setup.c
index a81f5e10fc8c..603e1ee83e89 100644
--- a/arch/arm64/kvm/hyp/s2-setup.c
+++ b/arch/arm64/kvm/hyp/s2-setup.c
@@ -32,6 +32,8 @@ u32 __hyp_text __init_stage2_translation(void)
 	 * PS is only 3. Fortunately, bit 19 is RES0 in VTCR_EL2...
 	 */
 	parange = read_sysreg(id_aa64mmfr0_el1) & 7;
+	if (parange > ID_AA64MMFR0_PARANGE_MAX)
+		parange = ID_AA64MMFR0_PARANGE_MAX;
 	val |= parange << 16;
 
 	/* Compute the actual PARange... */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 877d42fb0df6..9f16cfa89dac 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -229,11 +229,9 @@ ENTRY(__cpu_setup)
 	tcr_set_idmap_t0sz	x10, x9
 
 	/*
-	 * Read the PARange bits from ID_AA64MMFR0_EL1 and set the IPS bits in
-	 * TCR_EL1.
+	 * Set the IPS bits in TCR_EL1.
 	 */
-	mrs	x9, ID_AA64MMFR0_EL1
-	bfi	x10, x9, #32, #3
+	tcr_set_pa_size x10, #32, x5, x6
 #ifdef CONFIG_ARM64_HW_AFDBM
 	/*
 	 * Hardware update of the Access and Dirty bits.
-- 
2.1.4

[RFC 3/9] arm64: handle 52-bit addresses in TTBR

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits 2..5
in the TTBR registers. Introduce a couple of macros to move the bits
there, and change all TTBR writers to use them.

Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
system without 52-bit PA can only use up to 48-bit PAs. Add a kconfig
dependency to ensure PAN is configured.

In addition, when using 52-bit PA there is a special alignment
requirement on the top-level table. We don't currently have any VA_BITS
configuration that would violate the requirement, but one could be added
in the future, so add a compile-time BUG_ON to check for it.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h       |  2 ++
 arch/arm64/Kconfig                   |  1 +
 arch/arm64/include/asm/assembler.h   | 17 +++++++++++++++++
 arch/arm64/include/asm/kvm_mmu.h     |  2 ++
 arch/arm64/include/asm/mmu_context.h |  2 +-
 arch/arm64/include/asm/pgtable.h     |  6 ++++++
 arch/arm64/kernel/head.S             |  6 ++++--
 arch/arm64/kernel/hibernate-asm.S    | 12 +++++++-----
 arch/arm64/kernel/hibernate.c        |  2 +-
 arch/arm64/kvm/hyp-init.S            |  3 ++-
 arch/arm64/mm/pgd.c                  |  8 ++++++++
 arch/arm64/mm/proc.S                 | 13 ++++++++-----
 virt/kvm/arm/arm.c                   |  2 +-
 13 files changed, 60 insertions(+), 16 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index fa6f2174276b..8dbec683638b 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -221,6 +221,8 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return 8;
 }
 
+#define kvm_phys_to_vttbr(addr)		(addr)
+
 #endif	/* !__ASSEMBLY__ */
 
 #endif /* __ARM_KVM_MMU_H__ */
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 30d0cc272903..7e63048e3425 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -622,6 +622,7 @@ config ARM64_PA_BITS_48
 config ARM64_PA_BITS_52
 	bool "52-bit (ARMv8.2)"
 	depends on ARM64_64K_PAGES
+	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
 	help
 	  Enable support for a 52-bit physical address space, introduced as
 	  part of the ARMv8.2-LPA extension.
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 04cf94766b78..ba3c796b9fe1 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -512,4 +512,21 @@ alternative_else_nop_endif
 #endif
 	.endm
 
+/*
+ * Arrange a physical address in a TTBR register, taking care of 52-bit
+ * addresses.
+ *
+ * 	phys:	physical address, preserved
+ * 	ttbr:	returns the TTBR value
+ */
+	.macro	phys_to_ttbr, phys, ttbr
+#ifdef CONFIG_ARM64_PA_BITS_52
+	and	\ttbr, \phys, #(1 << 48) - 1
+	orr	\ttbr, \ttbr, \phys, lsr #48 - 2
+	bic	\ttbr, \ttbr, #(1 << 2) - 1
+#else
+	mov	\ttbr, \phys
+#endif
+	.endm
+
 #endif	/* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 672c8684d5c2..747bfff92948 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -309,5 +309,7 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
 }
 
+#define kvm_phys_to_vttbr(addr)		phys_to_ttbr(addr)
+
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 3257895a9b5e..c0aa2b221769 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -51,7 +51,7 @@ static inline void contextidr_thread_switch(struct task_struct *next)
  */
 static inline void cpu_set_reserved_ttbr0(void)
 {
-	unsigned long ttbr = __pa_symbol(empty_zero_page);
+	unsigned long ttbr = phys_to_ttbr(__pa_symbol(empty_zero_page));
 
 	write_sysreg(ttbr, ttbr0_el1);
 	isb();
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index b46e54c2399b..dcca52feaea2 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -720,6 +720,12 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
 #define kc_vaddr_to_offset(v)	((v) & ~VA_START)
 #define kc_offset_to_vaddr(o)	((o) | VA_START)
 
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define phys_to_ttbr(addr)	(((addr) & GENMASK(47, 6)) | (((addr) & GENMASK(51, 48)) >> 46))
+#else
+#define phys_to_ttbr(addr)	(addr)
+#endif
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __ASM_PGTABLE_H */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 0b243ecaf7ac..7fcbe23d9ce8 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -661,8 +661,10 @@ ENTRY(__enable_mmu)
 	update_early_cpu_boot_status 0, x1, x2
 	adrp	x1, idmap_pg_dir
 	adrp	x2, swapper_pg_dir
-	msr	ttbr0_el1, x1			// load TTBR0
-	msr	ttbr1_el1, x2			// load TTBR1
+	phys_to_ttbr x1, x3
+	phys_to_ttbr x2, x4
+	msr	ttbr0_el1, x3			// load TTBR0
+	msr	ttbr1_el1, x4			// load TTBR1
 	isb
 	msr	sctlr_el1, x0
 	isb
diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
index e56d848b6466..84f5d52fddda 100644
--- a/arch/arm64/kernel/hibernate-asm.S
+++ b/arch/arm64/kernel/hibernate-asm.S
@@ -33,12 +33,14 @@
  * Even switching to our copied tables will cause a changed output address at
  * each stage of the walk.
  */
-.macro break_before_make_ttbr_switch zero_page, page_table
-	msr	ttbr1_el1, \zero_page
+.macro break_before_make_ttbr_switch zero_page, page_table, tmp
+	phys_to_ttbr \zero_page, \tmp
+	msr	ttbr1_el1, \tmp
 	isb
 	tlbi	vmalle1
 	dsb	nsh
-	msr	ttbr1_el1, \page_table
+	phys_to_ttbr \page_table, \tmp
+	msr	ttbr1_el1, \tmp
 	isb
 .endm
 
@@ -78,7 +80,7 @@ ENTRY(swsusp_arch_suspend_exit)
 	 * We execute from ttbr0, change ttbr1 to our copied linear map tables
 	 * with a break-before-make via the zero page
 	 */
-	break_before_make_ttbr_switch	x5, x0
+	break_before_make_ttbr_switch	x5, x0, x6
 
 	mov	x21, x1
 	mov	x30, x2
@@ -109,7 +111,7 @@ ENTRY(swsusp_arch_suspend_exit)
 	dsb	ish		/* wait for PoU cleaning to finish */
 
 	/* switch to the restored kernels page tables */
-	break_before_make_ttbr_switch	x25, x21
+	break_before_make_ttbr_switch	x25, x21, x6
 
 	ic	ialluis
 	dsb	ish
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 095d3c170f5d..1ef660ebf049 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -263,7 +263,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
 	 */
 	cpu_set_reserved_ttbr0();
 	local_flush_tlb_all();
-	write_sysreg(virt_to_phys(pgd), ttbr0_el1);
+	write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1);
 	isb();
 
 	*phys_dst_addr = virt_to_phys((void *)dst);
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index f731a48bd9f1..a99718f32af9 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -63,7 +63,8 @@ __do_hyp_init:
 	cmp	x0, #HVC_STUB_HCALL_NR
 	b.lo	__kvm_handle_stub_hvc
 
-	msr	ttbr0_el2, x0
+	phys_to_ttbr x0, x4
+	msr	ttbr0_el2, x4
 
 	mrs	x4, tcr_el1
 	ldr	x5, =TCR_EL2_MASK
diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c
index 371c5f03a170..77919e615dfc 100644
--- a/arch/arm64/mm/pgd.c
+++ b/arch/arm64/mm/pgd.c
@@ -49,6 +49,14 @@ void __init pgd_cache_init(void)
 	if (PGD_SIZE == PAGE_SIZE)
 		return;
 
+#ifdef CONFIG_ARM64_PA_BITS_52
+	/*
+	 * With 52-bit physical addresses, the architecture requires the
+	 * top-level table to be aligned to at least 64 bytes.
+	 */
+	BUILD_BUG_ON(PGD_SIZE < 64);
+#endif
+
 	/*
 	 * Naturally aligned pgds required by the architecture.
 	 */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 9f16cfa89dac..5a29dbc703e2 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -138,10 +138,11 @@ ENDPROC(cpu_do_resume)
  *	- pgd_phys - physical address of new TTB
  */
 ENTRY(cpu_do_switch_mm)
-	pre_ttbr0_update_workaround x0, x2, x3
+	phys_to_ttbr x0, x2
+	pre_ttbr0_update_workaround x2, x3, x4
 	mmid	x1, x1				// get mm->context.id
-	bfi	x0, x1, #48, #16		// set the ASID
-	msr	ttbr0_el1, x0			// set TTBR0
+	bfi	x2, x1, #48, #16		// set the ASID
+	msr	ttbr0_el1, x2			// set TTBR0
 	isb
 	post_ttbr0_update_workaround
 	ret
@@ -159,14 +160,16 @@ ENTRY(idmap_cpu_replace_ttbr1)
 	msr	daifset, #0xf
 
 	adrp	x1, empty_zero_page
-	msr	ttbr1_el1, x1
+	phys_to_ttbr x1, x3
+	msr	ttbr1_el1, x3
 	isb
 
 	tlbi	vmalle1
 	dsb	nsh
 	isb
 
-	msr	ttbr1_el1, x0
+	phys_to_ttbr x0, x3
+	msr	ttbr1_el1, x3
 	isb
 
 	msr	daif, x2
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 95cba0799828..c208420aa11e 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -506,7 +506,7 @@ static void update_vttbr(struct kvm *kvm)
 	pgd_phys = virt_to_phys(kvm->arch.pgd);
 	BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK);
 	vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits);
-	kvm->arch.vttbr = pgd_phys | vmid;
+	kvm->arch.vttbr = kvm_phys_to_vttbr(pgd_phys) | vmid;
 
 	spin_unlock(&kvm_vmid_lock);
 }
-- 
2.1.4

[RFC 4/9] arm64: head.S: handle 52-bit PAs in PTEs in early page table setup

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits
12..15 in page table entries. Introduce a macro to move the bits there,
and change the early ID map and swapper table setup code to use it.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/kernel/head.S | 37 ++++++++++++++++++++++++++++---------
 1 file changed, 28 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 7fcbe23d9ce8..09f03bac53c5 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -148,6 +148,23 @@ preserve_boot_args:
 ENDPROC(preserve_boot_args)
 
 /*
+ * Macro to arrange a physical address in a page table entry, taking care of
+ * 52-bit addresses.
+ *
+ * Preserves:	phys
+ * Returns:	pte
+ */
+	.macro	phys_to_pte, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+	and	\pte, \phys, #(1 << 48) - 1
+	orr	\pte, \pte, \phys, lsr #48 - 12
+	bic	\pte, \pte, #(1 << 12) - 1
+#else
+	mov	\pte, \phys
+#endif
+	.endm
+
+/*
  * Macro to create a table entry to the next page.
  *
  *	tbl:	page table address
@@ -160,10 +177,11 @@ ENDPROC(preserve_boot_args)
  * Returns:	tbl -> next level table page address
  */
 	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
+	add	\tmp1, \tbl, #PAGE_SIZE
+	phys_to_pte \tmp1, \tmp2
+	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	lsr	\tmp1, \virt, #\shift
 	and	\tmp1, \tmp1, #\ptrs - 1	// table index
-	add	\tmp2, \tbl, #PAGE_SIZE
-	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	str	\tmp2, [\tbl, \tmp1, lsl #3]
 	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
 	.endm
@@ -190,16 +208,17 @@ ENDPROC(preserve_boot_args)
  * virtual range (inclusive).
  *
  * Preserves:	tbl, flags
- * Corrupts:	phys, start, end, pstate
+ * Corrupts:	phys, start, end, tmp
  */
-	.macro	create_block_map, tbl, flags, phys, start, end
-	lsr	\phys, \phys, #SWAPPER_BLOCK_SHIFT
+	.macro	create_block_map, tbl, flags, phys, start, end, tmp
 	lsr	\start, \start, #SWAPPER_BLOCK_SHIFT
 	and	\start, \start, #PTRS_PER_PTE - 1	// table index
-	orr	\phys, \flags, \phys, lsl #SWAPPER_BLOCK_SHIFT	// table entry
+	bic	\phys, \phys, #SWAPPER_BLOCK_SIZE - 1
 	lsr	\end, \end, #SWAPPER_BLOCK_SHIFT
 	and	\end, \end, #PTRS_PER_PTE - 1		// table end index
-9999:	str	\phys, [\tbl, \start, lsl #3]		// store the entry
+9999:	phys_to_pte \phys, \tmp
+	orr	\tmp, \tmp, \flags			// table entry
+	str	\tmp, [\tbl, \start, lsl #3]		// store the entry
 	add	\start, \start, #1			// next entry
 	add	\phys, \phys, #SWAPPER_BLOCK_SIZE		// next block
 	cmp	\start, \end
@@ -286,7 +305,7 @@ __create_page_tables:
 	create_pgd_entry x0, x3, x5, x6
 	mov	x5, x3				// __pa(__idmap_text_start)
 	adr_l	x6, __idmap_text_end		// __pa(__idmap_text_end)
-	create_block_map x0, x7, x3, x5, x6
+	create_block_map x0, x7, x3, x5, x6, x4
 
 	/*
 	 * Map the kernel image (starting with PHYS_OFFSET).
@@ -299,7 +318,7 @@ __create_page_tables:
 	adrp	x3, _text			// runtime __pa(_text)
 	sub	x6, x6, x3			// _end - _text
 	add	x6, x6, x5			// runtime __va(_end)
-	create_block_map x0, x7, x3, x5, x6
+	create_block_map x0, x7, x3, x5, x6, x4
 
 	/*
 	 * Since the page tables have been populated with non-cacheable
-- 
2.1.4

[RFC 5/9] arm64: don't open code page table entry creation

[Kristina Martsenko]

Instead of open coding the generation of page table entries, use the
macros/functions that exist for this - pfn_p*d and p*d_populate. Most
code in the kernel already uses these macros, this patch tries to fix
up the few places that don't. This is useful for the next patch in this
series, which needs to change the page table entry logic, and it's
better to have that logic in one place.

The KVM extended ID map is special, since we're creating a level above
CONFIG_PGTABLE_LEVELS and the required function isn't available. (The
normal kernel ID map code doesn't need this change because its page
tables are created in assembly (__create_page_tables)).

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 11 +++++++++--
 arch/arm64/kernel/hibernate.c    |  3 +--
 arch/arm64/mm/mmu.c              | 14 +++++++++-----
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 747bfff92948..5a11af32d1ff 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -276,6 +276,13 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
 	return __cpu_uses_extended_idmap();
 }
 
+/*
+ * Can't use pgd_populate here, because the extended idmap adds an extra level
+ * above CONFIG_PGTABLE_LEVELS (which is 2 or 3 if we're using the extended
+ * idmap), and pgd_populate is only available if CONFIG_PGTABLE_LEVELS = 4.
+ */
+#define kvm_extended_idmap_pgd(phys)	__pgd((phys) | PMD_TYPE_TABLE)
+
 static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       pgd_t *hyp_pgd,
 				       pgd_t *merged_hyp_pgd,
@@ -289,7 +296,7 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 	 * extended idmap.
 	 */
 	VM_BUG_ON(pgd_val(merged_hyp_pgd[0]));
-	merged_hyp_pgd[0] = __pgd(__pa(hyp_pgd) | PMD_TYPE_TABLE);
+	merged_hyp_pgd[0] = kvm_extended_idmap_pgd(__pa(hyp_pgd));
 
 	/*
 	 * Create another extended level entry that points to the boot HYP map,
@@ -299,7 +306,7 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 	 */
 	idmap_idx = hyp_idmap_start >> VA_BITS;
 	VM_BUG_ON(pgd_val(merged_hyp_pgd[idmap_idx]));
-	merged_hyp_pgd[idmap_idx] = __pgd(__pa(boot_hyp_pgd) | PMD_TYPE_TABLE);
+	merged_hyp_pgd[idmap_idx] = kvm_extended_idmap_pgd(__pa(boot_hyp_pgd));
 }
 
 static inline unsigned int kvm_get_vmid_bits(void)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 1ef660ebf049..18b9695d93f9 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -246,8 +246,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
 	}
 
 	pte = pte_offset_kernel(pmd, dst_addr);
-	set_pte(pte, __pte(virt_to_phys((void *)dst) |
-			 pgprot_val(PAGE_KERNEL_EXEC)));
+	set_pte(pte, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC));
 
 	/*
 	 * Load our new page tables. A strict BBM approach requires that we
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index f1eb15e0e864..4152b36033a2 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -570,8 +570,8 @@ static void __init map_kernel(pgd_t *pgd)
 		 * entry instead.
 		 */
 		BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES));
-		set_pud(pud_set_fixmap_offset(pgd, FIXADDR_START),
-			__pud(__pa_symbol(bm_pmd) | PUD_TYPE_TABLE));
+		pud_populate(&init_mm, pud_set_fixmap_offset(pgd, FIXADDR_START),
+			     lm_alias(bm_pmd));
 		pud_clear_fixmap();
 	} else {
 		BUG();
@@ -686,7 +686,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node)
 			if (!p)
 				return -ENOMEM;
 
-			set_pmd(pmd, __pmd(__pa(p) | PROT_SECT_NORMAL));
+			pmd_set_huge(pmd, __pa(p), __pgprot(PROT_SECT_NORMAL));
 		} else
 			vmemmap_verify((pte_t *)pmd, node, addr, next);
 	} while (addr = next, addr != end);
@@ -875,15 +875,19 @@ int __init arch_ioremap_pmd_supported(void)
 
 int pud_set_huge(pud_t *pud, phys_addr_t phys, pgprot_t prot)
 {
+	pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
+					pgprot_val(mk_sect_prot(prot)));
 	BUG_ON(phys & ~PUD_MASK);
-	set_pud(pud, __pud(phys | PUD_TYPE_SECT | pgprot_val(mk_sect_prot(prot))));
+	set_pud(pud, pfn_pud(__phys_to_pfn(phys), sect_prot));
 	return 1;
 }
 
 int pmd_set_huge(pmd_t *pmd, phys_addr_t phys, pgprot_t prot)
 {
+	pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
+					pgprot_val(mk_sect_prot(prot)));
 	BUG_ON(phys & ~PMD_MASK);
-	set_pmd(pmd, __pmd(phys | PMD_TYPE_SECT | pgprot_val(mk_sect_prot(prot))));
+	set_pmd(pmd, pfn_pmd(__phys_to_pfn(phys), sect_prot));
 	return 1;
 }
 
-- 
2.1.4

[RFC 6/9] arm64: handle 52-bit physical addresses in page table entries

[Kristina Martsenko]

The top 4 bits of a 52-bit physical address are positioned at bits
12..15 of a page table entry. Introduce macros to convert between a
physical address and its placement in a table entry, and change all
macros/functions that access PTEs to use them.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h       |  3 +-
 arch/arm64/include/asm/pgalloc.h       |  6 ++--
 arch/arm64/include/asm/pgtable-hwdef.h |  3 ++
 arch/arm64/include/asm/pgtable.h       | 57 +++++++++++++++++++++++++++-------
 4 files changed, 54 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 5a11af32d1ff..2705f5bd9abe 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -281,7 +281,8 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
  * above CONFIG_PGTABLE_LEVELS (which is 2 or 3 if we're using the extended
  * idmap), and pgd_populate is only available if CONFIG_PGTABLE_LEVELS = 4.
  */
-#define kvm_extended_idmap_pgd(phys)	__pgd((phys) | PMD_TYPE_TABLE)
+#define kvm_extended_idmap_pgd(phys) \
+	__pgd(pgd_val(__phys_to_pgd(phys)) | PMD_TYPE_TABLE)
 
 static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       pgd_t *hyp_pgd,
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index d25f4f137c2a..b57b83c34fbf 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -44,7 +44,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
 
 static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot)
 {
-	set_pud(pud, __pud(pmd | prot));
+	set_pud(pud, __pud(pud_val(__phys_to_pud(pmd)) | prot));
 }
 
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
@@ -73,7 +73,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
 
 static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot)
 {
-	set_pgd(pgdp, __pgd(pud | prot));
+	set_pgd(pgdp, __pgd(pgd_val(__phys_to_pgd(pud)) | prot));
 }
 
 static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
@@ -129,7 +129,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pte)
 static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t pte,
 				  pmdval_t prot)
 {
-	set_pmd(pmdp, __pmd(pte | prot));
+	set_pmd(pmdp, __pmd(pmd_val(__phys_to_pmd(pte)) | prot));
 }
 
 /*
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index eb0c2bd90de9..69cc40695164 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -166,6 +166,9 @@
 #define PTE_UXN			(_AT(pteval_t, 1) << 54)	/* User XN */
 #define PTE_HYP_XN		(_AT(pteval_t, 1) << 54)	/* HYP XN */
 
+#define PTE_ADDR_LOW		(((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
+#define PTE_ADDR_HIGH		(_AT(pteval_t, 0xf) << 12)
+
 /*
  * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).
  */
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index dcca52feaea2..558838624445 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -57,9 +57,20 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
 
 #define pte_ERROR(pte)		__pte_error(__FILE__, __LINE__, pte_val(pte))
 
-#define pte_pfn(pte)		((pte_val(pte) & PHYS_MASK) >> PAGE_SHIFT)
+/*
+ * Macros to convert between a physical address and its placement in a
+ * page table entry, taking care of 52-bit addresses.
+ */
+#ifdef CONFIG_ARM64_PA_BITS_52
+#define __pte_to_phys(pte)	((pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << 36))
+#define __phys_to_pte(phys)	__pte(((phys) & PTE_ADDR_LOW) | (((phys) & GENMASK(51, 48)) >> 36))
+#else
+#define __pte_to_phys(pte)	(pte_val(pte) & PTE_ADDR_LOW)
+#define __phys_to_pte(phys)	__pte(phys)
+#endif
 
-#define pfn_pte(pfn,prot)	(__pte(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+#define pte_pfn(pte)		(__pte_to_phys(pte) >> PAGE_SHIFT)
+#define pfn_pte(pfn,prot)	__pte(pte_val(__phys_to_pte((phys_addr_t)(pfn) << PAGE_SHIFT)) | pgprot_val(prot))
 
 #define pte_none(pte)		(!pte_val(pte))
 #define pte_clear(mm,addr,ptep)	set_pte(ptep, __pte(0))
@@ -270,11 +281,26 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
 
 #define __HAVE_ARCH_PTE_SPECIAL
 
+static inline pte_t pgd_pte(pgd_t pgd)
+{
+	return __pte(pgd_val(pgd));
+}
+
+static inline pgd_t pte_pgd(pte_t pte)
+{
+	return __pgd(pte_val(pte));
+}
+
 static inline pte_t pud_pte(pud_t pud)
 {
 	return __pte(pud_val(pud));
 }
 
+static inline pud_t pte_pud(pte_t pte)
+{
+	return __pud(pte_val(pte));
+}
+
 static inline pmd_t pud_pmd(pud_t pud)
 {
 	return __pmd(pud_val(pud));
@@ -336,15 +362,24 @@ static inline int pmd_protnone(pmd_t pmd)
 
 #define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
 
-#define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
-#define pfn_pmd(pfn,prot)	(__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+#define __pmd_to_phys(pmd)	__pte_to_phys(pmd_pte(pmd))
+#define __phys_to_pmd(phys)	pte_pmd(__phys_to_pte(phys))
+#define pmd_pfn(pmd)		((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT)
+#define pfn_pmd(pfn,prot)	__pmd(pmd_val(__phys_to_pmd((phys_addr_t)(pfn) << PAGE_SHIFT)) | pgprot_val(prot))
 #define mk_pmd(page,prot)	pfn_pmd(page_to_pfn(page),prot)
 
 #define pud_write(pud)		pte_write(pud_pte(pud))
-#define pud_pfn(pud)		(((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
+
+#define __pud_to_phys(pud)	__pte_to_phys(pud_pte(pud))
+#define __phys_to_pud(phys)	pte_pud(__phys_to_pte(phys))
+#define pud_pfn(pud)		((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT)
+#define pfn_pud(pfn,prot)	__pud(pud_val(__phys_to_pud((phys_addr_t)(pfn) << PAGE_SHIFT)) | pgprot_val(prot))
 
 #define set_pmd_at(mm, addr, pmdp, pmd)	set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
 
+#define __pgd_to_phys(pgd)	__pte_to_phys(pgd_pte(pgd))
+#define __phys_to_pgd(phys)	pte_pgd(__phys_to_pte(phys))
+
 #define __pgprot_modify(prot,mask,bits) \
 	__pgprot((pgprot_val(prot) & ~(mask)) | (bits))
 
@@ -395,7 +430,7 @@ static inline void pmd_clear(pmd_t *pmdp)
 
 static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
 {
-	return pmd_val(pmd) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pmd_to_phys(pmd);
 }
 
 /* Find an entry in the third-level page table. */
@@ -413,7 +448,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd)
 #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
 #define pte_clear_fixmap()		clear_fixmap(FIX_PTE)
 
-#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK))
+#define pmd_page(pmd)		pfn_to_page(__phys_to_pfn(__pmd_to_phys(pmd)))
 
 /* use ONLY for statically allocated translation tables */
 #define pte_offset_kimg(dir,addr)	((pte_t *)__phys_to_kimg(pte_offset_phys((dir), (addr))))
@@ -446,7 +481,7 @@ static inline void pud_clear(pud_t *pudp)
 
 static inline phys_addr_t pud_page_paddr(pud_t pud)
 {
-	return pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pud_to_phys(pud);
 }
 
 /* Find an entry in the second-level page table. */
@@ -459,7 +494,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud)
 #define pmd_set_fixmap_offset(pud, addr)	pmd_set_fixmap(pmd_offset_phys(pud, addr))
 #define pmd_clear_fixmap()		clear_fixmap(FIX_PMD)
 
-#define pud_page(pud)		pfn_to_page(__phys_to_pfn(pud_val(pud) & PHYS_MASK))
+#define pud_page(pud)		pfn_to_page(__phys_to_pfn(__pud_to_phys(pud)))
 
 /* use ONLY for statically allocated translation tables */
 #define pmd_offset_kimg(dir,addr)	((pmd_t *)__phys_to_kimg(pmd_offset_phys((dir), (addr))))
@@ -498,7 +533,7 @@ static inline void pgd_clear(pgd_t *pgdp)
 
 static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 {
-	return pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK;
+	return __pgd_to_phys(pgd);
 }
 
 /* Find an entry in the frst-level page table. */
@@ -511,7 +546,7 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 #define pud_set_fixmap_offset(pgd, addr)	pud_set_fixmap(pud_offset_phys(pgd, addr))
 #define pud_clear_fixmap()		clear_fixmap(FIX_PUD)
 
-#define pgd_page(pgd)		pfn_to_page(__phys_to_pfn(pgd_val(pgd) & PHYS_MASK))
+#define pgd_page(pgd)		pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd)))
 
 /* use ONLY for statically allocated translation tables */
 #define pud_offset_kimg(dir,addr)	((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr))))
-- 
2.1.4

[RFC 7/9] arm64: increase PHYS_MASK to 52 bits

[Kristina Martsenko]

PHYS_MASK_SHIFT represents the highest possible physical address
supported by the kernel, and is used in a number of places. In order to
support 52-bit physical memory, increase PHYS_MASK_SHIFT to 52 when
52-bit physical memory is configured, and retain 48 if it is not, to
e.g. keep IDMAP_PGTABLE_LEVELS accurate.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/pgtable-hwdef.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 69cc40695164..ab3eddb2e21b 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -199,7 +199,7 @@
 /*
  * Highest possible physical address supported.
  */
-#define PHYS_MASK_SHIFT		(48)
+#define PHYS_MASK_SHIFT		(CONFIG_ARM64_PA_BITS)
 #define PHYS_MASK		((UL(1) << PHYS_MASK_SHIFT) - 1)
 
 /*
-- 
2.1.4

[RFC 8/9] arm64: increase sparsemem MAX_PHYSMEM_BITS to 52

[Kristina Martsenko]

Sparsemem is used to store page structs and other information for each
physical page in the system. In order to support 52-bit physical memory,
increase MAX_PHYSMEM_BITS to 52 when 52-bit physical memory is
configured. If it is not configured, then the kernel can't use 52-bit
memory, so leave MAX_PHYSMEM bits at 48 in that case.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
index 74a9d301819f..b299929fe56c 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -17,7 +17,7 @@
 #define __ASM_SPARSEMEM_H
 
 #ifdef CONFIG_SPARSEMEM
-#define MAX_PHYSMEM_BITS	48
+#define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
 #define SECTION_SIZE_BITS	30
 #endif
 
-- 
2.1.4

[RFC 9/9] arm64: allow ID map to be extended to 52 bits

[Kristina Martsenko]

Currently, when using VA_BITS < 48, if the ID map text happens to be
placed in physical memory above VA_BITS, we increase the VA size (up to
48) and create a new table level, in order to map in the ID map text.
This is okay because the system always supports 48 bits of VA.

This patch extends the code such that if the system supports 52 bits of
VA, and the ID map text is placed that high up, then we increase the VA
size accordingly, up to 52.

One difference from the current implementation is that so far the
condition of VA_BITS < 48 has meant that the top level table is always
"full", with the maximum number of entries, and an extra table level is
always needed. Now, when VA_BITS = 48 (and using 64k pages), the top
level table is not full, and we simply need to increase the number of
entries in it, instead of creating a new table level.

Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h       |  5 +++
 arch/arm64/include/asm/assembler.h   |  2 -
 arch/arm64/include/asm/kvm_mmu.h     |  7 +++-
 arch/arm64/include/asm/mmu_context.h | 14 ++++++-
 arch/arm64/kernel/head.S             | 76 +++++++++++++++++++++---------------
 arch/arm64/kvm/hyp-init.S            | 17 ++++----
 arch/arm64/mm/mmu.c                  |  1 +
 virt/kvm/arm/mmu.c                   | 12 +++---
 8 files changed, 83 insertions(+), 51 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 8dbec683638b..8c5643e2eea4 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -211,6 +211,11 @@ static inline bool __kvm_cpu_uses_extended_idmap(void)
 	return false;
 }
 
+static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
+{
+	return PTRS_PER_PGD;
+}
+
 static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd,
 				       pgd_t *hyp_pgd,
 				       pgd_t *merged_hyp_pgd,
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index ba3c796b9fe1..106bf1ab4b23 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -331,10 +331,8 @@ alternative_endif
  * tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map
  */
 	.macro	tcr_set_idmap_t0sz, valreg, tmpreg
-#ifndef CONFIG_ARM64_VA_BITS_48
 	ldr_l	\tmpreg, idmap_t0sz
 	bfi	\valreg, \tmpreg, #TCR_T0SZ_OFFSET, #TCR_TxSZ_WIDTH
-#endif
 	.endm
 
 /*
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 2705f5bd9abe..76f6d96a2762 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -273,7 +273,12 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled);
 
 static inline bool __kvm_cpu_uses_extended_idmap(void)
 {
-	return __cpu_uses_extended_idmap();
+	return __cpu_uses_extended_idmap_table();
+}
+
+static inline unsigned long __kvm_idmap_ptrs_per_pgd(void)
+{
+	return idmap_ptrs_per_pgd;
 }
 
 /*
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index c0aa2b221769..53fe5eac0e0d 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -63,11 +63,21 @@ static inline void cpu_set_reserved_ttbr0(void)
  * physical memory, in which case it will be smaller.
  */
 extern u64 idmap_t0sz;
+extern u64 idmap_ptrs_per_pgd;
 
 static inline bool __cpu_uses_extended_idmap(void)
 {
-	return (!IS_ENABLED(CONFIG_ARM64_VA_BITS_48) &&
-		unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS)));
+	return unlikely(idmap_t0sz != TCR_T0SZ(VA_BITS));
+}
+
+/*
+ * True if the extended ID map requires an extra level of translation table
+ * to be configured.
+ */
+static inline bool __cpu_uses_extended_idmap_table(void)
+{
+	return __cpu_uses_extended_idmap() &&
+		(idmap_ptrs_per_pgd == PTRS_PER_PGD);
 }
 
 /*
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 09f03bac53c5..ae1d8b7c4654 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -173,7 +173,7 @@ ENDPROC(preserve_boot_args)
  *	ptrs:	#imm pointers per table page
  *
  * Preserves:	virt
- * Corrupts:	tmp1, tmp2
+ * Corrupts:	ptrs, tmp1, tmp2
  * Returns:	tbl -> next level table page address
  */
 	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
@@ -181,7 +181,8 @@ ENDPROC(preserve_boot_args)
 	phys_to_pte \tmp1, \tmp2
 	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	lsr	\tmp1, \virt, #\shift
-	and	\tmp1, \tmp1, #\ptrs - 1	// table index
+	sub	\ptrs, \ptrs, #1
+	and	\tmp1, \tmp1, \ptrs		// table index
 	str	\tmp2, [\tbl, \tmp1, lsl #3]
 	add	\tbl, \tbl, #PAGE_SIZE		// next level table page
 	.endm
@@ -191,15 +192,17 @@ ENDPROC(preserve_boot_args)
  * block entry in the next level (tbl) for the given virtual address.
  *
  * Preserves:	tbl, next, virt
- * Corrupts:	tmp1, tmp2
+ * Corrupts:	ptrs_per_pgd, tmp1, tmp2
  */
-	.macro	create_pgd_entry, tbl, virt, tmp1, tmp2
-	create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
+	.macro	create_pgd_entry, tbl, virt, ptrs_per_pgd, tmp1, tmp2
+	create_table_entry \tbl, \virt, PGDIR_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #if SWAPPER_PGTABLE_LEVELS > 3
-	create_table_entry \tbl, \virt, PUD_SHIFT, PTRS_PER_PUD, \tmp1, \tmp2
+	mov	\ptrs_per_pgd, PTRS_PER_PUD
+	create_table_entry \tbl, \virt, PUD_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #endif
 #if SWAPPER_PGTABLE_LEVELS > 2
-	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, PTRS_PER_PTE, \tmp1, \tmp2
+	mov	\ptrs_per_pgd, PTRS_PER_PTE
+	create_table_entry \tbl, \virt, SWAPPER_TABLE_SHIFT, \ptrs_per_pgd, \tmp1, \tmp2
 #endif
 	.endm
 
@@ -263,26 +266,13 @@ __create_page_tables:
 	adrp	x0, idmap_pg_dir
 	adrp	x3, __idmap_text_start		// __pa(__idmap_text_start)
 
-#ifndef CONFIG_ARM64_VA_BITS_48
-#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
-#define EXTRA_PTRS	(1 << (48 - EXTRA_SHIFT))
-
-	/*
-	 * If VA_BITS < 48, it may be too small to allow for an ID mapping to be
-	 * created that covers system RAM if that is located sufficiently high
-	 * in the physical address space. So for the ID map, use an extended
-	 * virtual range in that case, by configuring an additional translation
-	 * level.
-	 * First, we have to verify our assumption that the current value of
-	 * VA_BITS was chosen such that all translation levels are fully
-	 * utilised, and that lowering T0SZ will always result in an additional
-	 * translation level to be configured.
-	 */
-#if VA_BITS != EXTRA_SHIFT
-#error "Mismatch between VA_BITS and page size/number of translation levels"
-#endif
-
 	/*
+	 * VA_BITS may be too small to allow for an ID mapping to be created
+	 * that covers system RAM if that is located sufficiently high in the
+	 * physical address space. So for the ID map, use an extended virtual
+	 * range in that case, and configure an additional translation level
+	 * if needed.
+	 *
 	 * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the
 	 * entire ID map region can be mapped. As T0SZ == (64 - #bits used),
 	 * this number conveniently equals the number of leading zeroes in
@@ -291,18 +281,41 @@ __create_page_tables:
 	adrp	x5, __idmap_text_end
 	clz	x5, x5
 	cmp	x5, TCR_T0SZ(VA_BITS)	// default T0SZ small enough?
-	b.ge	1f			// .. then skip additional level
+	b.ge	1f			// .. then skip VA range extension
 
 	adr_l	x6, idmap_t0sz
 	str	x5, [x6]
 	dmb	sy
 	dc	ivac, x6		// Invalidate potentially stale cache line
 
-	create_table_entry x0, x3, EXTRA_SHIFT, EXTRA_PTRS, x5, x6
-1:
+#if (VA_BITS < 48)
+#define EXTRA_SHIFT	(PGDIR_SHIFT + PAGE_SHIFT - 3)
+#define EXTRA_PTRS	(1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT))
+
+	/*
+	 * If VA_BITS < 48, we have to configure an additional table level.
+	 * First, we have to verify our assumption that the current value of
+	 * VA_BITS was chosen such that all translation levels are fully
+	 * utilised, and that lowering T0SZ will always result in an additional
+	 * translation level to be configured.
+	 */
+#if VA_BITS != EXTRA_SHIFT
+#error "Mismatch between VA_BITS and page size/number of translation levels"
 #endif
 
-	create_pgd_entry x0, x3, x5, x6
+	mov	x4, EXTRA_PTRS
+	create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6
+#else
+	/*
+	 * If VA_BITS == 48, we don't have to configure an additional
+	 * translation level, but the top-level table has more entries.
+	 */
+	mov	x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT)
+	str_l	x4, idmap_ptrs_per_pgd, x5
+#endif
+1:
+	ldr_l	x4, idmap_ptrs_per_pgd
+	create_pgd_entry x0, x3, x4, x5, x6
 	mov	x5, x3				// __pa(__idmap_text_start)
 	adr_l	x6, __idmap_text_end		// __pa(__idmap_text_end)
 	create_block_map x0, x7, x3, x5, x6, x4
@@ -313,7 +326,8 @@ __create_page_tables:
 	adrp	x0, swapper_pg_dir
 	mov_q	x5, KIMAGE_VADDR + TEXT_OFFSET	// compile time __va(_text)
 	add	x5, x5, x23			// add KASLR displacement
-	create_pgd_entry x0, x5, x3, x6
+	mov	x4, PTRS_PER_PGD
+	create_pgd_entry x0, x5, x4, x3, x6
 	adrp	x6, _end			// runtime __pa(_end)
 	adrp	x3, _text			// runtime __pa(_text)
 	sub	x6, x6, x3			// _end - _text
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index a99718f32af9..c2ebe4e992df 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -72,24 +72,23 @@ __do_hyp_init:
 	mov	x5, #TCR_EL2_RES1
 	orr	x4, x4, x5
 
-#ifndef CONFIG_ARM64_VA_BITS_48
 	/*
-	 * If we are running with VA_BITS < 48, we may be running with an extra
-	 * level of translation in the ID map. This is only the case if system
-	 * RAM is out of range for the currently configured page size and number
-	 * of translation levels, in which case we will also need the extra
-	 * level for the HYP ID map, or we won't be able to enable the EL2 MMU.
+	 * The ID map may be configured to use an extended virtual address
+	 * range. This is only the case if system RAM is out of range for the
+	 * currently configured page size and VA_BITS, in which case we will
+	 * also need the extended virtual range for the HYP ID map, or we won't
+	 * be able to enable the EL2 MMU.
 	 *
 	 * However, at EL2, there is only one TTBR register, and we can't switch
 	 * between translation tables *and* update TCR_EL2.T0SZ@the same
-	 * time. Bottom line: we need the extra level in *both* our translation
-	 * tables.
+	 * time. Bottom line: we need to use the extended range with *both* our
+	 * translation tables.
 	 *
 	 * So use the same T0SZ value we use for the ID map.
 	 */
 	ldr_l	x5, idmap_t0sz
 	bfi	x4, x5, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
-#endif
+
 	/*
 	 * Set the PS bits in TCR_EL2.
 	 */
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 4152b36033a2..ece47a2b2e68 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -50,6 +50,7 @@
 #define NO_CONT_MAPPINGS	BIT(1)
 
 u64 idmap_t0sz = TCR_T0SZ(VA_BITS);
+u64 idmap_ptrs_per_pgd = PTRS_PER_PGD;
 
 u64 kimage_voffset __ro_after_init;
 EXPORT_SYMBOL(kimage_voffset);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b36945d49986..876caf531d32 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -623,7 +623,7 @@ static int create_hyp_pud_mappings(pgd_t *pgd, unsigned long start,
 	return 0;
 }
 
-static int __create_hyp_mappings(pgd_t *pgdp,
+static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
 				 unsigned long start, unsigned long end,
 				 unsigned long pfn, pgprot_t prot)
 {
@@ -636,7 +636,7 @@ static int __create_hyp_mappings(pgd_t *pgdp,
 	addr = start & PAGE_MASK;
 	end = PAGE_ALIGN(end);
 	do {
-		pgd = pgdp + pgd_index(addr);
+		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
 
 		if (pgd_none(*pgd)) {
 			pud = pud_alloc_one(NULL, addr);
@@ -699,8 +699,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 		int err;
 
 		phys_addr = kvm_kaddr_to_phys(from + virt_addr - start);
-		err = __create_hyp_mappings(hyp_pgd, virt_addr,
-					    virt_addr + PAGE_SIZE,
+		err = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD,
+					    virt_addr, virt_addr + PAGE_SIZE,
 					    __phys_to_pfn(phys_addr),
 					    prot);
 		if (err)
@@ -731,7 +731,7 @@ int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
 	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
 		return -EINVAL;
 
-	return __create_hyp_mappings(hyp_pgd, start, end,
+	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
 }
 
@@ -1737,7 +1737,7 @@ static int kvm_map_idmap_text(pgd_t *pgd)
 	int err;
 
 	/* Create the idmap in the boot page tables */
-	err = 	__create_hyp_mappings(pgd,
+	err = 	__create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
 				      hyp_idmap_start, hyp_idmap_end,
 				      __phys_to_pfn(hyp_idmap_start),
 				      PAGE_HYP_EXEC);
-- 
2.1.4

[RFC 3/9] arm64: handle 52-bit addresses in TTBR

[Robin Murphy]

Hi Kristina,

On 21/11/17 11:57, Kristina Martsenko wrote:
> The top 4 bits of a 52-bit physical address are positioned at bits 2..5
> in the TTBR registers. Introduce a couple of macros to move the bits
> there, and change all TTBR writers to use them.
> 
> Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
> 52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
> system without 52-bit PA can only use up to 48-bit PAs. Add a kconfig
> dependency to ensure PAN is configured.
> 
> In addition, when using 52-bit PA there is a special alignment
> requirement on the top-level table. We don't currently have any VA_BITS
> configuration that would violate the requirement, but one could be added
> in the future, so add a compile-time BUG_ON to check for it.
> 
> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
> ---
>   arch/arm/include/asm/kvm_mmu.h       |  2 ++
>   arch/arm64/Kconfig                   |  1 +
>   arch/arm64/include/asm/assembler.h   | 17 +++++++++++++++++
>   arch/arm64/include/asm/kvm_mmu.h     |  2 ++
>   arch/arm64/include/asm/mmu_context.h |  2 +-
>   arch/arm64/include/asm/pgtable.h     |  6 ++++++
>   arch/arm64/kernel/head.S             |  6 ++++--
>   arch/arm64/kernel/hibernate-asm.S    | 12 +++++++-----
>   arch/arm64/kernel/hibernate.c        |  2 +-
>   arch/arm64/kvm/hyp-init.S            |  3 ++-
>   arch/arm64/mm/pgd.c                  |  8 ++++++++
>   arch/arm64/mm/proc.S                 | 13 ++++++++-----
>   virt/kvm/arm/arm.c                   |  2 +-
>   13 files changed, 60 insertions(+), 16 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index fa6f2174276b..8dbec683638b 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -221,6 +221,8 @@ static inline unsigned int kvm_get_vmid_bits(void)
>   	return 8;
>   }
>   
> +#define kvm_phys_to_vttbr(addr)		(addr)
> +
>   #endif	/* !__ASSEMBLY__ */
>   
>   #endif /* __ARM_KVM_MMU_H__ */
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 30d0cc272903..7e63048e3425 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -622,6 +622,7 @@ config ARM64_PA_BITS_48
>   config ARM64_PA_BITS_52
>   	bool "52-bit (ARMv8.2)"
>   	depends on ARM64_64K_PAGES
> +	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
>   	help
>   	  Enable support for a 52-bit physical address space, introduced as
>   	  part of the ARMv8.2-LPA extension.
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 04cf94766b78..ba3c796b9fe1 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -512,4 +512,21 @@ alternative_else_nop_endif
>   #endif
>   	.endm
>   
> +/*
> + * Arrange a physical address in a TTBR register, taking care of 52-bit
> + * addresses.
> + *
> + * 	phys:	physical address, preserved
> + * 	ttbr:	returns the TTBR value
> + */
> +	.macro	phys_to_ttbr, phys, ttbr
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +	and	\ttbr, \phys, #(1 << 48) - 1
> +	orr	\ttbr, \ttbr, \phys, lsr #48 - 2
> +	bic	\ttbr, \ttbr, #(1 << 2) - 1

Is there any reason for masking off each end of the result separately 
like this, or could we just do it more straightforwardly?

#define TTBR_BADDR_MASK ((1 << 46) - 1) << 2

	orr	\ttbr, \phys, \phys, lsr #46
	and	\ttbr, \ttbr, #TTBR_BADDR_MASK

(and equivalently for phys_to_pte in patch 4)

Even better if there's a convenient place to define the mask such that 
it can be shared with KVM's existing VTTBR stuff too.

Robin.

> +#else
> +	mov	\ttbr, \phys
> +#endif
> +	.endm
> +
>   #endif	/* __ASM_ASSEMBLER_H */
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 672c8684d5c2..747bfff92948 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -309,5 +309,7 @@ static inline unsigned int kvm_get_vmid_bits(void)
>   	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>   }
>   
> +#define kvm_phys_to_vttbr(addr)		phys_to_ttbr(addr)
> +
>   #endif /* __ASSEMBLY__ */
>   #endif /* __ARM64_KVM_MMU_H__ */
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index 3257895a9b5e..c0aa2b221769 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -51,7 +51,7 @@ static inline void contextidr_thread_switch(struct task_struct *next)
>    */
>   static inline void cpu_set_reserved_ttbr0(void)
>   {
> -	unsigned long ttbr = __pa_symbol(empty_zero_page);
> +	unsigned long ttbr = phys_to_ttbr(__pa_symbol(empty_zero_page));
>   
>   	write_sysreg(ttbr, ttbr0_el1);
>   	isb();
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index b46e54c2399b..dcca52feaea2 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -720,6 +720,12 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
>   #define kc_vaddr_to_offset(v)	((v) & ~VA_START)
>   #define kc_offset_to_vaddr(o)	((o) | VA_START)
>   
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +#define phys_to_ttbr(addr)	(((addr) & GENMASK(47, 6)) | (((addr) & GENMASK(51, 48)) >> 46))
> +#else
> +#define phys_to_ttbr(addr)	(addr)
> +#endif
> +
>   #endif /* !__ASSEMBLY__ */
>   
>   #endif /* __ASM_PGTABLE_H */
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 0b243ecaf7ac..7fcbe23d9ce8 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -661,8 +661,10 @@ ENTRY(__enable_mmu)
>   	update_early_cpu_boot_status 0, x1, x2
>   	adrp	x1, idmap_pg_dir
>   	adrp	x2, swapper_pg_dir
> -	msr	ttbr0_el1, x1			// load TTBR0
> -	msr	ttbr1_el1, x2			// load TTBR1
> +	phys_to_ttbr x1, x3
> +	phys_to_ttbr x2, x4
> +	msr	ttbr0_el1, x3			// load TTBR0
> +	msr	ttbr1_el1, x4			// load TTBR1
>   	isb
>   	msr	sctlr_el1, x0
>   	isb
> diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
> index e56d848b6466..84f5d52fddda 100644
> --- a/arch/arm64/kernel/hibernate-asm.S
> +++ b/arch/arm64/kernel/hibernate-asm.S
> @@ -33,12 +33,14 @@
>    * Even switching to our copied tables will cause a changed output address at
>    * each stage of the walk.
>    */
> -.macro break_before_make_ttbr_switch zero_page, page_table
> -	msr	ttbr1_el1, \zero_page
> +.macro break_before_make_ttbr_switch zero_page, page_table, tmp
> +	phys_to_ttbr \zero_page, \tmp
> +	msr	ttbr1_el1, \tmp
>   	isb
>   	tlbi	vmalle1
>   	dsb	nsh
> -	msr	ttbr1_el1, \page_table
> +	phys_to_ttbr \page_table, \tmp
> +	msr	ttbr1_el1, \tmp
>   	isb
>   .endm
>   
> @@ -78,7 +80,7 @@ ENTRY(swsusp_arch_suspend_exit)
>   	 * We execute from ttbr0, change ttbr1 to our copied linear map tables
>   	 * with a break-before-make via the zero page
>   	 */
> -	break_before_make_ttbr_switch	x5, x0
> +	break_before_make_ttbr_switch	x5, x0, x6
>   
>   	mov	x21, x1
>   	mov	x30, x2
> @@ -109,7 +111,7 @@ ENTRY(swsusp_arch_suspend_exit)
>   	dsb	ish		/* wait for PoU cleaning to finish */
>   
>   	/* switch to the restored kernels page tables */
> -	break_before_make_ttbr_switch	x25, x21
> +	break_before_make_ttbr_switch	x25, x21, x6
>   
>   	ic	ialluis
>   	dsb	ish
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 095d3c170f5d..1ef660ebf049 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -263,7 +263,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
>   	 */
>   	cpu_set_reserved_ttbr0();
>   	local_flush_tlb_all();
> -	write_sysreg(virt_to_phys(pgd), ttbr0_el1);
> +	write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1);
>   	isb();
>   
>   	*phys_dst_addr = virt_to_phys((void *)dst);
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index f731a48bd9f1..a99718f32af9 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -63,7 +63,8 @@ __do_hyp_init:
>   	cmp	x0, #HVC_STUB_HCALL_NR
>   	b.lo	__kvm_handle_stub_hvc
>   
> -	msr	ttbr0_el2, x0
> +	phys_to_ttbr x0, x4
> +	msr	ttbr0_el2, x4
>   
>   	mrs	x4, tcr_el1
>   	ldr	x5, =TCR_EL2_MASK
> diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c
> index 371c5f03a170..77919e615dfc 100644
> --- a/arch/arm64/mm/pgd.c
> +++ b/arch/arm64/mm/pgd.c
> @@ -49,6 +49,14 @@ void __init pgd_cache_init(void)
>   	if (PGD_SIZE == PAGE_SIZE)
>   		return;
>   
> +#ifdef CONFIG_ARM64_PA_BITS_52
> +	/*
> +	 * With 52-bit physical addresses, the architecture requires the
> +	 * top-level table to be aligned to at least 64 bytes.
> +	 */
> +	BUILD_BUG_ON(PGD_SIZE < 64);
> +#endif
> +
>   	/*
>   	 * Naturally aligned pgds required by the architecture.
>   	 */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 9f16cfa89dac..5a29dbc703e2 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -138,10 +138,11 @@ ENDPROC(cpu_do_resume)
>    *	- pgd_phys - physical address of new TTB
>    */
>   ENTRY(cpu_do_switch_mm)
> -	pre_ttbr0_update_workaround x0, x2, x3
> +	phys_to_ttbr x0, x2
> +	pre_ttbr0_update_workaround x2, x3, x4
>   	mmid	x1, x1				// get mm->context.id
> -	bfi	x0, x1, #48, #16		// set the ASID
> -	msr	ttbr0_el1, x0			// set TTBR0
> +	bfi	x2, x1, #48, #16		// set the ASID
> +	msr	ttbr0_el1, x2			// set TTBR0
>   	isb
>   	post_ttbr0_update_workaround
>   	ret
> @@ -159,14 +160,16 @@ ENTRY(idmap_cpu_replace_ttbr1)
>   	msr	daifset, #0xf
>   
>   	adrp	x1, empty_zero_page
> -	msr	ttbr1_el1, x1
> +	phys_to_ttbr x1, x3
> +	msr	ttbr1_el1, x3
>   	isb
>   
>   	tlbi	vmalle1
>   	dsb	nsh
>   	isb
>   
> -	msr	ttbr1_el1, x0
> +	phys_to_ttbr x0, x3
> +	msr	ttbr1_el1, x3
>   	isb
>   
>   	msr	daif, x2
> diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
> index 95cba0799828..c208420aa11e 100644
> --- a/virt/kvm/arm/arm.c
> +++ b/virt/kvm/arm/arm.c
> @@ -506,7 +506,7 @@ static void update_vttbr(struct kvm *kvm)
>   	pgd_phys = virt_to_phys(kvm->arch.pgd);
>   	BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK);
>   	vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits);
> -	kvm->arch.vttbr = pgd_phys | vmid;
> +	kvm->arch.vttbr = kvm_phys_to_vttbr(pgd_phys) | vmid;
>   
>   	spin_unlock(&kvm_vmid_lock);
>   }
> 

[RFC 3/9] arm64: handle 52-bit addresses in TTBR

[Kristina Martsenko]

On 21/11/17 14:39, Robin Murphy wrote:
> Hi Kristina,
> 
> On 21/11/17 11:57, Kristina Martsenko wrote:
>> The top 4 bits of a 52-bit physical address are positioned at bits 2..5
>> in the TTBR registers. Introduce a couple of macros to move the bits
>> there, and change all TTBR writers to use them.
>>
>> Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
>> 52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
>> system without 52-bit PA can only use up to 48-bit PAs. Add a kconfig
>> dependency to ensure PAN is configured.
>>
>> In addition, when using 52-bit PA there is a special alignment
>> requirement on the top-level table. We don't currently have any VA_BITS
>> configuration that would violate the requirement, but one could be added
>> in the future, so add a compile-time BUG_ON to check for it.
>>
>> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
>> ---

[...]

>> diff --git a/arch/arm64/include/asm/assembler.h
>> b/arch/arm64/include/asm/assembler.h
>> index 04cf94766b78..ba3c796b9fe1 100644
>> --- a/arch/arm64/include/asm/assembler.h
>> +++ b/arch/arm64/include/asm/assembler.h
>> @@ -512,4 +512,21 @@ alternative_else_nop_endif
>> ? #endif
>> ????? .endm
>> ? +/*
>> + * Arrange a physical address in a TTBR register, taking care of 52-bit
>> + * addresses.
>> + *
>> + *???? phys:??? physical address, preserved
>> + *???? ttbr:??? returns the TTBR value
>> + */
>> +??? .macro??? phys_to_ttbr, phys, ttbr
>> +#ifdef CONFIG_ARM64_PA_BITS_52
>> +??? and??? \ttbr, \phys, #(1 << 48) - 1
>> +??? orr??? \ttbr, \ttbr, \phys, lsr #48 - 2
>> +??? bic??? \ttbr, \ttbr, #(1 << 2) - 1
> 
> Is there any reason for masking off each end of the result separately
> like this, or could we just do it more straightforwardly?

I don't recall any reason, maybe just to keep it simple, to avoid having
a separate mask macro.

> #define TTBR_BADDR_MASK ((1 << 46) - 1) << 2
> 
> ????orr??? \ttbr, \phys, \phys, lsr #46
> ????and??? \ttbr, \ttbr, #TTBR_BADDR_MASK
> 
> (and equivalently for phys_to_pte in patch 4)

Ok, I'll rewrite it like this. (Note that mask is 52-bit-specific though.)

> Even better if there's a convenient place to define the mask such that
> it can be shared with KVM's existing VTTBR stuff too.

Do you mean VTTBR_BADDR_MASK? I don't think this would be useful there,
VTTBR_BADDR_MASK checks the alignment of the address that goes into
VTTBR (not the VTTBR value itself), and takes into account specifically
the 40-bit IPA and concatenated page tables.

Kristina

>> +#else
>> +??? mov??? \ttbr, \phys
>> +#endif
>> +??? .endm
>> +
>> ? #endif??? /* __ASM_ASSEMBLER_H */

[RFC 3/9] arm64: handle 52-bit addresses in TTBR

[Robin Murphy]

On 07/12/17 12:29, Kristina Martsenko wrote:
> On 21/11/17 14:39, Robin Murphy wrote:
>> Hi Kristina,
>>
>> On 21/11/17 11:57, Kristina Martsenko wrote:
>>> The top 4 bits of a 52-bit physical address are positioned at bits 2..5
>>> in the TTBR registers. Introduce a couple of macros to move the bits
>>> there, and change all TTBR writers to use them.
>>>
>>> Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
>>> 52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
>>> system without 52-bit PA can only use up to 48-bit PAs. Add a kconfig
>>> dependency to ensure PAN is configured.
>>>
>>> In addition, when using 52-bit PA there is a special alignment
>>> requirement on the top-level table. We don't currently have any VA_BITS
>>> configuration that would violate the requirement, but one could be added
>>> in the future, so add a compile-time BUG_ON to check for it.
>>>
>>> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
>>> ---
> 
> [...]
> 
>>> diff --git a/arch/arm64/include/asm/assembler.h
>>> b/arch/arm64/include/asm/assembler.h
>>> index 04cf94766b78..ba3c796b9fe1 100644
>>> --- a/arch/arm64/include/asm/assembler.h
>>> +++ b/arch/arm64/include/asm/assembler.h
>>> @@ -512,4 +512,21 @@ alternative_else_nop_endif
>>>  ? #endif
>>>  ????? .endm
>>>  ? +/*
>>> + * Arrange a physical address in a TTBR register, taking care of 52-bit
>>> + * addresses.
>>> + *
>>> + *???? phys:??? physical address, preserved
>>> + *???? ttbr:??? returns the TTBR value
>>> + */
>>> +??? .macro??? phys_to_ttbr, phys, ttbr
>>> +#ifdef CONFIG_ARM64_PA_BITS_52
>>> +??? and??? \ttbr, \phys, #(1 << 48) - 1
>>> +??? orr??? \ttbr, \ttbr, \phys, lsr #48 - 2
>>> +??? bic??? \ttbr, \ttbr, #(1 << 2) - 1
>>
>> Is there any reason for masking off each end of the result separately
>> like this, or could we just do it more straightforwardly?
> 
> I don't recall any reason, maybe just to keep it simple, to avoid having
> a separate mask macro.
> 
>> #define TTBR_BADDR_MASK ((1 << 46) - 1) << 2
>>
>>  ????orr??? \ttbr, \phys, \phys, lsr #46
>>  ????and??? \ttbr, \ttbr, #TTBR_BADDR_MASK
>>
>> (and equivalently for phys_to_pte in patch 4)
> 
> Ok, I'll rewrite it like this. (Note that mask is 52-bit-specific though.)

I don't see that it need be 52-bit specific - true the BADDR field 
itself is strictly bits 47:1, but AFAICS bit 1 is always going to be 
RES0: either explicitly in the 52-bit case, or from the (x-1):1 
definition otherwise, since the requirement that a table must be aligned 
to its size infers an absolute lower bound of x >= 3 (it may be larger 
still due to other reasons, but I'm finding this area of the ARM ARM 
obnoxiously difficult to read). Thus defining the mask as covering 47:2 
should still be reasonable in all cases.

>> Even better if there's a convenient place to define the mask such that
>> it can be shared with KVM's existing VTTBR stuff too.
> 
> Do you mean VTTBR_BADDR_MASK? I don't think this would be useful there,
> VTTBR_BADDR_MASK checks the alignment of the address that goes into
> VTTBR (not the VTTBR value itself), and takes into account specifically
> the 40-bit IPA and concatenated page tables.

Ah, I see - from skimming the code I managed to assume VTTBR_BADDR_MASK 
was a mask for the actual VTTBR.BADDR register field; I hadn't delved 
into all the VTTBR_X gubbins (yuck). Fair enough, scratch that idea. At 
least TTBR_ASID_MASK[1] gets to have a friend :)

Robin.

[1] 
https://www.mail-archive.com/linux-kernel at vger.kernel.org/msg1555176.html

> 
> Kristina
> 
>>> +#else
>>> +??? mov??? \ttbr, \phys
>>> +#endif
>>> +??? .endm
>>> +
>>>  ? #endif??? /* __ASM_ASSEMBLER_H */

[RFC 3/9] arm64: handle 52-bit addresses in TTBR

[Kristina Martsenko]

On 07/12/17 14:51, Robin Murphy wrote:
> On 07/12/17 12:29, Kristina Martsenko wrote:
>> On 21/11/17 14:39, Robin Murphy wrote:
>>> Hi Kristina,
>>>
>>> On 21/11/17 11:57, Kristina Martsenko wrote:
>>>> The top 4 bits of a 52-bit physical address are positioned at bits 2..5
>>>> in the TTBR registers. Introduce a couple of macros to move the bits
>>>> there, and change all TTBR writers to use them.
>>>>
>>>> Leave TTBR0 PAN code unchanged, to avoid complicating it. A system with
>>>> 52-bit PA will have PAN anyway (because it's ARMv8.1 or later), and a
>>>> system without 52-bit PA can only use up to 48-bit PAs. Add a kconfig
>>>> dependency to ensure PAN is configured.
>>>>
>>>> In addition, when using 52-bit PA there is a special alignment
>>>> requirement on the top-level table. We don't currently have any VA_BITS
>>>> configuration that would violate the requirement, but one could be
>>>> added
>>>> in the future, so add a compile-time BUG_ON to check for it.
>>>>
>>>> Signed-off-by: Kristina Martsenko <kristina.martsenko@arm.com>
>>>> ---
>>
>> [...]
>>
>>>> diff --git a/arch/arm64/include/asm/assembler.h
>>>> b/arch/arm64/include/asm/assembler.h
>>>> index 04cf94766b78..ba3c796b9fe1 100644
>>>> --- a/arch/arm64/include/asm/assembler.h
>>>> +++ b/arch/arm64/include/asm/assembler.h
>>>> @@ -512,4 +512,21 @@ alternative_else_nop_endif
>>>> ?? #endif
>>>> ?????? .endm
>>>> ?? +/*
>>>> + * Arrange a physical address in a TTBR register, taking care of 52-bit
>>>> + * addresses.
>>>> + *
>>>> + *???? phys:??? physical address, preserved
>>>> + *???? ttbr:??? returns the TTBR value
>>>> + */
>>>> +??? .macro??? phys_to_ttbr, phys, ttbr
>>>> +#ifdef CONFIG_ARM64_PA_BITS_52
>>>> +??? and??? \ttbr, \phys, #(1 << 48) - 1
>>>> +??? orr??? \ttbr, \ttbr, \phys, lsr #48 - 2
>>>> +??? bic??? \ttbr, \ttbr, #(1 << 2) - 1
>>>
>>> Is there any reason for masking off each end of the result separately
>>> like this, or could we just do it more straightforwardly?
>>
>> I don't recall any reason, maybe just to keep it simple, to avoid having
>> a separate mask macro.
>>
>>> #define TTBR_BADDR_MASK ((1 << 46) - 1) << 2
>>>
>>> ?????orr??? \ttbr, \phys, \phys, lsr #46
>>> ?????and??? \ttbr, \ttbr, #TTBR_BADDR_MASK
>>>
>>> (and equivalently for phys_to_pte in patch 4)
>>
>> Ok, I'll rewrite it like this. (Note that mask is 52-bit-specific
>> though.)
> 
> I don't see that it need be 52-bit specific - true the BADDR field
> itself is strictly bits 47:1, but AFAICS bit 1 is always going to be
> RES0: either explicitly in the 52-bit case, or from the (x-1):1
> definition otherwise, since the requirement that a table must be aligned
> to its size infers an absolute lower bound of x >= 3 (it may be larger
> still due to other reasons, but I'm finding this area of the ARM ARM
> obnoxiously difficult to read). Thus defining the mask as covering 47:2
> should still be reasonable in all cases.

Yes BADDR is bits 47:1, and AFAICS bits 3:1 will be RES0 in the
non-52-bit case, since x >= 4 (since minimum 2 entries in a table). So I
think a non-52-bit mask should be 47:1 or 47:4. But in this patch, we
don't need a non-52-bit macro anyway, just one for the 52-bit case. I
will send a v2 today, please respond there if you're not happy with the
approach.

>>> Even better if there's a convenient place to define the mask such that
>>> it can be shared with KVM's existing VTTBR stuff too.
>>
>> Do you mean VTTBR_BADDR_MASK? I don't think this would be useful there,
>> VTTBR_BADDR_MASK checks the alignment of the address that goes into
>> VTTBR (not the VTTBR value itself), and takes into account specifically
>> the 40-bit IPA and concatenated page tables.
> 
> Ah, I see - from skimming the code I managed to assume VTTBR_BADDR_MASK
> was a mask for the actual VTTBR.BADDR register field; I hadn't delved
> into all the VTTBR_X gubbins (yuck). Fair enough, scratch that idea. At
> least TTBR_ASID_MASK[1] gets to have a friend :)

Yep, although TTBR_BADDR_MASK will go into pgtable-hwdef.h (like
TTBR_CNP_BIT [1]).

Kristina

[1] https://patchwork.kernel.org/patch/9992927/

> 
> Robin.
> 
> [1] https://www.mail-archive.com/linux-kernel at vger.kernel.org/msg1555176.html
> 
>>>> +#else
>>>> +??? mov??? \ttbr, \phys
>>>> +#endif
>>>> +??? .endm
>>>> +
>>>> ?? #endif??? /* __ASM_ASSEMBLER_H */
>