* [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX
@ 2018-01-26 12:03 Will Deacon
2018-01-26 12:03 ` [PATCH 1/7] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
` (6 more replies)
0 siblings, 7 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
Hi all,
Cavium ThunderX suffers from a mysterious hardware erratum where a broadcast
TLBI can lead to I-cache corruption when executing from non-global mappings.
This is somehow resolved for userspace by invalidating the entire I-cache
on context-switch.
http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
With kpti, the kernel always runs with non-global mappings which means that the
I-cache can become corrupted when running at EL1, causing weird crashes and
even attempts to execute user code.
This patch series tries to address this by making the transition from Global to
Non-global mappings a runtime decision after we've brought up all the CPUs. The
requirement for break-before-make in the presence of contiguous mappings make
this horribly complicated, so the page table manipulation runs mostly with the
MMU off and no stack.
I'd value feedback from people lucky enough to own a ThunderX as to whether
this fixes stability problems with their platform when running a kpti-enabled
kernel.
Patches based on for-next/core, which is why I've included Shanker's patch
introducing pre_disable_mmu_workaround as the first in the series (since
this is needed when we disable the MMU in idmap_kpti_install_ng_mappings).
Cheers,
Will
--->8
Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderX
Shanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041
Will Deacon (5):
arm64: kpti: Make use of nG dependent on
arm64_kernel_unmapped_at_el0()
arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments in phys_to_ttbr
arm64: entry: Reword comment about post_ttbr_update_workaround
Documentation/arm64/silicon-errata.txt | 1 +
arch/arm64/Kconfig | 12 +-
arch/arm64/include/asm/assembler.h | 22 +++-
arch/arm64/include/asm/kernel-pgtable.h | 12 +-
arch/arm64/include/asm/pgtable-prot.h | 30 +++--
arch/arm64/kernel/cpu-reset.S | 1 +
arch/arm64/kernel/cpufeature.c | 42 ++++++-
arch/arm64/kernel/efi-entry.S | 2 +
arch/arm64/kernel/entry.S | 12 +-
arch/arm64/kernel/head.S | 5 +-
arch/arm64/kernel/hibernate-asm.S | 4 +-
arch/arm64/kernel/relocate_kernel.S | 1 +
arch/arm64/kvm/hyp-init.S | 3 +-
arch/arm64/mm/mmu.c | 4 +
arch/arm64/mm/proc.S | 206 ++++++++++++++++++++++++++++++--
15 files changed, 302 insertions(+), 55 deletions(-)
--
2.1.4
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/7] arm64: Add software workaround for Falkor erratum 1041
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 12:03 ` [PATCH 2/7] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
` (5 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
From: Shanker Donthineni <shankerd@codeaurora.org>
The ARM architecture defines the memory locations that are permitted
to be accessed as the result of a speculative instruction fetch from
an exception level for which all stages of translation are disabled.
Specifically, the core is permitted to speculatively fetch from the
4KB region containing the current program counter 4K and next 4K.
When translation is changed from enabled to disabled for the running
exception level (SCTLR_ELn[M] changed from a value of 1 to 0), the
Falkor core may errantly speculatively access memory locations outside
of the 4KB region permitted by the architecture. The errant memory
access may lead to one of the following unexpected behaviors.
1) A System Error Interrupt (SEI) being raised by the Falkor core due
to the errant memory access attempting to access a region of memory
that is protected by a slave-side memory protection unit.
2) Unpredictable device behavior due to a speculative read from device
memory. This behavior may only occur if the instruction cache is
disabled prior to or coincident with translation being changed from
enabled to disabled.
The conditions leading to this erratum will not occur when either of the
following occur:
1) A higher exception level disables translation of a lower exception level
(e.g. EL2 changing SCTLR_EL1[M] from a value of 1 to 0).
2) An exception level disabling its stage-1 translation if its stage-2
translation is enabled (e.g. EL1 changing SCTLR_EL1[M] from a value of 1
to 0 when HCR_EL2[VM] has a value of 1).
To avoid the errant behavior, software must execute an ISB immediately
prior to executing the MSR that will change SCTLR_ELn[M] from 1 to 0.
Signed-off-by: Shanker Donthineni <shankerd@codeaurora.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
Documentation/arm64/silicon-errata.txt | 1 +
arch/arm64/Kconfig | 12 +++++++++++-
arch/arm64/include/asm/assembler.h | 10 ++++++++++
arch/arm64/kernel/cpu-reset.S | 1 +
arch/arm64/kernel/efi-entry.S | 2 ++
arch/arm64/kernel/head.S | 1 +
arch/arm64/kernel/relocate_kernel.S | 1 +
arch/arm64/kvm/hyp-init.S | 1 +
8 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/Documentation/arm64/silicon-errata.txt b/Documentation/arm64/silicon-errata.txt
index b9d93e981a05..c1d520de6dfe 100644
--- a/Documentation/arm64/silicon-errata.txt
+++ b/Documentation/arm64/silicon-errata.txt
@@ -75,3 +75,4 @@ stable kernels.
| Qualcomm Tech. | Kryo/Falkor v1 | E1003 | QCOM_FALKOR_ERRATUM_1003 |
| Qualcomm Tech. | Falkor v1 | E1009 | QCOM_FALKOR_ERRATUM_1009 |
| Qualcomm Tech. | QDF2400 ITS | E0065 | QCOM_QDF2400_ERRATUM_0065 |
+| Qualcomm Tech. | Falkor v{1,2} | E1041 | QCOM_FALKOR_ERRATUM_1041 |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1d51c8edf34b..b488076d63c2 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -550,7 +550,6 @@ config QCOM_QDF2400_ERRATUM_0065
If unsure, say Y.
-
config SOCIONEXT_SYNQUACER_PREITS
bool "Socionext Synquacer: Workaround for GICv3 pre-ITS"
default y
@@ -569,6 +568,17 @@ config HISILICON_ERRATUM_161600802
a 128kB offset to be applied to the target address in this commands.
If unsure, say Y.
+
+config QCOM_FALKOR_ERRATUM_E1041
+ bool "Falkor E1041: Speculative instruction fetches might cause errant memory access"
+ default y
+ help
+ Falkor CPU may speculatively fetch instructions from an improper
+ memory location when MMU translation is changed from SCTLR_ELn[M]=1
+ to SCTLR_ELn[M]=0. Prefix an ISB instruction to fix the problem.
+
+ If unsure, say Y.
+
endmenu
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 794fe8122602..3873dd7b5a32 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -523,4 +523,14 @@ alternative_endif
#endif
.endm
+/**
+ * Errata workaround prior to disable MMU. Insert an ISB immediately prior
+ * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
+ */
+ .macro pre_disable_mmu_workaround
+#ifdef CONFIG_QCOM_FALKOR_ERRATUM_E1041
+ isb
+#endif
+ .endm
+
#endif /* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 65f42d257414..2a752cb2a0f3 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -37,6 +37,7 @@ ENTRY(__cpu_soft_restart)
mrs x12, sctlr_el1
ldr x13, =SCTLR_ELx_FLAGS
bic x12, x12, x13
+ pre_disable_mmu_workaround
msr sctlr_el1, x12
isb
diff --git a/arch/arm64/kernel/efi-entry.S b/arch/arm64/kernel/efi-entry.S
index 4e6ad355bd05..6b9736c3fb56 100644
--- a/arch/arm64/kernel/efi-entry.S
+++ b/arch/arm64/kernel/efi-entry.S
@@ -96,6 +96,7 @@ ENTRY(entry)
mrs x0, sctlr_el2
bic x0, x0, #1 << 0 // clear SCTLR.M
bic x0, x0, #1 << 2 // clear SCTLR.C
+ pre_disable_mmu_workaround
msr sctlr_el2, x0
isb
b 2f
@@ -103,6 +104,7 @@ ENTRY(entry)
mrs x0, sctlr_el1
bic x0, x0, #1 << 0 // clear SCTLR.M
bic x0, x0, #1 << 2 // clear SCTLR.C
+ pre_disable_mmu_workaround
msr sctlr_el1, x0
isb
2:
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index c3b241b8b659..ba3ab04788dc 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -849,6 +849,7 @@ __primary_switch:
* to take into account by discarding the current kernel mapping and
* creating a new one.
*/
+ pre_disable_mmu_workaround
msr sctlr_el1, x20 // disable the MMU
isb
bl __create_page_tables // recreate kernel mapping
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index ce704a4aeadd..f407e422a720 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -45,6 +45,7 @@ ENTRY(arm64_relocate_new_kernel)
mrs x0, sctlr_el2
ldr x1, =SCTLR_ELx_FLAGS
bic x0, x0, x1
+ pre_disable_mmu_workaround
msr sctlr_el2, x0
isb
1:
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 8a00de187e56..e086c6eff8c6 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -153,6 +153,7 @@ reset:
mrs x5, sctlr_el2
ldr x6, =SCTLR_ELx_FLAGS
bic x5, x5, x6 // Clear SCTL_M and etc
+ pre_disable_mmu_workaround
msr sctlr_el2, x5
isb
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/7] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0()
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-26 12:03 ` [PATCH 1/7] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 12:03 ` [PATCH 3/7] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
` (4 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
To allow systems which do not require kpti to continue running with
global kernel mappings (which appears to be a requirement for Cavium
ThunderX due to a CPU erratum), make the use of nG in the kernel page
tables dependent on arm64_kernel_unmapped_at_el0(), which is resolved
at runtime.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/kernel-pgtable.h | 12 ++----------
arch/arm64/include/asm/pgtable-prot.h | 30 ++++++++++++++----------------
2 files changed, 16 insertions(+), 26 deletions(-)
diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h
index 82386e860dd2..a780f6714b44 100644
--- a/arch/arm64/include/asm/kernel-pgtable.h
+++ b/arch/arm64/include/asm/kernel-pgtable.h
@@ -123,16 +123,8 @@
/*
* Initial memory map attributes.
*/
-#define _SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
-#define _SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
-
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define SWAPPER_PTE_FLAGS (_SWAPPER_PTE_FLAGS | PTE_NG)
-#define SWAPPER_PMD_FLAGS (_SWAPPER_PMD_FLAGS | PMD_SECT_NG)
-#else
-#define SWAPPER_PTE_FLAGS _SWAPPER_PTE_FLAGS
-#define SWAPPER_PMD_FLAGS _SWAPPER_PMD_FLAGS
-#endif
+#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
+#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
#if ARM64_SWAPPER_USES_SECTION_MAPS
#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 22a926825e3f..2db84df5eb42 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -37,13 +37,11 @@
#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define PROT_DEFAULT (_PROT_DEFAULT | PTE_NG)
-#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_SECT_NG)
-#else
-#define PROT_DEFAULT _PROT_DEFAULT
-#define PROT_SECT_DEFAULT _PROT_SECT_DEFAULT
-#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
+#define PTE_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PTE_NG : 0)
+#define PMD_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PMD_SECT_NG : 0)
+
+#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG)
+#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG)
#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))
#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))
@@ -55,22 +53,22 @@
#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
-#define _PAGE_DEFAULT (PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
-#define _HYP_PAGE_DEFAULT (_PAGE_DEFAULT & ~PTE_NG)
+#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
+#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT
-#define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_RO __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_ROX __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT)
+#define PAGE_KERNEL __pgprot(PROT_NORMAL)
+#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)
+#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY)
+#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
+#define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)
#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
#define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
-#define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
-#define PAGE_S2_DEVICE __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
+#define PAGE_S2 __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
+#define PAGE_S2_DEVICE __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN)
#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/7] arm64: mm: Permit transitioning from Global to Non-Global without BBM
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-26 12:03 ` [PATCH 1/7] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
2018-01-26 12:03 ` [PATCH 2/7] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 12:03 ` [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
` (3 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
Break-before-make is not needed when transitioning from Global to
Non-Global mappings, provided that the contiguous hint is not being used.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/mm/mmu.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index b44992ec9643..fc7902bda02b 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -118,6 +118,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new)
if ((old | new) & PTE_CONT)
return false;
+ /* Transitioning from Global to Non-Global is safe */
+ if (((old ^ new) == PTE_NG) && (new & PTE_NG))
+ return true;
+
return ((old ^ new) & ~mask) == 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (2 preceding siblings ...)
2018-01-26 12:03 ` [PATCH 3/7] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 13:40 ` Marc Zyngier
2018-01-26 12:03 ` [PATCH 5/7] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
` (2 subsequent siblings)
6 siblings, 1 reply; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
Defaulting to global mappings for kernel space is generally good for
performance and appears to be necessary for Cavium ThunderX. If we
subsequently decide that we need to enable kpti, then we need to rewrite
our existing page table entries to be non-global. This is fiddly, and
made worse by the possible use of contiguous mappings, which require
a strict break-before-make sequence.
Since the enable callback runs on each online CPU from stop_machine
context, we can have all CPUs enter the idmap, where secondaries can
wait for the primary CPU to rewrite swapper with its MMU off. It's all
fairly horrible, but at least it only runs once.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/assembler.h | 10 ++
arch/arm64/kernel/cpufeature.c | 25 +++++
arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++--
3 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 3873dd7b5a32..23251eae6e8a 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -523,6 +523,16 @@ alternative_endif
#endif
.endm
+ .macro pte_to_phys, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+ ror \phys, \pte, #16
+ bfi \phys, \phys, #(16 + 12), #32
+ lsr \phys, \phys, #12
+#else
+ and \phys, \pte, #PTE_ADDR_MASK
+#endif
+ .endm
+
/**
* Errata workaround prior to disable MMU. Insert an ISB immediately prior
* to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 4b6f9051cf0c..cd9f46952db3 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -880,6 +880,30 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
ID_AA64PFR0_CSV3_SHIFT);
}
+static int kpti_install_ng_mappings(void *__unused)
+{
+ typedef void (kpti_remap_fn)(int, int, phys_addr_t);
+ extern kpti_remap_fn idmap_kpti_install_ng_mappings;
+ kpti_remap_fn *remap_fn;
+
+ static bool kpti_applied = false;
+ int cpu = smp_processor_id();
+
+ if (kpti_applied)
+ return 0;
+
+ remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);
+
+ cpu_install_idmap();
+ remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir));
+ cpu_uninstall_idmap();
+
+ if (!cpu)
+ kpti_applied = true;
+
+ return 0;
+}
+
static int __init parse_kpti(char *str)
{
bool enabled;
@@ -1003,6 +1027,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.capability = ARM64_UNMAP_KERNEL_AT_EL0,
.def_scope = SCOPE_SYSTEM,
.matches = unmap_kernel_at_el0,
+ .enable = kpti_install_ng_mappings,
},
#endif
{
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 9f177aac6390..cc7d2389edc8 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -166,6 +166,17 @@ ENTRY(cpu_do_switch_mm)
ENDPROC(cpu_do_switch_mm)
.pushsection ".idmap.text", "ax"
+
+.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
+ adrp \tmp1, empty_zero_page
+ phys_to_ttbr \tmp1, \tmp2
+ msr ttbr1_el1, \tmp2
+ isb
+ tlbi vmalle1
+ dsb nsh
+ isb
+.endm
+
/*
* void idmap_cpu_replace_ttbr1(phys_addr_t new_pgd)
*
@@ -175,14 +186,7 @@ ENDPROC(cpu_do_switch_mm)
ENTRY(idmap_cpu_replace_ttbr1)
save_and_disable_daif flags=x2
- adrp x1, empty_zero_page
- phys_to_ttbr x1, x3
- msr ttbr1_el1, x3
- isb
-
- tlbi vmalle1
- dsb nsh
- isb
+ __idmap_cpu_set_reserved_ttbr1 x1, x3
phys_to_ttbr x0, x3
msr ttbr1_el1, x3
@@ -194,6 +198,190 @@ ENTRY(idmap_cpu_replace_ttbr1)
ENDPROC(idmap_cpu_replace_ttbr1)
.popsection
+#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
+ .pushsection ".idmap.text", "ax"
+
+ .macro __idmap_kpti_get_pgtable_ent, type
+ dc cvac, cur_\()\type\()p // Ensure any existing dirty
+ dmb sy // lines are written back before
+ ldr \type, [cur_\()\type\()p] // loading the entry
+ tbz \type, #0, next_\()\type // Skip invalid entries
+ .endm
+
+ .macro __idmap_kpti_put_pgtable_ent_ng, type
+ orr \type, \type, #PTE_NG // Same bit for blocks and pages
+ str \type, [cur_\()\type\()p] // Update the entry and ensure it
+ dc civac, cur_\()\type\()p // is visible to all CPUs.
+ .endm
+
+/*
+ * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)
+ *
+ * Called exactly once from stop_machine context by each CPU found during boot.
+ */
+__idmap_kpti_flag:
+ .long 1
+ENTRY(idmap_kpti_install_ng_mappings)
+ cpu .req w0
+ num_cpus .req w1
+ swapper_pa .req x2
+ swapper_ttb .req x3
+ flag_ptr .req x4
+ cur_pgdp .req x5
+ end_pgdp .req x6
+ pgd .req x7
+ cur_pudp .req x8
+ end_pudp .req x9
+ pud .req x10
+ cur_pmdp .req x11
+ end_pmdp .req x12
+ pmd .req x13
+ cur_ptep .req x14
+ end_ptep .req x15
+ pte .req x16
+
+ mrs swapper_ttb, ttbr1_el1
+ adr flag_ptr, __idmap_kpti_flag
+
+ cbnz cpu, __idmap_kpti_secondary
+
+ /* We're the boot CPU. Wait for the others to catch up */
+ sevl
+1: wfe
+ ldaxr w18, [flag_ptr]
+ eor w18, w18, num_cpus
+ cbnz w18, 1b
+
+ /* We need to walk swapper, so turn off the MMU. */
+ pre_disable_mmu_workaround
+ mrs x18, sctlr_el1
+ bic x18, x18, #1
+ msr sctlr_el1, x18
+ isb
+
+ /* Everybody is enjoying the idmap, so we can rewrite swapper. */
+ /* PGD */
+ mov cur_pgdp, swapper_pa
+ add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8)
+do_pgd: __idmap_kpti_get_pgtable_ent pgd
+ tbnz pgd, #1, walk_puds
+ __idmap_kpti_put_pgtable_ent_ng pgd
+next_pgd:
+ add cur_pgdp, cur_pgdp, #8
+ cmp cur_pgdp, end_pgdp
+ b.ne do_pgd
+
+ /* Publish the updated tables and nuke all the TLBs */
+ dsb sy
+ tlbi vmalle1is
+ dsb ish
+ isb
+
+ /* We're done: fire up the MMU again */
+ mrs x18, sctlr_el1
+ orr x18, x18, #1
+ msr sctlr_el1, x18
+ isb
+
+ /* Set the flag to zero to indicate that we're all done */
+ str wzr, [flag_ptr]
+ ret
+
+ /* PUD */
+walk_puds:
+ .if CONFIG_PGTABLE_LEVELS > 3
+ pte_to_phys cur_pudp, pgd
+ add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8)
+do_pud: __idmap_kpti_get_pgtable_ent pud
+ tbnz pud, #1, walk_pmds
+ __idmap_kpti_put_pgtable_ent_ng pud
+next_pud:
+ add cur_pudp, cur_pudp, 8
+ cmp cur_pudp, end_pudp
+ b.ne do_pud
+ b next_pgd
+ .else /* CONFIG_PGTABLE_LEVELS <= 3 */
+ mov pud, pgd
+ b walk_pmds
+next_pud:
+ b next_pgd
+ .endif
+
+ /* PMD */
+walk_pmds:
+ .if CONFIG_PGTABLE_LEVELS > 2
+ pte_to_phys cur_pmdp, pud
+ add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8)
+do_pmd: __idmap_kpti_get_pgtable_ent pmd
+ tbnz pmd, #1, walk_ptes
+ __idmap_kpti_put_pgtable_ent_ng pmd
+next_pmd:
+ add cur_pmdp, cur_pmdp, #8
+ cmp cur_pmdp, end_pmdp
+ b.ne do_pmd
+ b next_pud
+ .else /* CONFIG_PGTABLE_LEVELS <= 2 */
+ mov pmd, pud
+ b walk_ptes
+next_pmd:
+ b next_pud
+ .endif
+
+ /* PTE */
+walk_ptes:
+ pte_to_phys cur_ptep, pmd
+ add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8)
+do_pte: __idmap_kpti_get_pgtable_ent pte
+ __idmap_kpti_put_pgtable_ent_ng pte
+next_pte:
+ add cur_ptep, cur_ptep, #8
+ cmp cur_ptep, end_ptep
+ b.ne do_pte
+ b next_pmd
+
+ /* Secondary CPUs end up here */
+__idmap_kpti_secondary:
+ /* Uninstall swapper before surgery begins */
+ __idmap_cpu_set_reserved_ttbr1 x18, x17
+
+ /* Increment the flag to let the boot CPU we're ready */
+1: ldxr w18, [flag_ptr]
+ add w18, w18, #1
+ stxr w17, w18, [flag_ptr]
+ cbnz w17, 1b
+
+ /* Wait for the boot CPU to finish messing around with swapper */
+ sevl
+1: wfe
+ ldxr w18, [flag_ptr]
+ cbnz w18, 1b
+
+ /* All done, act like nothing happened */
+ msr ttbr1_el1, swapper_ttb
+ isb
+ ret
+
+ .unreq cpu
+ .unreq num_cpus
+ .unreq swapper_pa
+ .unreq swapper_ttb
+ .unreq flag_ptr
+ .unreq cur_pgdp
+ .unreq end_pgdp
+ .unreq pgd
+ .unreq cur_pudp
+ .unreq end_pudp
+ .unreq pud
+ .unreq cur_pmdp
+ .unreq end_pmdp
+ .unreq pmd
+ .unreq cur_ptep
+ .unreq end_ptep
+ .unreq pte
+ENDPROC(idmap_kpti_install_ng_mappings)
+ .popsection
+#endif
+
/*
* __cpu_setup
*
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 5/7] arm64: Force KPTI to be disabled on Cavium ThunderX
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (3 preceding siblings ...)
2018-01-26 12:03 ` [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 12:03 ` [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
2018-01-26 12:03 ` [PATCH 7/7] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
6 siblings, 0 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
From: Marc Zyngier <marc.zyngier@arm.com>
Cavium ThunderX's erratum 27456 results in a corruption of icache
entries that are loaded from memory that is mapped as non-global
(i.e. ASID-tagged).
As KPTI is based on memory being mapped non-global, let's prevent
it from kicking in if this erratum is detected.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
[will: Update comment]
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/kernel/cpufeature.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index cd9f46952db3..0ff8ab79352c 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -855,12 +855,23 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */
static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
int __unused)
{
+ char const *str = "command line option";
u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
- /* Forced on command line? */
+ /*
+ * For reasons that aren't entirely clear, enabling KPTI on Cavium
+ * ThunderX leads to apparent I-cache corruption of kernel text, which
+ * ends as well as you might imagine. Don't even try.
+ */
+ if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) {
+ str = "ARM64_WORKAROUND_CAVIUM_27456";
+ __kpti_forced = -1;
+ }
+
+ /* Forced? */
if (__kpti_forced) {
- pr_info_once("kernel page table isolation forced %s by command line option\n",
- __kpti_forced > 0 ? "ON" : "OFF");
+ pr_info_once("kernel page table isolation forced %s by %s\n",
+ __kpti_forced > 0 ? "ON" : "OFF", str);
return __kpti_forced > 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (4 preceding siblings ...)
2018-01-26 12:03 ` [PATCH 5/7] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
2018-01-26 14:17 ` Robin Murphy
2018-01-26 12:03 ` [PATCH 7/7] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
6 siblings, 1 reply; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
Since AArch64 assembly instructions take the destination register as
their first operand, do the same thing for the phys_to_ttbr macro.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/assembler.h | 2 +-
arch/arm64/kernel/head.S | 4 ++--
arch/arm64/kernel/hibernate-asm.S | 4 ++--
arch/arm64/kvm/hyp-init.S | 2 +-
arch/arm64/mm/proc.S | 6 +++---
5 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 23251eae6e8a..e4495ef96058 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -514,7 +514,7 @@ alternative_endif
* phys: physical address, preserved
* ttbr: returns the TTBR value
*/
- .macro phys_to_ttbr, phys, ttbr
+ .macro phys_to_ttbr, ttbr, phys
#ifdef CONFIG_ARM64_PA_BITS_52
orr \ttbr, \phys, \phys, lsr #46
and \ttbr, \ttbr, #TTBR_BADDR_MASK_52
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index ba3ab04788dc..341649c08337 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -776,8 +776,8 @@ ENTRY(__enable_mmu)
update_early_cpu_boot_status 0, x1, x2
adrp x1, idmap_pg_dir
adrp x2, swapper_pg_dir
- phys_to_ttbr x1, x3
- phys_to_ttbr x2, x4
+ phys_to_ttbr x3, x1
+ phys_to_ttbr x4, x2
msr ttbr0_el1, x3 // load TTBR0
msr ttbr1_el1, x4 // load TTBR1
isb
diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
index 84f5d52fddda..dd14ab8c9f72 100644
--- a/arch/arm64/kernel/hibernate-asm.S
+++ b/arch/arm64/kernel/hibernate-asm.S
@@ -34,12 +34,12 @@
* each stage of the walk.
*/
.macro break_before_make_ttbr_switch zero_page, page_table, tmp
- phys_to_ttbr \zero_page, \tmp
+ phys_to_ttbr \tmp, \zero_page
msr ttbr1_el1, \tmp
isb
tlbi vmalle1
dsb nsh
- phys_to_ttbr \page_table, \tmp
+ phys_to_ttbr \tmp, \page_table
msr ttbr1_el1, \tmp
isb
.endm
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index e086c6eff8c6..5aa9ccf6db99 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -63,7 +63,7 @@ __do_hyp_init:
cmp x0, #HVC_STUB_HCALL_NR
b.lo __kvm_handle_stub_hvc
- phys_to_ttbr x0, x4
+ phys_to_ttbr x4, x0
msr ttbr0_el2, x4
mrs x4, tcr_el1
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index cc7d2389edc8..42204b2525ea 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -153,7 +153,7 @@ ENDPROC(cpu_do_resume)
ENTRY(cpu_do_switch_mm)
mrs x2, ttbr1_el1
mmid x1, x1 // get mm->context.id
- phys_to_ttbr x0, x3
+ phys_to_ttbr x3, x0
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
bfi x3, x1, #48, #16 // set the ASID field in TTBR0
#endif
@@ -169,7 +169,7 @@ ENDPROC(cpu_do_switch_mm)
.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
adrp \tmp1, empty_zero_page
- phys_to_ttbr \tmp1, \tmp2
+ phys_to_ttbr \tmp2, \tmp1
msr ttbr1_el1, \tmp2
isb
tlbi vmalle1
@@ -188,7 +188,7 @@ ENTRY(idmap_cpu_replace_ttbr1)
__idmap_cpu_set_reserved_ttbr1 x1, x3
- phys_to_ttbr x0, x3
+ phys_to_ttbr x3, x0
msr ttbr1_el1, x3
isb
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 7/7] arm64: entry: Reword comment about post_ttbr_update_workaround
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (5 preceding siblings ...)
2018-01-26 12:03 ` [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
@ 2018-01-26 12:03 ` Will Deacon
6 siblings, 0 replies; 12+ messages in thread
From: Will Deacon @ 2018-01-26 12:03 UTC (permalink / raw)
To: linux-arm-kernel
We don't fully understand the Cavium ThunderX erratum, but it appears
that mapping the kernel as nG can lead to horrible consequences such as
attempting to execute userspace from kernel context. Since kpti isn't
enabled for these CPUs anyway, simplify the comment justifying the lack
of post_ttbr_update_workaround in the exception trampoline.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/kernel/entry.S | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index b34e717d7597..fbe1444324b3 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif
orr \tmp, \tmp, #USER_ASID_FLAG
msr ttbr1_el1, \tmp
/*
- * We avoid running the post_ttbr_update_workaround here because the
- * user and kernel ASIDs don't have conflicting mappings, so any
- * "blessing" as described in:
- *
- * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
- *
- * will not hurt correctness. Whilst this may partially defeat the
- * point of using split ASIDs in the first place, it avoids
- * the hit of invalidating the entire I-cache on every return to
- * userspace.
+ * We avoid running the post_ttbr_update_workaround here because
+ * it's only needed by Cavium ThunderX, which doesn't require kpti.
*/
.endm
--
2.1.4
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-26 12:03 ` [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
@ 2018-01-26 13:40 ` Marc Zyngier
2018-01-26 14:05 ` Will Deacon
0 siblings, 1 reply; 12+ messages in thread
From: Marc Zyngier @ 2018-01-26 13:40 UTC (permalink / raw)
To: linux-arm-kernel
Hey Will,
On 26/01/18 12:03, Will Deacon wrote:
> Defaulting to global mappings for kernel space is generally good for
> performance and appears to be necessary for Cavium ThunderX. If we
> subsequently decide that we need to enable kpti, then we need to rewrite
> our existing page table entries to be non-global. This is fiddly, and
> made worse by the possible use of contiguous mappings, which require
> a strict break-before-make sequence.
>
> Since the enable callback runs on each online CPU from stop_machine
> context, we can have all CPUs enter the idmap, where secondaries can
> wait for the primary CPU to rewrite swapper with its MMU off. It's all
> fairly horrible, but at least it only runs once.
>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
> arch/arm64/include/asm/assembler.h | 10 ++
> arch/arm64/kernel/cpufeature.c | 25 +++++
> arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++--
> 3 files changed, 231 insertions(+), 8 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 3873dd7b5a32..23251eae6e8a 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -523,6 +523,16 @@ alternative_endif
> #endif
> .endm
>
> + .macro pte_to_phys, phys, pte
> +#ifdef CONFIG_ARM64_PA_BITS_52
> + ror \phys, \pte, #16
> + bfi \phys, \phys, #(16 + 12), #32
> + lsr \phys, \phys, #12
> +#else
> + and \phys, \pte, #PTE_ADDR_MASK
> +#endif
> + .endm
> +
> /**
> * Errata workaround prior to disable MMU. Insert an ISB immediately prior
> * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 4b6f9051cf0c..cd9f46952db3 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -880,6 +880,30 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
> ID_AA64PFR0_CSV3_SHIFT);
> }
>
> +static int kpti_install_ng_mappings(void *__unused)
> +{
> + typedef void (kpti_remap_fn)(int, int, phys_addr_t);
> + extern kpti_remap_fn idmap_kpti_install_ng_mappings;
> + kpti_remap_fn *remap_fn;
> +
> + static bool kpti_applied = false;
> + int cpu = smp_processor_id();
> +
> + if (kpti_applied)
> + return 0;
> +
> + remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);
> +
> + cpu_install_idmap();
> + remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir));
> + cpu_uninstall_idmap();
> +
> + if (!cpu)
> + kpti_applied = true;
> +
> + return 0;
> +}
> +
> static int __init parse_kpti(char *str)
> {
> bool enabled;
> @@ -1003,6 +1027,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
> .capability = ARM64_UNMAP_KERNEL_AT_EL0,
> .def_scope = SCOPE_SYSTEM,
> .matches = unmap_kernel_at_el0,
> + .enable = kpti_install_ng_mappings,
> },
> #endif
> {
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 9f177aac6390..cc7d2389edc8 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -166,6 +166,17 @@ ENTRY(cpu_do_switch_mm)
> ENDPROC(cpu_do_switch_mm)
>
> .pushsection ".idmap.text", "ax"
> +
> +.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
> + adrp \tmp1, empty_zero_page
> + phys_to_ttbr \tmp1, \tmp2
> + msr ttbr1_el1, \tmp2
> + isb
> + tlbi vmalle1
> + dsb nsh
> + isb
> +.endm
> +
> /*
> * void idmap_cpu_replace_ttbr1(phys_addr_t new_pgd)
> *
> @@ -175,14 +186,7 @@ ENDPROC(cpu_do_switch_mm)
> ENTRY(idmap_cpu_replace_ttbr1)
> save_and_disable_daif flags=x2
>
> - adrp x1, empty_zero_page
> - phys_to_ttbr x1, x3
> - msr ttbr1_el1, x3
> - isb
> -
> - tlbi vmalle1
> - dsb nsh
> - isb
> + __idmap_cpu_set_reserved_ttbr1 x1, x3
>
> phys_to_ttbr x0, x3
> msr ttbr1_el1, x3
> @@ -194,6 +198,190 @@ ENTRY(idmap_cpu_replace_ttbr1)
> ENDPROC(idmap_cpu_replace_ttbr1)
> .popsection
>
> +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
> + .pushsection ".idmap.text", "ax"
> +
> + .macro __idmap_kpti_get_pgtable_ent, type
> + dc cvac, cur_\()\type\()p // Ensure any existing dirty
> + dmb sy // lines are written back before
> + ldr \type, [cur_\()\type\()p] // loading the entry
> + tbz \type, #0, next_\()\type // Skip invalid entries
> + .endm
> +
> + .macro __idmap_kpti_put_pgtable_ent_ng, type
> + orr \type, \type, #PTE_NG // Same bit for blocks and pages
> + str \type, [cur_\()\type\()p] // Update the entry and ensure it
> + dc civac, cur_\()\type\()p // is visible to all CPUs.
> + .endm
> +
> +/*
> + * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)
> + *
> + * Called exactly once from stop_machine context by each CPU found during boot.
> + */
> +__idmap_kpti_flag:
> + .long 1
I'm a bit concerned that we're placing the counter in a section that is
not writeable (at least in theory). We can either have the idmap
writeable entirely, or have a idmap data section (which means extending
it to be more than a single page).
Not sure if that's a concern, but I thought I'd raise it. The core works
pretty well as is, so it cannot be completely wrong... ;-)
> +ENTRY(idmap_kpti_install_ng_mappings)
> + cpu .req w0
> + num_cpus .req w1
> + swapper_pa .req x2
> + swapper_ttb .req x3
> + flag_ptr .req x4
> + cur_pgdp .req x5
> + end_pgdp .req x6
> + pgd .req x7
> + cur_pudp .req x8
> + end_pudp .req x9
> + pud .req x10
> + cur_pmdp .req x11
> + end_pmdp .req x12
> + pmd .req x13
> + cur_ptep .req x14
> + end_ptep .req x15
> + pte .req x16
> +
> + mrs swapper_ttb, ttbr1_el1
> + adr flag_ptr, __idmap_kpti_flag
> +
> + cbnz cpu, __idmap_kpti_secondary
> +
> + /* We're the boot CPU. Wait for the others to catch up */
> + sevl
> +1: wfe
> + ldaxr w18, [flag_ptr]
> + eor w18, w18, num_cpus
> + cbnz w18, 1b
> +
> + /* We need to walk swapper, so turn off the MMU. */
> + pre_disable_mmu_workaround
> + mrs x18, sctlr_el1
> + bic x18, x18, #1
nit: You should be able to use SCTLR_ELx_M, as we already include
sysreg.h (indirectly).
> + msr sctlr_el1, x18
> + isb
> +
> + /* Everybody is enjoying the idmap, so we can rewrite swapper. */
> + /* PGD */
> + mov cur_pgdp, swapper_pa
> + add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8)
> +do_pgd: __idmap_kpti_get_pgtable_ent pgd
> + tbnz pgd, #1, walk_puds
> + __idmap_kpti_put_pgtable_ent_ng pgd
> +next_pgd:
> + add cur_pgdp, cur_pgdp, #8
> + cmp cur_pgdp, end_pgdp
> + b.ne do_pgd
> +
> + /* Publish the updated tables and nuke all the TLBs */
> + dsb sy
> + tlbi vmalle1is
> + dsb ish
> + isb
> +
> + /* We're done: fire up the MMU again */
> + mrs x18, sctlr_el1
> + orr x18, x18, #1
Same here.
> + msr sctlr_el1, x18
> + isb
> +
> + /* Set the flag to zero to indicate that we're all done */
> + str wzr, [flag_ptr]
> + ret
> +
> + /* PUD */
> +walk_puds:
> + .if CONFIG_PGTABLE_LEVELS > 3
> + pte_to_phys cur_pudp, pgd
> + add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8)
> +do_pud: __idmap_kpti_get_pgtable_ent pud
> + tbnz pud, #1, walk_pmds
> + __idmap_kpti_put_pgtable_ent_ng pud
> +next_pud:
> + add cur_pudp, cur_pudp, 8
> + cmp cur_pudp, end_pudp
> + b.ne do_pud
> + b next_pgd
> + .else /* CONFIG_PGTABLE_LEVELS <= 3 */
> + mov pud, pgd
> + b walk_pmds
> +next_pud:
> + b next_pgd
> + .endif
> +
> + /* PMD */
> +walk_pmds:
> + .if CONFIG_PGTABLE_LEVELS > 2
> + pte_to_phys cur_pmdp, pud
> + add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8)
> +do_pmd: __idmap_kpti_get_pgtable_ent pmd
> + tbnz pmd, #1, walk_ptes
> + __idmap_kpti_put_pgtable_ent_ng pmd
> +next_pmd:
> + add cur_pmdp, cur_pmdp, #8
> + cmp cur_pmdp, end_pmdp
> + b.ne do_pmd
> + b next_pud
> + .else /* CONFIG_PGTABLE_LEVELS <= 2 */
> + mov pmd, pud
> + b walk_ptes
> +next_pmd:
> + b next_pud
> + .endif
> +
> + /* PTE */
> +walk_ptes:
> + pte_to_phys cur_ptep, pmd
> + add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8)
> +do_pte: __idmap_kpti_get_pgtable_ent pte
> + __idmap_kpti_put_pgtable_ent_ng pte
> +next_pte:
> + add cur_ptep, cur_ptep, #8
> + cmp cur_ptep, end_ptep
> + b.ne do_pte
> + b next_pmd
> +
> + /* Secondary CPUs end up here */
> +__idmap_kpti_secondary:
> + /* Uninstall swapper before surgery begins */
> + __idmap_cpu_set_reserved_ttbr1 x18, x17
> +
> + /* Increment the flag to let the boot CPU we're ready */
> +1: ldxr w18, [flag_ptr]
> + add w18, w18, #1
> + stxr w17, w18, [flag_ptr]
> + cbnz w17, 1b
> +
> + /* Wait for the boot CPU to finish messing around with swapper */
> + sevl
> +1: wfe
> + ldxr w18, [flag_ptr]
> + cbnz w18, 1b
> +
> + /* All done, act like nothing happened */
> + msr ttbr1_el1, swapper_ttb
> + isb
> + ret
> +
> + .unreq cpu
> + .unreq num_cpus
> + .unreq swapper_pa
> + .unreq swapper_ttb
> + .unreq flag_ptr
> + .unreq cur_pgdp
> + .unreq end_pgdp
> + .unreq pgd
> + .unreq cur_pudp
> + .unreq end_pudp
> + .unreq pud
> + .unreq cur_pmdp
> + .unreq end_pmdp
> + .unreq pmd
> + .unreq cur_ptep
> + .unreq end_ptep
> + .unreq pte
> +ENDPROC(idmap_kpti_install_ng_mappings)
> + .popsection
> +#endif
> +
> /*
> * __cpu_setup
> *
>
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-26 13:40 ` Marc Zyngier
@ 2018-01-26 14:05 ` Will Deacon
2018-01-26 14:23 ` Marc Zyngier
0 siblings, 1 reply; 12+ messages in thread
From: Will Deacon @ 2018-01-26 14:05 UTC (permalink / raw)
To: linux-arm-kernel
Hi Marc,
On Fri, Jan 26, 2018 at 01:40:50PM +0000, Marc Zyngier wrote:
> On 26/01/18 12:03, Will Deacon wrote:
> > +/*
> > + * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)
> > + *
> > + * Called exactly once from stop_machine context by each CPU found during boot.
> > + */
> > +__idmap_kpti_flag:
> > + .long 1
>
> I'm a bit concerned that we're placing the counter in a section that is
> not writeable (at least in theory). We can either have the idmap
> writeable entirely, or have a idmap data section (which means extending
> it to be more than a single page).
The idmap is mapped using the same attributes as the initial swapper
mappings (SWAPPER_MM_MMUFLAGS) but, unlike swapper, it is not subsequently
remapped and therefore remains writeable.
I could update the .pushsection flags to reflect this with "awx"?
Also, whilst looking around in head.S, I noticed phys_to_pte. I'll move
that into assembler.h and switch the argument order as a separate patch.
> Not sure if that's a concern, but I thought I'd raise it. The core works
> pretty well as is, so it cannot be completely wrong... ;-)
Is that a Tested-by? ;)
> > +ENTRY(idmap_kpti_install_ng_mappings)
> > + cpu .req w0
> > + num_cpus .req w1
> > + swapper_pa .req x2
> > + swapper_ttb .req x3
> > + flag_ptr .req x4
> > + cur_pgdp .req x5
> > + end_pgdp .req x6
> > + pgd .req x7
> > + cur_pudp .req x8
> > + end_pudp .req x9
> > + pud .req x10
> > + cur_pmdp .req x11
> > + end_pmdp .req x12
> > + pmd .req x13
> > + cur_ptep .req x14
> > + end_ptep .req x15
> > + pte .req x16
> > +
> > + mrs swapper_ttb, ttbr1_el1
> > + adr flag_ptr, __idmap_kpti_flag
> > +
> > + cbnz cpu, __idmap_kpti_secondary
> > +
> > + /* We're the boot CPU. Wait for the others to catch up */
> > + sevl
> > +1: wfe
> > + ldaxr w18, [flag_ptr]
> > + eor w18, w18, num_cpus
> > + cbnz w18, 1b
> > +
> > + /* We need to walk swapper, so turn off the MMU. */
> > + pre_disable_mmu_workaround
> > + mrs x18, sctlr_el1
> > + bic x18, x18, #1
>
> nit: You should be able to use SCTLR_ELx_M, as we already include
> sysreg.h (indirectly).
Aha, I looked for that and failed to find it. Thanks.
Will
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr
2018-01-26 12:03 ` [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
@ 2018-01-26 14:17 ` Robin Murphy
0 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2018-01-26 14:17 UTC (permalink / raw)
To: linux-arm-kernel
On 26/01/18 12:03, Will Deacon wrote:
> Since AArch64 assembly instructions take the destination register as
> their first operand, do the same thing for the phys_to_ttbr macro.
Anything which makes the macro soup of our assembly files slightly more
intuitive is good by me!
Acked-by: Robin Murphy <robin.murphy@arm.com>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
> arch/arm64/include/asm/assembler.h | 2 +-
> arch/arm64/kernel/head.S | 4 ++--
> arch/arm64/kernel/hibernate-asm.S | 4 ++--
> arch/arm64/kvm/hyp-init.S | 2 +-
> arch/arm64/mm/proc.S | 6 +++---
> 5 files changed, 9 insertions(+), 9 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 23251eae6e8a..e4495ef96058 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -514,7 +514,7 @@ alternative_endif
> * phys: physical address, preserved
> * ttbr: returns the TTBR value
> */
> - .macro phys_to_ttbr, phys, ttbr
> + .macro phys_to_ttbr, ttbr, phys
> #ifdef CONFIG_ARM64_PA_BITS_52
> orr \ttbr, \phys, \phys, lsr #46
> and \ttbr, \ttbr, #TTBR_BADDR_MASK_52
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index ba3ab04788dc..341649c08337 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -776,8 +776,8 @@ ENTRY(__enable_mmu)
> update_early_cpu_boot_status 0, x1, x2
> adrp x1, idmap_pg_dir
> adrp x2, swapper_pg_dir
> - phys_to_ttbr x1, x3
> - phys_to_ttbr x2, x4
> + phys_to_ttbr x3, x1
> + phys_to_ttbr x4, x2
> msr ttbr0_el1, x3 // load TTBR0
> msr ttbr1_el1, x4 // load TTBR1
> isb
> diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
> index 84f5d52fddda..dd14ab8c9f72 100644
> --- a/arch/arm64/kernel/hibernate-asm.S
> +++ b/arch/arm64/kernel/hibernate-asm.S
> @@ -34,12 +34,12 @@
> * each stage of the walk.
> */
> .macro break_before_make_ttbr_switch zero_page, page_table, tmp
> - phys_to_ttbr \zero_page, \tmp
> + phys_to_ttbr \tmp, \zero_page
> msr ttbr1_el1, \tmp
> isb
> tlbi vmalle1
> dsb nsh
> - phys_to_ttbr \page_table, \tmp
> + phys_to_ttbr \tmp, \page_table
> msr ttbr1_el1, \tmp
> isb
> .endm
> diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
> index e086c6eff8c6..5aa9ccf6db99 100644
> --- a/arch/arm64/kvm/hyp-init.S
> +++ b/arch/arm64/kvm/hyp-init.S
> @@ -63,7 +63,7 @@ __do_hyp_init:
> cmp x0, #HVC_STUB_HCALL_NR
> b.lo __kvm_handle_stub_hvc
>
> - phys_to_ttbr x0, x4
> + phys_to_ttbr x4, x0
> msr ttbr0_el2, x4
>
> mrs x4, tcr_el1
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index cc7d2389edc8..42204b2525ea 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -153,7 +153,7 @@ ENDPROC(cpu_do_resume)
> ENTRY(cpu_do_switch_mm)
> mrs x2, ttbr1_el1
> mmid x1, x1 // get mm->context.id
> - phys_to_ttbr x0, x3
> + phys_to_ttbr x3, x0
> #ifdef CONFIG_ARM64_SW_TTBR0_PAN
> bfi x3, x1, #48, #16 // set the ASID field in TTBR0
> #endif
> @@ -169,7 +169,7 @@ ENDPROC(cpu_do_switch_mm)
>
> .macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
> adrp \tmp1, empty_zero_page
> - phys_to_ttbr \tmp1, \tmp2
> + phys_to_ttbr \tmp2, \tmp1
> msr ttbr1_el1, \tmp2
> isb
> tlbi vmalle1
> @@ -188,7 +188,7 @@ ENTRY(idmap_cpu_replace_ttbr1)
>
> __idmap_cpu_set_reserved_ttbr1 x1, x3
>
> - phys_to_ttbr x0, x3
> + phys_to_ttbr x3, x0
> msr ttbr1_el1, x3
> isb
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-26 14:05 ` Will Deacon
@ 2018-01-26 14:23 ` Marc Zyngier
0 siblings, 0 replies; 12+ messages in thread
From: Marc Zyngier @ 2018-01-26 14:23 UTC (permalink / raw)
To: linux-arm-kernel
On 26/01/18 14:05, Will Deacon wrote:
> Hi Marc,
>
> On Fri, Jan 26, 2018 at 01:40:50PM +0000, Marc Zyngier wrote:
>> On 26/01/18 12:03, Will Deacon wrote:
>>> +/*
>>> + * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)
>>> + *
>>> + * Called exactly once from stop_machine context by each CPU found during boot.
>>> + */
>>> +__idmap_kpti_flag:
>>> + .long 1
>>
>> I'm a bit concerned that we're placing the counter in a section that is
>> not writeable (at least in theory). We can either have the idmap
>> writeable entirely, or have a idmap data section (which means extending
>> it to be more than a single page).
>
> The idmap is mapped using the same attributes as the initial swapper
> mappings (SWAPPER_MM_MMUFLAGS) but, unlike swapper, it is not subsequently
> remapped and therefore remains writeable.
>
> I could update the .pushsection flags to reflect this with "awx"?
I think it would at least reflect both the expectations of this code and
what the mapping code does.
>
> Also, whilst looking around in head.S, I noticed phys_to_pte. I'll move
> that into assembler.h and switch the argument order as a separate patch.
>
>> Not sure if that's a concern, but I thought I'd raise it. The core works
>> pretty well as is, so it cannot be completely wrong... ;-)
>
> Is that a Tested-by? ;)
Since I'm lucky enough to have access to a ThunderX, here's one ;-)
Tested-by: Marc Zyngier <marc.zyngier@arm.com>
Also:
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2018-01-26 14:23 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-26 12:03 [PATCH 0/7] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-26 12:03 ` [PATCH 1/7] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
2018-01-26 12:03 ` [PATCH 2/7] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
2018-01-26 12:03 ` [PATCH 3/7] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
2018-01-26 12:03 ` [PATCH 4/7] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
2018-01-26 13:40 ` Marc Zyngier
2018-01-26 14:05 ` Will Deacon
2018-01-26 14:23 ` Marc Zyngier
2018-01-26 12:03 ` [PATCH 5/7] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
2018-01-26 12:03 ` [PATCH 6/7] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
2018-01-26 14:17 ` Robin Murphy
2018-01-26 12:03 ` [PATCH 7/7] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.