* [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX
@ 2018-01-29 11:59 Will Deacon
2018-01-29 11:59 ` [PATCH v2 1/9] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
` (10 more replies)
0 siblings, 11 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
Hi all,
This is version two of the patches I posted on Friday:
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags
Cheers,
Will
--->8
Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderX
Shanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041
Will Deacon (7):
arm64: kpti: Make use of nG dependent on
arm64_kernel_unmapped_at_el0()
arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments in phys_to_ttbr
arm64: entry: Reword comment about post_ttbr_update_workaround
arm64: assembler: Align phys_to_pte with pte_to_phys
arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives
Documentation/arm64/silicon-errata.txt | 1 +
arch/arm64/Kconfig | 12 +-
arch/arm64/include/asm/assembler.h | 36 +++++-
arch/arm64/include/asm/kernel-pgtable.h | 12 +-
arch/arm64/include/asm/pgtable-prot.h | 30 +++--
arch/arm64/kernel/cpu-reset.S | 3 +-
arch/arm64/kernel/cpufeature.c | 42 ++++++-
arch/arm64/kernel/efi-entry.S | 2 +
arch/arm64/kernel/entry.S | 12 +-
arch/arm64/kernel/head.S | 31 +----
arch/arm64/kernel/hibernate-asm.S | 4 +-
arch/arm64/kernel/relocate_kernel.S | 1 +
arch/arm64/kernel/sleep.S | 2 +-
arch/arm64/kvm/hyp-init.S | 3 +-
arch/arm64/mm/mmu.c | 4 +
arch/arm64/mm/proc.S | 212 ++++++++++++++++++++++++++++++--
16 files changed, 324 insertions(+), 83 deletions(-)
--
2.1.4
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 1/9] arm64: Add software workaround for Falkor erratum 1041
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 2/9] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
` (9 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
From: Shanker Donthineni <shankerd@codeaurora.org>
The ARM architecture defines the memory locations that are permitted
to be accessed as the result of a speculative instruction fetch from
an exception level for which all stages of translation are disabled.
Specifically, the core is permitted to speculatively fetch from the
4KB region containing the current program counter 4K and next 4K.
When translation is changed from enabled to disabled for the running
exception level (SCTLR_ELn[M] changed from a value of 1 to 0), the
Falkor core may errantly speculatively access memory locations outside
of the 4KB region permitted by the architecture. The errant memory
access may lead to one of the following unexpected behaviors.
1) A System Error Interrupt (SEI) being raised by the Falkor core due
to the errant memory access attempting to access a region of memory
that is protected by a slave-side memory protection unit.
2) Unpredictable device behavior due to a speculative read from device
memory. This behavior may only occur if the instruction cache is
disabled prior to or coincident with translation being changed from
enabled to disabled.
The conditions leading to this erratum will not occur when either of the
following occur:
1) A higher exception level disables translation of a lower exception level
(e.g. EL2 changing SCTLR_EL1[M] from a value of 1 to 0).
2) An exception level disabling its stage-1 translation if its stage-2
translation is enabled (e.g. EL1 changing SCTLR_EL1[M] from a value of 1
to 0 when HCR_EL2[VM] has a value of 1).
To avoid the errant behavior, software must execute an ISB immediately
prior to executing the MSR that will change SCTLR_ELn[M] from 1 to 0.
Signed-off-by: Shanker Donthineni <shankerd@codeaurora.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
Documentation/arm64/silicon-errata.txt | 1 +
arch/arm64/Kconfig | 12 +++++++++++-
arch/arm64/include/asm/assembler.h | 10 ++++++++++
arch/arm64/kernel/cpu-reset.S | 1 +
arch/arm64/kernel/efi-entry.S | 2 ++
arch/arm64/kernel/head.S | 1 +
arch/arm64/kernel/relocate_kernel.S | 1 +
arch/arm64/kvm/hyp-init.S | 1 +
8 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/Documentation/arm64/silicon-errata.txt b/Documentation/arm64/silicon-errata.txt
index b9d93e981a05..c1d520de6dfe 100644
--- a/Documentation/arm64/silicon-errata.txt
+++ b/Documentation/arm64/silicon-errata.txt
@@ -75,3 +75,4 @@ stable kernels.
| Qualcomm Tech. | Kryo/Falkor v1 | E1003 | QCOM_FALKOR_ERRATUM_1003 |
| Qualcomm Tech. | Falkor v1 | E1009 | QCOM_FALKOR_ERRATUM_1009 |
| Qualcomm Tech. | QDF2400 ITS | E0065 | QCOM_QDF2400_ERRATUM_0065 |
+| Qualcomm Tech. | Falkor v{1,2} | E1041 | QCOM_FALKOR_ERRATUM_1041 |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1d51c8edf34b..b488076d63c2 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -550,7 +550,6 @@ config QCOM_QDF2400_ERRATUM_0065
If unsure, say Y.
-
config SOCIONEXT_SYNQUACER_PREITS
bool "Socionext Synquacer: Workaround for GICv3 pre-ITS"
default y
@@ -569,6 +568,17 @@ config HISILICON_ERRATUM_161600802
a 128kB offset to be applied to the target address in this commands.
If unsure, say Y.
+
+config QCOM_FALKOR_ERRATUM_E1041
+ bool "Falkor E1041: Speculative instruction fetches might cause errant memory access"
+ default y
+ help
+ Falkor CPU may speculatively fetch instructions from an improper
+ memory location when MMU translation is changed from SCTLR_ELn[M]=1
+ to SCTLR_ELn[M]=0. Prefix an ISB instruction to fix the problem.
+
+ If unsure, say Y.
+
endmenu
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 794fe8122602..3873dd7b5a32 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -523,4 +523,14 @@ alternative_endif
#endif
.endm
+/**
+ * Errata workaround prior to disable MMU. Insert an ISB immediately prior
+ * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
+ */
+ .macro pre_disable_mmu_workaround
+#ifdef CONFIG_QCOM_FALKOR_ERRATUM_E1041
+ isb
+#endif
+ .endm
+
#endif /* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 65f42d257414..2a752cb2a0f3 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -37,6 +37,7 @@ ENTRY(__cpu_soft_restart)
mrs x12, sctlr_el1
ldr x13, =SCTLR_ELx_FLAGS
bic x12, x12, x13
+ pre_disable_mmu_workaround
msr sctlr_el1, x12
isb
diff --git a/arch/arm64/kernel/efi-entry.S b/arch/arm64/kernel/efi-entry.S
index 4e6ad355bd05..6b9736c3fb56 100644
--- a/arch/arm64/kernel/efi-entry.S
+++ b/arch/arm64/kernel/efi-entry.S
@@ -96,6 +96,7 @@ ENTRY(entry)
mrs x0, sctlr_el2
bic x0, x0, #1 << 0 // clear SCTLR.M
bic x0, x0, #1 << 2 // clear SCTLR.C
+ pre_disable_mmu_workaround
msr sctlr_el2, x0
isb
b 2f
@@ -103,6 +104,7 @@ ENTRY(entry)
mrs x0, sctlr_el1
bic x0, x0, #1 << 0 // clear SCTLR.M
bic x0, x0, #1 << 2 // clear SCTLR.C
+ pre_disable_mmu_workaround
msr sctlr_el1, x0
isb
2:
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index c3b241b8b659..ba3ab04788dc 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -849,6 +849,7 @@ __primary_switch:
* to take into account by discarding the current kernel mapping and
* creating a new one.
*/
+ pre_disable_mmu_workaround
msr sctlr_el1, x20 // disable the MMU
isb
bl __create_page_tables // recreate kernel mapping
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index ce704a4aeadd..f407e422a720 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -45,6 +45,7 @@ ENTRY(arm64_relocate_new_kernel)
mrs x0, sctlr_el2
ldr x1, =SCTLR_ELx_FLAGS
bic x0, x0, x1
+ pre_disable_mmu_workaround
msr sctlr_el2, x0
isb
1:
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 8a00de187e56..e086c6eff8c6 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -153,6 +153,7 @@ reset:
mrs x5, sctlr_el2
ldr x6, =SCTLR_ELx_FLAGS
bic x5, x5, x6 // Clear SCTL_M and etc
+ pre_disable_mmu_workaround
msr sctlr_el2, x5
isb
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 2/9] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0()
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-29 11:59 ` [PATCH v2 1/9] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 3/9] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
` (8 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
To allow systems which do not require kpti to continue running with
global kernel mappings (which appears to be a requirement for Cavium
ThunderX due to a CPU erratum), make the use of nG in the kernel page
tables dependent on arm64_kernel_unmapped_at_el0(), which is resolved
at runtime.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/kernel-pgtable.h | 12 ++----------
arch/arm64/include/asm/pgtable-prot.h | 30 ++++++++++++++----------------
2 files changed, 16 insertions(+), 26 deletions(-)
diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h
index 82386e860dd2..a780f6714b44 100644
--- a/arch/arm64/include/asm/kernel-pgtable.h
+++ b/arch/arm64/include/asm/kernel-pgtable.h
@@ -123,16 +123,8 @@
/*
* Initial memory map attributes.
*/
-#define _SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
-#define _SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
-
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define SWAPPER_PTE_FLAGS (_SWAPPER_PTE_FLAGS | PTE_NG)
-#define SWAPPER_PMD_FLAGS (_SWAPPER_PMD_FLAGS | PMD_SECT_NG)
-#else
-#define SWAPPER_PTE_FLAGS _SWAPPER_PTE_FLAGS
-#define SWAPPER_PMD_FLAGS _SWAPPER_PMD_FLAGS
-#endif
+#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
+#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
#if ARM64_SWAPPER_USES_SECTION_MAPS
#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 22a926825e3f..2db84df5eb42 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -37,13 +37,11 @@
#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define PROT_DEFAULT (_PROT_DEFAULT | PTE_NG)
-#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_SECT_NG)
-#else
-#define PROT_DEFAULT _PROT_DEFAULT
-#define PROT_SECT_DEFAULT _PROT_SECT_DEFAULT
-#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
+#define PTE_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PTE_NG : 0)
+#define PMD_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PMD_SECT_NG : 0)
+
+#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG)
+#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG)
#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))
#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))
@@ -55,22 +53,22 @@
#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
-#define _PAGE_DEFAULT (PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
-#define _HYP_PAGE_DEFAULT (_PAGE_DEFAULT & ~PTE_NG)
+#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
+#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT
-#define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_RO __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_ROX __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT)
+#define PAGE_KERNEL __pgprot(PROT_NORMAL)
+#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)
+#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY)
+#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN)
+#define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)
#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
#define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
-#define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
-#define PAGE_S2_DEVICE __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
+#define PAGE_S2 __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
+#define PAGE_S2_DEVICE __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN)
#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 3/9] arm64: mm: Permit transitioning from Global to Non-Global without BBM
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-29 11:59 ` [PATCH v2 1/9] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
2018-01-29 11:59 ` [PATCH v2 2/9] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
` (7 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
Break-before-make is not needed when transitioning from Global to
Non-Global mappings, provided that the contiguous hint is not being used.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/mm/mmu.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index b44992ec9643..fc7902bda02b 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -118,6 +118,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new)
if ((old | new) & PTE_CONT)
return false;
+ /* Transitioning from Global to Non-Global is safe */
+ if (((old ^ new) == PTE_NG) && (new & PTE_NG))
+ return true;
+
return ((old ^ new) & ~mask) == 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (2 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 3/9] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-30 10:30 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 5/9] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
` (6 subsequent siblings)
10 siblings, 1 reply; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
Defaulting to global mappings for kernel space is generally good for
performance and appears to be necessary for Cavium ThunderX. If we
subsequently decide that we need to enable kpti, then we need to rewrite
our existing page table entries to be non-global. This is fiddly, and
made worse by the possible use of contiguous mappings, which require
a strict break-before-make sequence.
Since the enable callback runs on each online CPU from stop_machine
context, we can have all CPUs enter the idmap, where secondaries can
wait for the primary CPU to rewrite swapper with its MMU off. It's all
fairly horrible, but at least it only runs once.
Tested-by: Marc Zyngier <marc.zyngier@arm.com>
Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/assembler.h | 10 ++
arch/arm64/kernel/cpufeature.c | 25 +++++
arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++--
3 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 3873dd7b5a32..23251eae6e8a 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -523,6 +523,16 @@ alternative_endif
#endif
.endm
+ .macro pte_to_phys, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+ ror \phys, \pte, #16
+ bfi \phys, \phys, #(16 + 12), #32
+ lsr \phys, \phys, #12
+#else
+ and \phys, \pte, #PTE_ADDR_MASK
+#endif
+ .endm
+
/**
* Errata workaround prior to disable MMU. Insert an ISB immediately prior
* to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 4b6f9051cf0c..cd9f46952db3 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -880,6 +880,30 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
ID_AA64PFR0_CSV3_SHIFT);
}
+static int kpti_install_ng_mappings(void *__unused)
+{
+ typedef void (kpti_remap_fn)(int, int, phys_addr_t);
+ extern kpti_remap_fn idmap_kpti_install_ng_mappings;
+ kpti_remap_fn *remap_fn;
+
+ static bool kpti_applied = false;
+ int cpu = smp_processor_id();
+
+ if (kpti_applied)
+ return 0;
+
+ remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);
+
+ cpu_install_idmap();
+ remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir));
+ cpu_uninstall_idmap();
+
+ if (!cpu)
+ kpti_applied = true;
+
+ return 0;
+}
+
static int __init parse_kpti(char *str)
{
bool enabled;
@@ -1003,6 +1027,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.capability = ARM64_UNMAP_KERNEL_AT_EL0,
.def_scope = SCOPE_SYSTEM,
.matches = unmap_kernel_at_el0,
+ .enable = kpti_install_ng_mappings,
},
#endif
{
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 9f177aac6390..ab8660eb55ca 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -166,6 +166,17 @@ ENTRY(cpu_do_switch_mm)
ENDPROC(cpu_do_switch_mm)
.pushsection ".idmap.text", "ax"
+
+.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
+ adrp \tmp1, empty_zero_page
+ phys_to_ttbr \tmp1, \tmp2
+ msr ttbr1_el1, \tmp2
+ isb
+ tlbi vmalle1
+ dsb nsh
+ isb
+.endm
+
/*
* void idmap_cpu_replace_ttbr1(phys_addr_t new_pgd)
*
@@ -175,14 +186,7 @@ ENDPROC(cpu_do_switch_mm)
ENTRY(idmap_cpu_replace_ttbr1)
save_and_disable_daif flags=x2
- adrp x1, empty_zero_page
- phys_to_ttbr x1, x3
- msr ttbr1_el1, x3
- isb
-
- tlbi vmalle1
- dsb nsh
- isb
+ __idmap_cpu_set_reserved_ttbr1 x1, x3
phys_to_ttbr x0, x3
msr ttbr1_el1, x3
@@ -194,6 +198,190 @@ ENTRY(idmap_cpu_replace_ttbr1)
ENDPROC(idmap_cpu_replace_ttbr1)
.popsection
+#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
+ .pushsection ".idmap.text", "ax"
+
+ .macro __idmap_kpti_get_pgtable_ent, type
+ dc cvac, cur_\()\type\()p // Ensure any existing dirty
+ dmb sy // lines are written back before
+ ldr \type, [cur_\()\type\()p] // loading the entry
+ tbz \type, #0, next_\()\type // Skip invalid entries
+ .endm
+
+ .macro __idmap_kpti_put_pgtable_ent_ng, type
+ orr \type, \type, #PTE_NG // Same bit for blocks and pages
+ str \type, [cur_\()\type\()p] // Update the entry and ensure it
+ dc civac, cur_\()\type\()p // is visible to all CPUs.
+ .endm
+
+/*
+ * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper)
+ *
+ * Called exactly once from stop_machine context by each CPU found during boot.
+ */
+__idmap_kpti_flag:
+ .long 1
+ENTRY(idmap_kpti_install_ng_mappings)
+ cpu .req w0
+ num_cpus .req w1
+ swapper_pa .req x2
+ swapper_ttb .req x3
+ flag_ptr .req x4
+ cur_pgdp .req x5
+ end_pgdp .req x6
+ pgd .req x7
+ cur_pudp .req x8
+ end_pudp .req x9
+ pud .req x10
+ cur_pmdp .req x11
+ end_pmdp .req x12
+ pmd .req x13
+ cur_ptep .req x14
+ end_ptep .req x15
+ pte .req x16
+
+ mrs swapper_ttb, ttbr1_el1
+ adr flag_ptr, __idmap_kpti_flag
+
+ cbnz cpu, __idmap_kpti_secondary
+
+ /* We're the boot CPU. Wait for the others to catch up */
+ sevl
+1: wfe
+ ldaxr w18, [flag_ptr]
+ eor w18, w18, num_cpus
+ cbnz w18, 1b
+
+ /* We need to walk swapper, so turn off the MMU. */
+ pre_disable_mmu_workaround
+ mrs x18, sctlr_el1
+ bic x18, x18, #SCTLR_ELx_M
+ msr sctlr_el1, x18
+ isb
+
+ /* Everybody is enjoying the idmap, so we can rewrite swapper. */
+ /* PGD */
+ mov cur_pgdp, swapper_pa
+ add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8)
+do_pgd: __idmap_kpti_get_pgtable_ent pgd
+ tbnz pgd, #1, walk_puds
+ __idmap_kpti_put_pgtable_ent_ng pgd
+next_pgd:
+ add cur_pgdp, cur_pgdp, #8
+ cmp cur_pgdp, end_pgdp
+ b.ne do_pgd
+
+ /* Publish the updated tables and nuke all the TLBs */
+ dsb sy
+ tlbi vmalle1is
+ dsb ish
+ isb
+
+ /* We're done: fire up the MMU again */
+ mrs x18, sctlr_el1
+ orr x18, x18, #SCTLR_ELx_M
+ msr sctlr_el1, x18
+ isb
+
+ /* Set the flag to zero to indicate that we're all done */
+ str wzr, [flag_ptr]
+ ret
+
+ /* PUD */
+walk_puds:
+ .if CONFIG_PGTABLE_LEVELS > 3
+ pte_to_phys cur_pudp, pgd
+ add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8)
+do_pud: __idmap_kpti_get_pgtable_ent pud
+ tbnz pud, #1, walk_pmds
+ __idmap_kpti_put_pgtable_ent_ng pud
+next_pud:
+ add cur_pudp, cur_pudp, 8
+ cmp cur_pudp, end_pudp
+ b.ne do_pud
+ b next_pgd
+ .else /* CONFIG_PGTABLE_LEVELS <= 3 */
+ mov pud, pgd
+ b walk_pmds
+next_pud:
+ b next_pgd
+ .endif
+
+ /* PMD */
+walk_pmds:
+ .if CONFIG_PGTABLE_LEVELS > 2
+ pte_to_phys cur_pmdp, pud
+ add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8)
+do_pmd: __idmap_kpti_get_pgtable_ent pmd
+ tbnz pmd, #1, walk_ptes
+ __idmap_kpti_put_pgtable_ent_ng pmd
+next_pmd:
+ add cur_pmdp, cur_pmdp, #8
+ cmp cur_pmdp, end_pmdp
+ b.ne do_pmd
+ b next_pud
+ .else /* CONFIG_PGTABLE_LEVELS <= 2 */
+ mov pmd, pud
+ b walk_ptes
+next_pmd:
+ b next_pud
+ .endif
+
+ /* PTE */
+walk_ptes:
+ pte_to_phys cur_ptep, pmd
+ add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8)
+do_pte: __idmap_kpti_get_pgtable_ent pte
+ __idmap_kpti_put_pgtable_ent_ng pte
+next_pte:
+ add cur_ptep, cur_ptep, #8
+ cmp cur_ptep, end_ptep
+ b.ne do_pte
+ b next_pmd
+
+ /* Secondary CPUs end up here */
+__idmap_kpti_secondary:
+ /* Uninstall swapper before surgery begins */
+ __idmap_cpu_set_reserved_ttbr1 x18, x17
+
+ /* Increment the flag to let the boot CPU we're ready */
+1: ldxr w18, [flag_ptr]
+ add w18, w18, #1
+ stxr w17, w18, [flag_ptr]
+ cbnz w17, 1b
+
+ /* Wait for the boot CPU to finish messing around with swapper */
+ sevl
+1: wfe
+ ldxr w18, [flag_ptr]
+ cbnz w18, 1b
+
+ /* All done, act like nothing happened */
+ msr ttbr1_el1, swapper_ttb
+ isb
+ ret
+
+ .unreq cpu
+ .unreq num_cpus
+ .unreq swapper_pa
+ .unreq swapper_ttb
+ .unreq flag_ptr
+ .unreq cur_pgdp
+ .unreq end_pgdp
+ .unreq pgd
+ .unreq cur_pudp
+ .unreq end_pudp
+ .unreq pud
+ .unreq cur_pmdp
+ .unreq end_pmdp
+ .unreq pmd
+ .unreq cur_ptep
+ .unreq end_ptep
+ .unreq pte
+ENDPROC(idmap_kpti_install_ng_mappings)
+ .popsection
+#endif
+
/*
* __cpu_setup
*
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 5/9] arm64: Force KPTI to be disabled on Cavium ThunderX
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (3 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 6/9] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
` (5 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
From: Marc Zyngier <marc.zyngier@arm.com>
Cavium ThunderX's erratum 27456 results in a corruption of icache
entries that are loaded from memory that is mapped as non-global
(i.e. ASID-tagged).
As KPTI is based on memory being mapped non-global, let's prevent
it from kicking in if this erratum is detected.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
[will: Update comment]
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/kernel/cpufeature.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index cd9f46952db3..0ff8ab79352c 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -855,12 +855,23 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */
static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
int __unused)
{
+ char const *str = "command line option";
u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
- /* Forced on command line? */
+ /*
+ * For reasons that aren't entirely clear, enabling KPTI on Cavium
+ * ThunderX leads to apparent I-cache corruption of kernel text, which
+ * ends as well as you might imagine. Don't even try.
+ */
+ if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) {
+ str = "ARM64_WORKAROUND_CAVIUM_27456";
+ __kpti_forced = -1;
+ }
+
+ /* Forced? */
if (__kpti_forced) {
- pr_info_once("kernel page table isolation forced %s by command line option\n",
- __kpti_forced > 0 ? "ON" : "OFF");
+ pr_info_once("kernel page table isolation forced %s by %s\n",
+ __kpti_forced > 0 ? "ON" : "OFF", str);
return __kpti_forced > 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 6/9] arm64: assembler: Change order of macro arguments in phys_to_ttbr
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (4 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 5/9] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
` (4 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
Since AArch64 assembly instructions take the destination register as
their first operand, do the same thing for the phys_to_ttbr macro.
Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/assembler.h | 2 +-
arch/arm64/kernel/head.S | 4 ++--
arch/arm64/kernel/hibernate-asm.S | 4 ++--
arch/arm64/kvm/hyp-init.S | 2 +-
arch/arm64/mm/proc.S | 6 +++---
5 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 23251eae6e8a..e4495ef96058 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -514,7 +514,7 @@ alternative_endif
* phys: physical address, preserved
* ttbr: returns the TTBR value
*/
- .macro phys_to_ttbr, phys, ttbr
+ .macro phys_to_ttbr, ttbr, phys
#ifdef CONFIG_ARM64_PA_BITS_52
orr \ttbr, \phys, \phys, lsr #46
and \ttbr, \ttbr, #TTBR_BADDR_MASK_52
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index ba3ab04788dc..341649c08337 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -776,8 +776,8 @@ ENTRY(__enable_mmu)
update_early_cpu_boot_status 0, x1, x2
adrp x1, idmap_pg_dir
adrp x2, swapper_pg_dir
- phys_to_ttbr x1, x3
- phys_to_ttbr x2, x4
+ phys_to_ttbr x3, x1
+ phys_to_ttbr x4, x2
msr ttbr0_el1, x3 // load TTBR0
msr ttbr1_el1, x4 // load TTBR1
isb
diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
index 84f5d52fddda..dd14ab8c9f72 100644
--- a/arch/arm64/kernel/hibernate-asm.S
+++ b/arch/arm64/kernel/hibernate-asm.S
@@ -34,12 +34,12 @@
* each stage of the walk.
*/
.macro break_before_make_ttbr_switch zero_page, page_table, tmp
- phys_to_ttbr \zero_page, \tmp
+ phys_to_ttbr \tmp, \zero_page
msr ttbr1_el1, \tmp
isb
tlbi vmalle1
dsb nsh
- phys_to_ttbr \page_table, \tmp
+ phys_to_ttbr \tmp, \page_table
msr ttbr1_el1, \tmp
isb
.endm
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index e086c6eff8c6..5aa9ccf6db99 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -63,7 +63,7 @@ __do_hyp_init:
cmp x0, #HVC_STUB_HCALL_NR
b.lo __kvm_handle_stub_hvc
- phys_to_ttbr x0, x4
+ phys_to_ttbr x4, x0
msr ttbr0_el2, x4
mrs x4, tcr_el1
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index ab8660eb55ca..cfd22ba9b510 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -153,7 +153,7 @@ ENDPROC(cpu_do_resume)
ENTRY(cpu_do_switch_mm)
mrs x2, ttbr1_el1
mmid x1, x1 // get mm->context.id
- phys_to_ttbr x0, x3
+ phys_to_ttbr x3, x0
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
bfi x3, x1, #48, #16 // set the ASID field in TTBR0
#endif
@@ -169,7 +169,7 @@ ENDPROC(cpu_do_switch_mm)
.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
adrp \tmp1, empty_zero_page
- phys_to_ttbr \tmp1, \tmp2
+ phys_to_ttbr \tmp2, \tmp1
msr ttbr1_el1, \tmp2
isb
tlbi vmalle1
@@ -188,7 +188,7 @@ ENTRY(idmap_cpu_replace_ttbr1)
__idmap_cpu_set_reserved_ttbr1 x1, x3
- phys_to_ttbr x0, x3
+ phys_to_ttbr x3, x0
msr ttbr1_el1, x3
isb
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (5 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 6/9] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-02-03 11:15 ` Ard Biesheuvel
2018-01-29 11:59 ` [PATCH v2 8/9] arm64: assembler: Align phys_to_pte with pte_to_phys Will Deacon
` (3 subsequent siblings)
10 siblings, 1 reply; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
We don't fully understand the Cavium ThunderX erratum, but it appears
that mapping the kernel as nG can lead to horrible consequences such as
attempting to execute userspace from kernel context. Since kpti isn't
enabled for these CPUs anyway, simplify the comment justifying the lack
of post_ttbr_update_workaround in the exception trampoline.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/kernel/entry.S | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index b34e717d7597..fbe1444324b3 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif
orr \tmp, \tmp, #USER_ASID_FLAG
msr ttbr1_el1, \tmp
/*
- * We avoid running the post_ttbr_update_workaround here because the
- * user and kernel ASIDs don't have conflicting mappings, so any
- * "blessing" as described in:
- *
- * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
- *
- * will not hurt correctness. Whilst this may partially defeat the
- * point of using split ASIDs in the first place, it avoids
- * the hit of invalidating the entire I-cache on every return to
- * userspace.
+ * We avoid running the post_ttbr_update_workaround here because
+ * it's only needed by Cavium ThunderX, which doesn't require kpti.
*/
.endm
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 8/9] arm64: assembler: Align phys_to_pte with pte_to_phys
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (6 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
@ 2018-01-29 11:59 ` Will Deacon
2018-01-29 12:00 ` [PATCH v2 9/9] arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives Will Deacon
` (2 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 11:59 UTC (permalink / raw)
To: linux-arm-kernel
pte_to_phys lives in assembler.h and takes its destination register as
the first argument. Move phys_to_pte out of head.S to sit with its
counterpart and rejig it to follow the same calling convention.
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/assembler.h | 14 ++++++++++++++
arch/arm64/kernel/head.S | 24 ++----------------------
2 files changed, 16 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index e4495ef96058..f311a1ed34d0 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -523,6 +523,20 @@ alternative_endif
#endif
.endm
+ .macro phys_to_pte, pte, phys
+#ifdef CONFIG_ARM64_PA_BITS_52
+ /*
+ * We assume \phys is 64K aligned and this is guaranteed by only
+ * supporting this configuration with 64K pages.
+ */
+ orr \pte, \phys, \phys, lsr #36
+ and \pte, \pte, #PTE_ADDR_MASK
+#else
+ mov \pte, \phys
+#endif
+ .endm
+
+
.macro pte_to_phys, phys, pte
#ifdef CONFIG_ARM64_PA_BITS_52
ror \phys, \pte, #16
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 341649c08337..25f5b2e400fb 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -148,26 +148,6 @@ preserve_boot_args:
ENDPROC(preserve_boot_args)
/*
- * Macro to arrange a physical address in a page table entry, taking care of
- * 52-bit addresses.
- *
- * Preserves: phys
- * Returns: pte
- */
- .macro phys_to_pte, phys, pte
-#ifdef CONFIG_ARM64_PA_BITS_52
- /*
- * We assume \phys is 64K aligned and this is guaranteed by only
- * supporting this configuration with 64K pages.
- */
- orr \pte, \phys, \phys, lsr #36
- and \pte, \pte, #PTE_ADDR_MASK
-#else
- mov \pte, \phys
-#endif
- .endm
-
-/*
* Macro to create a table entry to the next page.
*
* tbl: page table address
@@ -181,7 +161,7 @@ ENDPROC(preserve_boot_args)
*/
.macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
add \tmp1, \tbl, #PAGE_SIZE
- phys_to_pte \tmp1, \tmp2
+ phys_to_pte \tmp2, \tmp1
orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type
lsr \tmp1, \virt, #\shift
sub \ptrs, \ptrs, #1
@@ -207,7 +187,7 @@ ENDPROC(preserve_boot_args)
* Returns: rtbl
*/
.macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1
-.Lpe\@: phys_to_pte \rtbl, \tmp1
+.Lpe\@: phys_to_pte \tmp1, \rtbl
orr \tmp1, \tmp1, \flags // tmp1 = table entry
str \tmp1, [\tbl, \index, lsl #3]
add \rtbl, \rtbl, \inc // rtbl = pa next level
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 9/9] arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (7 preceding siblings ...)
2018-01-29 11:59 ` [PATCH v2 8/9] arm64: assembler: Align phys_to_pte with pte_to_phys Will Deacon
@ 2018-01-29 12:00 ` Will Deacon
2018-02-03 11:36 ` [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Ard Biesheuvel
2018-02-06 22:31 ` Catalin Marinas
10 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-29 12:00 UTC (permalink / raw)
To: linux-arm-kernel
The identity map is mapped as both writeable and executable by the
SWAPPER_MM_MMUFLAGS and this is relied upon by the kpti code to manage
a synchronisation flag. Update the .pushsection flags to reflect the
actual mapping attributes.
Reported-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/kernel/cpu-reset.S | 2 +-
arch/arm64/kernel/head.S | 2 +-
arch/arm64/kernel/sleep.S | 2 +-
arch/arm64/mm/proc.S | 8 ++++----
4 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 2a752cb2a0f3..8021b46c9743 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -16,7 +16,7 @@
#include <asm/virt.h>
.text
-.pushsection .idmap.text, "ax"
+.pushsection .idmap.text, "awx"
/*
* __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 25f5b2e400fb..2b6b8b24e5ab 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -455,7 +455,7 @@ ENDPROC(__primary_switched)
* end early head section, begin head code that is also used for
* hotplug and needs to have the same protections as the text region
*/
- .section ".idmap.text","ax"
+ .section ".idmap.text","awx"
ENTRY(kimage_vaddr)
.quad _text - TEXT_OFFSET
diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index 10dd16d7902d..bebec8ef9372 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -96,7 +96,7 @@ ENTRY(__cpu_suspend_enter)
ret
ENDPROC(__cpu_suspend_enter)
- .pushsection ".idmap.text", "ax"
+ .pushsection ".idmap.text", "awx"
ENTRY(cpu_resume)
bl el2_setup // if in EL2 drop to EL1 cleanly
bl __cpu_setup
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index cfd22ba9b510..71baed7e592a 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -90,7 +90,7 @@ ENDPROC(cpu_do_suspend)
*
* x0: Address of context pointer
*/
- .pushsection ".idmap.text", "ax"
+ .pushsection ".idmap.text", "awx"
ENTRY(cpu_do_resume)
ldp x2, x3, [x0]
ldp x4, x5, [x0, #16]
@@ -165,7 +165,7 @@ ENTRY(cpu_do_switch_mm)
b post_ttbr_update_workaround // Back to C code...
ENDPROC(cpu_do_switch_mm)
- .pushsection ".idmap.text", "ax"
+ .pushsection ".idmap.text", "awx"
.macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
adrp \tmp1, empty_zero_page
@@ -199,7 +199,7 @@ ENDPROC(idmap_cpu_replace_ttbr1)
.popsection
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
- .pushsection ".idmap.text", "ax"
+ .pushsection ".idmap.text", "awx"
.macro __idmap_kpti_get_pgtable_ent, type
dc cvac, cur_\()\type\()p // Ensure any existing dirty
@@ -388,7 +388,7 @@ ENDPROC(idmap_kpti_install_ng_mappings)
* Initialise the processor for turning the MMU on. Return in x0 the
* value of the SCTLR_EL1 register.
*/
- .pushsection ".idmap.text", "ax"
+ .pushsection ".idmap.text", "awx"
ENTRY(__cpu_setup)
tlbi vmalle1 // Invalidate local TLB
dsb nsh
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings
2018-01-29 11:59 ` [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
@ 2018-01-30 10:30 ` Will Deacon
0 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-01-30 10:30 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Jan 29, 2018 at 11:59:55AM +0000, Will Deacon wrote:
> Defaulting to global mappings for kernel space is generally good for
> performance and appears to be necessary for Cavium ThunderX. If we
> subsequently decide that we need to enable kpti, then we need to rewrite
> our existing page table entries to be non-global. This is fiddly, and
> made worse by the possible use of contiguous mappings, which require
> a strict break-before-make sequence.
>
> Since the enable callback runs on each online CPU from stop_machine
> context, we can have all CPUs enter the idmap, where secondaries can
> wait for the primary CPU to rewrite swapper with its MMU off. It's all
> fairly horrible, but at least it only runs once.
>
> Tested-by: Marc Zyngier <marc.zyngier@arm.com>
> Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
> arch/arm64/include/asm/assembler.h | 10 ++
> arch/arm64/kernel/cpufeature.c | 25 +++++
> arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++--
> 3 files changed, 231 insertions(+), 8 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 3873dd7b5a32..23251eae6e8a 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -523,6 +523,16 @@ alternative_endif
> #endif
> .endm
>
> + .macro pte_to_phys, phys, pte
> +#ifdef CONFIG_ARM64_PA_BITS_52
> + ror \phys, \pte, #16
> + bfi \phys, \phys, #(16 + 12), #32
> + lsr \phys, \phys, #12
I spun this up on a model with 52-bit PA support and it doesn't work,
unfortunately, because the lsr of #12 leaves 4 bits of the OA in the bottom
nybble. Instead, we need the following on top:
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index f311a1ed34d0..f4f3350f697e 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -539,9 +539,9 @@ alternative_endif
.macro pte_to_phys, phys, pte
#ifdef CONFIG_ARM64_PA_BITS_52
- ror \phys, \pte, #16
- bfi \phys, \phys, #(16 + 12), #32
- lsr \phys, \phys, #12
+ ubfiz \phys, \pte, #(48 - 16 - 12), #16
+ bfxil \phys, \pte, #16, #32
+ lsl \phys, \phys, #16
#else
and \phys, \pte, #PTE_ADDR_MASK
#endif
Will
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround
2018-01-29 11:59 ` [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
@ 2018-02-03 11:15 ` Ard Biesheuvel
2018-02-05 16:41 ` Will Deacon
0 siblings, 1 reply; 17+ messages in thread
From: Ard Biesheuvel @ 2018-02-03 11:15 UTC (permalink / raw)
To: linux-arm-kernel
On 29 January 2018 at 11:59, Will Deacon <will.deacon@arm.com> wrote:
> We don't fully understand the Cavium ThunderX erratum, but it appears
> that mapping the kernel as nG can lead to horrible consequences such as
> attempting to execute userspace from kernel context. Since kpti isn't
> enabled for these CPUs anyway, simplify the comment justifying the lack
> of post_ttbr_update_workaround in the exception trampoline.
>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
> arch/arm64/kernel/entry.S | 12 ++----------
> 1 file changed, 2 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index b34e717d7597..fbe1444324b3 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -1013,16 +1013,8 @@ alternative_else_nop_endif
> orr \tmp, \tmp, #USER_ASID_FLAG
> msr ttbr1_el1, \tmp
> /*
> - * We avoid running the post_ttbr_update_workaround here because the
> - * user and kernel ASIDs don't have conflicting mappings, so any
> - * "blessing" as described in:
> - *
> - * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
> - *
> - * will not hurt correctness. Whilst this may partially defeat the
> - * point of using split ASIDs in the first place, it avoids
> - * the hit of invalidating the entire I-cache on every return to
> - * userspace.
> + * We avoid running the post_ttbr_update_workaround here because
> + * it's only needed by Cavium ThunderX, which doesn't require kpti.
'requires KPTI to be disabled' sounds more accurate to me
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (8 preceding siblings ...)
2018-01-29 12:00 ` [PATCH v2 9/9] arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives Will Deacon
@ 2018-02-03 11:36 ` Ard Biesheuvel
2018-02-05 16:41 ` Will Deacon
2018-02-06 22:31 ` Catalin Marinas
10 siblings, 1 reply; 17+ messages in thread
From: Ard Biesheuvel @ 2018-02-03 11:36 UTC (permalink / raw)
To: linux-arm-kernel
On 29 January 2018 at 11:59, Will Deacon <will.deacon@arm.com> wrote:
> Hi all,
>
> This is version two of the patches I posted on Friday:
>
> http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
>
> Changes since v1 include:
>
> * Use SCTLR_ELx_M instead of open-coded #1
> * Changed section attributes for .idmap.text to reflect actual mappings
> * Rejigged phys_to_pte for consistency with pte_to_phys
> * Added reviewer tags
>
Any chance we could base this on the arm64/kpti branch rather than
for-next/core? Any backports of kpti will need to include this series
as well, or we break ThunderX, and I'd rather have you merge it into
for-next/core rather than the various backporters (whose level of
familiarity with this code is invariably lower than yours) into the
stable trees.
For reference, my v4.14 backport of KPTI is here:
https://git.linaro.org/kernel/speculation-fixes-staging.git/log/?h=v4.14-kpti-only
containing just the KPTI pieces that were covered by the arm64 PR for v4.16
Once Marc's recent Spectre variant 2 stuff hits mainline as well, I
will pull that into a separate branch (which may just be based on this
one if they are too difficult to disentangle)
> --->8
>
> Marc Zyngier (1):
> arm64: Force KPTI to be disabled on Cavium ThunderX
>
> Shanker Donthineni (1):
> arm64: Add software workaround for Falkor erratum 1041
>
> Will Deacon (7):
> arm64: kpti: Make use of nG dependent on
> arm64_kernel_unmapped_at_el0()
> arm64: mm: Permit transitioning from Global to Non-Global without BBM
> arm64: kpti: Add ->enable callback to remap swapper using nG mappings
> arm64: assembler: Change order of macro arguments in phys_to_ttbr
> arm64: entry: Reword comment about post_ttbr_update_workaround
> arm64: assembler: Align phys_to_pte with pte_to_phys
> arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives
>
> Documentation/arm64/silicon-errata.txt | 1 +
> arch/arm64/Kconfig | 12 +-
> arch/arm64/include/asm/assembler.h | 36 +++++-
> arch/arm64/include/asm/kernel-pgtable.h | 12 +-
> arch/arm64/include/asm/pgtable-prot.h | 30 +++--
> arch/arm64/kernel/cpu-reset.S | 3 +-
> arch/arm64/kernel/cpufeature.c | 42 ++++++-
> arch/arm64/kernel/efi-entry.S | 2 +
> arch/arm64/kernel/entry.S | 12 +-
> arch/arm64/kernel/head.S | 31 +----
> arch/arm64/kernel/hibernate-asm.S | 4 +-
> arch/arm64/kernel/relocate_kernel.S | 1 +
> arch/arm64/kernel/sleep.S | 2 +-
> arch/arm64/kvm/hyp-init.S | 3 +-
> arch/arm64/mm/mmu.c | 4 +
> arch/arm64/mm/proc.S | 212 ++++++++++++++++++++++++++++++--
> 16 files changed, 324 insertions(+), 83 deletions(-)
>
> --
> 2.1.4
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround
2018-02-03 11:15 ` Ard Biesheuvel
@ 2018-02-05 16:41 ` Will Deacon
0 siblings, 0 replies; 17+ messages in thread
From: Will Deacon @ 2018-02-05 16:41 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Feb 03, 2018 at 11:15:59AM +0000, Ard Biesheuvel wrote:
> On 29 January 2018 at 11:59, Will Deacon <will.deacon@arm.com> wrote:
> > We don't fully understand the Cavium ThunderX erratum, but it appears
> > that mapping the kernel as nG can lead to horrible consequences such as
> > attempting to execute userspace from kernel context. Since kpti isn't
> > enabled for these CPUs anyway, simplify the comment justifying the lack
> > of post_ttbr_update_workaround in the exception trampoline.
> >
> > Signed-off-by: Will Deacon <will.deacon@arm.com>
> > ---
> > arch/arm64/kernel/entry.S | 12 ++----------
> > 1 file changed, 2 insertions(+), 10 deletions(-)
> >
> > diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> > index b34e717d7597..fbe1444324b3 100644
> > --- a/arch/arm64/kernel/entry.S
> > +++ b/arch/arm64/kernel/entry.S
> > @@ -1013,16 +1013,8 @@ alternative_else_nop_endif
> > orr \tmp, \tmp, #USER_ASID_FLAG
> > msr ttbr1_el1, \tmp
> > /*
> > - * We avoid running the post_ttbr_update_workaround here because the
> > - * user and kernel ASIDs don't have conflicting mappings, so any
> > - * "blessing" as described in:
> > - *
> > - * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
> > - *
> > - * will not hurt correctness. Whilst this may partially defeat the
> > - * point of using split ASIDs in the first place, it avoids
> > - * the hit of invalidating the entire I-cache on every return to
> > - * userspace.
> > + * We avoid running the post_ttbr_update_workaround here because
> > + * it's only needed by Cavium ThunderX, which doesn't require kpti.
>
> 'requires KPTI to be disabled' sounds more accurate to me
Fair enough. I'll leave this for Catalin to merge in.
Will
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX
2018-02-03 11:36 ` [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Ard Biesheuvel
@ 2018-02-05 16:41 ` Will Deacon
2018-02-05 19:21 ` Ard Biesheuvel
0 siblings, 1 reply; 17+ messages in thread
From: Will Deacon @ 2018-02-05 16:41 UTC (permalink / raw)
To: linux-arm-kernel
Hi Ard,
On Sat, Feb 03, 2018 at 11:36:04AM +0000, Ard Biesheuvel wrote:
> On 29 January 2018 at 11:59, Will Deacon <will.deacon@arm.com> wrote:
> > Hi all,
> >
> > This is version two of the patches I posted on Friday:
> >
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
> >
> > Changes since v1 include:
> >
> > * Use SCTLR_ELx_M instead of open-coded #1
> > * Changed section attributes for .idmap.text to reflect actual mappings
> > * Rejigged phys_to_pte for consistency with pte_to_phys
> > * Added reviewer tags
> >
>
> Any chance we could base this on the arm64/kpti branch rather than
> for-next/core? Any backports of kpti will need to include this series
> as well, or we break ThunderX, and I'd rather have you merge it into
> for-next/core rather than the various backporters (whose level of
> familiarity with this code is invariably lower than yours) into the
> stable trees.
I'd prefer to have this based on for-next/core for ease of merging, but I'll
do a version for the kpti branch too (probably at -rc1). Once mainline has
settled down, I plan to rebuild kpti with references to upstream commits
etc.
Will
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX
2018-02-05 16:41 ` Will Deacon
@ 2018-02-05 19:21 ` Ard Biesheuvel
0 siblings, 0 replies; 17+ messages in thread
From: Ard Biesheuvel @ 2018-02-05 19:21 UTC (permalink / raw)
To: linux-arm-kernel
On 5 February 2018 at 17:41, Will Deacon <will.deacon@arm.com> wrote:
> Hi Ard,
>
> On Sat, Feb 03, 2018 at 11:36:04AM +0000, Ard Biesheuvel wrote:
>> On 29 January 2018 at 11:59, Will Deacon <will.deacon@arm.com> wrote:
>> > Hi all,
>> >
>> > This is version two of the patches I posted on Friday:
>> >
>> > http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
>> >
>> > Changes since v1 include:
>> >
>> > * Use SCTLR_ELx_M instead of open-coded #1
>> > * Changed section attributes for .idmap.text to reflect actual mappings
>> > * Rejigged phys_to_pte for consistency with pte_to_phys
>> > * Added reviewer tags
>> >
>>
>> Any chance we could base this on the arm64/kpti branch rather than
>> for-next/core? Any backports of kpti will need to include this series
>> as well, or we break ThunderX, and I'd rather have you merge it into
>> for-next/core rather than the various backporters (whose level of
>> familiarity with this code is invariably lower than yours) into the
>> stable trees.
>
> I'd prefer to have this based on for-next/core for ease of merging, but I'll
> do a version for the kpti branch too (probably at -rc1). Once mainline has
> settled down, I plan to rebuild kpti with references to upstream commits
> etc.
>
I am already doing that for v4.14. Perhaps it is simplest to merge
that branch into v4.15 as well (unless you need to rebuild the branch
for other reasons as well)
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
` (9 preceding siblings ...)
2018-02-03 11:36 ` [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Ard Biesheuvel
@ 2018-02-06 22:31 ` Catalin Marinas
10 siblings, 0 replies; 17+ messages in thread
From: Catalin Marinas @ 2018-02-06 22:31 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Jan 29, 2018 at 11:59:51AM +0000, Will Deacon wrote:
> Hi all,
>
> This is version two of the patches I posted on Friday:
>
> http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
>
> Changes since v1 include:
>
> * Use SCTLR_ELx_M instead of open-coded #1
> * Changed section attributes for .idmap.text to reflect actual mappings
> * Rejigged phys_to_pte for consistency with pte_to_phys
> * Added reviewer tags
Queued for 4.16. Thanks.
--
Catalin
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2018-02-06 22:31 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-29 11:59 [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Will Deacon
2018-01-29 11:59 ` [PATCH v2 1/9] arm64: Add software workaround for Falkor erratum 1041 Will Deacon
2018-01-29 11:59 ` [PATCH v2 2/9] arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0() Will Deacon
2018-01-29 11:59 ` [PATCH v2 3/9] arm64: mm: Permit transitioning from Global to Non-Global without BBM Will Deacon
2018-01-29 11:59 ` [PATCH v2 4/9] arm64: kpti: Add ->enable callback to remap swapper using nG mappings Will Deacon
2018-01-30 10:30 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 5/9] arm64: Force KPTI to be disabled on Cavium ThunderX Will Deacon
2018-01-29 11:59 ` [PATCH v2 6/9] arm64: assembler: Change order of macro arguments in phys_to_ttbr Will Deacon
2018-01-29 11:59 ` [PATCH v2 7/9] arm64: entry: Reword comment about post_ttbr_update_workaround Will Deacon
2018-02-03 11:15 ` Ard Biesheuvel
2018-02-05 16:41 ` Will Deacon
2018-01-29 11:59 ` [PATCH v2 8/9] arm64: assembler: Align phys_to_pte with pte_to_phys Will Deacon
2018-01-29 12:00 ` [PATCH v2 9/9] arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives Will Deacon
2018-02-03 11:36 ` [PATCH v2 0/9] Fix kpti-enabled kernels for Cavium ThunderX Ard Biesheuvel
2018-02-05 16:41 ` Will Deacon
2018-02-05 19:21 ` Ard Biesheuvel
2018-02-06 22:31 ` Catalin Marinas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.