* [PATCH v2 00/12] powerpc: Optimise KUAP on book3s/32
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
This series is a rework of KUAP on book3s/32.
On book3s32, KUAP is heavier than on other platform because it can't
be opened globaly at once, it must be done for each 256Mb segment.
Instead of opening access to all necessary segments via a heavy logic,
only open access to the segment matching the start of the range.
99.999% of the time, the range doesn't cross segment limit so
it is not worth checking and handling that corner case at the
first place. If that happens, take a fault and do the additional
segments opening in the fault handler. See patch 9 for details.
This series also adds the capability to disable KUAP and KUEP at boot
time via the "nosmap" and "nosmep" kernel parameters as PPC64 does.
Last 2 patches are cleanup of generic KUAP code once book3s/32 has
been simplified.
Changes in v2:
- Incorporated the conversion to C of switch_mmu_context()
- Converted initial KUAP/KUEP setup to C
- Added capability to disable KUAP and KUEP at boottime
- Enable KUAP and KUEP by default
Christophe Leroy (12):
powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
powerpc/32s: Refactor update of user segment registers
powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
powerpc/32s: Convert switch_mmu_context() to C
powerpc/32s: Simplify calculation of segment register content
powerpc/32s: Initialise KUAP and KUEP in C
powerpc/32s: Allow disabling KUEP at boot time
powerpc/32s: Allow disabling KUAP at boot time
powerpc/32s: Rework Kernel Userspace Access Protection
powerpc/32s: Activate KUAP and KUEP by default
powerpc/kuap: Remove KUAP_CURRENT_XXX
powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
arch/powerpc/include/asm/book3s/32/kup.h | 199 ++++++++++++------
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 41 ++++
arch/powerpc/include/asm/book3s/64/kup.h | 3 +-
arch/powerpc/include/asm/kup.h | 28 +--
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +-
arch/powerpc/include/asm/processor.h | 10 +-
arch/powerpc/kernel/asm-offsets.c | 5 -
arch/powerpc/kernel/head_book3s_32.S | 64 ------
arch/powerpc/kernel/process.c | 3 +
arch/powerpc/kernel/smp.c | 4 +
arch/powerpc/kvm/book3s_32_mmu_host.c | 3 -
arch/powerpc/mm/book3s32/Makefile | 1 +
arch/powerpc/mm/book3s32/kuap.c | 30 +++
arch/powerpc/mm/book3s32/kuep.c | 42 +---
arch/powerpc/mm/book3s32/mmu.c | 20 --
arch/powerpc/mm/book3s32/mmu_context.c | 48 +++--
arch/powerpc/platforms/Kconfig.cputype | 4 +-
17 files changed, 276 insertions(+), 232 deletions(-)
create mode 100644 arch/powerpc/mm/book3s32/kuap.c
--
2.25.0
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v2 00/12] powerpc: Optimise KUAP on book3s/32
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
This series is a rework of KUAP on book3s/32.
On book3s32, KUAP is heavier than on other platform because it can't
be opened globaly at once, it must be done for each 256Mb segment.
Instead of opening access to all necessary segments via a heavy logic,
only open access to the segment matching the start of the range.
99.999% of the time, the range doesn't cross segment limit so
it is not worth checking and handling that corner case at the
first place. If that happens, take a fault and do the additional
segments opening in the fault handler. See patch 9 for details.
This series also adds the capability to disable KUAP and KUEP at boot
time via the "nosmap" and "nosmep" kernel parameters as PPC64 does.
Last 2 patches are cleanup of generic KUAP code once book3s/32 has
been simplified.
Changes in v2:
- Incorporated the conversion to C of switch_mmu_context()
- Converted initial KUAP/KUEP setup to C
- Added capability to disable KUAP and KUEP at boottime
- Enable KUAP and KUEP by default
Christophe Leroy (12):
powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
powerpc/32s: Refactor update of user segment registers
powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
powerpc/32s: Convert switch_mmu_context() to C
powerpc/32s: Simplify calculation of segment register content
powerpc/32s: Initialise KUAP and KUEP in C
powerpc/32s: Allow disabling KUEP at boot time
powerpc/32s: Allow disabling KUAP at boot time
powerpc/32s: Rework Kernel Userspace Access Protection
powerpc/32s: Activate KUAP and KUEP by default
powerpc/kuap: Remove KUAP_CURRENT_XXX
powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
arch/powerpc/include/asm/book3s/32/kup.h | 199 ++++++++++++------
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 41 ++++
arch/powerpc/include/asm/book3s/64/kup.h | 3 +-
arch/powerpc/include/asm/kup.h | 28 +--
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +-
arch/powerpc/include/asm/processor.h | 10 +-
arch/powerpc/kernel/asm-offsets.c | 5 -
arch/powerpc/kernel/head_book3s_32.S | 64 ------
arch/powerpc/kernel/process.c | 3 +
arch/powerpc/kernel/smp.c | 4 +
arch/powerpc/kvm/book3s_32_mmu_host.c | 3 -
arch/powerpc/mm/book3s32/Makefile | 1 +
arch/powerpc/mm/book3s32/kuap.c | 30 +++
arch/powerpc/mm/book3s32/kuep.c | 42 +---
arch/powerpc/mm/book3s32/mmu.c | 20 --
arch/powerpc/mm/book3s32/mmu_context.c | 48 +++--
arch/powerpc/platforms/Kconfig.cputype | 4 +-
17 files changed, 276 insertions(+), 232 deletions(-)
create mode 100644 arch/powerpc/mm/book3s32/kuap.c
--
2.25.0
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v2 01/12] powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
Avoids the #ifdef in mmu.c
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/mm/book3s32/Makefile | 1 +
arch/powerpc/mm/book3s32/kuap.c | 11 +++++++++++
arch/powerpc/mm/book3s32/kuep.c | 8 ++++++++
arch/powerpc/mm/book3s32/mmu.c | 20 --------------------
4 files changed, 20 insertions(+), 20 deletions(-)
create mode 100644 arch/powerpc/mm/book3s32/kuap.c
diff --git a/arch/powerpc/mm/book3s32/Makefile b/arch/powerpc/mm/book3s32/Makefile
index 7f0c8a78ba0c..15f4773643d2 100644
--- a/arch/powerpc/mm/book3s32/Makefile
+++ b/arch/powerpc/mm/book3s32/Makefile
@@ -10,3 +10,4 @@ obj-y += mmu.o mmu_context.o
obj-$(CONFIG_PPC_BOOK3S_603) += nohash_low.o
obj-$(CONFIG_PPC_BOOK3S_604) += hash_low.o tlb.o
obj-$(CONFIG_PPC_KUEP) += kuep.o
+obj-$(CONFIG_PPC_KUAP) += kuap.o
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
new file mode 100644
index 000000000000..1df55392878e
--- /dev/null
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <asm/kup.h>
+
+void __init setup_kuap(bool disabled)
+{
+ pr_info("Activating Kernel Userspace Access Protection\n");
+
+ if (disabled)
+ pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
+}
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 8ed1b8634839..6eafe7b2b031 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -38,3 +38,11 @@ void kuep_unlock(void)
{
kuep_update(mfsr(0) & ~SR_NX);
}
+
+void __init setup_kuep(bool disabled)
+{
+ pr_info("Activating Kernel Userspace Execution Prevention\n");
+
+ if (disabled)
+ pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
+}
diff --git a/arch/powerpc/mm/book3s32/mmu.c b/arch/powerpc/mm/book3s32/mmu.c
index 159930351d9f..27061583a010 100644
--- a/arch/powerpc/mm/book3s32/mmu.c
+++ b/arch/powerpc/mm/book3s32/mmu.c
@@ -445,26 +445,6 @@ void __init print_system_hash_info(void)
pr_info("Hash_mask = 0x%lx\n", Hash_mask);
}
-#ifdef CONFIG_PPC_KUEP
-void __init setup_kuep(bool disabled)
-{
- pr_info("Activating Kernel Userspace Execution Prevention\n");
-
- if (disabled)
- pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
-}
-#endif
-
-#ifdef CONFIG_PPC_KUAP
-void __init setup_kuap(bool disabled)
-{
- pr_info("Activating Kernel Userspace Access Protection\n");
-
- if (disabled)
- pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
-}
-#endif
-
void __init early_init_mmu(void)
{
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 01/12] powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
Avoids the #ifdef in mmu.c
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/mm/book3s32/Makefile | 1 +
arch/powerpc/mm/book3s32/kuap.c | 11 +++++++++++
arch/powerpc/mm/book3s32/kuep.c | 8 ++++++++
arch/powerpc/mm/book3s32/mmu.c | 20 --------------------
4 files changed, 20 insertions(+), 20 deletions(-)
create mode 100644 arch/powerpc/mm/book3s32/kuap.c
diff --git a/arch/powerpc/mm/book3s32/Makefile b/arch/powerpc/mm/book3s32/Makefile
index 7f0c8a78ba0c..15f4773643d2 100644
--- a/arch/powerpc/mm/book3s32/Makefile
+++ b/arch/powerpc/mm/book3s32/Makefile
@@ -10,3 +10,4 @@ obj-y += mmu.o mmu_context.o
obj-$(CONFIG_PPC_BOOK3S_603) += nohash_low.o
obj-$(CONFIG_PPC_BOOK3S_604) += hash_low.o tlb.o
obj-$(CONFIG_PPC_KUEP) += kuep.o
+obj-$(CONFIG_PPC_KUAP) += kuap.o
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
new file mode 100644
index 000000000000..1df55392878e
--- /dev/null
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <asm/kup.h>
+
+void __init setup_kuap(bool disabled)
+{
+ pr_info("Activating Kernel Userspace Access Protection\n");
+
+ if (disabled)
+ pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
+}
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 8ed1b8634839..6eafe7b2b031 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -38,3 +38,11 @@ void kuep_unlock(void)
{
kuep_update(mfsr(0) & ~SR_NX);
}
+
+void __init setup_kuep(bool disabled)
+{
+ pr_info("Activating Kernel Userspace Execution Prevention\n");
+
+ if (disabled)
+ pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
+}
diff --git a/arch/powerpc/mm/book3s32/mmu.c b/arch/powerpc/mm/book3s32/mmu.c
index 159930351d9f..27061583a010 100644
--- a/arch/powerpc/mm/book3s32/mmu.c
+++ b/arch/powerpc/mm/book3s32/mmu.c
@@ -445,26 +445,6 @@ void __init print_system_hash_info(void)
pr_info("Hash_mask = 0x%lx\n", Hash_mask);
}
-#ifdef CONFIG_PPC_KUEP
-void __init setup_kuep(bool disabled)
-{
- pr_info("Activating Kernel Userspace Execution Prevention\n");
-
- if (disabled)
- pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
-}
-#endif
-
-#ifdef CONFIG_PPC_KUAP
-void __init setup_kuap(bool disabled)
-{
- pr_info("Activating Kernel Userspace Access Protection\n");
-
- if (disabled)
- pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
-}
-#endif
-
void __init early_init_mmu(void)
{
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 02/12] powerpc/32s: Refactor update of user segment registers
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
KUEP implements the update of user segment registers.
Move it into mmu-hash.h in order to use it from other places.
And inline kuep_lock() and kuep_unlock(). Inlining kuep_lock() is
important for system_call_exception(), otherwise system_call_exception()
has to save into stack the system call parameters that are used just
after, and doing that takes more instructions than kuep_lock() itself.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 21 +++++++++++
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 27 ++++++++++++++
arch/powerpc/include/asm/kup.h | 5 +--
arch/powerpc/mm/book3s32/kuep.c | 37 -------------------
4 files changed, 49 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index c1f7c2e625a6..83aa0dde50d6 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,27 @@
#ifndef __ASSEMBLY__
+static __always_inline bool kuep_is_disabled(void)
+{
+ return !IS_ENABLED(CONFIG_PPC_KUEP);
+}
+
+static inline void kuep_lock(void)
+{
+ if (kuep_is_disabled())
+ return;
+
+ update_user_segments(mfsr(0) | SR_NX);
+}
+
+static inline void kuep_unlock(void)
+{
+ if (kuep_is_disabled())
+ return;
+
+ update_user_segments(mfsr(0) & ~SR_NX);
+}
+
#ifdef CONFIG_PPC_KUAP
#include <linux/sched.h>
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index b85f8e114a9c..cc0284bbac86 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -102,6 +102,33 @@ extern s32 patch__hash_page_B, patch__hash_page_C;
extern s32 patch__flush_hash_A0, patch__flush_hash_A1, patch__flush_hash_A2;
extern s32 patch__flush_hash_B;
+#include <asm/reg.h>
+#include <asm/task_size_32.h>
+
+#define UPDATE_TWO_USER_SEGMENTS(n) do { \
+ if (TASK_SIZE > ((n) << 28)) \
+ mtsr(val1, (n) << 28); \
+ if (TASK_SIZE > (((n) + 1) << 28)) \
+ mtsr(val2, ((n) + 1) << 28); \
+ val1 = (val1 + 0x222) & 0xf0ffffff; \
+ val2 = (val2 + 0x222) & 0xf0ffffff; \
+} while (0)
+
+static __always_inline void update_user_segments(u32 val)
+{
+ int val1 = val;
+ int val2 = (val + 0x111) & 0xf0ffffff;
+
+ UPDATE_TWO_USER_SEGMENTS(0);
+ UPDATE_TWO_USER_SEGMENTS(2);
+ UPDATE_TWO_USER_SEGMENTS(4);
+ UPDATE_TWO_USER_SEGMENTS(6);
+ UPDATE_TWO_USER_SEGMENTS(8);
+ UPDATE_TWO_USER_SEGMENTS(10);
+ UPDATE_TWO_USER_SEGMENTS(12);
+ UPDATE_TWO_USER_SEGMENTS(14);
+}
+
#endif /* !__ASSEMBLY__ */
/* We happily ignore the smaller BATs on 601, we don't actually use
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index ec96232529ac..4b94d4293777 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -46,10 +46,7 @@ void setup_kuep(bool disabled);
static inline void setup_kuep(bool disabled) { }
#endif /* CONFIG_PPC_KUEP */
-#if defined(CONFIG_PPC_KUEP) && defined(CONFIG_PPC_BOOK3S_32)
-void kuep_lock(void);
-void kuep_unlock(void);
-#else
+#ifndef CONFIG_PPC_BOOK3S_32
static inline void kuep_lock(void) { }
static inline void kuep_unlock(void) { }
#endif
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 6eafe7b2b031..919595f47e25 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -1,43 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
-#include <asm/reg.h>
-#include <asm/task_size_32.h>
-#include <asm/mmu.h>
-
-#define KUEP_UPDATE_TWO_USER_SEGMENTS(n) do { \
- if (TASK_SIZE > ((n) << 28)) \
- mtsr(val1, (n) << 28); \
- if (TASK_SIZE > (((n) + 1) << 28)) \
- mtsr(val2, ((n) + 1) << 28); \
- val1 = (val1 + 0x222) & 0xf0ffffff; \
- val2 = (val2 + 0x222) & 0xf0ffffff; \
-} while (0)
-
-static __always_inline void kuep_update(u32 val)
-{
- int val1 = val;
- int val2 = (val + 0x111) & 0xf0ffffff;
-
- KUEP_UPDATE_TWO_USER_SEGMENTS(0);
- KUEP_UPDATE_TWO_USER_SEGMENTS(2);
- KUEP_UPDATE_TWO_USER_SEGMENTS(4);
- KUEP_UPDATE_TWO_USER_SEGMENTS(6);
- KUEP_UPDATE_TWO_USER_SEGMENTS(8);
- KUEP_UPDATE_TWO_USER_SEGMENTS(10);
- KUEP_UPDATE_TWO_USER_SEGMENTS(12);
- KUEP_UPDATE_TWO_USER_SEGMENTS(14);
-}
-
-void kuep_lock(void)
-{
- kuep_update(mfsr(0) | SR_NX);
-}
-
-void kuep_unlock(void)
-{
- kuep_update(mfsr(0) & ~SR_NX);
-}
void __init setup_kuep(bool disabled)
{
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 02/12] powerpc/32s: Refactor update of user segment registers
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
KUEP implements the update of user segment registers.
Move it into mmu-hash.h in order to use it from other places.
And inline kuep_lock() and kuep_unlock(). Inlining kuep_lock() is
important for system_call_exception(), otherwise system_call_exception()
has to save into stack the system call parameters that are used just
after, and doing that takes more instructions than kuep_lock() itself.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 21 +++++++++++
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 27 ++++++++++++++
arch/powerpc/include/asm/kup.h | 5 +--
arch/powerpc/mm/book3s32/kuep.c | 37 -------------------
4 files changed, 49 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index c1f7c2e625a6..83aa0dde50d6 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,27 @@
#ifndef __ASSEMBLY__
+static __always_inline bool kuep_is_disabled(void)
+{
+ return !IS_ENABLED(CONFIG_PPC_KUEP);
+}
+
+static inline void kuep_lock(void)
+{
+ if (kuep_is_disabled())
+ return;
+
+ update_user_segments(mfsr(0) | SR_NX);
+}
+
+static inline void kuep_unlock(void)
+{
+ if (kuep_is_disabled())
+ return;
+
+ update_user_segments(mfsr(0) & ~SR_NX);
+}
+
#ifdef CONFIG_PPC_KUAP
#include <linux/sched.h>
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index b85f8e114a9c..cc0284bbac86 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -102,6 +102,33 @@ extern s32 patch__hash_page_B, patch__hash_page_C;
extern s32 patch__flush_hash_A0, patch__flush_hash_A1, patch__flush_hash_A2;
extern s32 patch__flush_hash_B;
+#include <asm/reg.h>
+#include <asm/task_size_32.h>
+
+#define UPDATE_TWO_USER_SEGMENTS(n) do { \
+ if (TASK_SIZE > ((n) << 28)) \
+ mtsr(val1, (n) << 28); \
+ if (TASK_SIZE > (((n) + 1) << 28)) \
+ mtsr(val2, ((n) + 1) << 28); \
+ val1 = (val1 + 0x222) & 0xf0ffffff; \
+ val2 = (val2 + 0x222) & 0xf0ffffff; \
+} while (0)
+
+static __always_inline void update_user_segments(u32 val)
+{
+ int val1 = val;
+ int val2 = (val + 0x111) & 0xf0ffffff;
+
+ UPDATE_TWO_USER_SEGMENTS(0);
+ UPDATE_TWO_USER_SEGMENTS(2);
+ UPDATE_TWO_USER_SEGMENTS(4);
+ UPDATE_TWO_USER_SEGMENTS(6);
+ UPDATE_TWO_USER_SEGMENTS(8);
+ UPDATE_TWO_USER_SEGMENTS(10);
+ UPDATE_TWO_USER_SEGMENTS(12);
+ UPDATE_TWO_USER_SEGMENTS(14);
+}
+
#endif /* !__ASSEMBLY__ */
/* We happily ignore the smaller BATs on 601, we don't actually use
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index ec96232529ac..4b94d4293777 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -46,10 +46,7 @@ void setup_kuep(bool disabled);
static inline void setup_kuep(bool disabled) { }
#endif /* CONFIG_PPC_KUEP */
-#if defined(CONFIG_PPC_KUEP) && defined(CONFIG_PPC_BOOK3S_32)
-void kuep_lock(void);
-void kuep_unlock(void);
-#else
+#ifndef CONFIG_PPC_BOOK3S_32
static inline void kuep_lock(void) { }
static inline void kuep_unlock(void) { }
#endif
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 6eafe7b2b031..919595f47e25 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -1,43 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
-#include <asm/reg.h>
-#include <asm/task_size_32.h>
-#include <asm/mmu.h>
-
-#define KUEP_UPDATE_TWO_USER_SEGMENTS(n) do { \
- if (TASK_SIZE > ((n) << 28)) \
- mtsr(val1, (n) << 28); \
- if (TASK_SIZE > (((n) + 1) << 28)) \
- mtsr(val2, ((n) + 1) << 28); \
- val1 = (val1 + 0x222) & 0xf0ffffff; \
- val2 = (val2 + 0x222) & 0xf0ffffff; \
-} while (0)
-
-static __always_inline void kuep_update(u32 val)
-{
- int val1 = val;
- int val2 = (val + 0x111) & 0xf0ffffff;
-
- KUEP_UPDATE_TWO_USER_SEGMENTS(0);
- KUEP_UPDATE_TWO_USER_SEGMENTS(2);
- KUEP_UPDATE_TWO_USER_SEGMENTS(4);
- KUEP_UPDATE_TWO_USER_SEGMENTS(6);
- KUEP_UPDATE_TWO_USER_SEGMENTS(8);
- KUEP_UPDATE_TWO_USER_SEGMENTS(10);
- KUEP_UPDATE_TWO_USER_SEGMENTS(12);
- KUEP_UPDATE_TWO_USER_SEGMENTS(14);
-}
-
-void kuep_lock(void)
-{
- kuep_update(mfsr(0) | SR_NX);
-}
-
-void kuep_unlock(void)
-{
- kuep_update(mfsr(0) & ~SR_NX);
-}
void __init setup_kuep(bool disabled)
{
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 03/12] powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
From: Christophe Leroy <christophe.leroy@c-s.fr>
In order to reuse it in switch_mmu_context(), this
patch moves CTX_TO_VSID() macro into asm/book3s/32/mmu-hash.h
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 10 ++++++++++
arch/powerpc/kvm/book3s_32_mmu_host.c | 3 ---
arch/powerpc/mm/book3s32/mmu_context.c | 13 -------------
3 files changed, 10 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index cc0284bbac86..583388399b1b 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -66,6 +66,16 @@ struct ppc_bat {
#ifndef __ASSEMBLY__
+/*
+ * This macro defines the mapping from contexts to VSIDs (virtual
+ * segment IDs). We use a skew on both the context and the high 4 bits
+ * of the 32-bit virtual address (the "effective segment ID") in order
+ * to spread out the entries in the MMU hash table. Note, if this
+ * function is changed then hash functions will have to be
+ * changed to correspond.
+ */
+#define CTX_TO_VSID(c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff)
+
/*
* Hardware Page Table Entry
* Note that the xpn and x bitfields are used only by processors that
diff --git a/arch/powerpc/kvm/book3s_32_mmu_host.c b/arch/powerpc/kvm/book3s_32_mmu_host.c
index e8e7b2c530d1..4b3a8d80cfa3 100644
--- a/arch/powerpc/kvm/book3s_32_mmu_host.c
+++ b/arch/powerpc/kvm/book3s_32_mmu_host.c
@@ -353,9 +353,6 @@ void kvmppc_mmu_destroy_pr(struct kvm_vcpu *vcpu)
preempt_enable();
}
-/* From mm/mmu_context_hash32.c */
-#define CTX_TO_VSID(c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff)
-
int kvmppc_mmu_init_pr(struct kvm_vcpu *vcpu)
{
struct kvmppc_vcpu_book3s *vcpu3s = to_book3s(vcpu);
diff --git a/arch/powerpc/mm/book3s32/mmu_context.c b/arch/powerpc/mm/book3s32/mmu_context.c
index 218996e40a8e..c949363f315f 100644
--- a/arch/powerpc/mm/book3s32/mmu_context.c
+++ b/arch/powerpc/mm/book3s32/mmu_context.c
@@ -39,19 +39,6 @@
#define LAST_CONTEXT 32767
#define FIRST_CONTEXT 1
-/*
- * This function defines the mapping from contexts to VSIDs (virtual
- * segment IDs). We use a skew on both the context and the high 4 bits
- * of the 32-bit virtual address (the "effective segment ID") in order
- * to spread out the entries in the MMU hash table. Note, if this
- * function is changed then arch/ppc/mm/hashtable.S will have to be
- * changed to correspond.
- *
- *
- * CTX_TO_VSID(ctx, va) (((ctx) * (897 * 16) + ((va) >> 28) * 0x111) \
- * & 0xffffff)
- */
-
static unsigned long next_mmu_context;
static unsigned long context_map[LAST_CONTEXT / BITS_PER_LONG + 1];
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 03/12] powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
From: Christophe Leroy <christophe.leroy@c-s.fr>
In order to reuse it in switch_mmu_context(), this
patch moves CTX_TO_VSID() macro into asm/book3s/32/mmu-hash.h
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 10 ++++++++++
arch/powerpc/kvm/book3s_32_mmu_host.c | 3 ---
arch/powerpc/mm/book3s32/mmu_context.c | 13 -------------
3 files changed, 10 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index cc0284bbac86..583388399b1b 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -66,6 +66,16 @@ struct ppc_bat {
#ifndef __ASSEMBLY__
+/*
+ * This macro defines the mapping from contexts to VSIDs (virtual
+ * segment IDs). We use a skew on both the context and the high 4 bits
+ * of the 32-bit virtual address (the "effective segment ID") in order
+ * to spread out the entries in the MMU hash table. Note, if this
+ * function is changed then hash functions will have to be
+ * changed to correspond.
+ */
+#define CTX_TO_VSID(c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff)
+
/*
* Hardware Page Table Entry
* Note that the xpn and x bitfields are used only by processors that
diff --git a/arch/powerpc/kvm/book3s_32_mmu_host.c b/arch/powerpc/kvm/book3s_32_mmu_host.c
index e8e7b2c530d1..4b3a8d80cfa3 100644
--- a/arch/powerpc/kvm/book3s_32_mmu_host.c
+++ b/arch/powerpc/kvm/book3s_32_mmu_host.c
@@ -353,9 +353,6 @@ void kvmppc_mmu_destroy_pr(struct kvm_vcpu *vcpu)
preempt_enable();
}
-/* From mm/mmu_context_hash32.c */
-#define CTX_TO_VSID(c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff)
-
int kvmppc_mmu_init_pr(struct kvm_vcpu *vcpu)
{
struct kvmppc_vcpu_book3s *vcpu3s = to_book3s(vcpu);
diff --git a/arch/powerpc/mm/book3s32/mmu_context.c b/arch/powerpc/mm/book3s32/mmu_context.c
index 218996e40a8e..c949363f315f 100644
--- a/arch/powerpc/mm/book3s32/mmu_context.c
+++ b/arch/powerpc/mm/book3s32/mmu_context.c
@@ -39,19 +39,6 @@
#define LAST_CONTEXT 32767
#define FIRST_CONTEXT 1
-/*
- * This function defines the mapping from contexts to VSIDs (virtual
- * segment IDs). We use a skew on both the context and the high 4 bits
- * of the 32-bit virtual address (the "effective segment ID") in order
- * to spread out the entries in the MMU hash table. Note, if this
- * function is changed then arch/ppc/mm/hashtable.S will have to be
- * changed to correspond.
- *
- *
- * CTX_TO_VSID(ctx, va) (((ctx) * (897 * 16) + ((va) >> 28) * 0x111) \
- * & 0xffffff)
- */
-
static unsigned long next_mmu_context;
static unsigned long context_map[LAST_CONTEXT / BITS_PER_LONG + 1];
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 04/12] powerpc/32s: Convert switch_mmu_context() to C
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
switch_mmu_context() does things that can easily be done in C.
For updating user segments, we have update_user_segments().
As mentionned in commit b5efec00b671 ("powerpc/32s: Move KUEP
locking/unlocking in C"), update_user_segments() has the loop
unrolled which is a significant performance gain.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 5 ++
arch/powerpc/kernel/asm-offsets.c | 5 --
arch/powerpc/kernel/head_book3s_32.S | 58 ------------------------
arch/powerpc/mm/book3s32/mmu_context.c | 35 ++++++++++++++
4 files changed, 40 insertions(+), 63 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 83aa0dde50d6..618ffc8e4ee9 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,11 @@
#ifndef __ASSEMBLY__
+static __always_inline bool kuap_is_disabled(void)
+{
+ return !IS_ENABLED(CONFIG_PPC_KUAP);
+}
+
static __always_inline bool kuep_is_disabled(void)
{
return !IS_ENABLED(CONFIG_PPC_KUEP);
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 0480f4006e0c..14d0e646cb7a 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -86,7 +86,6 @@ int main(void)
OFFSET(PACA_CANARY, paca_struct, canary);
#endif
#endif
- OFFSET(MMCONTEXTID, mm_struct, context.id);
#ifdef CONFIG_PPC32
#ifdef CONFIG_PPC_RTAS
OFFSET(RTAS_SP, thread_struct, rtas_sp);
@@ -325,10 +324,6 @@ int main(void)
#endif
#endif
-#ifndef CONFIG_PPC64
- OFFSET(MM_PGD, mm_struct, pgd);
-#endif /* ! CONFIG_PPC64 */
-
/* About the CPU features table */
OFFSET(CPU_SPEC_FEATURES, cpu_spec, cpu_features);
OFFSET(CPU_SPEC_SETUP, cpu_spec, cpu_setup);
diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S
index 326262030279..db0e2dc25f86 100644
--- a/arch/powerpc/kernel/head_book3s_32.S
+++ b/arch/powerpc/kernel/head_book3s_32.S
@@ -1033,58 +1033,6 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_HPTE_TABLE)
mtspr SPRN_SRR1,r4
rfi
-/*
- * void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next);
- *
- * Set up the segment registers for a new context.
- */
-_ENTRY(switch_mmu_context)
- lwz r3,MMCONTEXTID(r4)
- cmpwi cr0,r3,0
- blt- 4f
- mulli r3,r3,897 /* multiply context by skew factor */
- rlwinm r3,r3,4,8,27 /* VSID = (context & 0xfffff) << 4 */
-#ifdef CONFIG_PPC_KUEP
- oris r3, r3, SR_NX@h /* Set Nx */
-#endif
-#ifdef CONFIG_PPC_KUAP
- oris r3, r3, SR_KS@h /* Set Ks */
-#endif
- li r0,NUM_USER_SEGMENTS
- mtctr r0
-
-#ifdef CONFIG_BDI_SWITCH
- /* Context switch the PTE pointer for the Abatron BDI2000.
- * The PGDIR is passed as second argument.
- */
- lwz r4, MM_PGD(r4)
- lis r5, abatron_pteptrs@ha
- stw r4, abatron_pteptrs@l + 0x4(r5)
-#endif
-BEGIN_MMU_FTR_SECTION
-#ifndef CONFIG_BDI_SWITCH
- lwz r4, MM_PGD(r4)
-#endif
- tophys(r4, r4)
- rlwinm r4, r4, 4, 0xffff01ff
- mtspr SPRN_SDR1, r4
-END_MMU_FTR_SECTION_IFCLR(MMU_FTR_HPTE_TABLE)
- li r4,0
- isync
-3:
- mtsrin r3,r4
- addi r3,r3,0x111 /* next VSID */
- rlwinm r3,r3,0,8,3 /* clear out any overflow from VSID field */
- addis r4,r4,0x1000 /* address of next segment */
- bdnz 3b
- sync
- isync
- blr
-4: trap
- EMIT_BUG_ENTRY 4b,__FILE__,__LINE__,0
- blr
-EXPORT_SYMBOL(switch_mmu_context)
-
/*
* An undocumented "feature" of 604e requires that the v bit
* be cleared before changing BAT values.
@@ -1282,9 +1230,3 @@ EXPORT_SYMBOL(empty_zero_page)
.globl swapper_pg_dir
swapper_pg_dir:
.space PGD_TABLE_SIZE
-
-/* Room for two PTE pointers, usually the kernel and current user pointers
- * to their respective root page table.
- */
-abatron_pteptrs:
- .space 8
diff --git a/arch/powerpc/mm/book3s32/mmu_context.c b/arch/powerpc/mm/book3s32/mmu_context.c
index c949363f315f..e2708e387dc3 100644
--- a/arch/powerpc/mm/book3s32/mmu_context.c
+++ b/arch/powerpc/mm/book3s32/mmu_context.c
@@ -23,6 +23,12 @@
#include <asm/mmu_context.h>
+/*
+ * Room for two PTE pointers, usually the kernel and current user pointers
+ * to their respective root page table.
+ */
+void *abatron_pteptrs[2];
+
/*
* On 32-bit PowerPC 6xx/7xx/7xxx CPUs, we use a set of 16 VSIDs
* (virtual segment identifiers) for each context. Although the
@@ -98,3 +104,32 @@ void __init mmu_context_init(void)
context_map[0] = (1 << FIRST_CONTEXT) - 1;
next_mmu_context = FIRST_CONTEXT;
}
+
+void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk)
+{
+ long id = next->context.id;
+ unsigned long val;
+
+ if (id < 0)
+ panic("mm_struct %p has no context ID", next);
+
+ isync();
+
+ val = CTX_TO_VSID(id, 0);
+ if (!kuep_is_disabled())
+ val |= SR_NX;
+ if (!kuap_is_disabled())
+ val |= SR_KS;
+
+ update_user_segments(val);
+
+ if (IS_ENABLED(CONFIG_BDI_SWITCH))
+ abatron_pteptrs[1] = next->pgd;
+
+ if (!mmu_has_feature(MMU_FTR_HPTE_TABLE))
+ mtspr(SPRN_SDR1, rol32(__pa(next->pgd), 4) & 0xffff01ff);
+
+ mb(); /* sync */
+ isync();
+}
+EXPORT_SYMBOL(switch_mmu_context);
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 04/12] powerpc/32s: Convert switch_mmu_context() to C
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
switch_mmu_context() does things that can easily be done in C.
For updating user segments, we have update_user_segments().
As mentionned in commit b5efec00b671 ("powerpc/32s: Move KUEP
locking/unlocking in C"), update_user_segments() has the loop
unrolled which is a significant performance gain.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 5 ++
arch/powerpc/kernel/asm-offsets.c | 5 --
arch/powerpc/kernel/head_book3s_32.S | 58 ------------------------
arch/powerpc/mm/book3s32/mmu_context.c | 35 ++++++++++++++
4 files changed, 40 insertions(+), 63 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 83aa0dde50d6..618ffc8e4ee9 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,11 @@
#ifndef __ASSEMBLY__
+static __always_inline bool kuap_is_disabled(void)
+{
+ return !IS_ENABLED(CONFIG_PPC_KUAP);
+}
+
static __always_inline bool kuep_is_disabled(void)
{
return !IS_ENABLED(CONFIG_PPC_KUEP);
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 0480f4006e0c..14d0e646cb7a 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -86,7 +86,6 @@ int main(void)
OFFSET(PACA_CANARY, paca_struct, canary);
#endif
#endif
- OFFSET(MMCONTEXTID, mm_struct, context.id);
#ifdef CONFIG_PPC32
#ifdef CONFIG_PPC_RTAS
OFFSET(RTAS_SP, thread_struct, rtas_sp);
@@ -325,10 +324,6 @@ int main(void)
#endif
#endif
-#ifndef CONFIG_PPC64
- OFFSET(MM_PGD, mm_struct, pgd);
-#endif /* ! CONFIG_PPC64 */
-
/* About the CPU features table */
OFFSET(CPU_SPEC_FEATURES, cpu_spec, cpu_features);
OFFSET(CPU_SPEC_SETUP, cpu_spec, cpu_setup);
diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S
index 326262030279..db0e2dc25f86 100644
--- a/arch/powerpc/kernel/head_book3s_32.S
+++ b/arch/powerpc/kernel/head_book3s_32.S
@@ -1033,58 +1033,6 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_HPTE_TABLE)
mtspr SPRN_SRR1,r4
rfi
-/*
- * void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next);
- *
- * Set up the segment registers for a new context.
- */
-_ENTRY(switch_mmu_context)
- lwz r3,MMCONTEXTID(r4)
- cmpwi cr0,r3,0
- blt- 4f
- mulli r3,r3,897 /* multiply context by skew factor */
- rlwinm r3,r3,4,8,27 /* VSID = (context & 0xfffff) << 4 */
-#ifdef CONFIG_PPC_KUEP
- oris r3, r3, SR_NX@h /* Set Nx */
-#endif
-#ifdef CONFIG_PPC_KUAP
- oris r3, r3, SR_KS@h /* Set Ks */
-#endif
- li r0,NUM_USER_SEGMENTS
- mtctr r0
-
-#ifdef CONFIG_BDI_SWITCH
- /* Context switch the PTE pointer for the Abatron BDI2000.
- * The PGDIR is passed as second argument.
- */
- lwz r4, MM_PGD(r4)
- lis r5, abatron_pteptrs@ha
- stw r4, abatron_pteptrs@l + 0x4(r5)
-#endif
-BEGIN_MMU_FTR_SECTION
-#ifndef CONFIG_BDI_SWITCH
- lwz r4, MM_PGD(r4)
-#endif
- tophys(r4, r4)
- rlwinm r4, r4, 4, 0xffff01ff
- mtspr SPRN_SDR1, r4
-END_MMU_FTR_SECTION_IFCLR(MMU_FTR_HPTE_TABLE)
- li r4,0
- isync
-3:
- mtsrin r3,r4
- addi r3,r3,0x111 /* next VSID */
- rlwinm r3,r3,0,8,3 /* clear out any overflow from VSID field */
- addis r4,r4,0x1000 /* address of next segment */
- bdnz 3b
- sync
- isync
- blr
-4: trap
- EMIT_BUG_ENTRY 4b,__FILE__,__LINE__,0
- blr
-EXPORT_SYMBOL(switch_mmu_context)
-
/*
* An undocumented "feature" of 604e requires that the v bit
* be cleared before changing BAT values.
@@ -1282,9 +1230,3 @@ EXPORT_SYMBOL(empty_zero_page)
.globl swapper_pg_dir
swapper_pg_dir:
.space PGD_TABLE_SIZE
-
-/* Room for two PTE pointers, usually the kernel and current user pointers
- * to their respective root page table.
- */
-abatron_pteptrs:
- .space 8
diff --git a/arch/powerpc/mm/book3s32/mmu_context.c b/arch/powerpc/mm/book3s32/mmu_context.c
index c949363f315f..e2708e387dc3 100644
--- a/arch/powerpc/mm/book3s32/mmu_context.c
+++ b/arch/powerpc/mm/book3s32/mmu_context.c
@@ -23,6 +23,12 @@
#include <asm/mmu_context.h>
+/*
+ * Room for two PTE pointers, usually the kernel and current user pointers
+ * to their respective root page table.
+ */
+void *abatron_pteptrs[2];
+
/*
* On 32-bit PowerPC 6xx/7xx/7xxx CPUs, we use a set of 16 VSIDs
* (virtual segment identifiers) for each context. Although the
@@ -98,3 +104,32 @@ void __init mmu_context_init(void)
context_map[0] = (1 << FIRST_CONTEXT) - 1;
next_mmu_context = FIRST_CONTEXT;
}
+
+void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk)
+{
+ long id = next->context.id;
+ unsigned long val;
+
+ if (id < 0)
+ panic("mm_struct %p has no context ID", next);
+
+ isync();
+
+ val = CTX_TO_VSID(id, 0);
+ if (!kuep_is_disabled())
+ val |= SR_NX;
+ if (!kuap_is_disabled())
+ val |= SR_KS;
+
+ update_user_segments(val);
+
+ if (IS_ENABLED(CONFIG_BDI_SWITCH))
+ abatron_pteptrs[1] = next->pgd;
+
+ if (!mmu_has_feature(MMU_FTR_HPTE_TABLE))
+ mtspr(SPRN_SDR1, rol32(__pa(next->pgd), 4) & 0xffff01ff);
+
+ mb(); /* sync */
+ isync();
+}
+EXPORT_SYMBOL(switch_mmu_context);
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 05/12] powerpc/32s: Simplify calculation of segment register content
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
segment register has VSID on bits 8-31.
Bits 4-7 are reserved, there is no requirement to set them to 0.
VSIDs are calculated from VSID of SR0 by adding 0x111.
Even with highest possible VSID which would be 0xFFFFF0,
adding 16 times 0x111 results in 0x1001100.
So, the reserved bits are never overflowed, no need to clear
the reserved bits after each calculation.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 42 ++++++++++---------
1 file changed, 23 insertions(+), 19 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index 583388399b1b..f5be185cbdf8 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -115,28 +115,32 @@ extern s32 patch__flush_hash_B;
#include <asm/reg.h>
#include <asm/task_size_32.h>
-#define UPDATE_TWO_USER_SEGMENTS(n) do { \
- if (TASK_SIZE > ((n) << 28)) \
- mtsr(val1, (n) << 28); \
- if (TASK_SIZE > (((n) + 1) << 28)) \
- mtsr(val2, ((n) + 1) << 28); \
- val1 = (val1 + 0x222) & 0xf0ffffff; \
- val2 = (val2 + 0x222) & 0xf0ffffff; \
-} while (0)
+static __always_inline void update_user_segment(u32 n, u32 val)
+{
+ if (n << 28 < TASK_SIZE)
+ mtsr(val + n * 0x111, n << 28);
+}
static __always_inline void update_user_segments(u32 val)
{
- int val1 = val;
- int val2 = (val + 0x111) & 0xf0ffffff;
-
- UPDATE_TWO_USER_SEGMENTS(0);
- UPDATE_TWO_USER_SEGMENTS(2);
- UPDATE_TWO_USER_SEGMENTS(4);
- UPDATE_TWO_USER_SEGMENTS(6);
- UPDATE_TWO_USER_SEGMENTS(8);
- UPDATE_TWO_USER_SEGMENTS(10);
- UPDATE_TWO_USER_SEGMENTS(12);
- UPDATE_TWO_USER_SEGMENTS(14);
+ val &= 0xf0ffffff;
+
+ update_user_segment(0, val);
+ update_user_segment(1, val);
+ update_user_segment(2, val);
+ update_user_segment(3, val);
+ update_user_segment(4, val);
+ update_user_segment(5, val);
+ update_user_segment(6, val);
+ update_user_segment(7, val);
+ update_user_segment(8, val);
+ update_user_segment(9, val);
+ update_user_segment(10, val);
+ update_user_segment(11, val);
+ update_user_segment(12, val);
+ update_user_segment(13, val);
+ update_user_segment(14, val);
+ update_user_segment(15, val);
}
#endif /* !__ASSEMBLY__ */
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 05/12] powerpc/32s: Simplify calculation of segment register content
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
segment register has VSID on bits 8-31.
Bits 4-7 are reserved, there is no requirement to set them to 0.
VSIDs are calculated from VSID of SR0 by adding 0x111.
Even with highest possible VSID which would be 0xFFFFF0,
adding 16 times 0x111 results in 0x1001100.
So, the reserved bits are never overflowed, no need to clear
the reserved bits after each calculation.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 42 ++++++++++---------
1 file changed, 23 insertions(+), 19 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/mmu-hash.h b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
index 583388399b1b..f5be185cbdf8 100644
--- a/arch/powerpc/include/asm/book3s/32/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/32/mmu-hash.h
@@ -115,28 +115,32 @@ extern s32 patch__flush_hash_B;
#include <asm/reg.h>
#include <asm/task_size_32.h>
-#define UPDATE_TWO_USER_SEGMENTS(n) do { \
- if (TASK_SIZE > ((n) << 28)) \
- mtsr(val1, (n) << 28); \
- if (TASK_SIZE > (((n) + 1) << 28)) \
- mtsr(val2, ((n) + 1) << 28); \
- val1 = (val1 + 0x222) & 0xf0ffffff; \
- val2 = (val2 + 0x222) & 0xf0ffffff; \
-} while (0)
+static __always_inline void update_user_segment(u32 n, u32 val)
+{
+ if (n << 28 < TASK_SIZE)
+ mtsr(val + n * 0x111, n << 28);
+}
static __always_inline void update_user_segments(u32 val)
{
- int val1 = val;
- int val2 = (val + 0x111) & 0xf0ffffff;
-
- UPDATE_TWO_USER_SEGMENTS(0);
- UPDATE_TWO_USER_SEGMENTS(2);
- UPDATE_TWO_USER_SEGMENTS(4);
- UPDATE_TWO_USER_SEGMENTS(6);
- UPDATE_TWO_USER_SEGMENTS(8);
- UPDATE_TWO_USER_SEGMENTS(10);
- UPDATE_TWO_USER_SEGMENTS(12);
- UPDATE_TWO_USER_SEGMENTS(14);
+ val &= 0xf0ffffff;
+
+ update_user_segment(0, val);
+ update_user_segment(1, val);
+ update_user_segment(2, val);
+ update_user_segment(3, val);
+ update_user_segment(4, val);
+ update_user_segment(5, val);
+ update_user_segment(6, val);
+ update_user_segment(7, val);
+ update_user_segment(8, val);
+ update_user_segment(9, val);
+ update_user_segment(10, val);
+ update_user_segment(11, val);
+ update_user_segment(12, val);
+ update_user_segment(13, val);
+ update_user_segment(14, val);
+ update_user_segment(15, val);
}
#endif /* !__ASSEMBLY__ */
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 06/12] powerpc/32s: Initialise KUAP and KUEP in C
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
In order to selectively activate KUAP and KUEP in a following patch,
perform KUAP and KUEP initialisation in C.
Unlike PPC64, PPC32 doesn't have an early_setup_secondary(),
so do it in start_secondary().
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/head_book3s_32.S | 6 ------
arch/powerpc/kernel/smp.c | 4 ++++
arch/powerpc/mm/book3s32/kuap.c | 6 ++++++
arch/powerpc/mm/book3s32/kuep.c | 6 ++++++
4 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S
index db0e2dc25f86..0b82672ff7a6 100644
--- a/arch/powerpc/kernel/head_book3s_32.S
+++ b/arch/powerpc/kernel/head_book3s_32.S
@@ -934,12 +934,6 @@ _GLOBAL(load_segment_registers)
li r0, NUM_USER_SEGMENTS /* load up user segment register values */
mtctr r0 /* for context 0 */
li r3, 0 /* Kp = 0, Ks = 0, VSID = 0 */
-#ifdef CONFIG_PPC_KUEP
- oris r3, r3, SR_NX@h /* Set Nx */
-#endif
-#ifdef CONFIG_PPC_KUAP
- oris r3, r3, SR_KS@h /* Set Ks */
-#endif
li r4, 0
3: mtsrin r3, r4
addi r3, r3, 0x111 /* increment VSID */
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 2e05c783440a..820ae31e0231 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1541,6 +1541,10 @@ void start_secondary(void *unused)
{
unsigned int cpu = raw_smp_processor_id();
+ /* PPC64 calls setup_kup() in early_setup_secondary() */
+ if (IS_ENABLED(CONFIG_PPC32))
+ setup_kup();
+
mmgrab(&init_mm);
current->active_mm = &init_mm;
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index 1df55392878e..5533ed92ab3d 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -1,9 +1,15 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
+#include <asm/smp.h>
void __init setup_kuap(bool disabled)
{
+ kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+
+ if (smp_processor_id() != boot_cpuid)
+ return;
+
pr_info("Activating Kernel Userspace Access Protection\n");
if (disabled)
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 919595f47e25..3147e2edcf63 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -1,9 +1,15 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
+#include <asm/smp.h>
void __init setup_kuep(bool disabled)
{
+ kuep_lock();
+
+ if (smp_processor_id() != boot_cpuid)
+ return;
+
pr_info("Activating Kernel Userspace Execution Prevention\n");
if (disabled)
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 06/12] powerpc/32s: Initialise KUAP and KUEP in C
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In order to selectively activate KUAP and KUEP in a following patch,
perform KUAP and KUEP initialisation in C.
Unlike PPC64, PPC32 doesn't have an early_setup_secondary(),
so do it in start_secondary().
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/head_book3s_32.S | 6 ------
arch/powerpc/kernel/smp.c | 4 ++++
arch/powerpc/mm/book3s32/kuap.c | 6 ++++++
arch/powerpc/mm/book3s32/kuep.c | 6 ++++++
4 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S
index db0e2dc25f86..0b82672ff7a6 100644
--- a/arch/powerpc/kernel/head_book3s_32.S
+++ b/arch/powerpc/kernel/head_book3s_32.S
@@ -934,12 +934,6 @@ _GLOBAL(load_segment_registers)
li r0, NUM_USER_SEGMENTS /* load up user segment register values */
mtctr r0 /* for context 0 */
li r3, 0 /* Kp = 0, Ks = 0, VSID = 0 */
-#ifdef CONFIG_PPC_KUEP
- oris r3, r3, SR_NX@h /* Set Nx */
-#endif
-#ifdef CONFIG_PPC_KUAP
- oris r3, r3, SR_KS@h /* Set Ks */
-#endif
li r4, 0
3: mtsrin r3, r4
addi r3, r3, 0x111 /* increment VSID */
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 2e05c783440a..820ae31e0231 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1541,6 +1541,10 @@ void start_secondary(void *unused)
{
unsigned int cpu = raw_smp_processor_id();
+ /* PPC64 calls setup_kup() in early_setup_secondary() */
+ if (IS_ENABLED(CONFIG_PPC32))
+ setup_kup();
+
mmgrab(&init_mm);
current->active_mm = &init_mm;
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index 1df55392878e..5533ed92ab3d 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -1,9 +1,15 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
+#include <asm/smp.h>
void __init setup_kuap(bool disabled)
{
+ kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+
+ if (smp_processor_id() != boot_cpuid)
+ return;
+
pr_info("Activating Kernel Userspace Access Protection\n");
if (disabled)
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 919595f47e25..3147e2edcf63 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -1,9 +1,15 @@
// SPDX-License-Identifier: GPL-2.0-or-later
#include <asm/kup.h>
+#include <asm/smp.h>
void __init setup_kuep(bool disabled)
{
+ kuep_lock();
+
+ if (smp_processor_id() != boot_cpuid)
+ return;
+
pr_info("Activating Kernel Userspace Execution Prevention\n");
if (disabled)
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 07/12] powerpc/32s: Allow disabling KUEP at boot time
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
PPC64 uses MMU features to enable/disable KUEP at boot time.
But feature fixups are applied way too early on PPC32.
Now that all KUEP related actions are in C following the
conversion of KUEP initial setup and context switch in C,
static branches can be used to enable/disable KUEP.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 6 +++++-
arch/powerpc/mm/book3s32/kuep.c | 11 +++++++----
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 618ffc8e4ee9..2854d970dabe 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,10 @@
#ifndef __ASSEMBLY__
+#include <linux/jump_label.h>
+
+extern struct static_key_false disable_kuep_key;
+
static __always_inline bool kuap_is_disabled(void)
{
return !IS_ENABLED(CONFIG_PPC_KUAP);
@@ -14,7 +18,7 @@ static __always_inline bool kuap_is_disabled(void)
static __always_inline bool kuep_is_disabled(void)
{
- return !IS_ENABLED(CONFIG_PPC_KUEP);
+ return !IS_ENABLED(CONFIG_PPC_KUEP) || static_branch_unlikely(&disable_kuep_key);
}
static inline void kuep_lock(void)
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 3147e2edcf63..3f6eb6e23fca 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -3,15 +3,18 @@
#include <asm/kup.h>
#include <asm/smp.h>
+struct static_key_false disable_kuep_key;
+
void __init setup_kuep(bool disabled)
{
- kuep_lock();
+ if (!disabled)
+ kuep_lock();
if (smp_processor_id() != boot_cpuid)
return;
- pr_info("Activating Kernel Userspace Execution Prevention\n");
-
if (disabled)
- pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
+ static_branch_enable(&disable_kuep_key);
+ else
+ pr_info("Activating Kernel Userspace Execution Prevention\n");
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 07/12] powerpc/32s: Allow disabling KUEP at boot time
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
PPC64 uses MMU features to enable/disable KUEP at boot time.
But feature fixups are applied way too early on PPC32.
Now that all KUEP related actions are in C following the
conversion of KUEP initial setup and context switch in C,
static branches can be used to enable/disable KUEP.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 6 +++++-
arch/powerpc/mm/book3s32/kuep.c | 11 +++++++----
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 618ffc8e4ee9..2854d970dabe 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -7,6 +7,10 @@
#ifndef __ASSEMBLY__
+#include <linux/jump_label.h>
+
+extern struct static_key_false disable_kuep_key;
+
static __always_inline bool kuap_is_disabled(void)
{
return !IS_ENABLED(CONFIG_PPC_KUAP);
@@ -14,7 +18,7 @@ static __always_inline bool kuap_is_disabled(void)
static __always_inline bool kuep_is_disabled(void)
{
- return !IS_ENABLED(CONFIG_PPC_KUEP);
+ return !IS_ENABLED(CONFIG_PPC_KUEP) || static_branch_unlikely(&disable_kuep_key);
}
static inline void kuep_lock(void)
diff --git a/arch/powerpc/mm/book3s32/kuep.c b/arch/powerpc/mm/book3s32/kuep.c
index 3147e2edcf63..3f6eb6e23fca 100644
--- a/arch/powerpc/mm/book3s32/kuep.c
+++ b/arch/powerpc/mm/book3s32/kuep.c
@@ -3,15 +3,18 @@
#include <asm/kup.h>
#include <asm/smp.h>
+struct static_key_false disable_kuep_key;
+
void __init setup_kuep(bool disabled)
{
- kuep_lock();
+ if (!disabled)
+ kuep_lock();
if (smp_processor_id() != boot_cpuid)
return;
- pr_info("Activating Kernel Userspace Execution Prevention\n");
-
if (disabled)
- pr_warn("KUEP cannot be disabled yet on 6xx when compiled in\n");
+ static_branch_enable(&disable_kuep_key);
+ else
+ pr_info("Activating Kernel Userspace Execution Prevention\n");
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 08/12] powerpc/32s: Allow disabling KUAP at boot time
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
PPC64 uses MMU features to enable/disable KUAP at boot time.
But feature fixups are applied way too early on PPC32.
Now that all KUAP related actions are in C following the
conversion of KUAP initial setup and context switch in C,
static branches can be used to enable/disable KUAP.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 27 +++++++++++++++++++++++-
arch/powerpc/mm/book3s32/kuap.c | 11 ++++++----
2 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 2854d970dabe..68fbe28c6d7e 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -9,11 +9,12 @@
#include <linux/jump_label.h>
+extern struct static_key_false disable_kuap_key;
extern struct static_key_false disable_kuep_key;
static __always_inline bool kuap_is_disabled(void)
{
- return !IS_ENABLED(CONFIG_PPC_KUAP);
+ return !IS_ENABLED(CONFIG_PPC_KUAP) || static_branch_unlikely(&disable_kuap_key);
}
static __always_inline bool kuep_is_disabled(void)
@@ -62,6 +63,9 @@ static inline void kuap_save_and_lock(struct pt_regs *regs)
u32 addr = kuap & 0xf0000000;
u32 end = kuap << 28;
+ if (kuap_is_disabled())
+ return;
+
regs->kuap = kuap;
if (unlikely(!kuap))
return;
@@ -79,6 +83,9 @@ static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)
u32 addr = regs->kuap & 0xf0000000;
u32 end = regs->kuap << 28;
+ if (kuap_is_disabled())
+ return;
+
current->thread.kuap = regs->kuap;
if (unlikely(regs->kuap == kuap))
@@ -91,6 +98,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)
{
unsigned long kuap = current->thread.kuap;
+ if (kuap_is_disabled())
+ return 0;
+
WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != 0);
return kuap;
@@ -106,6 +116,9 @@ static __always_inline void allow_user_access(void __user *to, const void __user
{
u32 addr, end;
+ if (kuap_is_disabled())
+ return;
+
BUILD_BUG_ON(!__builtin_constant_p(dir));
BUILD_BUG_ON(dir & ~KUAP_READ_WRITE);
@@ -128,6 +141,9 @@ static __always_inline void prevent_user_access(void __user *to, const void __us
{
u32 addr, end;
+ if (kuap_is_disabled())
+ return;
+
BUILD_BUG_ON(!__builtin_constant_p(dir));
if (dir & KUAP_CURRENT_WRITE) {
@@ -159,6 +175,9 @@ static inline unsigned long prevent_user_access_return(void)
unsigned long end = flags << 28;
void __user *to = (__force void __user *)addr;
+ if (kuap_is_disabled())
+ return 0;
+
if (flags)
prevent_user_access(to, to, end - addr, KUAP_READ_WRITE);
@@ -171,6 +190,9 @@ static inline void restore_user_access(unsigned long flags)
unsigned long end = flags << 28;
void __user *to = (__force void __user *)addr;
+ if (kuap_is_disabled())
+ return;
+
if (flags)
allow_user_access(to, to, end - addr, KUAP_READ_WRITE);
}
@@ -181,6 +203,9 @@ bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)
unsigned long begin = regs->kuap & 0xf0000000;
unsigned long end = regs->kuap << 28;
+ if (kuap_is_disabled())
+ return false;
+
return is_write && (address < begin || address >= end);
}
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index 5533ed92ab3d..a4ce6cdc28e5 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -3,15 +3,18 @@
#include <asm/kup.h>
#include <asm/smp.h>
+struct static_key_false disable_kuap_key;
+
void __init setup_kuap(bool disabled)
{
- kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+ if (!disabled)
+ kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
if (smp_processor_id() != boot_cpuid)
return;
- pr_info("Activating Kernel Userspace Access Protection\n");
-
if (disabled)
- pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
+ static_branch_enable(&disable_kuap_key);
+ else
+ pr_info("Activating Kernel Userspace Access Protection\n");
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 08/12] powerpc/32s: Allow disabling KUAP at boot time
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
PPC64 uses MMU features to enable/disable KUAP at boot time.
But feature fixups are applied way too early on PPC32.
Now that all KUAP related actions are in C following the
conversion of KUAP initial setup and context switch in C,
static branches can be used to enable/disable KUAP.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 27 +++++++++++++++++++++++-
arch/powerpc/mm/book3s32/kuap.c | 11 ++++++----
2 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 2854d970dabe..68fbe28c6d7e 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -9,11 +9,12 @@
#include <linux/jump_label.h>
+extern struct static_key_false disable_kuap_key;
extern struct static_key_false disable_kuep_key;
static __always_inline bool kuap_is_disabled(void)
{
- return !IS_ENABLED(CONFIG_PPC_KUAP);
+ return !IS_ENABLED(CONFIG_PPC_KUAP) || static_branch_unlikely(&disable_kuap_key);
}
static __always_inline bool kuep_is_disabled(void)
@@ -62,6 +63,9 @@ static inline void kuap_save_and_lock(struct pt_regs *regs)
u32 addr = kuap & 0xf0000000;
u32 end = kuap << 28;
+ if (kuap_is_disabled())
+ return;
+
regs->kuap = kuap;
if (unlikely(!kuap))
return;
@@ -79,6 +83,9 @@ static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)
u32 addr = regs->kuap & 0xf0000000;
u32 end = regs->kuap << 28;
+ if (kuap_is_disabled())
+ return;
+
current->thread.kuap = regs->kuap;
if (unlikely(regs->kuap == kuap))
@@ -91,6 +98,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)
{
unsigned long kuap = current->thread.kuap;
+ if (kuap_is_disabled())
+ return 0;
+
WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != 0);
return kuap;
@@ -106,6 +116,9 @@ static __always_inline void allow_user_access(void __user *to, const void __user
{
u32 addr, end;
+ if (kuap_is_disabled())
+ return;
+
BUILD_BUG_ON(!__builtin_constant_p(dir));
BUILD_BUG_ON(dir & ~KUAP_READ_WRITE);
@@ -128,6 +141,9 @@ static __always_inline void prevent_user_access(void __user *to, const void __us
{
u32 addr, end;
+ if (kuap_is_disabled())
+ return;
+
BUILD_BUG_ON(!__builtin_constant_p(dir));
if (dir & KUAP_CURRENT_WRITE) {
@@ -159,6 +175,9 @@ static inline unsigned long prevent_user_access_return(void)
unsigned long end = flags << 28;
void __user *to = (__force void __user *)addr;
+ if (kuap_is_disabled())
+ return 0;
+
if (flags)
prevent_user_access(to, to, end - addr, KUAP_READ_WRITE);
@@ -171,6 +190,9 @@ static inline void restore_user_access(unsigned long flags)
unsigned long end = flags << 28;
void __user *to = (__force void __user *)addr;
+ if (kuap_is_disabled())
+ return;
+
if (flags)
allow_user_access(to, to, end - addr, KUAP_READ_WRITE);
}
@@ -181,6 +203,9 @@ bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)
unsigned long begin = regs->kuap & 0xf0000000;
unsigned long end = regs->kuap << 28;
+ if (kuap_is_disabled())
+ return false;
+
return is_write && (address < begin || address >= end);
}
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index 5533ed92ab3d..a4ce6cdc28e5 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -3,15 +3,18 @@
#include <asm/kup.h>
#include <asm/smp.h>
+struct static_key_false disable_kuap_key;
+
void __init setup_kuap(bool disabled)
{
- kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+ if (!disabled)
+ kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
if (smp_processor_id() != boot_cpuid)
return;
- pr_info("Activating Kernel Userspace Access Protection\n");
-
if (disabled)
- pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
+ static_branch_enable(&disable_kuap_key);
+ else
+ pr_info("Activating Kernel Userspace Access Protection\n");
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 09/12] powerpc/32s: Rework Kernel Userspace Access Protection
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
On book3s/32, KUAP is provided by toggling Ks bit in segment registers.
One segment register addresses 256M of virtual memory.
At the time being, KUAP implements a complex logic to apply the
unlock/lock on the exact number of segments covering the user range
to access, with saving the boundaries of the range of segments in
a member of thread struct.
But most if not all user accesses are within a single segment.
Rework KUAP with a different approach:
- Open only one segment, the one corresponding to the starting
address of the range to be accessed.
- If a second segment is involved, it will generate a page fault. The
segment will then be open by the page fault handler.
The kuap member of thread struct will now contain:
- The start address of the current on going user access, that will be
used to know which segment to lock at the end of the user access.
- ~0 when no user access is open
- ~1 when additionnal segments are opened by a page fault.
Then, at lock time
- When only one segment is open, close it.
- When several segments are open, close all user segments.
Almost 100% of the time, only one segment will be involved.
In interrupts, inline the function that unlock/lock all segments,
because not inlining them implies a lot of register save/restore.
With the patch, writing value 128 in userspace in perf_copy_attr() is
done with 16 instructions:
3890: 93 82 04 dc stw r28,1244(r2)
3894: 7d 20 e5 26 mfsrin r9,r28
3898: 55 29 00 80 rlwinm r9,r9,0,2,0
389c: 7d 20 e1 e4 mtsrin r9,r28
38a0: 4c 00 01 2c isync
38a4: 39 20 00 80 li r9,128
38a8: 91 3c 00 00 stw r9,0(r28)
38ac: 81 42 04 dc lwz r10,1244(r2)
38b0: 39 00 ff ff li r8,-1
38b4: 91 02 04 dc stw r8,1244(r2)
38b8: 2c 0a ff fe cmpwi r10,-2
38bc: 41 82 00 88 beq 3944 <perf_copy_attr+0x36c>
38c0: 7d 20 55 26 mfsrin r9,r10
38c4: 65 29 40 00 oris r9,r9,16384
38c8: 7d 20 51 e4 mtsrin r9,r10
38cc: 4c 00 01 2c isync
...
3944: 48 00 00 01 bl 3944 <perf_copy_attr+0x36c>
3944: R_PPC_REL24 kuap_lock_all_ool
Before the patch it was 118 instructions. In reality only 42 are
executed in most cases, but GCC is not able to see that a properly
aligned user access cannot involve more than one segment.
5060: 39 1d 00 04 addi r8,r29,4
5064: 3d 20 b0 00 lis r9,-20480
5068: 7c 08 48 40 cmplw r8,r9
506c: 40 81 00 08 ble 5074 <perf_copy_attr+0x2cc>
5070: 3d 00 b0 00 lis r8,-20480
5074: 39 28 ff ff addi r9,r8,-1
5078: 57 aa 00 06 rlwinm r10,r29,0,0,3
507c: 55 29 27 3e rlwinm r9,r9,4,28,31
5080: 39 29 00 01 addi r9,r9,1
5084: 7d 29 53 78 or r9,r9,r10
5088: 91 22 04 dc stw r9,1244(r2)
508c: 7d 20 ed 26 mfsrin r9,r29
5090: 55 29 00 80 rlwinm r9,r9,0,2,0
5094: 7c 08 50 40 cmplw r8,r10
5098: 40 81 00 c0 ble 5158 <perf_copy_attr+0x3b0>
509c: 7d 46 50 f8 not r6,r10
50a0: 7c c6 42 14 add r6,r6,r8
50a4: 54 c6 27 be rlwinm r6,r6,4,30,31
50a8: 7d 20 51 e4 mtsrin r9,r10
50ac: 3c ea 10 00 addis r7,r10,4096
50b0: 39 29 01 11 addi r9,r9,273
50b4: 7f 88 38 40 cmplw cr7,r8,r7
50b8: 55 29 02 06 rlwinm r9,r9,0,8,3
50bc: 40 9d 00 9c ble cr7,5158 <perf_copy_attr+0x3b0>
50c0: 2f 86 00 00 cmpwi cr7,r6,0
50c4: 41 9e 00 4c beq cr7,5110 <perf_copy_attr+0x368>
50c8: 2f 86 00 01 cmpwi cr7,r6,1
50cc: 41 9e 00 2c beq cr7,50f8 <perf_copy_attr+0x350>
50d0: 2f 86 00 02 cmpwi cr7,r6,2
50d4: 41 9e 00 14 beq cr7,50e8 <perf_copy_attr+0x340>
50d8: 7d 20 39 e4 mtsrin r9,r7
50dc: 39 29 01 11 addi r9,r9,273
50e0: 3c e7 10 00 addis r7,r7,4096
50e4: 55 29 02 06 rlwinm r9,r9,0,8,3
50e8: 7d 20 39 e4 mtsrin r9,r7
50ec: 39 29 01 11 addi r9,r9,273
50f0: 3c e7 10 00 addis r7,r7,4096
50f4: 55 29 02 06 rlwinm r9,r9,0,8,3
50f8: 7d 20 39 e4 mtsrin r9,r7
50fc: 3c e7 10 00 addis r7,r7,4096
5100: 39 29 01 11 addi r9,r9,273
5104: 7f 88 38 40 cmplw cr7,r8,r7
5108: 55 29 02 06 rlwinm r9,r9,0,8,3
510c: 40 9d 00 4c ble cr7,5158 <perf_copy_attr+0x3b0>
5110: 7d 20 39 e4 mtsrin r9,r7
5114: 39 29 01 11 addi r9,r9,273
5118: 3c c7 10 00 addis r6,r7,4096
511c: 55 29 02 06 rlwinm r9,r9,0,8,3
5120: 7d 20 31 e4 mtsrin r9,r6
5124: 39 29 01 11 addi r9,r9,273
5128: 3c c6 10 00 addis r6,r6,4096
512c: 55 29 02 06 rlwinm r9,r9,0,8,3
5130: 7d 20 31 e4 mtsrin r9,r6
5134: 39 29 01 11 addi r9,r9,273
5138: 3c c7 30 00 addis r6,r7,12288
513c: 55 29 02 06 rlwinm r9,r9,0,8,3
5140: 7d 20 31 e4 mtsrin r9,r6
5144: 3c e7 40 00 addis r7,r7,16384
5148: 39 29 01 11 addi r9,r9,273
514c: 7f 88 38 40 cmplw cr7,r8,r7
5150: 55 29 02 06 rlwinm r9,r9,0,8,3
5154: 41 9d ff bc bgt cr7,5110 <perf_copy_attr+0x368>
5158: 4c 00 01 2c isync
515c: 39 20 00 80 li r9,128
5160: 91 3d 00 00 stw r9,0(r29)
5164: 38 e0 00 00 li r7,0
5168: 90 e2 04 dc stw r7,1244(r2)
516c: 7d 20 ed 26 mfsrin r9,r29
5170: 65 29 40 00 oris r9,r9,16384
5174: 40 81 00 c0 ble 5234 <perf_copy_attr+0x48c>
5178: 7d 47 50 f8 not r7,r10
517c: 7c e7 42 14 add r7,r7,r8
5180: 54 e7 27 be rlwinm r7,r7,4,30,31
5184: 7d 20 51 e4 mtsrin r9,r10
5188: 3d 4a 10 00 addis r10,r10,4096
518c: 39 29 01 11 addi r9,r9,273
5190: 7c 08 50 40 cmplw r8,r10
5194: 55 29 02 06 rlwinm r9,r9,0,8,3
5198: 40 81 00 9c ble 5234 <perf_copy_attr+0x48c>
519c: 2c 07 00 00 cmpwi r7,0
51a0: 41 82 00 4c beq 51ec <perf_copy_attr+0x444>
51a4: 2c 07 00 01 cmpwi r7,1
51a8: 41 82 00 2c beq 51d4 <perf_copy_attr+0x42c>
51ac: 2c 07 00 02 cmpwi r7,2
51b0: 41 82 00 14 beq 51c4 <perf_copy_attr+0x41c>
51b4: 7d 20 51 e4 mtsrin r9,r10
51b8: 39 29 01 11 addi r9,r9,273
51bc: 3d 4a 10 00 addis r10,r10,4096
51c0: 55 29 02 06 rlwinm r9,r9,0,8,3
51c4: 7d 20 51 e4 mtsrin r9,r10
51c8: 39 29 01 11 addi r9,r9,273
51cc: 3d 4a 10 00 addis r10,r10,4096
51d0: 55 29 02 06 rlwinm r9,r9,0,8,3
51d4: 7d 20 51 e4 mtsrin r9,r10
51d8: 3d 4a 10 00 addis r10,r10,4096
51dc: 39 29 01 11 addi r9,r9,273
51e0: 7c 08 50 40 cmplw r8,r10
51e4: 55 29 02 06 rlwinm r9,r9,0,8,3
51e8: 40 81 00 4c ble 5234 <perf_copy_attr+0x48c>
51ec: 7d 20 51 e4 mtsrin r9,r10
51f0: 39 29 01 11 addi r9,r9,273
51f4: 3c ea 10 00 addis r7,r10,4096
51f8: 55 29 02 06 rlwinm r9,r9,0,8,3
51fc: 7d 20 39 e4 mtsrin r9,r7
5200: 39 29 01 11 addi r9,r9,273
5204: 3c e7 10 00 addis r7,r7,4096
5208: 55 29 02 06 rlwinm r9,r9,0,8,3
520c: 7d 20 39 e4 mtsrin r9,r7
5210: 39 29 01 11 addi r9,r9,273
5214: 3c ea 30 00 addis r7,r10,12288
5218: 55 29 02 06 rlwinm r9,r9,0,8,3
521c: 7d 20 39 e4 mtsrin r9,r7
5220: 3d 4a 40 00 addis r10,r10,16384
5224: 39 29 01 11 addi r9,r9,273
5228: 7c 08 50 40 cmplw r8,r10
522c: 55 29 02 06 rlwinm r9,r9,0,8,3
5230: 41 81 ff bc bgt 51ec <perf_copy_attr+0x444>
5234: 4c 00 01 2c isync
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 150 ++++++++++++-----------
arch/powerpc/include/asm/processor.h | 10 +-
arch/powerpc/kernel/process.c | 3 +
arch/powerpc/mm/book3s32/kuap.c | 12 +-
4 files changed, 102 insertions(+), 73 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 68fbe28c6d7e..bac7edae64bf 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -42,36 +42,69 @@ static inline void kuep_unlock(void)
#include <linux/sched.h>
-static inline void kuap_update_sr(u32 sr, u32 addr, u32 end)
-{
- addr &= 0xf0000000; /* align addr to start of segment */
- barrier(); /* make sure thread.kuap is updated before playing with SRs */
- for (;;) {
- mtsr(sr, addr);
- addr += 0x10000000; /* address of next segment */
- if (addr >= end)
- break;
- sr += 0x111; /* next VSID */
- sr &= 0xf0ffffff; /* clear VSID overflow */
- }
+#define KUAP_NONE (~0UL)
+#define KUAP_ALL (~1UL)
+
+static inline void kuap_lock_one(unsigned long addr)
+{
+ mtsr(mfsr(addr) | SR_KS, addr);
+ isync(); /* Context sync required after mtsr() */
+}
+
+static inline void kuap_unlock_one(unsigned long addr)
+{
+ mtsr(mfsr(addr) & ~SR_KS, addr);
+ isync(); /* Context sync required after mtsr() */
+}
+
+static inline void kuap_lock_all(void)
+{
+ update_user_segments(mfsr(0) | SR_KS);
isync(); /* Context sync required after mtsr() */
}
+static inline void kuap_unlock_all(void)
+{
+ update_user_segments(mfsr(0) & ~SR_KS);
+ isync(); /* Context sync required after mtsr() */
+}
+
+void kuap_lock_all_ool(void);
+void kuap_unlock_all_ool(void);
+
+static inline void kuap_lock(unsigned long addr, bool ool)
+{
+ if (likely(addr != KUAP_ALL))
+ kuap_lock_one(addr);
+ else if (!ool)
+ kuap_lock_all();
+ else
+ kuap_lock_all_ool();
+}
+
+static inline void kuap_unlock(unsigned long addr, bool ool)
+{
+ if (likely(addr != KUAP_ALL))
+ kuap_unlock_one(addr);
+ else if (!ool)
+ kuap_unlock_all();
+ else
+ kuap_unlock_all_ool();
+}
+
static inline void kuap_save_and_lock(struct pt_regs *regs)
{
unsigned long kuap = current->thread.kuap;
- u32 addr = kuap & 0xf0000000;
- u32 end = kuap << 28;
if (kuap_is_disabled())
return;
regs->kuap = kuap;
- if (unlikely(!kuap))
+ if (unlikely(kuap == KUAP_NONE))
return;
- current->thread.kuap = 0;
- kuap_update_sr(mfsr(addr) | SR_KS, addr, end); /* Set Ks */
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(kuap, false);
}
static inline void kuap_user_restore(struct pt_regs *regs)
@@ -80,18 +113,12 @@ static inline void kuap_user_restore(struct pt_regs *regs)
static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)
{
- u32 addr = regs->kuap & 0xf0000000;
- u32 end = regs->kuap << 28;
-
if (kuap_is_disabled())
return;
current->thread.kuap = regs->kuap;
- if (unlikely(regs->kuap == kuap))
- return;
-
- kuap_update_sr(mfsr(addr) & ~SR_KS, addr, end); /* Clear Ks */
+ kuap_unlock(regs->kuap, false);
}
static inline unsigned long kuap_get_and_assert_locked(void)
@@ -99,9 +126,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)
unsigned long kuap = current->thread.kuap;
if (kuap_is_disabled())
- return 0;
+ return KUAP_NONE;
- WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != 0);
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != KUAP_NONE);
return kuap;
}
@@ -114,8 +141,6 @@ static inline void kuap_assert_locked(void)
static __always_inline void allow_user_access(void __user *to, const void __user *from,
u32 size, unsigned long dir)
{
- u32 addr, end;
-
if (kuap_is_disabled())
return;
@@ -125,88 +150,71 @@ static __always_inline void allow_user_access(void __user *to, const void __user
if (!(dir & KUAP_WRITE))
return;
- addr = (__force u32)to;
-
- if (unlikely(addr >= TASK_SIZE || !size))
- return;
-
- end = min(addr + size, TASK_SIZE);
-
- current->thread.kuap = (addr & 0xf0000000) | ((((end - 1) >> 28) + 1) & 0xf);
- kuap_update_sr(mfsr(addr) & ~SR_KS, addr, end); /* Clear Ks */
+ current->thread.kuap = (__force u32)to;
+ kuap_unlock_one((__force u32)to);
}
static __always_inline void prevent_user_access(void __user *to, const void __user *from,
u32 size, unsigned long dir)
{
- u32 addr, end;
+ u32 kuap = current->thread.kuap;
if (kuap_is_disabled())
return;
BUILD_BUG_ON(!__builtin_constant_p(dir));
- if (dir & KUAP_CURRENT_WRITE) {
- u32 kuap = current->thread.kuap;
-
- if (unlikely(!kuap))
- return;
-
- addr = kuap & 0xf0000000;
- end = kuap << 28;
- } else if (dir & KUAP_WRITE) {
- addr = (__force u32)to;
- end = min(addr + size, TASK_SIZE);
-
- if (unlikely(addr >= TASK_SIZE || !size))
- return;
- } else {
+ if (!(dir & KUAP_WRITE))
return;
- }
- current->thread.kuap = 0;
- kuap_update_sr(mfsr(addr) | SR_KS, addr, end); /* set Ks */
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(kuap, true);
}
static inline unsigned long prevent_user_access_return(void)
{
unsigned long flags = current->thread.kuap;
- unsigned long addr = flags & 0xf0000000;
- unsigned long end = flags << 28;
- void __user *to = (__force void __user *)addr;
if (kuap_is_disabled())
- return 0;
+ return KUAP_NONE;
- if (flags)
- prevent_user_access(to, to, end - addr, KUAP_READ_WRITE);
+ if (flags != KUAP_NONE) {
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(flags, true);
+ }
return flags;
}
static inline void restore_user_access(unsigned long flags)
{
- unsigned long addr = flags & 0xf0000000;
- unsigned long end = flags << 28;
- void __user *to = (__force void __user *)addr;
-
if (kuap_is_disabled())
return;
- if (flags)
- allow_user_access(to, to, end - addr, KUAP_READ_WRITE);
+ if (flags != KUAP_NONE) {
+ current->thread.kuap = flags;
+ kuap_unlock(flags, true);
+ }
}
static inline bool
bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)
{
- unsigned long begin = regs->kuap & 0xf0000000;
- unsigned long end = regs->kuap << 28;
+ unsigned long kuap = regs->kuap;
if (kuap_is_disabled())
return false;
- return is_write && (address < begin || address >= end);
+ if (!is_write || kuap == KUAP_ALL)
+ return false;
+ if (kuap == KUAP_NONE)
+ return true;
+
+ /* If faulting address doesn't match unlocked segment, unlock all */
+ if ((kuap ^ address) & 0xf0000000)
+ regs->kuap = KUAP_ALL;
+
+ return false;
}
#endif /* CONFIG_PPC_KUAP */
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 7bf8a15af224..aeb1a35163d1 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -276,7 +276,15 @@ struct thread_struct {
#define SPEFSCR_INIT
#endif
-#ifdef CONFIG_PPC32
+#if defined(CONFIG_PPC_BOOK3S_32) && defined(CONFIG_PPC_KUAP)
+#define INIT_THREAD { \
+ .ksp = INIT_SP, \
+ .pgdir = swapper_pg_dir, \
+ .kuap = ~0UL, /* KUAP_NONE */ \
+ .fpexc_mode = MSR_FE0 | MSR_FE1, \
+ SPEFSCR_INIT \
+}
+#elif defined(CONFIG_PPC32)
#define INIT_THREAD { \
.ksp = INIT_SP, \
.pgdir = swapper_pg_dir, \
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 89e34aa273e2..c6d21604457d 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1736,6 +1736,9 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
#ifdef CONFIG_ALTIVEC
p->thread.vr_save_area = NULL;
#endif
+#if defined(CONFIG_PPC_BOOK3S_32) && defined(CONFIG_PPC_KUAP)
+ p->thread.kuap = KUAP_NONE;
+#endif
setup_ksp_vsid(p, sp);
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index a4ce6cdc28e5..1a764367039e 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -5,10 +5,20 @@
struct static_key_false disable_kuap_key;
+void kuap_lock_all_ool(void)
+{
+ kuap_lock_all();
+}
+
+void kuap_unlock_all_ool(void)
+{
+ kuap_unlock_all();
+}
+
void __init setup_kuap(bool disabled)
{
if (!disabled)
- kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+ kuap_lock_all_ool();
if (smp_processor_id() != boot_cpuid)
return;
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 09/12] powerpc/32s: Rework Kernel Userspace Access Protection
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
On book3s/32, KUAP is provided by toggling Ks bit in segment registers.
One segment register addresses 256M of virtual memory.
At the time being, KUAP implements a complex logic to apply the
unlock/lock on the exact number of segments covering the user range
to access, with saving the boundaries of the range of segments in
a member of thread struct.
But most if not all user accesses are within a single segment.
Rework KUAP with a different approach:
- Open only one segment, the one corresponding to the starting
address of the range to be accessed.
- If a second segment is involved, it will generate a page fault. The
segment will then be open by the page fault handler.
The kuap member of thread struct will now contain:
- The start address of the current on going user access, that will be
used to know which segment to lock at the end of the user access.
- ~0 when no user access is open
- ~1 when additionnal segments are opened by a page fault.
Then, at lock time
- When only one segment is open, close it.
- When several segments are open, close all user segments.
Almost 100% of the time, only one segment will be involved.
In interrupts, inline the function that unlock/lock all segments,
because not inlining them implies a lot of register save/restore.
With the patch, writing value 128 in userspace in perf_copy_attr() is
done with 16 instructions:
3890: 93 82 04 dc stw r28,1244(r2)
3894: 7d 20 e5 26 mfsrin r9,r28
3898: 55 29 00 80 rlwinm r9,r9,0,2,0
389c: 7d 20 e1 e4 mtsrin r9,r28
38a0: 4c 00 01 2c isync
38a4: 39 20 00 80 li r9,128
38a8: 91 3c 00 00 stw r9,0(r28)
38ac: 81 42 04 dc lwz r10,1244(r2)
38b0: 39 00 ff ff li r8,-1
38b4: 91 02 04 dc stw r8,1244(r2)
38b8: 2c 0a ff fe cmpwi r10,-2
38bc: 41 82 00 88 beq 3944 <perf_copy_attr+0x36c>
38c0: 7d 20 55 26 mfsrin r9,r10
38c4: 65 29 40 00 oris r9,r9,16384
38c8: 7d 20 51 e4 mtsrin r9,r10
38cc: 4c 00 01 2c isync
...
3944: 48 00 00 01 bl 3944 <perf_copy_attr+0x36c>
3944: R_PPC_REL24 kuap_lock_all_ool
Before the patch it was 118 instructions. In reality only 42 are
executed in most cases, but GCC is not able to see that a properly
aligned user access cannot involve more than one segment.
5060: 39 1d 00 04 addi r8,r29,4
5064: 3d 20 b0 00 lis r9,-20480
5068: 7c 08 48 40 cmplw r8,r9
506c: 40 81 00 08 ble 5074 <perf_copy_attr+0x2cc>
5070: 3d 00 b0 00 lis r8,-20480
5074: 39 28 ff ff addi r9,r8,-1
5078: 57 aa 00 06 rlwinm r10,r29,0,0,3
507c: 55 29 27 3e rlwinm r9,r9,4,28,31
5080: 39 29 00 01 addi r9,r9,1
5084: 7d 29 53 78 or r9,r9,r10
5088: 91 22 04 dc stw r9,1244(r2)
508c: 7d 20 ed 26 mfsrin r9,r29
5090: 55 29 00 80 rlwinm r9,r9,0,2,0
5094: 7c 08 50 40 cmplw r8,r10
5098: 40 81 00 c0 ble 5158 <perf_copy_attr+0x3b0>
509c: 7d 46 50 f8 not r6,r10
50a0: 7c c6 42 14 add r6,r6,r8
50a4: 54 c6 27 be rlwinm r6,r6,4,30,31
50a8: 7d 20 51 e4 mtsrin r9,r10
50ac: 3c ea 10 00 addis r7,r10,4096
50b0: 39 29 01 11 addi r9,r9,273
50b4: 7f 88 38 40 cmplw cr7,r8,r7
50b8: 55 29 02 06 rlwinm r9,r9,0,8,3
50bc: 40 9d 00 9c ble cr7,5158 <perf_copy_attr+0x3b0>
50c0: 2f 86 00 00 cmpwi cr7,r6,0
50c4: 41 9e 00 4c beq cr7,5110 <perf_copy_attr+0x368>
50c8: 2f 86 00 01 cmpwi cr7,r6,1
50cc: 41 9e 00 2c beq cr7,50f8 <perf_copy_attr+0x350>
50d0: 2f 86 00 02 cmpwi cr7,r6,2
50d4: 41 9e 00 14 beq cr7,50e8 <perf_copy_attr+0x340>
50d8: 7d 20 39 e4 mtsrin r9,r7
50dc: 39 29 01 11 addi r9,r9,273
50e0: 3c e7 10 00 addis r7,r7,4096
50e4: 55 29 02 06 rlwinm r9,r9,0,8,3
50e8: 7d 20 39 e4 mtsrin r9,r7
50ec: 39 29 01 11 addi r9,r9,273
50f0: 3c e7 10 00 addis r7,r7,4096
50f4: 55 29 02 06 rlwinm r9,r9,0,8,3
50f8: 7d 20 39 e4 mtsrin r9,r7
50fc: 3c e7 10 00 addis r7,r7,4096
5100: 39 29 01 11 addi r9,r9,273
5104: 7f 88 38 40 cmplw cr7,r8,r7
5108: 55 29 02 06 rlwinm r9,r9,0,8,3
510c: 40 9d 00 4c ble cr7,5158 <perf_copy_attr+0x3b0>
5110: 7d 20 39 e4 mtsrin r9,r7
5114: 39 29 01 11 addi r9,r9,273
5118: 3c c7 10 00 addis r6,r7,4096
511c: 55 29 02 06 rlwinm r9,r9,0,8,3
5120: 7d 20 31 e4 mtsrin r9,r6
5124: 39 29 01 11 addi r9,r9,273
5128: 3c c6 10 00 addis r6,r6,4096
512c: 55 29 02 06 rlwinm r9,r9,0,8,3
5130: 7d 20 31 e4 mtsrin r9,r6
5134: 39 29 01 11 addi r9,r9,273
5138: 3c c7 30 00 addis r6,r7,12288
513c: 55 29 02 06 rlwinm r9,r9,0,8,3
5140: 7d 20 31 e4 mtsrin r9,r6
5144: 3c e7 40 00 addis r7,r7,16384
5148: 39 29 01 11 addi r9,r9,273
514c: 7f 88 38 40 cmplw cr7,r8,r7
5150: 55 29 02 06 rlwinm r9,r9,0,8,3
5154: 41 9d ff bc bgt cr7,5110 <perf_copy_attr+0x368>
5158: 4c 00 01 2c isync
515c: 39 20 00 80 li r9,128
5160: 91 3d 00 00 stw r9,0(r29)
5164: 38 e0 00 00 li r7,0
5168: 90 e2 04 dc stw r7,1244(r2)
516c: 7d 20 ed 26 mfsrin r9,r29
5170: 65 29 40 00 oris r9,r9,16384
5174: 40 81 00 c0 ble 5234 <perf_copy_attr+0x48c>
5178: 7d 47 50 f8 not r7,r10
517c: 7c e7 42 14 add r7,r7,r8
5180: 54 e7 27 be rlwinm r7,r7,4,30,31
5184: 7d 20 51 e4 mtsrin r9,r10
5188: 3d 4a 10 00 addis r10,r10,4096
518c: 39 29 01 11 addi r9,r9,273
5190: 7c 08 50 40 cmplw r8,r10
5194: 55 29 02 06 rlwinm r9,r9,0,8,3
5198: 40 81 00 9c ble 5234 <perf_copy_attr+0x48c>
519c: 2c 07 00 00 cmpwi r7,0
51a0: 41 82 00 4c beq 51ec <perf_copy_attr+0x444>
51a4: 2c 07 00 01 cmpwi r7,1
51a8: 41 82 00 2c beq 51d4 <perf_copy_attr+0x42c>
51ac: 2c 07 00 02 cmpwi r7,2
51b0: 41 82 00 14 beq 51c4 <perf_copy_attr+0x41c>
51b4: 7d 20 51 e4 mtsrin r9,r10
51b8: 39 29 01 11 addi r9,r9,273
51bc: 3d 4a 10 00 addis r10,r10,4096
51c0: 55 29 02 06 rlwinm r9,r9,0,8,3
51c4: 7d 20 51 e4 mtsrin r9,r10
51c8: 39 29 01 11 addi r9,r9,273
51cc: 3d 4a 10 00 addis r10,r10,4096
51d0: 55 29 02 06 rlwinm r9,r9,0,8,3
51d4: 7d 20 51 e4 mtsrin r9,r10
51d8: 3d 4a 10 00 addis r10,r10,4096
51dc: 39 29 01 11 addi r9,r9,273
51e0: 7c 08 50 40 cmplw r8,r10
51e4: 55 29 02 06 rlwinm r9,r9,0,8,3
51e8: 40 81 00 4c ble 5234 <perf_copy_attr+0x48c>
51ec: 7d 20 51 e4 mtsrin r9,r10
51f0: 39 29 01 11 addi r9,r9,273
51f4: 3c ea 10 00 addis r7,r10,4096
51f8: 55 29 02 06 rlwinm r9,r9,0,8,3
51fc: 7d 20 39 e4 mtsrin r9,r7
5200: 39 29 01 11 addi r9,r9,273
5204: 3c e7 10 00 addis r7,r7,4096
5208: 55 29 02 06 rlwinm r9,r9,0,8,3
520c: 7d 20 39 e4 mtsrin r9,r7
5210: 39 29 01 11 addi r9,r9,273
5214: 3c ea 30 00 addis r7,r10,12288
5218: 55 29 02 06 rlwinm r9,r9,0,8,3
521c: 7d 20 39 e4 mtsrin r9,r7
5220: 3d 4a 40 00 addis r10,r10,16384
5224: 39 29 01 11 addi r9,r9,273
5228: 7c 08 50 40 cmplw r8,r10
522c: 55 29 02 06 rlwinm r9,r9,0,8,3
5230: 41 81 ff bc bgt 51ec <perf_copy_attr+0x444>
5234: 4c 00 01 2c isync
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 150 ++++++++++++-----------
arch/powerpc/include/asm/processor.h | 10 +-
arch/powerpc/kernel/process.c | 3 +
arch/powerpc/mm/book3s32/kuap.c | 12 +-
4 files changed, 102 insertions(+), 73 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index 68fbe28c6d7e..bac7edae64bf 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -42,36 +42,69 @@ static inline void kuep_unlock(void)
#include <linux/sched.h>
-static inline void kuap_update_sr(u32 sr, u32 addr, u32 end)
-{
- addr &= 0xf0000000; /* align addr to start of segment */
- barrier(); /* make sure thread.kuap is updated before playing with SRs */
- for (;;) {
- mtsr(sr, addr);
- addr += 0x10000000; /* address of next segment */
- if (addr >= end)
- break;
- sr += 0x111; /* next VSID */
- sr &= 0xf0ffffff; /* clear VSID overflow */
- }
+#define KUAP_NONE (~0UL)
+#define KUAP_ALL (~1UL)
+
+static inline void kuap_lock_one(unsigned long addr)
+{
+ mtsr(mfsr(addr) | SR_KS, addr);
+ isync(); /* Context sync required after mtsr() */
+}
+
+static inline void kuap_unlock_one(unsigned long addr)
+{
+ mtsr(mfsr(addr) & ~SR_KS, addr);
+ isync(); /* Context sync required after mtsr() */
+}
+
+static inline void kuap_lock_all(void)
+{
+ update_user_segments(mfsr(0) | SR_KS);
isync(); /* Context sync required after mtsr() */
}
+static inline void kuap_unlock_all(void)
+{
+ update_user_segments(mfsr(0) & ~SR_KS);
+ isync(); /* Context sync required after mtsr() */
+}
+
+void kuap_lock_all_ool(void);
+void kuap_unlock_all_ool(void);
+
+static inline void kuap_lock(unsigned long addr, bool ool)
+{
+ if (likely(addr != KUAP_ALL))
+ kuap_lock_one(addr);
+ else if (!ool)
+ kuap_lock_all();
+ else
+ kuap_lock_all_ool();
+}
+
+static inline void kuap_unlock(unsigned long addr, bool ool)
+{
+ if (likely(addr != KUAP_ALL))
+ kuap_unlock_one(addr);
+ else if (!ool)
+ kuap_unlock_all();
+ else
+ kuap_unlock_all_ool();
+}
+
static inline void kuap_save_and_lock(struct pt_regs *regs)
{
unsigned long kuap = current->thread.kuap;
- u32 addr = kuap & 0xf0000000;
- u32 end = kuap << 28;
if (kuap_is_disabled())
return;
regs->kuap = kuap;
- if (unlikely(!kuap))
+ if (unlikely(kuap == KUAP_NONE))
return;
- current->thread.kuap = 0;
- kuap_update_sr(mfsr(addr) | SR_KS, addr, end); /* Set Ks */
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(kuap, false);
}
static inline void kuap_user_restore(struct pt_regs *regs)
@@ -80,18 +113,12 @@ static inline void kuap_user_restore(struct pt_regs *regs)
static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)
{
- u32 addr = regs->kuap & 0xf0000000;
- u32 end = regs->kuap << 28;
-
if (kuap_is_disabled())
return;
current->thread.kuap = regs->kuap;
- if (unlikely(regs->kuap == kuap))
- return;
-
- kuap_update_sr(mfsr(addr) & ~SR_KS, addr, end); /* Clear Ks */
+ kuap_unlock(regs->kuap, false);
}
static inline unsigned long kuap_get_and_assert_locked(void)
@@ -99,9 +126,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)
unsigned long kuap = current->thread.kuap;
if (kuap_is_disabled())
- return 0;
+ return KUAP_NONE;
- WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != 0);
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != KUAP_NONE);
return kuap;
}
@@ -114,8 +141,6 @@ static inline void kuap_assert_locked(void)
static __always_inline void allow_user_access(void __user *to, const void __user *from,
u32 size, unsigned long dir)
{
- u32 addr, end;
-
if (kuap_is_disabled())
return;
@@ -125,88 +150,71 @@ static __always_inline void allow_user_access(void __user *to, const void __user
if (!(dir & KUAP_WRITE))
return;
- addr = (__force u32)to;
-
- if (unlikely(addr >= TASK_SIZE || !size))
- return;
-
- end = min(addr + size, TASK_SIZE);
-
- current->thread.kuap = (addr & 0xf0000000) | ((((end - 1) >> 28) + 1) & 0xf);
- kuap_update_sr(mfsr(addr) & ~SR_KS, addr, end); /* Clear Ks */
+ current->thread.kuap = (__force u32)to;
+ kuap_unlock_one((__force u32)to);
}
static __always_inline void prevent_user_access(void __user *to, const void __user *from,
u32 size, unsigned long dir)
{
- u32 addr, end;
+ u32 kuap = current->thread.kuap;
if (kuap_is_disabled())
return;
BUILD_BUG_ON(!__builtin_constant_p(dir));
- if (dir & KUAP_CURRENT_WRITE) {
- u32 kuap = current->thread.kuap;
-
- if (unlikely(!kuap))
- return;
-
- addr = kuap & 0xf0000000;
- end = kuap << 28;
- } else if (dir & KUAP_WRITE) {
- addr = (__force u32)to;
- end = min(addr + size, TASK_SIZE);
-
- if (unlikely(addr >= TASK_SIZE || !size))
- return;
- } else {
+ if (!(dir & KUAP_WRITE))
return;
- }
- current->thread.kuap = 0;
- kuap_update_sr(mfsr(addr) | SR_KS, addr, end); /* set Ks */
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(kuap, true);
}
static inline unsigned long prevent_user_access_return(void)
{
unsigned long flags = current->thread.kuap;
- unsigned long addr = flags & 0xf0000000;
- unsigned long end = flags << 28;
- void __user *to = (__force void __user *)addr;
if (kuap_is_disabled())
- return 0;
+ return KUAP_NONE;
- if (flags)
- prevent_user_access(to, to, end - addr, KUAP_READ_WRITE);
+ if (flags != KUAP_NONE) {
+ current->thread.kuap = KUAP_NONE;
+ kuap_lock(flags, true);
+ }
return flags;
}
static inline void restore_user_access(unsigned long flags)
{
- unsigned long addr = flags & 0xf0000000;
- unsigned long end = flags << 28;
- void __user *to = (__force void __user *)addr;
-
if (kuap_is_disabled())
return;
- if (flags)
- allow_user_access(to, to, end - addr, KUAP_READ_WRITE);
+ if (flags != KUAP_NONE) {
+ current->thread.kuap = flags;
+ kuap_unlock(flags, true);
+ }
}
static inline bool
bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)
{
- unsigned long begin = regs->kuap & 0xf0000000;
- unsigned long end = regs->kuap << 28;
+ unsigned long kuap = regs->kuap;
if (kuap_is_disabled())
return false;
- return is_write && (address < begin || address >= end);
+ if (!is_write || kuap == KUAP_ALL)
+ return false;
+ if (kuap == KUAP_NONE)
+ return true;
+
+ /* If faulting address doesn't match unlocked segment, unlock all */
+ if ((kuap ^ address) & 0xf0000000)
+ regs->kuap = KUAP_ALL;
+
+ return false;
}
#endif /* CONFIG_PPC_KUAP */
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 7bf8a15af224..aeb1a35163d1 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -276,7 +276,15 @@ struct thread_struct {
#define SPEFSCR_INIT
#endif
-#ifdef CONFIG_PPC32
+#if defined(CONFIG_PPC_BOOK3S_32) && defined(CONFIG_PPC_KUAP)
+#define INIT_THREAD { \
+ .ksp = INIT_SP, \
+ .pgdir = swapper_pg_dir, \
+ .kuap = ~0UL, /* KUAP_NONE */ \
+ .fpexc_mode = MSR_FE0 | MSR_FE1, \
+ SPEFSCR_INIT \
+}
+#elif defined(CONFIG_PPC32)
#define INIT_THREAD { \
.ksp = INIT_SP, \
.pgdir = swapper_pg_dir, \
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 89e34aa273e2..c6d21604457d 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1736,6 +1736,9 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
#ifdef CONFIG_ALTIVEC
p->thread.vr_save_area = NULL;
#endif
+#if defined(CONFIG_PPC_BOOK3S_32) && defined(CONFIG_PPC_KUAP)
+ p->thread.kuap = KUAP_NONE;
+#endif
setup_ksp_vsid(p, sp);
diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
index a4ce6cdc28e5..1a764367039e 100644
--- a/arch/powerpc/mm/book3s32/kuap.c
+++ b/arch/powerpc/mm/book3s32/kuap.c
@@ -5,10 +5,20 @@
struct static_key_false disable_kuap_key;
+void kuap_lock_all_ool(void)
+{
+ kuap_lock_all();
+}
+
+void kuap_unlock_all_ool(void)
+{
+ kuap_unlock_all();
+}
+
void __init setup_kuap(bool disabled)
{
if (!disabled)
- kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
+ kuap_lock_all_ool();
if (smp_processor_id() != boot_cpuid)
return;
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 10/12] powerpc/32s: Activate KUAP and KUEP by default
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
Now that KUAP and KUEP have been significantly optimised and can be
disabled at boot time using 'nosmap' and 'nosmep' kernel parameters,
them can be active by default like in other powerpc platforms.
It is still possible to disable them completely in the configuration.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/platforms/Kconfig.cputype | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index 885140055b7a..52f533b19518 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -389,7 +389,7 @@ config PPC_HAVE_KUEP
config PPC_KUEP
bool "Kernel Userspace Execution Prevention"
depends on PPC_HAVE_KUEP
- default y if !PPC_BOOK3S_32
+ default y
help
Enable support for Kernel Userspace Execution Prevention (KUEP)
@@ -401,7 +401,7 @@ config PPC_HAVE_KUAP
config PPC_KUAP
bool "Kernel Userspace Access Protection"
depends on PPC_HAVE_KUAP
- default y if !PPC_BOOK3S_32
+ default y
help
Enable support for Kernel Userspace Access Protection (KUAP)
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 10/12] powerpc/32s: Activate KUAP and KUEP by default
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
Now that KUAP and KUEP have been significantly optimised and can be
disabled at boot time using 'nosmap' and 'nosmep' kernel parameters,
them can be active by default like in other powerpc platforms.
It is still possible to disable them completely in the configuration.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/platforms/Kconfig.cputype | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index 885140055b7a..52f533b19518 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -389,7 +389,7 @@ config PPC_HAVE_KUEP
config PPC_KUEP
bool "Kernel Userspace Execution Prevention"
depends on PPC_HAVE_KUEP
- default y if !PPC_BOOK3S_32
+ default y
help
Enable support for Kernel Userspace Execution Prevention (KUEP)
@@ -401,7 +401,7 @@ config PPC_HAVE_KUAP
config PPC_KUAP
bool "Kernel Userspace Access Protection"
depends on PPC_HAVE_KUAP
- default y if !PPC_BOOK3S_32
+ default y
help
Enable support for Kernel Userspace Access Protection (KUAP)
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 11/12] powerpc/kuap: Remove KUAP_CURRENT_XXX
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
book3s/32 was the only user of KUAP_CURRENT_XXX.
After rework of book3s/32 KUAP, it is not used anymore.
Remove them.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 1 -
arch/powerpc/include/asm/kup.h | 14 +++-----------
2 files changed, 3 insertions(+), 12 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index bac7edae64bf..eb5ef59a406b 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -145,7 +145,6 @@ static __always_inline void allow_user_access(void __user *to, const void __user
return;
BUILD_BUG_ON(!__builtin_constant_p(dir));
- BUILD_BUG_ON(dir & ~KUAP_READ_WRITE);
if (!(dir & KUAP_WRITE))
return;
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index 4b94d4293777..2c47feee9482 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -5,14 +5,6 @@
#define KUAP_READ 1
#define KUAP_WRITE 2
#define KUAP_READ_WRITE (KUAP_READ | KUAP_WRITE)
-/*
- * For prevent_user_access() only.
- * Use the current saved situation instead of the to/from/size params.
- * Used on book3s/32
- */
-#define KUAP_CURRENT_READ 4
-#define KUAP_CURRENT_WRITE 8
-#define KUAP_CURRENT (KUAP_CURRENT_READ | KUAP_CURRENT_WRITE)
#ifdef CONFIG_PPC_BOOK3S_64
#include <asm/book3s/64/kup.h>
@@ -129,17 +121,17 @@ static inline void prevent_read_write_user(void __user *to, const void __user *f
static inline void prevent_current_access_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_READ_WRITE);
}
static inline void prevent_current_read_from_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT_READ);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_READ);
}
static inline void prevent_current_write_to_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT_WRITE);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_WRITE);
}
#endif /* !__ASSEMBLY__ */
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 11/12] powerpc/kuap: Remove KUAP_CURRENT_XXX
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
book3s/32 was the only user of KUAP_CURRENT_XXX.
After rework of book3s/32 KUAP, it is not used anymore.
Remove them.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 1 -
arch/powerpc/include/asm/kup.h | 14 +++-----------
2 files changed, 3 insertions(+), 12 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index bac7edae64bf..eb5ef59a406b 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -145,7 +145,6 @@ static __always_inline void allow_user_access(void __user *to, const void __user
return;
BUILD_BUG_ON(!__builtin_constant_p(dir));
- BUILD_BUG_ON(dir & ~KUAP_READ_WRITE);
if (!(dir & KUAP_WRITE))
return;
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index 4b94d4293777..2c47feee9482 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -5,14 +5,6 @@
#define KUAP_READ 1
#define KUAP_WRITE 2
#define KUAP_READ_WRITE (KUAP_READ | KUAP_WRITE)
-/*
- * For prevent_user_access() only.
- * Use the current saved situation instead of the to/from/size params.
- * Used on book3s/32
- */
-#define KUAP_CURRENT_READ 4
-#define KUAP_CURRENT_WRITE 8
-#define KUAP_CURRENT (KUAP_CURRENT_READ | KUAP_CURRENT_WRITE)
#ifdef CONFIG_PPC_BOOK3S_64
#include <asm/book3s/64/kup.h>
@@ -129,17 +121,17 @@ static inline void prevent_read_write_user(void __user *to, const void __user *f
static inline void prevent_current_access_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_READ_WRITE);
}
static inline void prevent_current_read_from_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT_READ);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_READ);
}
static inline void prevent_current_write_to_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_CURRENT_WRITE);
+ prevent_user_access(NULL, NULL, ~0UL, KUAP_WRITE);
}
#endif /* !__ASSEMBLY__ */
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 12/12] powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
2021-06-03 8:41 ` Christophe Leroy
@ 2021-06-03 8:41 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linux-kernel, linuxppc-dev
prevent_user_access() doesn't use anymore to/from/size parameters.
Remove them.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 3 +--
arch/powerpc/include/asm/book3s/64/kup.h | 3 +--
arch/powerpc/include/asm/kup.h | 15 +++++++--------
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +--
4 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index eb5ef59a406b..64201125a287 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -153,8 +153,7 @@ static __always_inline void allow_user_access(void __user *to, const void __user
kuap_unlock_one((__force u32)to);
}
-static __always_inline void prevent_user_access(void __user *to, const void __user *from,
- u32 size, unsigned long dir)
+static __always_inline void prevent_user_access(unsigned long dir)
{
u32 kuap = current->thread.kuap;
diff --git a/arch/powerpc/include/asm/book3s/64/kup.h b/arch/powerpc/include/asm/book3s/64/kup.h
index 9700da3a4093..a1cc73a88710 100644
--- a/arch/powerpc/include/asm/book3s/64/kup.h
+++ b/arch/powerpc/include/asm/book3s/64/kup.h
@@ -398,8 +398,7 @@ static __always_inline void allow_user_access(void __user *to, const void __user
#endif /* !CONFIG_PPC_KUAP */
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir)
+static inline void prevent_user_access(unsigned long dir)
{
set_kuap(AMR_KUAP_BLOCKED);
if (static_branch_unlikely(&uaccess_flush_key))
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index 2c47feee9482..0202a7e84a49 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -72,8 +72,7 @@ static inline unsigned long kuap_get_and_assert_locked(void)
#ifndef CONFIG_PPC_BOOK3S_64
static inline void allow_user_access(void __user *to, const void __user *from,
unsigned long size, unsigned long dir) { }
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir) { }
+static inline void prevent_user_access(unsigned long dir) { }
static inline unsigned long prevent_user_access_return(void) { return 0UL; }
static inline void restore_user_access(unsigned long flags) { }
#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -105,33 +104,33 @@ static inline void allow_read_write_user(void __user *to, const void __user *fro
static inline void prevent_read_from_user(const void __user *from, unsigned long size)
{
- prevent_user_access(NULL, from, size, KUAP_READ);
+ prevent_user_access(KUAP_READ);
}
static inline void prevent_write_to_user(void __user *to, unsigned long size)
{
- prevent_user_access(to, NULL, size, KUAP_WRITE);
+ prevent_user_access(KUAP_WRITE);
}
static inline void prevent_read_write_user(void __user *to, const void __user *from,
unsigned long size)
{
- prevent_user_access(to, from, size, KUAP_READ_WRITE);
+ prevent_user_access(KUAP_READ_WRITE);
}
static inline void prevent_current_access_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_READ_WRITE);
+ prevent_user_access(KUAP_READ_WRITE);
}
static inline void prevent_current_read_from_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_READ);
+ prevent_user_access(KUAP_READ);
}
static inline void prevent_current_write_to_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_WRITE);
+ prevent_user_access(KUAP_WRITE);
}
#endif /* !__ASSEMBLY__ */
diff --git a/arch/powerpc/include/asm/nohash/32/kup-8xx.h b/arch/powerpc/include/asm/nohash/32/kup-8xx.h
index 295ef5639609..61333e3b5d2b 100644
--- a/arch/powerpc/include/asm/nohash/32/kup-8xx.h
+++ b/arch/powerpc/include/asm/nohash/32/kup-8xx.h
@@ -48,8 +48,7 @@ static inline void allow_user_access(void __user *to, const void __user *from,
mtspr(SPRN_MD_AP, MD_APG_INIT);
}
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir)
+static inline void prevent_user_access(unsigned long dir)
{
mtspr(SPRN_MD_AP, MD_APG_KUAP);
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v2 12/12] powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
@ 2021-06-03 8:41 ` Christophe Leroy
0 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-03 8:41 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
prevent_user_access() doesn't use anymore to/from/size parameters.
Remove them.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/include/asm/book3s/32/kup.h | 3 +--
arch/powerpc/include/asm/book3s/64/kup.h | 3 +--
arch/powerpc/include/asm/kup.h | 15 +++++++--------
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +--
4 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
index eb5ef59a406b..64201125a287 100644
--- a/arch/powerpc/include/asm/book3s/32/kup.h
+++ b/arch/powerpc/include/asm/book3s/32/kup.h
@@ -153,8 +153,7 @@ static __always_inline void allow_user_access(void __user *to, const void __user
kuap_unlock_one((__force u32)to);
}
-static __always_inline void prevent_user_access(void __user *to, const void __user *from,
- u32 size, unsigned long dir)
+static __always_inline void prevent_user_access(unsigned long dir)
{
u32 kuap = current->thread.kuap;
diff --git a/arch/powerpc/include/asm/book3s/64/kup.h b/arch/powerpc/include/asm/book3s/64/kup.h
index 9700da3a4093..a1cc73a88710 100644
--- a/arch/powerpc/include/asm/book3s/64/kup.h
+++ b/arch/powerpc/include/asm/book3s/64/kup.h
@@ -398,8 +398,7 @@ static __always_inline void allow_user_access(void __user *to, const void __user
#endif /* !CONFIG_PPC_KUAP */
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir)
+static inline void prevent_user_access(unsigned long dir)
{
set_kuap(AMR_KUAP_BLOCKED);
if (static_branch_unlikely(&uaccess_flush_key))
diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h
index 2c47feee9482..0202a7e84a49 100644
--- a/arch/powerpc/include/asm/kup.h
+++ b/arch/powerpc/include/asm/kup.h
@@ -72,8 +72,7 @@ static inline unsigned long kuap_get_and_assert_locked(void)
#ifndef CONFIG_PPC_BOOK3S_64
static inline void allow_user_access(void __user *to, const void __user *from,
unsigned long size, unsigned long dir) { }
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir) { }
+static inline void prevent_user_access(unsigned long dir) { }
static inline unsigned long prevent_user_access_return(void) { return 0UL; }
static inline void restore_user_access(unsigned long flags) { }
#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -105,33 +104,33 @@ static inline void allow_read_write_user(void __user *to, const void __user *fro
static inline void prevent_read_from_user(const void __user *from, unsigned long size)
{
- prevent_user_access(NULL, from, size, KUAP_READ);
+ prevent_user_access(KUAP_READ);
}
static inline void prevent_write_to_user(void __user *to, unsigned long size)
{
- prevent_user_access(to, NULL, size, KUAP_WRITE);
+ prevent_user_access(KUAP_WRITE);
}
static inline void prevent_read_write_user(void __user *to, const void __user *from,
unsigned long size)
{
- prevent_user_access(to, from, size, KUAP_READ_WRITE);
+ prevent_user_access(KUAP_READ_WRITE);
}
static inline void prevent_current_access_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_READ_WRITE);
+ prevent_user_access(KUAP_READ_WRITE);
}
static inline void prevent_current_read_from_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_READ);
+ prevent_user_access(KUAP_READ);
}
static inline void prevent_current_write_to_user(void)
{
- prevent_user_access(NULL, NULL, ~0UL, KUAP_WRITE);
+ prevent_user_access(KUAP_WRITE);
}
#endif /* !__ASSEMBLY__ */
diff --git a/arch/powerpc/include/asm/nohash/32/kup-8xx.h b/arch/powerpc/include/asm/nohash/32/kup-8xx.h
index 295ef5639609..61333e3b5d2b 100644
--- a/arch/powerpc/include/asm/nohash/32/kup-8xx.h
+++ b/arch/powerpc/include/asm/nohash/32/kup-8xx.h
@@ -48,8 +48,7 @@ static inline void allow_user_access(void __user *to, const void __user *from,
mtspr(SPRN_MD_AP, MD_APG_INIT);
}
-static inline void prevent_user_access(void __user *to, const void __user *from,
- unsigned long size, unsigned long dir)
+static inline void prevent_user_access(unsigned long dir)
{
mtspr(SPRN_MD_AP, MD_APG_KUAP);
}
--
2.25.0
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [PATCH v2 08/12] powerpc/32s: Allow disabling KUAP at boot time
2021-06-03 8:41 ` Christophe Leroy
(?)
@ 2021-06-12 6:18 ` Christophe Leroy
-1 siblings, 0 replies; 28+ messages in thread
From: Christophe Leroy @ 2021-06-12 6:18 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
Le 03/06/2021 à 10:41, Christophe Leroy a écrit :
> PPC64 uses MMU features to enable/disable KUAP at boot time.
> But feature fixups are applied way too early on PPC32.
>
> Now that all KUAP related actions are in C following the
> conversion of KUAP initial setup and context switch in C,
> static branches can be used to enable/disable KUAP.
>
> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
> arch/powerpc/include/asm/book3s/32/kup.h | 27 +++++++++++++++++++++++-
> arch/powerpc/mm/book3s32/kuap.c | 11 ++++++----
> 2 files changed, 33 insertions(+), 5 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/book3s/32/kup.h b/arch/powerpc/include/asm/book3s/32/kup.h
> index 2854d970dabe..68fbe28c6d7e 100644
> --- a/arch/powerpc/include/asm/book3s/32/kup.h
> +++ b/arch/powerpc/include/asm/book3s/32/kup.h
> @@ -9,11 +9,12 @@
>
> #include <linux/jump_label.h>
>
> +extern struct static_key_false disable_kuap_key;
Same as 8xx, this needs to be exported for modules.
> extern struct static_key_false disable_kuep_key;
>
> static __always_inline bool kuap_is_disabled(void)
> {
> - return !IS_ENABLED(CONFIG_PPC_KUAP);
> + return !IS_ENABLED(CONFIG_PPC_KUAP) || static_branch_unlikely(&disable_kuap_key);
> }
>
> static __always_inline bool kuep_is_disabled(void)
> @@ -62,6 +63,9 @@ static inline void kuap_save_and_lock(struct pt_regs *regs)
> u32 addr = kuap & 0xf0000000;
> u32 end = kuap << 28;
>
> + if (kuap_is_disabled())
> + return;
> +
> regs->kuap = kuap;
> if (unlikely(!kuap))
> return;
> @@ -79,6 +83,9 @@ static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)
> u32 addr = regs->kuap & 0xf0000000;
> u32 end = regs->kuap << 28;
>
> + if (kuap_is_disabled())
> + return;
> +
> current->thread.kuap = regs->kuap;
>
> if (unlikely(regs->kuap == kuap))
> @@ -91,6 +98,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)
> {
> unsigned long kuap = current->thread.kuap;
>
> + if (kuap_is_disabled())
> + return 0;
> +
> WARN_ON_ONCE(IS_ENABLED(CONFIG_PPC_KUAP_DEBUG) && kuap != 0);
>
> return kuap;
> @@ -106,6 +116,9 @@ static __always_inline void allow_user_access(void __user *to, const void __user
> {
> u32 addr, end;
>
> + if (kuap_is_disabled())
> + return;
> +
> BUILD_BUG_ON(!__builtin_constant_p(dir));
> BUILD_BUG_ON(dir & ~KUAP_READ_WRITE);
>
> @@ -128,6 +141,9 @@ static __always_inline void prevent_user_access(void __user *to, const void __us
> {
> u32 addr, end;
>
> + if (kuap_is_disabled())
> + return;
> +
> BUILD_BUG_ON(!__builtin_constant_p(dir));
>
> if (dir & KUAP_CURRENT_WRITE) {
> @@ -159,6 +175,9 @@ static inline unsigned long prevent_user_access_return(void)
> unsigned long end = flags << 28;
> void __user *to = (__force void __user *)addr;
>
> + if (kuap_is_disabled())
> + return 0;
> +
> if (flags)
> prevent_user_access(to, to, end - addr, KUAP_READ_WRITE);
>
> @@ -171,6 +190,9 @@ static inline void restore_user_access(unsigned long flags)
> unsigned long end = flags << 28;
> void __user *to = (__force void __user *)addr;
>
> + if (kuap_is_disabled())
> + return;
> +
> if (flags)
> allow_user_access(to, to, end - addr, KUAP_READ_WRITE);
> }
> @@ -181,6 +203,9 @@ bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)
> unsigned long begin = regs->kuap & 0xf0000000;
> unsigned long end = regs->kuap << 28;
>
> + if (kuap_is_disabled())
> + return false;
> +
> return is_write && (address < begin || address >= end);
> }
>
> diff --git a/arch/powerpc/mm/book3s32/kuap.c b/arch/powerpc/mm/book3s32/kuap.c
> index 5533ed92ab3d..a4ce6cdc28e5 100644
> --- a/arch/powerpc/mm/book3s32/kuap.c
> +++ b/arch/powerpc/mm/book3s32/kuap.c
> @@ -3,15 +3,18 @@
> #include <asm/kup.h>
> #include <asm/smp.h>
>
> +struct static_key_false disable_kuap_key;
> +
> void __init setup_kuap(bool disabled)
> {
> - kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
> + if (!disabled)
> + kuap_update_sr(mfsr(0) | SR_KS, 0, TASK_SIZE);
>
> if (smp_processor_id() != boot_cpuid)
> return;
>
> - pr_info("Activating Kernel Userspace Access Protection\n");
> -
> if (disabled)
> - pr_warn("KUAP cannot be disabled yet on 6xx when compiled in\n");
> + static_branch_enable(&disable_kuap_key);
> + else
> + pr_info("Activating Kernel Userspace Access Protection\n");
> }
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v2 00/12] powerpc: Optimise KUAP on book3s/32
2021-06-03 8:41 ` Christophe Leroy
` (12 preceding siblings ...)
(?)
@ 2021-06-18 3:51 ` Michael Ellerman
-1 siblings, 0 replies; 28+ messages in thread
From: Michael Ellerman @ 2021-06-18 3:51 UTC (permalink / raw)
To: Paul Mackerras, Michael Ellerman, Benjamin Herrenschmidt,
Christophe Leroy
Cc: linuxppc-dev, linux-kernel
On Thu, 3 Jun 2021 08:41:35 +0000 (UTC), Christophe Leroy wrote:
> This series is a rework of KUAP on book3s/32.
>
> On book3s32, KUAP is heavier than on other platform because it can't
> be opened globaly at once, it must be done for each 256Mb segment.
>
> Instead of opening access to all necessary segments via a heavy logic,
> only open access to the segment matching the start of the range.
>
> [...]
Applied to powerpc/next.
[01/12] powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
https://git.kernel.org/powerpc/c/91ec66719d4c5c0e7b4e32585b01881660d1bc53
[02/12] powerpc/32s: Refactor update of user segment registers
https://git.kernel.org/powerpc/c/91bb30822a2e1d7900f9f42e9e92647a9015f979
[03/12] powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
https://git.kernel.org/powerpc/c/7235bb3593781ed022d0714a73c2c0d8eb8a835f
[04/12] powerpc/32s: Convert switch_mmu_context() to C
https://git.kernel.org/powerpc/c/863771a28e27dc9eaeaa88cea300370d032f0e0f
[05/12] powerpc/32s: Simplify calculation of segment register content
https://git.kernel.org/powerpc/c/882136fb2f5208a35ddad9205b20e5791edd4782
[06/12] powerpc/32s: Initialise KUAP and KUEP in C
https://git.kernel.org/powerpc/c/86f46f3432727933be82f64b739712a6edb9d704
[07/12] powerpc/32s: Allow disabling KUEP at boot time
https://git.kernel.org/powerpc/c/50d2f104cd9572af476579eae9aa1b38de602ec7
[08/12] powerpc/32s: Allow disabling KUAP at boot time
https://git.kernel.org/powerpc/c/6b4d630068b0c5cdd6d8e599182b131448e0cb06
[09/12] powerpc/32s: Rework Kernel Userspace Access Protection
https://git.kernel.org/powerpc/c/16132529cee586ee9a058bb33cfbdcb5d884f6b3
[10/12] powerpc/32s: Activate KUAP and KUEP by default
https://git.kernel.org/powerpc/c/9f5bd8f1471d7498c934c0a686fd0997cf872653
[11/12] powerpc/kuap: Remove KUAP_CURRENT_XXX
https://git.kernel.org/powerpc/c/d008f8f8a0c3efe4fe1008a797f9497ea5965e27
[12/12] powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
https://git.kernel.org/powerpc/c/cb2f1fb205cc20695fcaef84baf80d9d3e54c88b
cheers
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2021-06-18 4:23 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-03 8:41 [PATCH v2 00/12] powerpc: Optimise KUAP on book3s/32 Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 01/12] powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 02/12] powerpc/32s: Refactor update of user segment registers Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 03/12] powerpc/32s: move CTX_TO_VSID() into mmu-hash.h Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 04/12] powerpc/32s: Convert switch_mmu_context() to C Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 05/12] powerpc/32s: Simplify calculation of segment register content Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 06/12] powerpc/32s: Initialise KUAP and KUEP in C Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 07/12] powerpc/32s: Allow disabling KUEP at boot time Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 08/12] powerpc/32s: Allow disabling KUAP " Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-12 6:18 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 09/12] powerpc/32s: Rework Kernel Userspace Access Protection Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 10/12] powerpc/32s: Activate KUAP and KUEP by default Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 11/12] powerpc/kuap: Remove KUAP_CURRENT_XXX Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-03 8:41 ` [PATCH v2 12/12] powerpc/kuap: Remove to/from/size parameters of prevent_user_access() Christophe Leroy
2021-06-03 8:41 ` Christophe Leroy
2021-06-18 3:51 ` [PATCH v2 00/12] powerpc: Optimise KUAP on book3s/32 Michael Ellerman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.