* [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch()
@ 2021-09-15 19:03 Peter Collingbourne
2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne
2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas
0 siblings, 2 replies; 5+ messages in thread
From: Peter Collingbourne @ 2021-09-15 19:03 UTC (permalink / raw)
To: Catalin Marinas, Vincenzo Frascino, Will Deacon, Andrey Konovalov
Cc: Peter Collingbourne, Evgenii Stepanov, linux-arm-kernel
This lets us avoid doing unnecessary work on hardware that does
not support MTE, and will allow us to freely use MTE instructions
in the code called by mte_thread_switch().
Since this would mean that we do a redundant check in
mte_check_tfsr_el1(), remove it and add two checks now required in
its callers.
Signed-off-by: Peter Collingbourne <pcc@google.com>
Link: https://linux-review.googlesource.com/id/I02fd000d1ef2c86c7d2952a7f099b254ec227a5d
---
v3:
- remove check from mte_check_tfsr_el1()
arch/arm64/include/asm/mte.h | 6 ++++++
arch/arm64/kernel/mte.c | 10 ++++------
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 3f93b9e0b339..02511650cffe 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -99,11 +99,17 @@ void mte_check_tfsr_el1(void);
static inline void mte_check_tfsr_entry(void)
{
+ if (!system_supports_mte())
+ return;
+
mte_check_tfsr_el1();
}
static inline void mte_check_tfsr_exit(void)
{
+ if (!system_supports_mte())
+ return;
+
/*
* The asynchronous faults are sync'ed automatically with
* TFSR_EL1 on kernel entry but for exit an explicit dsb()
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index 9d314a3bad3b..e5e801bc5312 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -142,12 +142,7 @@ void mte_enable_kernel_async(void)
#ifdef CONFIG_KASAN_HW_TAGS
void mte_check_tfsr_el1(void)
{
- u64 tfsr_el1;
-
- if (!system_supports_mte())
- return;
-
- tfsr_el1 = read_sysreg_s(SYS_TFSR_EL1);
+ u64 tfsr_el1 = read_sysreg_s(SYS_TFSR_EL1);
if (unlikely(tfsr_el1 & SYS_TFSR_EL1_TF1)) {
/*
@@ -199,6 +194,9 @@ void mte_thread_init_user(void)
void mte_thread_switch(struct task_struct *next)
{
+ if (!system_supports_mte())
+ return;
+
mte_update_sctlr_user(next);
/*
--
2.33.0.309.g3052b89438-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled
2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne
@ 2021-09-15 19:03 ` Peter Collingbourne
2021-09-21 12:39 ` Catalin Marinas
2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas
1 sibling, 1 reply; 5+ messages in thread
From: Peter Collingbourne @ 2021-09-15 19:03 UTC (permalink / raw)
To: Catalin Marinas, Vincenzo Frascino, Will Deacon, Andrey Konovalov
Cc: Peter Collingbourne, Evgenii Stepanov, linux-arm-kernel
It is not necessary to write to GCR_EL1 on every kernel entry and
exit when HW tag-based KASAN is disabled because the kernel will not
execute any IRG instructions in that mode. Since accessing GCR_EL1
can be expensive on some microarchitectures, avoid doing so by moving
the access to task switch when HW tag-based KASAN is disabled.
Signed-off-by: Peter Collingbourne <pcc@google.com>
Acked-by: Andrey Konovalov <andreyknvl@gmail.com>
Link: https://linux-review.googlesource.com/id/I78e90d60612a94c24344526f476ac4ff216e10d2
---
v2:
- add prerequisite patch to fix booting on non-MTE hardware
arch/arm64/kernel/entry.S | 10 +++++-----
arch/arm64/kernel/mte.c | 24 +++++++++++++++++++++---
include/linux/kasan.h | 9 +++++++--
3 files changed, 33 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index bc6d5a970a13..2f69ae43941d 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -168,9 +168,9 @@ alternative_else_nop_endif
.macro mte_set_kernel_gcr, tmp, tmp2
#ifdef CONFIG_KASAN_HW_TAGS
-alternative_if_not ARM64_MTE
+alternative_cb kasan_hw_tags_enable
b 1f
-alternative_else_nop_endif
+alternative_cb_end
mov \tmp, KERNEL_GCR_EL1
msr_s SYS_GCR_EL1, \tmp
1:
@@ -178,10 +178,10 @@ alternative_else_nop_endif
.endm
.macro mte_set_user_gcr, tsk, tmp, tmp2
-#ifdef CONFIG_ARM64_MTE
-alternative_if_not ARM64_MTE
+#ifdef CONFIG_KASAN_HW_TAGS
+alternative_cb kasan_hw_tags_enable
b 1f
-alternative_else_nop_endif
+alternative_cb_end
ldr \tmp, [\tsk, #THREAD_MTE_CTRL]
mte_set_gcr \tmp, \tmp2
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index e5e801bc5312..bb8204d4c5d3 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -157,7 +157,7 @@ void mte_check_tfsr_el1(void)
}
#endif
-static void mte_update_sctlr_user(struct task_struct *task)
+static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task)
{
/*
* This must be called with preemption disabled and can only be called
@@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task)
else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC)
sctlr |= SCTLR_EL1_TCF0_SYNC;
task->thread.sctlr_user = sctlr;
+
+ /*
+ * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by
+ * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled.
+ */
+ if (!kasan_hw_tags_enabled())
+ write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) &
+ SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND,
+ SYS_GCR_EL1);
+}
+
+void __init kasan_hw_tags_enable(struct alt_instr *alt, __le32 *origptr,
+ __le32 *updptr, int nr_inst)
+{
+ BUG_ON(nr_inst != 1); /* Branch -> NOP */
+
+ if (kasan_hw_tags_enabled())
+ *updptr = cpu_to_le32(aarch64_insn_gen_nop());
}
void mte_thread_init_user(void)
@@ -197,7 +215,7 @@ void mte_thread_switch(struct task_struct *next)
if (!system_supports_mte())
return;
- mte_update_sctlr_user(next);
+ mte_update_sctlr_user_and_gcr_excl(next);
/*
* Check if an async tag exception occurred at EL1.
@@ -242,7 +260,7 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg)
task->thread.mte_ctrl = mte_ctrl;
if (task == current) {
preempt_disable();
- mte_update_sctlr_user(task);
+ mte_update_sctlr_user_and_gcr_excl(task);
update_sctlr_el1(task->thread.sctlr_user);
preempt_enable();
}
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index dd874a1ee862..de5f5913374d 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -89,7 +89,7 @@ static __always_inline bool kasan_enabled(void)
return static_branch_likely(&kasan_flag_enabled);
}
-static inline bool kasan_has_integrated_init(void)
+static inline bool kasan_hw_tags_enabled(void)
{
return kasan_enabled();
}
@@ -104,7 +104,7 @@ static inline bool kasan_enabled(void)
return IS_ENABLED(CONFIG_KASAN);
}
-static inline bool kasan_has_integrated_init(void)
+static inline bool kasan_hw_tags_enabled(void)
{
return false;
}
@@ -125,6 +125,11 @@ static __always_inline void kasan_free_pages(struct page *page,
#endif /* CONFIG_KASAN_HW_TAGS */
+static inline bool kasan_has_integrated_init(void)
+{
+ return kasan_hw_tags_enabled();
+}
+
#ifdef CONFIG_KASAN
struct kasan_cache {
--
2.33.0.309.g3052b89438-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled
2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne
@ 2021-09-21 12:39 ` Catalin Marinas
2021-09-24 1:08 ` Peter Collingbourne
0 siblings, 1 reply; 5+ messages in thread
From: Catalin Marinas @ 2021-09-21 12:39 UTC (permalink / raw)
To: Peter Collingbourne
Cc: Vincenzo Frascino, Will Deacon, Andrey Konovalov,
Evgenii Stepanov, linux-arm-kernel
On Wed, Sep 15, 2021 at 12:03:36PM -0700, Peter Collingbourne wrote:
> -static void mte_update_sctlr_user(struct task_struct *task)
> +static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task)
> {
> /*
> * This must be called with preemption disabled and can only be called
> @@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task)
> else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC)
> sctlr |= SCTLR_EL1_TCF0_SYNC;
> task->thread.sctlr_user = sctlr;
> +
> + /*
> + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by
> + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled.
> + */
> + if (!kasan_hw_tags_enabled())
> + write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) &
> + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND,
> + SYS_GCR_EL1);
> +}
The only nitpick I have is that I'd like to keep the GCR_EL1 updating in
a separate function, mte_update_gcr_user() or something (it can be
static inline) that you'd explicitly call from mte_thread_switch() and
set_mte_ctrl(). The original mte_update_sctlr_user() was not writing any
system registers.
Maybe a better alternative would be to move the update_sctlr_el1() call
in mte_update_sctlr_user() as well (and rename it more generically to
mte_update_user_ctrl()) but with a new bool update_sctlr argument that's
false on the thread switch path.
Either way:
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch()
2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne
2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne
@ 2021-09-21 12:45 ` Catalin Marinas
1 sibling, 0 replies; 5+ messages in thread
From: Catalin Marinas @ 2021-09-21 12:45 UTC (permalink / raw)
To: Andrey Konovalov, Will Deacon, Peter Collingbourne, Vincenzo Frascino
Cc: linux-arm-kernel, Evgenii Stepanov
On Wed, 15 Sep 2021 12:03:35 -0700, Peter Collingbourne wrote:
> This lets us avoid doing unnecessary work on hardware that does
> not support MTE, and will allow us to freely use MTE instructions
> in the code called by mte_thread_switch().
>
> Since this would mean that we do a redundant check in
> mte_check_tfsr_el1(), remove it and add two checks now required in
> its callers.
I applied the first patch to arm64 (for-next/fixes), it should appear in
5.15-rc3. It looks to me more like a performance regression with the
additional dsb+isb on the syscall path for hw not supporting MTE (and I
added a Fixes tag). I'll leave the second patch to Will for 5.16.
Thanks!
[1/2] arm64: add an MTE support check to the top of mte_thread_switch()
https://git.kernel.org/arm64/c/8c8a3b5bd960
--
Catalin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled
2021-09-21 12:39 ` Catalin Marinas
@ 2021-09-24 1:08 ` Peter Collingbourne
0 siblings, 0 replies; 5+ messages in thread
From: Peter Collingbourne @ 2021-09-24 1:08 UTC (permalink / raw)
To: Catalin Marinas
Cc: Vincenzo Frascino, Will Deacon, Andrey Konovalov,
Evgenii Stepanov, Linux ARM
On Tue, Sep 21, 2021 at 5:40 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Sep 15, 2021 at 12:03:36PM -0700, Peter Collingbourne wrote:
> > -static void mte_update_sctlr_user(struct task_struct *task)
> > +static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task)
> > {
> > /*
> > * This must be called with preemption disabled and can only be called
> > @@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task)
> > else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC)
> > sctlr |= SCTLR_EL1_TCF0_SYNC;
> > task->thread.sctlr_user = sctlr;
> > +
> > + /*
> > + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by
> > + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled.
> > + */
> > + if (!kasan_hw_tags_enabled())
> > + write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) &
> > + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND,
> > + SYS_GCR_EL1);
> > +}
>
> The only nitpick I have is that I'd like to keep the GCR_EL1 updating in
> a separate function, mte_update_gcr_user() or something (it can be
> static inline) that you'd explicitly call from mte_thread_switch() and
> set_mte_ctrl(). The original mte_update_sctlr_user() was not writing any
> system registers.
Done in v4. (I mildly prefer the way I originally wrote it as I think
it's less error prone, but I don't mind that much.) Sorry, I forgot to
add your Reviewed-by.
Peter
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-09-24 1:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne
2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne
2021-09-21 12:39 ` Catalin Marinas
2021-09-24 1:08 ` Peter Collingbourne
2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.