* [PATCH RESEND] powerpc: add simd.h implementation specific to PowerPC
@ 2019-05-13 0:51 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-13 0:51 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Shawn Landden, Paul Mackerras, linux-kernel
It is safe to do SIMD in an interrupt on PowerPC.
Only disable when there is no SIMD available
(and this is a static branch).
Tested and works with the WireGuard (Zinc) patch I wrote that needs this.
Also improves performance of the crypto subsystem that checks this.
Re-sending because this linuxppc-dev didn't seem to accept it.
Buglink: https://bugzilla.kernel.org/show_bug.cgi?id=203571
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 15 +++++++++++++++
1 file changed, 15 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..b3fecb95a
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+#include <asm/cpu_has_feature.h>
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * As documented in Chapter 6.2.1 Machine Status Save/Restore Registers
+ * of Power ISA (2.07 and 3.0), all registers are saved/restored in an interrupt.
+ */
+static inline bool may_use_simd(void)
+{
+ return !cpu_has_feature(CPU_FTR_FPU_UNAVAILABLE);
+}
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH RESEND] powerpc: add simd.h implementation specific to PowerPC
2019-05-13 0:51 ` Shawn Landden
@ 2019-05-13 11:53 ` Michael Ellerman
-1 siblings, 0 replies; 27+ messages in thread
From: Michael Ellerman @ 2019-05-13 11:53 UTC (permalink / raw)
To: Shawn Landden, linuxppc-dev
Cc: Benjamin Herrenschmidt, Paul Mackerras, linux-kernel, Shawn Landden
Shawn Landden <shawn@git.icu> writes:
> It is safe to do SIMD in an interrupt on PowerPC.
No it's not sorry :)
> Only disable when there is no SIMD available
> (and this is a static branch).
>
> Tested and works with the WireGuard (Zinc) patch I wrote that needs this.
> Also improves performance of the crypto subsystem that checks this.
>
> Re-sending because this linuxppc-dev didn't seem to accept it.
It did but you were probably moderated as a non-subscriber? In future if
you just wait a while for the moderators to wake up it should come
through. Though having posted once and been approved I think you might
not get moderated at all in future (?).
> Buglink: https://bugzilla.kernel.org/show_bug.cgi?id=203571
> Signed-off-by: Shawn Landden <shawn@git.icu>
> ---
> arch/powerpc/include/asm/simd.h | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
> create mode 100644 arch/powerpc/include/asm/simd.h
>
> diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
> new file mode 100644
> index 000000000..b3fecb95a
> --- /dev/null
> +++ b/arch/powerpc/include/asm/simd.h
> @@ -0,0 +1,15 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +
> +#include <asm/cpu_has_feature.h>
> +
> +/*
> + * may_use_simd - whether it is allowable at this time to issue SIMD
> + * instructions or access the SIMD register file
> + *
> + * As documented in Chapter 6.2.1 Machine Status Save/Restore Registers
> + * of Power ISA (2.07 and 3.0), all registers are saved/restored in an interrupt.
I think the confusion here is that the ISA says:
When various interrupts occur, the state of the machine is saved in the
Machine Status Save/Restore registers (SRR0 and SRR1).
And you've read that to mean all "the state" of the machine, ie.
including GPRs, FPRs etc.
But unfortunately it's not that simple. All the hardware does is write
two 64-bit registers (SRR0 & SRR1) with the information required to be
able to return to the state the CPU was in prior to the interrupt. That
includes (obviously) the PC you were executing at, and what state the
CPU was in (ie. 64/32-bit, MMU on/off, FP on/off etc.). All the saving
of registers etc. is left up to software. It's the RISC way :)
I guess we need to work out why exactly may_use_simd() is returning
false in your kworker.
cheers
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH RESEND] powerpc: add simd.h implementation specific to PowerPC
@ 2019-05-13 11:53 ` Michael Ellerman
0 siblings, 0 replies; 27+ messages in thread
From: Michael Ellerman @ 2019-05-13 11:53 UTC (permalink / raw)
To: Shawn Landden, linuxppc-dev; +Cc: Shawn Landden, Paul Mackerras, linux-kernel
Shawn Landden <shawn@git.icu> writes:
> It is safe to do SIMD in an interrupt on PowerPC.
No it's not sorry :)
> Only disable when there is no SIMD available
> (and this is a static branch).
>
> Tested and works with the WireGuard (Zinc) patch I wrote that needs this.
> Also improves performance of the crypto subsystem that checks this.
>
> Re-sending because this linuxppc-dev didn't seem to accept it.
It did but you were probably moderated as a non-subscriber? In future if
you just wait a while for the moderators to wake up it should come
through. Though having posted once and been approved I think you might
not get moderated at all in future (?).
> Buglink: https://bugzilla.kernel.org/show_bug.cgi?id=203571
> Signed-off-by: Shawn Landden <shawn@git.icu>
> ---
> arch/powerpc/include/asm/simd.h | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
> create mode 100644 arch/powerpc/include/asm/simd.h
>
> diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
> new file mode 100644
> index 000000000..b3fecb95a
> --- /dev/null
> +++ b/arch/powerpc/include/asm/simd.h
> @@ -0,0 +1,15 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +
> +#include <asm/cpu_has_feature.h>
> +
> +/*
> + * may_use_simd - whether it is allowable at this time to issue SIMD
> + * instructions or access the SIMD register file
> + *
> + * As documented in Chapter 6.2.1 Machine Status Save/Restore Registers
> + * of Power ISA (2.07 and 3.0), all registers are saved/restored in an interrupt.
I think the confusion here is that the ISA says:
When various interrupts occur, the state of the machine is saved in the
Machine Status Save/Restore registers (SRR0 and SRR1).
And you've read that to mean all "the state" of the machine, ie.
including GPRs, FPRs etc.
But unfortunately it's not that simple. All the hardware does is write
two 64-bit registers (SRR0 & SRR1) with the information required to be
able to return to the state the CPU was in prior to the interrupt. That
includes (obviously) the PC you were executing at, and what state the
CPU was in (ie. 64/32-bit, MMU on/off, FP on/off etc.). All the saving
of registers etc. is left up to software. It's the RISC way :)
I guess we need to work out why exactly may_use_simd() is returning
false in your kworker.
cheers
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-13 0:51 ` Shawn Landden
(?)
(?)
@ 2019-05-14 1:44 ` Shawn Landden
2019-05-14 1:44 ` [PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
` (2 more replies)
-1 siblings, 3 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 1:44 UTC (permalink / raw)
Cc: Shawn Landden, Paul Mackerras, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 13 +++++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 43 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..2c02ad531
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+static __must_check inline bool may_use_simd(void)
+{
+ return irq_simd_usable();
+}
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..e436d708a 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use SIMD in kernel mode with the
+ * whole "kernel_altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool irq_simd_usable(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(irq_simd_usable);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-14 1:44 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Shawn Landden
@ 2019-05-14 1:44 ` Shawn Landden
2019-05-14 5:43 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Benjamin Herrenschmidt
2019-05-14 15:46 ` [v3 " Shawn Landden
2 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 1:44 UTC (permalink / raw)
Cc: Shawn Landden, Paul Mackerras, linuxppc-dev
This second patch is separate because it could be wrong,
like I am not sure about how kernel thread migration works,
and it is even allowing simd in preemptible kernel code.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 8 +++++
arch/powerpc/include/asm/switch_to.h | 10 ++----
arch/powerpc/kernel/process.c | 50 ++++++++++++++++++++++++++--
3 files changed, 57 insertions(+), 11 deletions(-)
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
index 2c02ad531..7b582b07e 100644
--- a/arch/powerpc/include/asm/simd.h
+++ b/arch/powerpc/include/asm/simd.h
@@ -7,7 +7,15 @@
* It's always ok in process context (ie "not interrupt")
* but it is sometimes ok even from an irq.
*/
+#ifdef CONFIG_ALTIVEC
+extern bool irq_simd_usable(void);
static __must_check inline bool may_use_simd(void)
{
return irq_simd_usable();
}
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h
index 5b03d8a82..537998997 100644
--- a/arch/powerpc/include/asm/switch_to.h
+++ b/arch/powerpc/include/asm/switch_to.h
@@ -44,10 +44,7 @@ extern void enable_kernel_altivec(void);
extern void flush_altivec_to_thread(struct task_struct *);
extern void giveup_altivec(struct task_struct *);
extern void save_altivec(struct task_struct *);
-static inline void disable_kernel_altivec(void)
-{
- msr_check_and_clear(MSR_VEC);
-}
+extern void disable_kernel_altivec(void);
#else
static inline void save_altivec(struct task_struct *t) { }
static inline void __giveup_altivec(struct task_struct *t) { }
@@ -56,10 +53,7 @@ static inline void __giveup_altivec(struct task_struct *t) { }
#ifdef CONFIG_VSX
extern void enable_kernel_vsx(void);
extern void flush_vsx_to_thread(struct task_struct *);
-static inline void disable_kernel_vsx(void)
-{
- msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
-}
+extern void disable_kernel_vsx(void);
#endif
#ifdef CONFIG_SPE
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index e436d708a..41a0ab500 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -267,6 +267,29 @@ static int restore_fp(struct task_struct *tsk) { return 0; }
#ifdef CONFIG_ALTIVEC
#define loadvec(thr) ((thr).load_vec)
+/*
+ * Track whether the kernel is using the SIMD state
+ * currently.
+ *
+ * This flag is used:
+ *
+ * - by IRQ context code to potentially use the FPU
+ * if it's unused.
+ *
+ * - to debug kernel_altivec/vsx_begin()/end() correctness
+ */
+static DEFINE_PER_CPU(bool, in_kernel_simd);
+
+static bool kernel_simd_disabled(void)
+{
+ return this_cpu_read(in_kernel_simd);
+}
+
+static bool interrupted_kernel_simd_idle(void)
+{
+ return !kernel_simd_disabled();
+}
+
static void __giveup_altivec(struct task_struct *tsk)
{
unsigned long msr;
@@ -295,7 +318,9 @@ void enable_kernel_altivec(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_simd));
+ this_cpu_write(in_kernel_simd, true);
cpumsr = msr_check_and_set(MSR_VEC);
@@ -316,6 +341,14 @@ void enable_kernel_altivec(void)
}
EXPORT_SYMBOL(enable_kernel_altivec);
+extern void disable_kernel_altivec(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_simd));
+ this_cpu_write(in_kernel_simd, false);
+ msr_check_and_clear(MSR_VEC);
+}
+EXPORT_SYMBOL(disable_kernel_altivec);
+
/*
* Make sure the VMX/Altivec register state in the
* the thread_struct is up to date for task tsk.
@@ -371,7 +404,8 @@ static bool interrupted_user_mode(void)
bool irq_simd_usable(void)
{
return !in_interrupt() ||
- interrupted_user_mode();
+ interrupted_user_mode() ||
+ interrupted_kernel_simd_idle();
}
EXPORT_SYMBOL(irq_simd_usable);
@@ -411,7 +445,9 @@ void enable_kernel_vsx(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_simd));
+ this_cpu_write(in_kernel_simd, true);
cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);
@@ -433,6 +469,14 @@ void enable_kernel_vsx(void)
}
EXPORT_SYMBOL(enable_kernel_vsx);
+void disable_kernel_vsx(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_simd));
+ this_cpu_write(in_kernel_simd, false);
+ msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
+}
+EXPORT_SYMBOL(disable_kernel_vsx);
+
void flush_vsx_to_thread(struct task_struct *tsk)
{
if (tsk->thread.regs) {
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-14 1:44 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Shawn Landden
2019-05-14 1:44 ` [PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
@ 2019-05-14 5:43 ` Benjamin Herrenschmidt
2019-05-14 15:44 ` Shawn Landden
2019-05-14 15:46 ` [v3 " Shawn Landden
2 siblings, 1 reply; 27+ messages in thread
From: Benjamin Herrenschmidt @ 2019-05-14 5:43 UTC (permalink / raw)
To: Shawn Landden; +Cc: Paul Mackerras, linuxppc-dev
On Mon, 2019-05-13 at 22:44 -0300, Shawn Landden wrote:
> +
> +/*
> + * Were we in user mode when we were
> + * interrupted?
> + *
> + * Doing kernel_altivec/vsx_begin/end() is ok if we are running
> + * in an interrupt context from user mode - we'll just
> + * save the FPU state as required.
> + */
> +static bool interrupted_user_mode(void)
> +{
> + struct pt_regs *regs = get_irq_regs();
> +
> + return regs && user_mode(regs);
> +}
> +
That's interesting .... it *could* work but we'll have to careful audit
the code to make sure thats ok.
We probably also want to handle the case where the CPU is in the idle
loop.
Do we always save the user state when switching out these days ? If
yes, then there's no "live" state to worry about...
Cheers,
Ben.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-14 5:43 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Benjamin Herrenschmidt
@ 2019-05-14 15:44 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 15:44 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: Paul Mackerras, linuxppc-dev
On Tue, May 14, 2019 at 12:43 AM Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
>
> On Mon, 2019-05-13 at 22:44 -0300, Shawn Landden wrote:
> > +
> > +/*
> > + * Were we in user mode when we were
> > + * interrupted?
> > + *
> > + * Doing kernel_altivec/vsx_begin/end() is ok if we are running
> > + * in an interrupt context from user mode - we'll just
> > + * save the FPU state as required.
> > + */
> > +static bool interrupted_user_mode(void)
> > +{
> > + struct pt_regs *regs = get_irq_regs();
> > +
> > + return regs && user_mode(regs);
> > +}
> > +
>
> That's interesting .... it *could* work but we'll have to careful audit
> the code to make sure thats ok.
>
> We probably also want to handle the case where the CPU is in the idle
> loop.
That is the next patch. It is best to split these up because then git
bisect works better, and these are higher-risk changes.
>
> Do we always save the user state when switching out these days ? If
> yes, then there's no "live" state to worry about...
>
> Cheers,
> Ben.
>
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* [v3 1/2] [PowerPC] Add simd.h implementation
2019-05-14 1:44 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Shawn Landden
2019-05-14 1:44 ` [PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
2019-05-14 5:43 ` [PATCH 1/2] [PowerPC] Add simd.h implementation Benjamin Herrenschmidt
@ 2019-05-14 15:46 ` Shawn Landden
2019-05-14 15:46 ` [v3 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
2 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 15:46 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 17 +++++++++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..2fe26f258
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+#ifdef CONFIG_PPC_FPU
+extern bool may_use_simd(void);
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..ef534831f 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [v3 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-14 15:46 ` [v3 " Shawn Landden
@ 2019-05-14 15:46 ` Shawn Landden
2019-05-15 1:03 ` kbuild test robot
0 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 15:46 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
This second patch is separate because it could be wrong,
like I am not sure about how kernel thread migration works,
and it is even allowing simd in preemptible kernel code.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/switch_to.h | 15 ++-----
arch/powerpc/kernel/process.c | 60 ++++++++++++++++++++++++++--
2 files changed, 59 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h
index 5b03d8a82..c79f7d24a 100644
--- a/arch/powerpc/include/asm/switch_to.h
+++ b/arch/powerpc/include/asm/switch_to.h
@@ -30,10 +30,7 @@ extern void enable_kernel_fp(void);
extern void flush_fp_to_thread(struct task_struct *);
extern void giveup_fpu(struct task_struct *);
extern void save_fpu(struct task_struct *);
-static inline void disable_kernel_fp(void)
-{
- msr_check_and_clear(MSR_FP);
-}
+extern void disable_kernel_fp(void);
#else
static inline void save_fpu(struct task_struct *t) { }
static inline void flush_fp_to_thread(struct task_struct *t) { }
@@ -44,10 +41,7 @@ extern void enable_kernel_altivec(void);
extern void flush_altivec_to_thread(struct task_struct *);
extern void giveup_altivec(struct task_struct *);
extern void save_altivec(struct task_struct *);
-static inline void disable_kernel_altivec(void)
-{
- msr_check_and_clear(MSR_VEC);
-}
+extern void disable_kernel_altivec(void);
#else
static inline void save_altivec(struct task_struct *t) { }
static inline void __giveup_altivec(struct task_struct *t) { }
@@ -56,10 +50,7 @@ static inline void __giveup_altivec(struct task_struct *t) { }
#ifdef CONFIG_VSX
extern void enable_kernel_vsx(void);
extern void flush_vsx_to_thread(struct task_struct *);
-static inline void disable_kernel_vsx(void)
-{
- msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
-}
+extern void disable_kernel_vsx(void);
#endif
#ifdef CONFIG_SPE
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index ef534831f..4ffc9c664 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -170,6 +170,29 @@ void __msr_check_and_clear(unsigned long bits)
EXPORT_SYMBOL(__msr_check_and_clear);
#ifdef CONFIG_PPC_FPU
+/*
+ * Track whether the kernel is using the FPU state
+ * currently.
+ *
+ * This flag is used:
+ *
+ * - by IRQ context code to potentially use the FPU
+ * if it's unused.
+ *
+ * - to debug kernel_fpu/altivec/vsx_begin()/end() correctness
+ */
+static DEFINE_PER_CPU(bool, in_kernel_fpu);
+
+static bool kernel_fpu_disabled(void)
+{
+ return this_cpu_read(in_kernel_fpu);
+}
+
+static bool interrupted_kernel_fpu_idle(void)
+{
+ return !kernel_fpu_disabled();
+}
+
static void __giveup_fpu(struct task_struct *tsk)
{
unsigned long msr;
@@ -230,7 +253,8 @@ void enable_kernel_fp(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP);
@@ -251,6 +275,15 @@ void enable_kernel_fp(void)
}
EXPORT_SYMBOL(enable_kernel_fp);
+void disable_kernel_fp(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+
+ msr_check_and_clear(MSR_FP);
+}
+EXPORT_SYMBOL(disable_kernel_fp);
+
static int restore_fp(struct task_struct *tsk)
{
if (tsk->thread.load_fp || tm_active_with_fp(tsk)) {
@@ -295,7 +328,8 @@ void enable_kernel_altivec(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_VEC);
@@ -316,6 +350,14 @@ void enable_kernel_altivec(void)
}
EXPORT_SYMBOL(enable_kernel_altivec);
+extern void disable_kernel_altivec(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_VEC);
+}
+EXPORT_SYMBOL(disable_kernel_altivec);
+
/*
* Make sure the VMX/Altivec register state in the
* the thread_struct is up to date for task tsk.
@@ -371,7 +413,8 @@ static bool interrupted_user_mode(void)
bool may_use_simd(void)
{
return !in_interrupt() ||
- interrupted_user_mode();
+ interrupted_user_mode() ||
+ interrupted_kernel_fpu_idle();
}
EXPORT_SYMBOL(may_use_simd);
@@ -411,7 +454,8 @@ void enable_kernel_vsx(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);
@@ -433,6 +477,14 @@ void enable_kernel_vsx(void)
}
EXPORT_SYMBOL(enable_kernel_vsx);
+void disable_kernel_vsx(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
+}
+EXPORT_SYMBOL(disable_kernel_vsx);
+
void flush_vsx_to_thread(struct task_struct *tsk)
{
if (tsk->thread.regs) {
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [v2 1/2] [PowerPC] Add simd.h implementation
2019-05-13 0:51 ` Shawn Landden
` (2 preceding siblings ...)
(?)
@ 2019-05-14 2:23 ` Shawn Landden
2019-05-14 2:23 ` [v2 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
-1 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 2:23 UTC (permalink / raw)
Cc: Shawn Landden, Paul Mackerras, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 10 ++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..9b066d633
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+extern bool may_use_simd(void);
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..ef534831f 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [v2 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-14 2:23 ` [v2 1/2] [PowerPC] Add simd.h implementation Shawn Landden
@ 2019-05-14 2:23 ` Shawn Landden
2019-05-14 7:22 ` Russell Currey
0 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 2:23 UTC (permalink / raw)
Cc: Shawn Landden, Paul Mackerras, linuxppc-dev
This second patch is separate because it could be wrong,
like I am not sure about how kernel thread migration works,
and it is even allowing simd in preemptible kernel code.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/switch_to.h | 15 ++-----
arch/powerpc/kernel/process.c | 60 ++++++++++++++++++++++++++--
2 files changed, 59 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h
index 5b03d8a82..c79f7d24a 100644
--- a/arch/powerpc/include/asm/switch_to.h
+++ b/arch/powerpc/include/asm/switch_to.h
@@ -30,10 +30,7 @@ extern void enable_kernel_fp(void);
extern void flush_fp_to_thread(struct task_struct *);
extern void giveup_fpu(struct task_struct *);
extern void save_fpu(struct task_struct *);
-static inline void disable_kernel_fp(void)
-{
- msr_check_and_clear(MSR_FP);
-}
+extern void disable_kernel_fp(void);
#else
static inline void save_fpu(struct task_struct *t) { }
static inline void flush_fp_to_thread(struct task_struct *t) { }
@@ -44,10 +41,7 @@ extern void enable_kernel_altivec(void);
extern void flush_altivec_to_thread(struct task_struct *);
extern void giveup_altivec(struct task_struct *);
extern void save_altivec(struct task_struct *);
-static inline void disable_kernel_altivec(void)
-{
- msr_check_and_clear(MSR_VEC);
-}
+extern void disable_kernel_altivec(void);
#else
static inline void save_altivec(struct task_struct *t) { }
static inline void __giveup_altivec(struct task_struct *t) { }
@@ -56,10 +50,7 @@ static inline void __giveup_altivec(struct task_struct *t) { }
#ifdef CONFIG_VSX
extern void enable_kernel_vsx(void);
extern void flush_vsx_to_thread(struct task_struct *);
-static inline void disable_kernel_vsx(void)
-{
- msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
-}
+extern void disable_kernel_vsx(void);
#endif
#ifdef CONFIG_SPE
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index ef534831f..d15f2cb51 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -170,6 +170,29 @@ void __msr_check_and_clear(unsigned long bits)
EXPORT_SYMBOL(__msr_check_and_clear);
#ifdef CONFIG_PPC_FPU
+/*
+ * Track whether the kernel is using the FPU state
+ * currently.
+ *
+ * This flag is used:
+ *
+ * - by IRQ context code to potentially use the FPU
+ * if it's unused.
+ *
+ * - to debug kernel_fpu/altivec/vsx_begin()/end() correctness
+ */
+static DEFINE_PER_CPU(bool, in_kernel_fpu);
+
+static bool kernel_fpu_disabled(void)
+{
+ return this_cpu_read(in_kernel_fpu);
+}
+
+static bool interrupted_kernel_fpu_idle(void)
+{
+ return !kernel_fpu_disabled();
+}
+
static void __giveup_fpu(struct task_struct *tsk)
{
unsigned long msr;
@@ -230,7 +253,8 @@ void enable_kernel_fp(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP);
@@ -251,6 +275,15 @@ void enable_kernel_fp(void)
}
EXPORT_SYMBOL(enable_kernel_fp);
+void disable_kernel_fp(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+
+ msr_check_and_clear(MSR_FP);
+}
+EXPORT_SYMBOL(disable_kernel_fp);
+
static int restore_fp(struct task_struct *tsk)
{
if (tsk->thread.load_fp || tm_active_with_fp(tsk)) {
@@ -295,7 +328,8 @@ void enable_kernel_altivec(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_VEC);
@@ -316,6 +350,14 @@ void enable_kernel_altivec(void)
}
EXPORT_SYMBOL(enable_kernel_altivec);
+extern void disable_kernel_altivec(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_VEC);
+}
+EXPORT_SYMBOL(disable_kernel_altivec);
+
/*
* Make sure the VMX/Altivec register state in the
* the thread_struct is up to date for task tsk.
@@ -371,7 +413,8 @@ static bool interrupted_user_mode(void)
bool may_use_simd(void)
{
return !in_interrupt() ||
- interrupted_user_mode();
+ interrupted_user_mode() ||
+ interrupted_kernel_fpu_idle();
}
EXPORT_SYMBOL(may_use_simd);
@@ -411,7 +454,8 @@ void enable_kernel_vsx(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);
@@ -433,6 +477,14 @@ void enable_kernel_vsx(void)
}
EXPORT_SYMBOL(enable_kernel_vsx);
+void disable_kernel_vsx(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
+}
+EXPORT_SYMBOL(disable_kernel_vsx);
+
void flush_vsx_to_thread(struct task_struct *tsk)
{
if (tsk->thread.regs) {
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [v2 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-14 2:23 ` [v2 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
@ 2019-05-14 7:22 ` Russell Currey
2019-05-14 15:35 ` Shawn Landden
0 siblings, 1 reply; 27+ messages in thread
From: Russell Currey @ 2019-05-14 7:22 UTC (permalink / raw)
To: Shawn Landden; +Cc: linuxppc-dev, Paul Mackerras
On Mon, 2019-05-13 at 23:23 -0300, Shawn Landden wrote:
> This second patch is separate because it could be wrong,
> like I am not sure about how kernel thread migration works,
> and it is even allowing simd in preemptible kernel code.
>
> Signed-off-by: Shawn Landden <shawn@git.icu>
> ---
Hi Shawn,
This patch doesn't build on 64-bit embedded (ppc64e_defconfig):
arch/powerpc/kernel/process.c:194:13: error: 'interrupted_kernel_fpu_idle' defined but not used [-Werror=unused-function]
static bool interrupted_kernel_fpu_idle(void)
^~~~~~~~~~~~~~~~~~~~~~~~~~~
and otherwise adds two sparse warnings:
+arch/powerpc/kernel/process.c:356:13: warning: function 'disable_kernel_altivec' with external linkage has definition
+arch/powerpc/kernel/process.c:416:6: warning: symbol 'may_use_simd' was not declared. Should it be static?
There's also some style issues (spaces instead of tabs).
Reported by snowpatch (see https://patchwork.ozlabs.org/patch/1099181/)
- Russell
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [v2 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-14 7:22 ` Russell Currey
@ 2019-05-14 15:35 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 15:35 UTC (permalink / raw)
To: Russell Currey; +Cc: linuxppc-dev, Paul Mackerras
On Tue, May 14, 2019 at 2:22 AM Russell Currey <ruscur@russell.cc> wrote:
>
> On Mon, 2019-05-13 at 23:23 -0300, Shawn Landden wrote:
> > This second patch is separate because it could be wrong,
> > like I am not sure about how kernel thread migration works,
> > and it is even allowing simd in preemptible kernel code.
> >
> > Signed-off-by: Shawn Landden <shawn@git.icu>
> > ---
>
> Hi Shawn,
>
> This patch doesn't build on 64-bit embedded (ppc64e_defconfig):
>
> arch/powerpc/kernel/process.c:194:13: error: 'interrupted_kernel_fpu_idle' defined but not used [-Werror=unused-function]
> static bool interrupted_kernel_fpu_idle(void)
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks for noticing this. I knew what I needed to do, and this was
just sloppiness on my end.
>
> and otherwise adds two sparse warnings:
>
> +arch/powerpc/kernel/process.c:356:13: warning: function 'disable_kernel_altivec' with external linkage has definition
> +arch/powerpc/kernel/process.c:416:6: warning: symbol 'may_use_simd' was not declared. Should it be static?
This is the same problem as above.
>
> There's also some style issues (spaces instead of tabs).
Yes, I have to be careful using nano on a VPS.
New patch coming in 1 sec.
>
> Reported by snowpatch (see https://patchwork.ozlabs.org/patch/1099181/)
>
> - Russell
>
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH] powerpc: Allow may_use_simd() to function as feature detection
2019-05-13 0:51 ` Shawn Landden
` (3 preceding siblings ...)
(?)
@ 2019-05-14 12:49 ` Shawn Landden
2019-05-14 18:06 ` Segher Boessenkool
-1 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 12:49 UTC (permalink / raw)
Cc: Shawn Landden, Paul Mackerras, linuxppc-dev
ARM does this, so we might as well too.
I am a bit confused however as CONFIG_ALTIVEC does not select
CONFIG_PPC_FPU. Would you ever have altivec without a fpu?
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
index 9b066d633..2fe26f258 100644
--- a/arch/powerpc/include/asm/simd.h
+++ b/arch/powerpc/include/asm/simd.h
@@ -7,4 +7,11 @@
* It's always ok in process context (ie "not interrupt")
* but it is sometimes ok even from an irq.
*/
+#ifdef CONFIG_PPC_FPU
extern bool may_use_simd(void);
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH] powerpc: Allow may_use_simd() to function as feature detection
2019-05-14 12:49 ` [PATCH] powerpc: Allow may_use_simd() to function as feature detection Shawn Landden
@ 2019-05-14 18:06 ` Segher Boessenkool
2019-05-14 19:00 ` Shawn Landden
0 siblings, 1 reply; 27+ messages in thread
From: Segher Boessenkool @ 2019-05-14 18:06 UTC (permalink / raw)
To: Shawn Landden; +Cc: linuxppc-dev, Paul Mackerras
On Tue, May 14, 2019 at 09:49:18AM -0300, Shawn Landden wrote:
> ARM does this, so we might as well too.
> I am a bit confused however as CONFIG_ALTIVEC does not select
> CONFIG_PPC_FPU. Would you ever have altivec without a fpu?
There is no hardware like that, none supported anyway. It does not make
very much sense, and it cannot happen with VSX, so no hardware like it
will ever show up most likely.
It is much simpler to just make a Kconfig dependency (or a select) between
the symbols than to have to add code like this patch.
Segher
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH] powerpc: Allow may_use_simd() to function as feature detection
2019-05-14 18:06 ` Segher Boessenkool
@ 2019-05-14 19:00 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-14 19:00 UTC (permalink / raw)
To: Segher Boessenkool; +Cc: linuxppc-dev, Paul Mackerras
[-- Attachment #1: Type: text/plain, Size: 884 bytes --]
Wireguard is not in the kernel (yet) and uses these symbols. (It is
IS_ENABLED so doesn't need it, but I think it is reasonable) I think these
enable/disable symbols should not be marked GPL-only.
El mar., 14 may. 2019 13:06, Segher Boessenkool <segher@kernel.crashing.org>
escribió:
> On Tue, May 14, 2019 at 09:49:18AM -0300, Shawn Landden wrote:
> > ARM does this, so we might as well too.
> > I am a bit confused however as CONFIG_ALTIVEC does not select
> > CONFIG_PPC_FPU. Would you ever have altivec without a fpu?
>
> There is no hardware like that, none supported anyway. It does not make
> very much sense, and it cannot happen with VSX, so no hardware like it
> will ever show up most likely.
>
> It is much simpler to just make a Kconfig dependency (or a select) between
> the symbols than to have to add code like this patch.
>
>
> Segher
>
[-- Attachment #2: Type: text/html, Size: 1437 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-13 0:51 ` Shawn Landden
` (4 preceding siblings ...)
(?)
@ 2019-05-15 1:36 ` Shawn Landden
-1 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-15 1:36 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 17 +++++++++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..2fe26f258
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+#ifdef CONFIG_PPC_FPU
+extern bool may_use_simd(void);
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..ef534831f 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [v4 PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-13 0:51 ` Shawn Landden
` (5 preceding siblings ...)
(?)
@ 2019-05-15 1:37 ` Shawn Landden
2019-05-15 1:37 ` [v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
` (2 more replies)
-1 siblings, 3 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-15 1:37 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 17 +++++++++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..2fe26f258
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+#ifdef CONFIG_PPC_FPU
+extern bool may_use_simd(void);
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..ef534831f 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-15 1:37 ` [v4 PATCH " Shawn Landden
@ 2019-05-15 1:37 ` Shawn Landden
2019-05-15 6:27 ` [v4 PATCH 1/2] [PowerPC] Add simd.h implementation Christophe Leroy
2019-05-18 16:04 ` [RESEND v4 " Shawn Landden
2 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-15 1:37 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
This second patch is separate because it could be wrong,
like I am not sure about how kernel thread migration works,
and it is even allowing simd in preemptible kernel code.
v4: fix build without CONFIG_AVX
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/switch_to.h | 15 +---
arch/powerpc/kernel/process.c | 117 +++++++++++++++++++--------
2 files changed, 88 insertions(+), 44 deletions(-)
diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h
index 5b03d8a82..c79f7d24a 100644
--- a/arch/powerpc/include/asm/switch_to.h
+++ b/arch/powerpc/include/asm/switch_to.h
@@ -30,10 +30,7 @@ extern void enable_kernel_fp(void);
extern void flush_fp_to_thread(struct task_struct *);
extern void giveup_fpu(struct task_struct *);
extern void save_fpu(struct task_struct *);
-static inline void disable_kernel_fp(void)
-{
- msr_check_and_clear(MSR_FP);
-}
+extern void disable_kernel_fp(void);
#else
static inline void save_fpu(struct task_struct *t) { }
static inline void flush_fp_to_thread(struct task_struct *t) { }
@@ -44,10 +41,7 @@ extern void enable_kernel_altivec(void);
extern void flush_altivec_to_thread(struct task_struct *);
extern void giveup_altivec(struct task_struct *);
extern void save_altivec(struct task_struct *);
-static inline void disable_kernel_altivec(void)
-{
- msr_check_and_clear(MSR_VEC);
-}
+extern void disable_kernel_altivec(void);
#else
static inline void save_altivec(struct task_struct *t) { }
static inline void __giveup_altivec(struct task_struct *t) { }
@@ -56,10 +50,7 @@ static inline void __giveup_altivec(struct task_struct *t) { }
#ifdef CONFIG_VSX
extern void enable_kernel_vsx(void);
extern void flush_vsx_to_thread(struct task_struct *);
-static inline void disable_kernel_vsx(void)
-{
- msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
-}
+extern void disable_kernel_vsx(void);
#endif
#ifdef CONFIG_SPE
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index ef534831f..0136fd132 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -170,6 +170,29 @@ void __msr_check_and_clear(unsigned long bits)
EXPORT_SYMBOL(__msr_check_and_clear);
#ifdef CONFIG_PPC_FPU
+/*
+ * Track whether the kernel is using the FPU state
+ * currently.
+ *
+ * This flag is used:
+ *
+ * - by IRQ context code to potentially use the FPU
+ * if it's unused.
+ *
+ * - to debug kernel_fpu/altivec/vsx_begin()/end() correctness
+ */
+static DEFINE_PER_CPU(bool, in_kernel_fpu);
+
+static bool kernel_fpu_disabled(void)
+{
+ return this_cpu_read(in_kernel_fpu);
+}
+
+static bool interrupted_kernel_fpu_idle(void)
+{
+ return !kernel_fpu_disabled();
+}
+
static void __giveup_fpu(struct task_struct *tsk)
{
unsigned long msr;
@@ -230,7 +253,8 @@ void enable_kernel_fp(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP);
@@ -251,6 +275,15 @@ void enable_kernel_fp(void)
}
EXPORT_SYMBOL(enable_kernel_fp);
+void disable_kernel_fp(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+
+ msr_check_and_clear(MSR_FP);
+}
+EXPORT_SYMBOL(disable_kernel_fp);
+
static int restore_fp(struct task_struct *tsk)
{
if (tsk->thread.load_fp || tm_active_with_fp(tsk)) {
@@ -260,6 +293,37 @@ static int restore_fp(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode() ||
+ interrupted_kernel_fpu_idle();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
static int restore_fp(struct task_struct *tsk) { return 0; }
#endif /* CONFIG_PPC_FPU */
@@ -295,7 +359,8 @@ void enable_kernel_altivec(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_VEC);
@@ -316,6 +381,14 @@ void enable_kernel_altivec(void)
}
EXPORT_SYMBOL(enable_kernel_altivec);
+extern void disable_kernel_altivec(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_VEC);
+}
+EXPORT_SYMBOL(disable_kernel_altivec);
+
/*
* Make sure the VMX/Altivec register state in the
* the thread_struct is up to date for task tsk.
@@ -346,35 +419,6 @@ static int restore_altivec(struct task_struct *tsk)
return 0;
}
-/*
- * Were we in user mode when we were
- * interrupted?
- *
- * Doing kernel_altivec/vsx_begin/end() is ok if we are running
- * in an interrupt context from user mode - we'll just
- * save the FPU state as required.
- */
-static bool interrupted_user_mode(void)
-{
- struct pt_regs *regs = get_irq_regs();
-
- return regs && user_mode(regs);
-}
-
-/*
- * Can we use FPU in kernel mode with the
- * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
- *
- * It's always ok in process context (ie "not interrupt")
- * but it is sometimes ok even from an irq.
- */
-bool may_use_simd(void)
-{
- return !in_interrupt() ||
- interrupted_user_mode();
-}
-EXPORT_SYMBOL(may_use_simd);
-
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
@@ -411,7 +455,8 @@ void enable_kernel_vsx(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);
@@ -433,6 +478,14 @@ void enable_kernel_vsx(void)
}
EXPORT_SYMBOL(enable_kernel_vsx);
+void disable_kernel_vsx(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
+}
+EXPORT_SYMBOL(disable_kernel_vsx);
+
void flush_vsx_to_thread(struct task_struct *tsk)
{
if (tsk->thread.regs) {
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [v4 PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-15 1:37 ` [v4 PATCH " Shawn Landden
2019-05-15 1:37 ` [v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
@ 2019-05-15 6:27 ` Christophe Leroy
2019-05-16 1:12 ` Shawn Landden
2019-05-18 16:04 ` [RESEND v4 " Shawn Landden
2 siblings, 1 reply; 27+ messages in thread
From: Christophe Leroy @ 2019-05-15 6:27 UTC (permalink / raw)
To: Shawn Landden; +Cc: linuxppc-dev, Paul Mackerras
Le 15/05/2019 à 03:37, Shawn Landden a écrit :
> Based off the x86 one.
>
> WireGuard really wants to be able to do SIMD in interrupts,
> so it can accelerate its in-bound path.
>
> Signed-off-by: Shawn Landden <shawn@git.icu>
> ---
Could you please as usual list here the changes provided by each version
to ease the review ?
Thanks
Christophe
> arch/powerpc/include/asm/simd.h | 17 +++++++++++++++++
> arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
> 2 files changed, 47 insertions(+)
> create mode 100644 arch/powerpc/include/asm/simd.h
>
> diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
> new file mode 100644
> index 000000000..2fe26f258
> --- /dev/null
> +++ b/arch/powerpc/include/asm/simd.h
> @@ -0,0 +1,17 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +
> +/*
> + * may_use_simd - whether it is allowable at this time to issue SIMD
> + * instructions or access the SIMD register file
> + *
> + * It's always ok in process context (ie "not interrupt")
> + * but it is sometimes ok even from an irq.
> + */
> +#ifdef CONFIG_PPC_FPU
> +extern bool may_use_simd(void);
> +#else
> +static inline bool may_use_simd(void)
> +{
> + return false;
> +}
> +#endif
> diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
> index dd9e0d538..ef534831f 100644
> --- a/arch/powerpc/kernel/process.c
> +++ b/arch/powerpc/kernel/process.c
> @@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
> }
> return 0;
> }
> +
> +/*
> + * Were we in user mode when we were
> + * interrupted?
> + *
> + * Doing kernel_altivec/vsx_begin/end() is ok if we are running
> + * in an interrupt context from user mode - we'll just
> + * save the FPU state as required.
> + */
> +static bool interrupted_user_mode(void)
> +{
> + struct pt_regs *regs = get_irq_regs();
> +
> + return regs && user_mode(regs);
> +}
> +
> +/*
> + * Can we use FPU in kernel mode with the
> + * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
> + *
> + * It's always ok in process context (ie "not interrupt")
> + * but it is sometimes ok even from an irq.
> + */
> +bool may_use_simd(void)
> +{
> + return !in_interrupt() ||
> + interrupted_user_mode();
> +}
> +EXPORT_SYMBOL(may_use_simd);
> +
> #else
> #define loadvec(thr) 0
> static inline int restore_altivec(struct task_struct *tsk) { return 0; }
>
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [v4 PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-15 6:27 ` [v4 PATCH 1/2] [PowerPC] Add simd.h implementation Christophe Leroy
@ 2019-05-16 1:12 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-16 1:12 UTC (permalink / raw)
To: Christophe Leroy; +Cc: linuxppc-dev, Paul Mackerras
On Wed, May 15, 2019 at 1:27 AM Christophe Leroy
<christophe.leroy@c-s.fr> wrote:
> Could you please as usual list here the changes provided by each version
> to ease the review ?
A bunch of embarrassing stuff that caused it not to build on some
set-ups (the functions were under the wrong include guards), and I
added include guards on simd.h so that you can use may_use_simd() even
if you don't have the FPU enabled (ARM's simd.h does this).
^ permalink raw reply [flat|nested] 27+ messages in thread
* [RESEND v4 PATCH 1/2] [PowerPC] Add simd.h implementation
2019-05-15 1:37 ` [v4 PATCH " Shawn Landden
2019-05-15 1:37 ` [v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
2019-05-15 6:27 ` [v4 PATCH 1/2] [PowerPC] Add simd.h implementation Christophe Leroy
@ 2019-05-18 16:04 ` Shawn Landden
2019-05-18 16:04 ` [RESEND v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code Shawn Landden
2 siblings, 1 reply; 27+ messages in thread
From: Shawn Landden @ 2019-05-18 16:04 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
Based off the x86 one.
WireGuard really wants to be able to do SIMD in interrupts,
so it can accelerate its in-bound path.
v4: allow using the may_use_simd symbol even when it always
returns false (via include guards)
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/simd.h | 17 +++++++++++++++++
arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
create mode 100644 arch/powerpc/include/asm/simd.h
diff --git a/arch/powerpc/include/asm/simd.h b/arch/powerpc/include/asm/simd.h
new file mode 100644
index 000000000..2fe26f258
--- /dev/null
+++ b/arch/powerpc/include/asm/simd.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue SIMD
+ * instructions or access the SIMD register file
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+#ifdef CONFIG_PPC_FPU
+extern bool may_use_simd(void);
+#else
+static inline bool may_use_simd(void)
+{
+ return false;
+}
+#endif
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index dd9e0d538..ef534831f 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -345,6 +345,36 @@ static int restore_altivec(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread* [RESEND v4 PATCH 2/2] [PowerPC] Allow use of SIMD in interrupts from kernel code
2019-05-18 16:04 ` [RESEND v4 " Shawn Landden
@ 2019-05-18 16:04 ` Shawn Landden
0 siblings, 0 replies; 27+ messages in thread
From: Shawn Landden @ 2019-05-18 16:04 UTC (permalink / raw)
Cc: Paul Mackerras, Shawn Landden, linuxppc-dev
This even allows simd in preemptible kernel code,
as does x86, although this is rarely safe (could be used with
kthread_create_on_cpu). All callers are disabling preemption.
v4: fix build without CONFIG_AVX
change commit message
Signed-off-by: Shawn Landden <shawn@git.icu>
---
arch/powerpc/include/asm/switch_to.h | 15 +---
arch/powerpc/kernel/process.c | 117 +++++++++++++++++++--------
2 files changed, 88 insertions(+), 44 deletions(-)
diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h
index 5b03d8a82..c79f7d24a 100644
--- a/arch/powerpc/include/asm/switch_to.h
+++ b/arch/powerpc/include/asm/switch_to.h
@@ -30,10 +30,7 @@ extern void enable_kernel_fp(void);
extern void flush_fp_to_thread(struct task_struct *);
extern void giveup_fpu(struct task_struct *);
extern void save_fpu(struct task_struct *);
-static inline void disable_kernel_fp(void)
-{
- msr_check_and_clear(MSR_FP);
-}
+extern void disable_kernel_fp(void);
#else
static inline void save_fpu(struct task_struct *t) { }
static inline void flush_fp_to_thread(struct task_struct *t) { }
@@ -44,10 +41,7 @@ extern void enable_kernel_altivec(void);
extern void flush_altivec_to_thread(struct task_struct *);
extern void giveup_altivec(struct task_struct *);
extern void save_altivec(struct task_struct *);
-static inline void disable_kernel_altivec(void)
-{
- msr_check_and_clear(MSR_VEC);
-}
+extern void disable_kernel_altivec(void);
#else
static inline void save_altivec(struct task_struct *t) { }
static inline void __giveup_altivec(struct task_struct *t) { }
@@ -56,10 +50,7 @@ static inline void __giveup_altivec(struct task_struct *t) { }
#ifdef CONFIG_VSX
extern void enable_kernel_vsx(void);
extern void flush_vsx_to_thread(struct task_struct *);
-static inline void disable_kernel_vsx(void)
-{
- msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
-}
+extern void disable_kernel_vsx(void);
#endif
#ifdef CONFIG_SPE
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index ef534831f..0136fd132 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -170,6 +170,29 @@ void __msr_check_and_clear(unsigned long bits)
EXPORT_SYMBOL(__msr_check_and_clear);
#ifdef CONFIG_PPC_FPU
+/*
+ * Track whether the kernel is using the FPU state
+ * currently.
+ *
+ * This flag is used:
+ *
+ * - by IRQ context code to potentially use the FPU
+ * if it's unused.
+ *
+ * - to debug kernel_fpu/altivec/vsx_begin()/end() correctness
+ */
+static DEFINE_PER_CPU(bool, in_kernel_fpu);
+
+static bool kernel_fpu_disabled(void)
+{
+ return this_cpu_read(in_kernel_fpu);
+}
+
+static bool interrupted_kernel_fpu_idle(void)
+{
+ return !kernel_fpu_disabled();
+}
+
static void __giveup_fpu(struct task_struct *tsk)
{
unsigned long msr;
@@ -230,7 +253,8 @@ void enable_kernel_fp(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP);
@@ -251,6 +275,15 @@ void enable_kernel_fp(void)
}
EXPORT_SYMBOL(enable_kernel_fp);
+void disable_kernel_fp(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+
+ msr_check_and_clear(MSR_FP);
+}
+EXPORT_SYMBOL(disable_kernel_fp);
+
static int restore_fp(struct task_struct *tsk)
{
if (tsk->thread.load_fp || tm_active_with_fp(tsk)) {
@@ -260,6 +293,37 @@ static int restore_fp(struct task_struct *tsk)
}
return 0;
}
+
+/*
+ * Were we in user mode when we were
+ * interrupted?
+ *
+ * Doing kernel_altivec/vsx_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+
+ return regs && user_mode(regs);
+}
+
+/*
+ * Can we use FPU in kernel mode with the
+ * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+bool may_use_simd(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode() ||
+ interrupted_kernel_fpu_idle();
+}
+EXPORT_SYMBOL(may_use_simd);
+
#else
static int restore_fp(struct task_struct *tsk) { return 0; }
#endif /* CONFIG_PPC_FPU */
@@ -295,7 +359,8 @@ void enable_kernel_altivec(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_VEC);
@@ -316,6 +381,14 @@ void enable_kernel_altivec(void)
}
EXPORT_SYMBOL(enable_kernel_altivec);
+extern void disable_kernel_altivec(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_VEC);
+}
+EXPORT_SYMBOL(disable_kernel_altivec);
+
/*
* Make sure the VMX/Altivec register state in the
* the thread_struct is up to date for task tsk.
@@ -346,35 +419,6 @@ static int restore_altivec(struct task_struct *tsk)
return 0;
}
-/*
- * Were we in user mode when we were
- * interrupted?
- *
- * Doing kernel_altivec/vsx_begin/end() is ok if we are running
- * in an interrupt context from user mode - we'll just
- * save the FPU state as required.
- */
-static bool interrupted_user_mode(void)
-{
- struct pt_regs *regs = get_irq_regs();
-
- return regs && user_mode(regs);
-}
-
-/*
- * Can we use FPU in kernel mode with the
- * whole "kernel_fpu/altivec/vsx_begin/end()" sequence?
- *
- * It's always ok in process context (ie "not interrupt")
- * but it is sometimes ok even from an irq.
- */
-bool may_use_simd(void)
-{
- return !in_interrupt() ||
- interrupted_user_mode();
-}
-EXPORT_SYMBOL(may_use_simd);
-
#else
#define loadvec(thr) 0
static inline int restore_altivec(struct task_struct *tsk) { return 0; }
@@ -411,7 +455,8 @@ void enable_kernel_vsx(void)
{
unsigned long cpumsr;
- WARN_ON(preemptible());
+ WARN_ON_ONCE(this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, true);
cpumsr = msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);
@@ -433,6 +478,14 @@ void enable_kernel_vsx(void)
}
EXPORT_SYMBOL(enable_kernel_vsx);
+void disable_kernel_vsx(void)
+{
+ WARN_ON_ONCE(!this_cpu_read(in_kernel_fpu));
+ this_cpu_write(in_kernel_fpu, false);
+ msr_check_and_clear(MSR_FP|MSR_VEC|MSR_VSX);
+}
+EXPORT_SYMBOL(disable_kernel_vsx);
+
void flush_vsx_to_thread(struct task_struct *tsk)
{
if (tsk->thread.regs) {
--
2.21.0.1020.gf2820cf01a
^ permalink raw reply related [flat|nested] 27+ messages in thread