From: Greentime Hu <greentime.hu@sifive.com> To: palmer@dabbelt.com, paul.walmsley@sifive.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, aou@eecs.berkeley.edu Subject: [PATCH v9 13/17] riscv: Add support for kernel mode vector Date: Tue, 9 Nov 2021 17:48:25 +0800 [thread overview] Message-ID: <5700882ca97cc9fb8ee44ce127f30ca04d81495e.1636362169.git.greentime.hu@sifive.com> (raw) In-Reply-To: <cover.1636362169.git.greentime.hu@sifive.com> Add <asm/vector.h> containing kernel_rvv_begin()/kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> --- arch/riscv/include/asm/switch_to.h | 1 + arch/riscv/include/asm/vector.h | 14 +++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 144 +++++++++++++++++++++++++ arch/riscv/kernel/vector.S | 9 ++ 5 files changed, 169 insertions(+) create mode 100644 arch/riscv/include/asm/vector.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index 576204217e0f..b02064a3effd 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -79,6 +79,7 @@ static __always_inline bool has_vector(void) extern unsigned long riscv_vsize; extern void __vstate_save(struct __riscv_v_state *save_to, void *datap); extern void __vstate_restore(struct __riscv_v_state *restore_from, void *datap); +void vector_flush_cpu_state(void); static inline void __vstate_clean(struct pt_regs *regs) { diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h new file mode 100644 index 000000000000..5d7f14453f68 --- /dev/null +++ b/arch/riscv/include/asm/vector.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020 SiFive + */ + +#ifndef __ASM_RISCV_VECTOR_H +#define __ASM_RISCV_VECTOR_H + +#include <linux/types.h> + +void kernel_rvv_begin(void); +void kernel_rvv_end(void); + +#endif /* ! __ASM_RISCV_VECTOR_H */ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 344078080839..8feda6312187 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -41,6 +41,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_VECTOR) += vector.o +obj-$(CONFIG_VECTOR) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..8d2e53ea25c1 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas <catalin.marinas@arm.com> + * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org> + * Copyright (C) 2021 SiFive + */ +#include <linux/compiler.h> +#include <linux/irqflags.h> +#include <linux/percpu.h> +#include <linux/preempt.h> +#include <linux/types.h> + +#include <asm/vector.h> +#include <asm/switch_to.h> + +DECLARE_PER_CPU(bool, vector_context_busy); +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_vector - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_vector(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + + + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +static void rvv_enable(void) +{ + csr_set(CSR_STATUS, SR_VS); +} + +static void rvv_disable(void) +{ + csr_clear(CSR_STATUS, SR_VS); +} + +/* + * kernel_rvv_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_vector() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_rvv_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_rvv_end() is + * called. + */ +void kernel_rvv_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + WARN_ON(!may_use_vector()); + + /* Acquire kernel mode vector */ + get_cpu_vector_context(); + + /* Save vector state, if any */ + vstate_save(current, task_pt_regs(current)); + + /* Enable vector */ + rvv_enable(); + + /* Invalidate vector regs */ + vector_flush_cpu_state(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_begin); + +/* + * kernel_rvv_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_rvv_begin() was previously + * called, with no call to kernel_rvv_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_rvv_begin() is called again in the meantime. + */ +void kernel_rvv_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + /* Invalidate vector regs */ + vector_flush_cpu_state(); + + /* Restore vector state, if any */ + vstate_restore(current, task_pt_regs(current)); + + /* disable vector */ + rvv_disable(); + + /* release kernel mode vector */ + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_end); diff --git a/arch/riscv/kernel/vector.S b/arch/riscv/kernel/vector.S index 7f3e3a838424..a973532d2e7f 100644 --- a/arch/riscv/kernel/vector.S +++ b/arch/riscv/kernel/vector.S @@ -79,3 +79,12 @@ ENTRY(__vstate_restore) csrc sstatus, status ret ENDPROC(__vstate_restore) + +ENTRY(vector_flush_cpu_state) + vsetvli t0, x0, e8, m8 + vmv.v.i v0, 0 + vmv.v.i v8, 0 + vmv.v.i v16, 0 + vmv.v.i v24, 0 + ret +ENDPROC(vector_flush_cpu_state) -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: Greentime Hu <greentime.hu@sifive.com> To: palmer@dabbelt.com, paul.walmsley@sifive.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, aou@eecs.berkeley.edu Subject: [PATCH v9 13/17] riscv: Add support for kernel mode vector Date: Tue, 9 Nov 2021 17:48:25 +0800 [thread overview] Message-ID: <5700882ca97cc9fb8ee44ce127f30ca04d81495e.1636362169.git.greentime.hu@sifive.com> (raw) In-Reply-To: <cover.1636362169.git.greentime.hu@sifive.com> Add <asm/vector.h> containing kernel_rvv_begin()/kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> --- arch/riscv/include/asm/switch_to.h | 1 + arch/riscv/include/asm/vector.h | 14 +++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 144 +++++++++++++++++++++++++ arch/riscv/kernel/vector.S | 9 ++ 5 files changed, 169 insertions(+) create mode 100644 arch/riscv/include/asm/vector.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index 576204217e0f..b02064a3effd 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -79,6 +79,7 @@ static __always_inline bool has_vector(void) extern unsigned long riscv_vsize; extern void __vstate_save(struct __riscv_v_state *save_to, void *datap); extern void __vstate_restore(struct __riscv_v_state *restore_from, void *datap); +void vector_flush_cpu_state(void); static inline void __vstate_clean(struct pt_regs *regs) { diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h new file mode 100644 index 000000000000..5d7f14453f68 --- /dev/null +++ b/arch/riscv/include/asm/vector.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020 SiFive + */ + +#ifndef __ASM_RISCV_VECTOR_H +#define __ASM_RISCV_VECTOR_H + +#include <linux/types.h> + +void kernel_rvv_begin(void); +void kernel_rvv_end(void); + +#endif /* ! __ASM_RISCV_VECTOR_H */ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 344078080839..8feda6312187 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -41,6 +41,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_VECTOR) += vector.o +obj-$(CONFIG_VECTOR) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..8d2e53ea25c1 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas <catalin.marinas@arm.com> + * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org> + * Copyright (C) 2021 SiFive + */ +#include <linux/compiler.h> +#include <linux/irqflags.h> +#include <linux/percpu.h> +#include <linux/preempt.h> +#include <linux/types.h> + +#include <asm/vector.h> +#include <asm/switch_to.h> + +DECLARE_PER_CPU(bool, vector_context_busy); +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_vector - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_vector(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + + + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +static void rvv_enable(void) +{ + csr_set(CSR_STATUS, SR_VS); +} + +static void rvv_disable(void) +{ + csr_clear(CSR_STATUS, SR_VS); +} + +/* + * kernel_rvv_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_vector() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_rvv_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_rvv_end() is + * called. + */ +void kernel_rvv_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + WARN_ON(!may_use_vector()); + + /* Acquire kernel mode vector */ + get_cpu_vector_context(); + + /* Save vector state, if any */ + vstate_save(current, task_pt_regs(current)); + + /* Enable vector */ + rvv_enable(); + + /* Invalidate vector regs */ + vector_flush_cpu_state(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_begin); + +/* + * kernel_rvv_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_rvv_begin() was previously + * called, with no call to kernel_rvv_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_rvv_begin() is called again in the meantime. + */ +void kernel_rvv_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + /* Invalidate vector regs */ + vector_flush_cpu_state(); + + /* Restore vector state, if any */ + vstate_restore(current, task_pt_regs(current)); + + /* disable vector */ + rvv_disable(); + + /* release kernel mode vector */ + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_end); diff --git a/arch/riscv/kernel/vector.S b/arch/riscv/kernel/vector.S index 7f3e3a838424..a973532d2e7f 100644 --- a/arch/riscv/kernel/vector.S +++ b/arch/riscv/kernel/vector.S @@ -79,3 +79,12 @@ ENTRY(__vstate_restore) csrc sstatus, status ret ENDPROC(__vstate_restore) + +ENTRY(vector_flush_cpu_state) + vsetvli t0, x0, e8, m8 + vmv.v.i v0, 0 + vmv.v.i v8, 0 + vmv.v.i v16, 0 + vmv.v.i v24, 0 + ret +ENDPROC(vector_flush_cpu_state) -- 2.31.1 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2021-11-09 9:49 UTC|newest] Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-09 9:48 [PATCH v9 00/17] riscv: Add vector ISA support Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 01/17] riscv: Separate patch for cflags and aflags Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 02/17] riscv: Rename __switch_to_aux -> fpu Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 03/17] riscv: Extending cpufeature.c to detect V-extension Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 04/17] riscv: Add new csr defines related to vector extension Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 05/17] riscv: Add vector feature to compile Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-12 0:54 ` kernel test robot 2021-11-12 0:54 ` kernel test robot 2021-11-12 0:54 ` kernel test robot 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 06/17] riscv: Add has_vector/riscv_vsize to save vector features Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 07/17] riscv: Reset vector register Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2022-01-04 6:02 ` Greentime Hu 2022-01-04 6:02 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 08/17] riscv: Add vector struct and assembler definitions Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-12-14 16:29 ` Palmer Dabbelt 2021-12-14 16:29 ` Palmer Dabbelt 2022-01-07 13:28 ` Greentime Hu 2022-01-07 13:28 ` Greentime Hu 2022-01-07 13:53 ` Jessica Clarke 2022-01-07 13:53 ` Jessica Clarke 2021-11-09 9:48 ` [PATCH v9 09/17] riscv: Add task switch support for vector Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-11 13:49 ` kernel test robot 2021-11-11 13:49 ` kernel test robot 2021-11-11 13:49 ` kernel test robot 2021-11-11 19:16 ` kernel test robot 2021-11-11 19:16 ` kernel test robot 2021-11-11 19:16 ` kernel test robot 2021-12-15 18:38 ` Palmer Dabbelt 2021-12-15 18:38 ` Palmer Dabbelt 2021-11-09 9:48 ` [PATCH v9 10/17] riscv: Add ptrace vector support Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 11/17] riscv: Add sigcontext save/restore for vector Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 12/17] riscv: signal: Report signal frame size to userspace via auxv Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` Greentime Hu [this message] 2021-11-09 9:48 ` [PATCH v9 13/17] riscv: Add support for kernel mode vector Greentime Hu 2021-11-09 9:48 ` [PATCH v9 14/17] riscv: Use CSR_STATUS to replace sstatus in vector.S Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 15/17] riscv: Add vector extension XOR implementation Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 16/17] riscv: Fix an illegal instruction exception when accessing vlenb without enable vector first Greentime Hu 2021-11-09 9:48 ` Greentime Hu 2021-11-10 10:38 ` Ben Dooks 2021-11-10 10:38 ` Ben Dooks 2021-11-16 13:14 ` Greentime Hu 2021-11-16 13:14 ` Greentime Hu 2021-11-09 9:48 ` [PATCH v9 17/17] riscv: Fix a kernel panic issue if $s2 is set to a specific value before entering Linux Greentime Hu 2021-11-09 9:48 ` Greentime Hu
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5700882ca97cc9fb8ee44ce127f30ca04d81495e.1636362169.git.greentime.hu@sifive.com \ --to=greentime.hu@sifive.com \ --cc=aou@eecs.berkeley.edu \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=palmer@dabbelt.com \ --cc=paul.walmsley@sifive.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.