From: Eric Biggers <ebiggers@kernel.org> To: linux-crypto@vger.kernel.org, linux-riscv@lists.infradead.org, Jerry Shih <jerry.shih@sifive.com> Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel <ardb@kernel.org>, Heiko Stuebner <heiko@sntech.de>, Phoebe Chen <phoebe.chen@sifive.com>, hongrong.hsu@sifive.com, Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>, Andy Chiu <andy.chiu@sifive.com> Subject: [PATCH v2 02/12] riscv: vector: make Vector always available for softirq context Date: Fri, 5 Jan 2024 10:49:38 -0800 [thread overview] Message-ID: <20240105184950.43181-3-ebiggers@kernel.org> (raw) In-Reply-To: <20240105184950.43181-1-ebiggers@kernel.org> From: Andy Chiu <andy.chiu@sifive.com> The goal of this patch is to provide full support of Vector in kernel softirq context. So that some of the crypto alogrithms won't need scalar fallbacks. By disabling bottom halves in active kernel-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. So, softirq context is able to use Vector whenever it runs. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Signed-off-by: Eric Biggers <ebiggers@google.com> --- arch/riscv/include/asm/processor.h | 5 +++-- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 14 ++++++++++++-- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 28d19aea24b1d..e768397890673 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -71,22 +71,23 @@ #ifndef __ASSEMBLY__ struct task_struct; struct pt_regs; /* * We use a flag to track in-kernel Vector context. Currently the flag has the * following meaning: * * - bit 0: indicates whether the in-kernel Vector context is active. The - * activation of this state disables the preemption. Currently only 0 and 1 - * are valid value for this field. Other values are reserved for future uses. + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. Currently only 0 and 1 are valid value for this field. + * Other values are reserved for future uses. */ #define RISCV_KERNEL_MODE_V 0x1 /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ unsigned long ra; unsigned long sp; /* Kernel mode stack */ unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef8af413a9fc7..4d699e16c9a96 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -21,22 +21,26 @@ * instructions or access the vector register file * * Callers must not assume that the result remains true beyond the next * preempt_enable() or return from softirq context. */ static __must_check inline bool may_use_simd(void) { /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temporarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ static __must_check inline bool may_use_simd(void) { return false; } #endif /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 114cf4f0a0eb6..2fc145edae3dd 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -39,37 +39,47 @@ static inline void riscv_v_stop(u32 flags) } /* * Claim ownership of the CPU vector context for use by the calling context. * * The caller may freely manipulate the vector context metadata until * put_cpu_vector_context() is called. */ void get_cpu_vector_context(void) { - preempt_disable(); + /* + * disable softirqs so it is impossible for softirqs to nest + * get_cpu_vector_context() when kernel is actively using Vector. + */ + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); riscv_v_start(RISCV_KERNEL_MODE_V); } /* * Release the CPU vector context. * * Must be called from a context in which get_cpu_vector_context() was * previously called, with no call to put_cpu_vector_context() in the * meantime. */ void put_cpu_vector_context(void) { riscv_v_stop(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context * * Must not be called unless may_use_simd() returns true. * Task context in the vector registers is saved back to memory as necessary. * * A matching call to kernel_vector_end() must be made before returning from the -- 2.43.0
WARNING: multiple messages have this Message-ID (diff)
From: Eric Biggers <ebiggers@kernel.org> To: linux-crypto@vger.kernel.org, linux-riscv@lists.infradead.org, Jerry Shih <jerry.shih@sifive.com> Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel <ardb@kernel.org>, Heiko Stuebner <heiko@sntech.de>, Phoebe Chen <phoebe.chen@sifive.com>, hongrong.hsu@sifive.com, Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>, Andy Chiu <andy.chiu@sifive.com> Subject: [PATCH v2 02/12] riscv: vector: make Vector always available for softirq context Date: Fri, 5 Jan 2024 10:49:38 -0800 [thread overview] Message-ID: <20240105184950.43181-3-ebiggers@kernel.org> (raw) In-Reply-To: <20240105184950.43181-1-ebiggers@kernel.org> From: Andy Chiu <andy.chiu@sifive.com> The goal of this patch is to provide full support of Vector in kernel softirq context. So that some of the crypto alogrithms won't need scalar fallbacks. By disabling bottom halves in active kernel-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. So, softirq context is able to use Vector whenever it runs. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Signed-off-by: Eric Biggers <ebiggers@google.com> --- arch/riscv/include/asm/processor.h | 5 +++-- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 14 ++++++++++++-- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 28d19aea24b1d..e768397890673 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -71,22 +71,23 @@ #ifndef __ASSEMBLY__ struct task_struct; struct pt_regs; /* * We use a flag to track in-kernel Vector context. Currently the flag has the * following meaning: * * - bit 0: indicates whether the in-kernel Vector context is active. The - * activation of this state disables the preemption. Currently only 0 and 1 - * are valid value for this field. Other values are reserved for future uses. + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. Currently only 0 and 1 are valid value for this field. + * Other values are reserved for future uses. */ #define RISCV_KERNEL_MODE_V 0x1 /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ unsigned long ra; unsigned long sp; /* Kernel mode stack */ unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef8af413a9fc7..4d699e16c9a96 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -21,22 +21,26 @@ * instructions or access the vector register file * * Callers must not assume that the result remains true beyond the next * preempt_enable() or return from softirq context. */ static __must_check inline bool may_use_simd(void) { /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temporarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ static __must_check inline bool may_use_simd(void) { return false; } #endif /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 114cf4f0a0eb6..2fc145edae3dd 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -39,37 +39,47 @@ static inline void riscv_v_stop(u32 flags) } /* * Claim ownership of the CPU vector context for use by the calling context. * * The caller may freely manipulate the vector context metadata until * put_cpu_vector_context() is called. */ void get_cpu_vector_context(void) { - preempt_disable(); + /* + * disable softirqs so it is impossible for softirqs to nest + * get_cpu_vector_context() when kernel is actively using Vector. + */ + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); riscv_v_start(RISCV_KERNEL_MODE_V); } /* * Release the CPU vector context. * * Must be called from a context in which get_cpu_vector_context() was * previously called, with no call to put_cpu_vector_context() in the * meantime. */ void put_cpu_vector_context(void) { riscv_v_stop(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context * * Must not be called unless may_use_simd() returns true. * Task context in the vector registers is saved back to memory as necessary. * * A matching call to kernel_vector_end() must be made before returning from the -- 2.43.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2024-01-05 18:51 UTC|newest] Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-01-05 18:49 [PATCH v2 00/12] RISC-V crypto with reworked asm files Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 01/12] riscv: Add support for kernel mode vector Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` Eric Biggers [this message] 2024-01-05 18:49 ` [PATCH v2 02/12] riscv: vector: make Vector always available for softirq context Eric Biggers 2024-01-05 18:49 ` [PATCH v2 03/12] RISC-V: add helper function to read the vector VLEN Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 04/12] RISC-V: add TOOLCHAIN_HAS_VECTOR_CRYPTO Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 05/12] RISC-V: hook new crypto subdir into build-system Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 06/12] crypto: riscv - add vector crypto accelerated AES-{ECB,CBC,CTR,XTS} Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 07/12] crypto: riscv - add vector crypto accelerated ChaCha20 Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 08/12] crypto: riscv - add vector crypto accelerated GHASH Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 09/12] crypto: riscv - add vector crypto accelerated SHA-{256,224} Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 10/12] crypto: riscv - add vector crypto accelerated SHA-{512,384} Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 11/12] crypto: riscv - add vector crypto accelerated SM3 Eric Biggers 2024-01-05 18:49 ` Eric Biggers 2024-01-05 18:49 ` [PATCH v2 12/12] crypto: riscv - add vector crypto accelerated SM4 Eric Biggers 2024-01-05 18:49 ` Eric Biggers
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240105184950.43181-3-ebiggers@kernel.org \ --to=ebiggers@kernel.org \ --cc=andy.chiu@sifive.com \ --cc=aou@eecs.berkeley.edu \ --cc=ardb@kernel.org \ --cc=heiko@sntech.de \ --cc=hongrong.hsu@sifive.com \ --cc=jerry.shih@sifive.com \ --cc=linux-crypto@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=palmer@dabbelt.com \ --cc=paul.walmsley@sifive.com \ --cc=phoebe.chen@sifive.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.