From: Ard Biesheuvel <ard.biesheuvel@linaro.org> To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel <ard.biesheuvel@linaro.org>, Dave Martin <Dave.Martin@arm.com>, Russell King - ARM Linux <linux@armlinux.org.uk>, Sebastian Andrzej Siewior <bigeasy@linutronix.de>, Mark Rutland <mark.rutland@arm.com>, linux-rt-users@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will.deacon@arm.com>, Steven Rostedt <rostedt@goodmis.org>, Thomas Gleixner <tglx@linutronix.de> Subject: [PATCH v3 11/20] arm64: assembler: add macros to conditionally yield the NEON under PREEMPT Date: Wed, 6 Dec 2017 19:43:37 +0000 [thread overview] Message-ID: <20171206194346.24393-12-ard.biesheuvel@linaro.org> (raw) In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Add support macros to conditionally yield the NEON (and thus the CPU) that may be called from the assembler code. In some cases, yielding the NEON involves saving and restoring a non trivial amount of context (especially in the CRC folding algorithms), and so the macro is split into three, and the code in between is only executed when the yield path is taken, allowing the context to be preserved. The third macro takes an optional label argument that marks the resume path after a yield has been performed. Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> --- arch/arm64/include/asm/assembler.h | 51 ++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 5f61487e9f93..c54e408fd5a7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -572,4 +572,55 @@ alternative_else_nop_endif #endif .endm +/* + * Check whether to yield to another runnable task from kernel mode NEON code + * (which runs with preemption disabled). + * + * if_will_cond_yield_neon + * // pre-yield patchup code + * do_cond_yield_neon + * // post-yield patchup code + * endif_yield_neon + * + * - Check whether the preempt count is exactly 1, in which case disabling + * preemption once will make the task preemptible. If this is not the case, + * yielding is pointless. + * - Check whether TIF_NEED_RESCHED is set, and if so, disable and re-enable + * kernel mode NEON (which will trigger a reschedule), and branch to the + * yield fixup code. + * + * This macro sequence clobbers x0, x1 and the flags register unconditionally, + * and may clobber x2 .. x18 if the yield path is taken. + */ + + .macro cond_yield_neon, lbl + if_will_cond_yield_neon + do_cond_yield_neon + endif_yield_neon \lbl + .endm + + .macro if_will_cond_yield_neon +#ifdef CONFIG_PREEMPT + get_thread_info x0 + ldr w1, [x0, #TSK_TI_PREEMPT] + ldr x0, [x0, #TSK_TI_FLAGS] + cmp w1, #1 // == PREEMPT_OFFSET + csel x0, x0, xzr, eq + tbnz x0, #TIF_NEED_RESCHED, 5555f // needs rescheduling? +#endif + .subsection 1 +5555: + .endm + + .macro do_cond_yield_neon + bl kernel_neon_end + bl kernel_neon_begin + .endm + + .macro endif_yield_neon, lbl=6666f + b \lbl + .previous +6666: + .endm + #endif /* __ASM_ASSEMBLER_H */ -- 2.11.0
WARNING: multiple messages have this Message-ID (diff)
From: ard.biesheuvel@linaro.org (Ard Biesheuvel) To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 11/20] arm64: assembler: add macros to conditionally yield the NEON under PREEMPT Date: Wed, 6 Dec 2017 19:43:37 +0000 [thread overview] Message-ID: <20171206194346.24393-12-ard.biesheuvel@linaro.org> (raw) In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Add support macros to conditionally yield the NEON (and thus the CPU) that may be called from the assembler code. In some cases, yielding the NEON involves saving and restoring a non trivial amount of context (especially in the CRC folding algorithms), and so the macro is split into three, and the code in between is only executed when the yield path is taken, allowing the context to be preserved. The third macro takes an optional label argument that marks the resume path after a yield has been performed. Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> --- arch/arm64/include/asm/assembler.h | 51 ++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 5f61487e9f93..c54e408fd5a7 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -572,4 +572,55 @@ alternative_else_nop_endif #endif .endm +/* + * Check whether to yield to another runnable task from kernel mode NEON code + * (which runs with preemption disabled). + * + * if_will_cond_yield_neon + * // pre-yield patchup code + * do_cond_yield_neon + * // post-yield patchup code + * endif_yield_neon + * + * - Check whether the preempt count is exactly 1, in which case disabling + * preemption once will make the task preemptible. If this is not the case, + * yielding is pointless. + * - Check whether TIF_NEED_RESCHED is set, and if so, disable and re-enable + * kernel mode NEON (which will trigger a reschedule), and branch to the + * yield fixup code. + * + * This macro sequence clobbers x0, x1 and the flags register unconditionally, + * and may clobber x2 .. x18 if the yield path is taken. + */ + + .macro cond_yield_neon, lbl + if_will_cond_yield_neon + do_cond_yield_neon + endif_yield_neon \lbl + .endm + + .macro if_will_cond_yield_neon +#ifdef CONFIG_PREEMPT + get_thread_info x0 + ldr w1, [x0, #TSK_TI_PREEMPT] + ldr x0, [x0, #TSK_TI_FLAGS] + cmp w1, #1 // == PREEMPT_OFFSET + csel x0, x0, xzr, eq + tbnz x0, #TIF_NEED_RESCHED, 5555f // needs rescheduling? +#endif + .subsection 1 +5555: + .endm + + .macro do_cond_yield_neon + bl kernel_neon_end + bl kernel_neon_begin + .endm + + .macro endif_yield_neon, lbl=6666f + b \lbl + .previous +6666: + .endm + #endif /* __ASM_ASSEMBLER_H */ -- 2.11.0
next prev parent reply other threads:[~2017-12-06 19:43 UTC|newest] Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-12-06 19:43 [PATCH v3 00/20] crypto: arm64 - play nice with CONFIG_PREEMPT Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 01/20] crypto: testmgr - add a new test case for CRC-T10DIF Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 02/20] crypto: arm64/aes-ce-ccm - move kernel mode neon en/disable into loop Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 03/20] crypto: arm64/aes-blk " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 04/20] crypto: arm64/aes-bs " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 05/20] crypto: arm64/chacha20 " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 06/20] crypto: arm64/aes-blk - remove configurable interleave Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 07/20] crypto: arm64/aes-blk - add 4 way interleave to CBC encrypt path Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 08/20] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 09/20] crypto: arm64/sha256-neon - play nice with CONFIG_PREEMPT kernels Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 10/20] arm64: assembler: add utility macros to push/pop stack frames Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-07 14:11 ` Dave Martin 2017-12-07 14:11 ` Dave Martin 2017-12-07 14:21 ` Ard Biesheuvel 2017-12-07 14:21 ` Ard Biesheuvel 2017-12-07 14:53 ` Dave Martin 2017-12-07 14:53 ` Dave Martin 2017-12-07 14:58 ` Ard Biesheuvel 2017-12-07 14:58 ` Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel [this message] 2017-12-06 19:43 ` [PATCH v3 11/20] arm64: assembler: add macros to conditionally yield the NEON under PREEMPT Ard Biesheuvel 2017-12-07 14:39 ` Dave Martin 2017-12-07 14:39 ` Dave Martin 2017-12-07 14:50 ` Ard Biesheuvel 2017-12-07 14:50 ` Ard Biesheuvel 2017-12-07 15:47 ` Ard Biesheuvel 2017-12-07 15:47 ` Ard Biesheuvel 2017-12-07 15:51 ` Ard Biesheuvel 2017-12-07 15:51 ` Ard Biesheuvel 2017-12-07 16:15 ` Dave Martin 2017-12-07 16:15 ` Dave Martin 2017-12-07 16:11 ` Dave Martin 2017-12-07 16:11 ` Dave Martin 2017-12-06 19:43 ` [PATCH v3 12/20] crypto: arm64/sha1-ce - yield NEON after every block of input Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 13/20] crypto: arm64/sha2-ce " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 14/20] crypto: arm64/aes-ccm " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 15/20] crypto: arm64/aes-blk " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 16/20] crypto: arm64/aes-bs " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 17/20] crypto: arm64/aes-ghash " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 18/20] crypto: arm64/crc32-ce " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 19/20] crypto: arm64/crct10dif-ce " Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel 2017-12-06 19:43 ` [PATCH v3 20/20] DO NOT MERGE Ard Biesheuvel 2017-12-06 19:43 ` Ard Biesheuvel
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20171206194346.24393-12-ard.biesheuvel@linaro.org \ --to=ard.biesheuvel@linaro.org \ --cc=Dave.Martin@arm.com \ --cc=bigeasy@linutronix.de \ --cc=catalin.marinas@arm.com \ --cc=herbert@gondor.apana.org.au \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-crypto@vger.kernel.org \ --cc=linux-rt-users@vger.kernel.org \ --cc=linux@armlinux.org.uk \ --cc=mark.rutland@arm.com \ --cc=peterz@infradead.org \ --cc=rostedt@goodmis.org \ --cc=tglx@linutronix.de \ --cc=will.deacon@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.