From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1620CC433E0 for ; Thu, 28 Jan 2021 13:08:03 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BF97764DCE for ; Thu, 28 Jan 2021 13:08:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF97764DCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=NlM+Ii2TK1V1RqoQF82MxBNChX2rlTUnPralehkSnSI=; b=H1BKgkwZ/Eqc/WteYR/Hctg3yC oDFZ0xlzlgF+PdxFl2HZ1JtK0ANrpj80Lh0VAU+0zD6wnTPtFhYKb/uib/SvST48GxKJNrwA0By8i GSatizkLTFTsrOv1aLnwBu3apu+9bkWthcCPFRleW1htA4IEg5BI6bvf37B81U34Q2Znrmy128nw3 MILbKCwl//nNJv3Kl5COR0qXYl+R0JdzTypEW/tbSARa7wG4jWUlfsSh7z9o07if5Vyxoc6HQgL7A F5rjg69ojYQXvHQMg+Z1X6Fz/GpAazfnxz2q9ieQtXFHRmHw4gr3HvAYEScwmw3R402st/A59N0ER y7B6zkgg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l570W-0005Jb-2F; Thu, 28 Jan 2021 13:06:44 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l570S-0005Il-Sb for linux-arm-kernel@lists.infradead.org; Thu, 28 Jan 2021 13:06:41 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id AFB7B64D9F; Thu, 28 Jan 2021 13:06:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1611839200; bh=+6OMNZcUUWaDvKg8NRL1a4tI1nBxsV+pPg6GaYsCbds=; h=From:To:Cc:Subject:Date:From; b=GXr7N4Nzg3nwJkB+78EsNwIdajoG/QMs9OlZ+IG+YIPqDeLfFcQFC9PmW7cdHTVYq fDm9y4VWeeOweyXI5vQaa817K2iI1veQyxXnA8xylOF40YEBOkzpYVcuT4hR7G4TPp k2fpEC6afKc1YdWg6ClcB0GqsUP4ID+/LMbdPAvDjzVMkI85sSIZMooGDquryDTCPG tx8lv8x9qIsDpUvhUEM1fQsLx2bk3F4DczZ09irGl/EKNM9DoCoVkfQ2gOR1mflM20 gAOPAxZcmkzmcswawU4RpwOPOB5qlVUdUaoo46syL3HT9fTJm23Ln5A64VIhwBVlo0 Rvf7Z/dKUx0Nw== From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Subject: [PATCH 0/9] arm64: rework NEON yielding to avoid scheduling from asm code Date: Thu, 28 Jan 2021 14:06:16 +0100 Message-Id: <20210128130625.54076-1-ardb@kernel.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210128_080641_053524_2BF58AF7 X-CRM114-Status: GOOD ( 13.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, herbert@gondor.apana.org.au, Eric Biggers , catalin.marinas@arm.com, Dave Martin , will@kernel.org, Ard Biesheuvel , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Given how kernel mode NEON code disables preemption (to ensure that the FP/SIMD register state is protected without having to context switch it), we need to take care not to let those algorithms operate on unbounded input data, or we may end up with excessive scheduling blackouts on CONFIG_PREEMPT kernels. This is currently handled by the cond_yield_neon macros, which check the preempt count and the TIF_NEED_RESCHED flag from assembler code, and call into kernel_neon_end()+kernel_neon_begin(), triggering a reschedule. This works as expected, but is a bit messy, given how much of the state preserve/restore code in the algorithm needs to be duplicated, as well as causing the need to manage the stack frame explicitly. All of this is better handled by the compiler, especially now that we have enabled features such as the shadow call stack and BTI, and are working to improve call stack validation. In some cases, yielding is not necessary at all: algoritms that implement skciphers and use the skcipher walk API will be invoked at page granularity, which is granular enough for our purpose. In other cases, it is better to simply return early from the assembler routine if a reschedule is pending, and let the C code handle with this, by retrying the call until it completes. This removes any voluntary schedule() calls from the call stack, making the code much easier to reason about in the context of stack validation, rcu_tasks synchronization, etc. Practical note: assuming there are no objections to these changes, it may be the most convenient to take patch #1 into the arm64 tree for v5.12, and postpone the rest for merging via the crypto tree. (Note that this series was created against the cryptodev tree, and so the arm64 maintainers are also welcome to take the whole set if it applies cleanly to the arm64 tree) Cc: Dave Martin Cc: Eric Biggers Ard Biesheuvel (9): arm64: assembler: add cond_yield macro crypto: arm64/sha1-ce - simplify NEON yield crypto: arm64/sha2-ce - simplify NEON yield crypto: arm64/sha3-ce - simplify NEON yield crypto: arm64/sha512-ce - simplify NEON yield crypto: arm64/aes-neonbs - remove NEON yield calls crypto: arm64/aes-ce-mac - simplify NEON yield crypto: arm64/crc-t10dif - move NEON yield to C code arm64: assembler: remove conditional NEON yield macros arch/arm64/crypto/aes-glue.c | 21 +++-- arch/arm64/crypto/aes-modes.S | 52 +++++-------- arch/arm64/crypto/aes-neonbs-core.S | 8 +- arch/arm64/crypto/crct10dif-ce-core.S | 43 +++-------- arch/arm64/crypto/crct10dif-ce-glue.c | 30 ++++++-- arch/arm64/crypto/sha1-ce-core.S | 47 ++++-------- arch/arm64/crypto/sha1-ce-glue.c | 22 +++--- arch/arm64/crypto/sha2-ce-core.S | 38 ++++----- arch/arm64/crypto/sha2-ce-glue.c | 22 +++--- arch/arm64/crypto/sha3-ce-core.S | 81 ++++++++------------ arch/arm64/crypto/sha3-ce-glue.c | 14 ++-- arch/arm64/crypto/sha512-ce-core.S | 29 ++----- arch/arm64/crypto/sha512-ce-glue.c | 53 +++++++------ arch/arm64/include/asm/assembler.h | 78 +++---------------- 14 files changed, 209 insertions(+), 329 deletions(-) -- 2.29.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel