* [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 7:50 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code Eric Biggers
` (8 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
The arm64 implementations of ChaCha and XChaCha are failing the extra
crypto self-tests following my patches to test the !may_use_simd() code
paths, which previously were untested. The problem is as follows:
When !may_use_simd(), the arm64 NEON implementations fall back to the
generic implementation, which uses the skcipher_walk API to iterate
through the src/dst scatterlists. Due to how the skcipher_walk API
works, walk.stride is set from the skcipher_alg actually being used,
which in this case is the arm64 NEON algorithm. Thus walk.stride is
5*CHACHA_BLOCK_SIZE, not CHACHA_BLOCK_SIZE.
This unnecessarily large stride shouldn't cause an actual problem.
However, the generic implementation computes round_down(nbytes,
walk.stride). round_down() assumes the round amount is a power of 2,
which 5*CHACHA_BLOCK_SIZE is not, so it gives the wrong result.
This causes the following case in skcipher_walk_done() to be hit,
causing a WARN() and failing the encryption operation:
if (WARN_ON(err)) {
/* unexpected case; didn't process all bytes */
err = -EINVAL;
goto finish;
}
Fix it by rounding down to CHACHA_BLOCK_SIZE instead of walk.stride.
(Or we could replace round_down() with rounddown(), but that would add a
slow division operation every time, which I think we should avoid.)
Fixes: 2fe55987b262 ("crypto: arm64/chacha - use combined SIMD/ALU routine for more speed")
Cc: <stable@vger.kernel.org> # v5.0+
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/chacha_generic.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/crypto/chacha_generic.c b/crypto/chacha_generic.c
index 35b583101f4f..90ec0ec1b4f7 100644
--- a/crypto/chacha_generic.c
+++ b/crypto/chacha_generic.c
@@ -52,7 +52,7 @@ static int chacha_stream_xor(struct skcipher_request *req,
unsigned int nbytes = walk.nbytes;
if (nbytes < walk.total)
- nbytes = round_down(nbytes, walk.stride);
+ nbytes = round_down(nbytes, CHACHA_BLOCK_SIZE);
chacha_docrypt(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes, ctx->nrounds);
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback
2019-03-13 5:12 ` [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback Eric Biggers
@ 2019-03-13 7:50 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 7:50 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> The arm64 implementations of ChaCha and XChaCha are failing the extra
> crypto self-tests following my patches to test the !may_use_simd() code
> paths, which previously were untested. The problem is as follows:
>
> When !may_use_simd(), the arm64 NEON implementations fall back to the
> generic implementation, which uses the skcipher_walk API to iterate
> through the src/dst scatterlists. Due to how the skcipher_walk API
> works, walk.stride is set from the skcipher_alg actually being used,
> which in this case is the arm64 NEON algorithm. Thus walk.stride is
> 5*CHACHA_BLOCK_SIZE, not CHACHA_BLOCK_SIZE.
>
> This unnecessarily large stride shouldn't cause an actual problem.
> However, the generic implementation computes round_down(nbytes,
> walk.stride). round_down() assumes the round amount is a power of 2,
> which 5*CHACHA_BLOCK_SIZE is not, so it gives the wrong result.
>
> This causes the following case in skcipher_walk_done() to be hit,
> causing a WARN() and failing the encryption operation:
>
> if (WARN_ON(err)) {
> /* unexpected case; didn't process all bytes */
> err = -EINVAL;
> goto finish;
> }
>
> Fix it by rounding down to CHACHA_BLOCK_SIZE instead of walk.stride.
>
> (Or we could replace round_down() with rounddown(), but that would add a
> slow division operation every time, which I think we should avoid.)
>
> Fixes: 2fe55987b262 ("crypto: arm64/chacha - use combined SIMD/ALU routine for more speed")
> Cc: <stable@vger.kernel.org> # v5.0+
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> crypto/chacha_generic.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/crypto/chacha_generic.c b/crypto/chacha_generic.c
> index 35b583101f4f..90ec0ec1b4f7 100644
> --- a/crypto/chacha_generic.c
> +++ b/crypto/chacha_generic.c
> @@ -52,7 +52,7 @@ static int chacha_stream_xor(struct skcipher_request *req,
> unsigned int nbytes = walk.nbytes;
>
> if (nbytes < walk.total)
> - nbytes = round_down(nbytes, walk.stride);
> + nbytes = round_down(nbytes, CHACHA_BLOCK_SIZE);
>
> chacha_docrypt(state, walk.dst.virt.addr, walk.src.virt.addr,
> nbytes, ctx->nrounds);
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
2019-03-13 5:12 ` [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:29 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable() Eric Biggers
` (7 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
The arm64 gcm-aes-ce algorithm is failing the extra crypto self-tests
following my patches to test the !may_use_simd() code paths, which
previously were untested. The problem is that in the !may_use_simd()
case, an odd number of AES blocks can be processed within each step of
the skcipher_walk. However, the skcipher_walk is being done with a
"stride" of 2 blocks and is advanced by an even number of blocks after
each step. This causes the encryption to produce the wrong ciphertext
and authentication tag, and causes the decryption to incorrectly fail.
Fix it by only processing an even number of blocks per step.
Fixes: c2b24c36e0a3 ("crypto: arm64/aes-gcm-ce - fix scatterwalk API violation")
Fixes: 71e52c278c54 ("crypto: arm64/aes-ce-gcm - operate on two input blocks at a time")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm64/crypto/ghash-ce-glue.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c
index 791ad422c427..089b09286da7 100644
--- a/arch/arm64/crypto/ghash-ce-glue.c
+++ b/arch/arm64/crypto/ghash-ce-glue.c
@@ -473,9 +473,11 @@ static int gcm_encrypt(struct aead_request *req)
put_unaligned_be32(2, iv + GCM_IV_SIZE);
while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) {
- int blocks = walk.nbytes / AES_BLOCK_SIZE;
+ const int blocks =
+ walk.nbytes / (2 * AES_BLOCK_SIZE) * 2;
u8 *dst = walk.dst.virt.addr;
u8 *src = walk.src.virt.addr;
+ int remaining = blocks;
do {
__aes_arm64_encrypt(ctx->aes_key.key_enc,
@@ -485,9 +487,9 @@ static int gcm_encrypt(struct aead_request *req)
dst += AES_BLOCK_SIZE;
src += AES_BLOCK_SIZE;
- } while (--blocks > 0);
+ } while (--remaining > 0);
- ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg,
+ ghash_do_update(blocks, dg,
walk.dst.virt.addr, &ctx->ghash_key,
NULL, pmull_ghash_update_p64);
@@ -609,7 +611,7 @@ static int gcm_decrypt(struct aead_request *req)
put_unaligned_be32(2, iv + GCM_IV_SIZE);
while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) {
- int blocks = walk.nbytes / AES_BLOCK_SIZE;
+ int blocks = walk.nbytes / (2 * AES_BLOCK_SIZE) * 2;
u8 *dst = walk.dst.virt.addr;
u8 *src = walk.src.virt.addr;
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
2019-03-13 5:12 ` [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code Eric Biggers
@ 2019-03-13 10:29 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:29 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> The arm64 gcm-aes-ce algorithm is failing the extra crypto self-tests
> following my patches to test the !may_use_simd() code paths, which
> previously were untested. The problem is that in the !may_use_simd()
> case, an odd number of AES blocks can be processed within each step of
> the skcipher_walk. However, the skcipher_walk is being done with a
> "stride" of 2 blocks and is advanced by an even number of blocks after
> each step. This causes the encryption to produce the wrong ciphertext
> and authentication tag, and causes the decryption to incorrectly fail.
>
> Fix it by only processing an even number of blocks per step.
>
> Fixes: c2b24c36e0a3 ("crypto: arm64/aes-gcm-ce - fix scatterwalk API violation")
> Fixes: 71e52c278c54 ("crypto: arm64/aes-ce-gcm - operate on two input blocks at a time")
> Cc: <stable@vger.kernel.org> # v4.19+
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> arch/arm64/crypto/ghash-ce-glue.c | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c
> index 791ad422c427..089b09286da7 100644
> --- a/arch/arm64/crypto/ghash-ce-glue.c
> +++ b/arch/arm64/crypto/ghash-ce-glue.c
> @@ -473,9 +473,11 @@ static int gcm_encrypt(struct aead_request *req)
> put_unaligned_be32(2, iv + GCM_IV_SIZE);
>
> while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) {
> - int blocks = walk.nbytes / AES_BLOCK_SIZE;
> + const int blocks =
> + walk.nbytes / (2 * AES_BLOCK_SIZE) * 2;
> u8 *dst = walk.dst.virt.addr;
> u8 *src = walk.src.virt.addr;
> + int remaining = blocks;
>
> do {
> __aes_arm64_encrypt(ctx->aes_key.key_enc,
> @@ -485,9 +487,9 @@ static int gcm_encrypt(struct aead_request *req)
>
> dst += AES_BLOCK_SIZE;
> src += AES_BLOCK_SIZE;
> - } while (--blocks > 0);
> + } while (--remaining > 0);
>
> - ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg,
> + ghash_do_update(blocks, dg,
> walk.dst.virt.addr, &ctx->ghash_key,
> NULL, pmull_ghash_update_p64);
>
> @@ -609,7 +611,7 @@ static int gcm_decrypt(struct aead_request *req)
> put_unaligned_be32(2, iv + GCM_IV_SIZE);
>
> while (walk.nbytes >= (2 * AES_BLOCK_SIZE)) {
> - int blocks = walk.nbytes / AES_BLOCK_SIZE;
> + int blocks = walk.nbytes / (2 * AES_BLOCK_SIZE) * 2;
> u8 *dst = walk.dst.virt.addr;
> u8 *src = walk.src.virt.addr;
>
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable()
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
2019-03-13 5:12 ` [PATCH 1/8] crypto: chacha-generic - fix use as arm64 no-NEON fallback Eric Biggers
2019-03-13 5:12 ` [PATCH 2/8] crypto: arm64/gcm-aes-ce - fix no-NEON fallback code Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:31 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable() Eric Biggers
` (6 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
So that the no-SIMD fallback code can be tested by the crypto
self-tests, add a macro crypto_simd_usable() which wraps may_use_simd(),
but also returns false if the crypto self-tests have set a per-CPU bool
to disable SIMD in crypto code on the current CPU.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/testmgr.c | 26 +++++++++++++++++++++++++-
include/crypto/internal/simd.h | 24 ++++++++++++++++++++++++
2 files changed, 49 insertions(+), 1 deletion(-)
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 5d56b2990762..52417dde811f 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -37,6 +37,7 @@
#include <crypto/akcipher.h>
#include <crypto/kpp.h>
#include <crypto/acompress.h>
+#include <crypto/internal/simd.h>
#include "internal.h"
@@ -52,6 +53,9 @@ MODULE_PARM_DESC(noextratests, "disable expensive crypto self-tests");
static unsigned int fuzz_iterations = 100;
module_param(fuzz_iterations, uint, 0644);
MODULE_PARM_DESC(fuzz_iterations, "number of fuzz test iterations");
+
+DEFINE_PER_CPU(bool, crypto_simd_disabled_for_test);
+EXPORT_PER_CPU_SYMBOL_GPL(crypto_simd_disabled_for_test);
#endif
#ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS
@@ -838,7 +842,27 @@ static void generate_random_testvec_config(struct testvec_config *cfg,
WARN_ON_ONCE(!valid_testvec_config(cfg));
}
-#endif /* CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
+
+static void crypto_disable_simd_for_test(void)
+{
+ preempt_disable();
+ __this_cpu_write(crypto_simd_disabled_for_test, true);
+}
+
+static void crypto_reenable_simd_for_test(void)
+{
+ __this_cpu_write(crypto_simd_disabled_for_test, false);
+ preempt_enable();
+}
+#else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
+static void crypto_disable_simd_for_test(void)
+{
+}
+
+static void crypto_reenable_simd_for_test(void)
+{
+}
+#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
static int check_nonfinal_hash_op(const char *op, int err,
u8 *result, unsigned int digestsize,
diff --git a/include/crypto/internal/simd.h b/include/crypto/internal/simd.h
index a23b18b6ad61..d2316242a988 100644
--- a/include/crypto/internal/simd.h
+++ b/include/crypto/internal/simd.h
@@ -6,6 +6,9 @@
#ifndef _CRYPTO_INTERNAL_SIMD_H
#define _CRYPTO_INTERNAL_SIMD_H
+#include <linux/percpu.h>
+#include <linux/types.h>
+
/* skcipher support */
struct simd_skcipher_alg;
@@ -42,4 +45,25 @@ int simd_register_aeads_compat(struct aead_alg *algs, int count,
void simd_unregister_aeads(struct aead_alg *algs, int count,
struct simd_aead_alg **simd_algs);
+/*
+ * crypto_simd_usable() - is it allowed at this time to use SIMD instructions or
+ * access the SIMD register file?
+ *
+ * This delegates to may_use_simd(), except that this also returns false if SIMD
+ * in crypto code has been temporarily disabled on this CPU by the crypto
+ * self-tests, in order to test the no-SIMD fallback code. This override is
+ * currently limited to configurations where the extra self-tests are enabled,
+ * because it might be a bit too invasive to be part of the regular self-tests.
+ *
+ * This is a macro so that <asm/simd.h>, which some architectures don't have,
+ * doesn't have to be included directly here.
+ */
+#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS
+DECLARE_PER_CPU(bool, crypto_simd_disabled_for_test);
+#define crypto_simd_usable() \
+ (may_use_simd() && !this_cpu_read(crypto_simd_disabled_for_test))
+#else
+#define crypto_simd_usable() may_use_simd()
+#endif
+
#endif /* _CRYPTO_INTERNAL_SIMD_H */
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable()
2019-03-13 5:12 ` [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable() Eric Biggers
@ 2019-03-13 10:31 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:31 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> So that the no-SIMD fallback code can be tested by the crypto
> self-tests, add a macro crypto_simd_usable() which wraps may_use_simd(),
> but also returns false if the crypto self-tests have set a per-CPU bool
> to disable SIMD in crypto code on the current CPU.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> crypto/testmgr.c | 26 +++++++++++++++++++++++++-
> include/crypto/internal/simd.h | 24 ++++++++++++++++++++++++
> 2 files changed, 49 insertions(+), 1 deletion(-)
>
> diff --git a/crypto/testmgr.c b/crypto/testmgr.c
> index 5d56b2990762..52417dde811f 100644
> --- a/crypto/testmgr.c
> +++ b/crypto/testmgr.c
> @@ -37,6 +37,7 @@
> #include <crypto/akcipher.h>
> #include <crypto/kpp.h>
> #include <crypto/acompress.h>
> +#include <crypto/internal/simd.h>
>
> #include "internal.h"
>
> @@ -52,6 +53,9 @@ MODULE_PARM_DESC(noextratests, "disable expensive crypto self-tests");
> static unsigned int fuzz_iterations = 100;
> module_param(fuzz_iterations, uint, 0644);
> MODULE_PARM_DESC(fuzz_iterations, "number of fuzz test iterations");
> +
> +DEFINE_PER_CPU(bool, crypto_simd_disabled_for_test);
> +EXPORT_PER_CPU_SYMBOL_GPL(crypto_simd_disabled_for_test);
> #endif
>
> #ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS
> @@ -838,7 +842,27 @@ static void generate_random_testvec_config(struct testvec_config *cfg,
>
> WARN_ON_ONCE(!valid_testvec_config(cfg));
> }
> -#endif /* CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
> +
> +static void crypto_disable_simd_for_test(void)
> +{
> + preempt_disable();
> + __this_cpu_write(crypto_simd_disabled_for_test, true);
> +}
> +
> +static void crypto_reenable_simd_for_test(void)
> +{
> + __this_cpu_write(crypto_simd_disabled_for_test, false);
> + preempt_enable();
> +}
> +#else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
> +static void crypto_disable_simd_for_test(void)
> +{
> +}
> +
> +static void crypto_reenable_simd_for_test(void)
> +{
> +}
> +#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
>
> static int check_nonfinal_hash_op(const char *op, int err,
> u8 *result, unsigned int digestsize,
> diff --git a/include/crypto/internal/simd.h b/include/crypto/internal/simd.h
> index a23b18b6ad61..d2316242a988 100644
> --- a/include/crypto/internal/simd.h
> +++ b/include/crypto/internal/simd.h
> @@ -6,6 +6,9 @@
> #ifndef _CRYPTO_INTERNAL_SIMD_H
> #define _CRYPTO_INTERNAL_SIMD_H
>
> +#include <linux/percpu.h>
> +#include <linux/types.h>
> +
> /* skcipher support */
>
> struct simd_skcipher_alg;
> @@ -42,4 +45,25 @@ int simd_register_aeads_compat(struct aead_alg *algs, int count,
> void simd_unregister_aeads(struct aead_alg *algs, int count,
> struct simd_aead_alg **simd_algs);
>
> +/*
> + * crypto_simd_usable() - is it allowed at this time to use SIMD instructions or
> + * access the SIMD register file?
> + *
> + * This delegates to may_use_simd(), except that this also returns false if SIMD
> + * in crypto code has been temporarily disabled on this CPU by the crypto
> + * self-tests, in order to test the no-SIMD fallback code. This override is
> + * currently limited to configurations where the extra self-tests are enabled,
> + * because it might be a bit too invasive to be part of the regular self-tests.
> + *
> + * This is a macro so that <asm/simd.h>, which some architectures don't have,
> + * doesn't have to be included directly here.
> + */
> +#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS
> +DECLARE_PER_CPU(bool, crypto_simd_disabled_for_test);
> +#define crypto_simd_usable() \
> + (may_use_simd() && !this_cpu_read(crypto_simd_disabled_for_test))
> +#else
> +#define crypto_simd_usable() may_use_simd()
> +#endif
> +
> #endif /* _CRYPTO_INTERNAL_SIMD_H */
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable()
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (2 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 3/8] crypto: simd,testmgr - introduce crypto_simd_usable() Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:32 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 5/8] crypto: arm " Eric Biggers
` (5 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
Replace all calls to irq_fpu_usable() in the x86 crypto code with
crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/x86/crypto/aesni-intel_glue.c | 8 ++++----
arch/x86/crypto/chacha_glue.c | 6 +++---
arch/x86/crypto/crc32-pclmul_glue.c | 5 +++--
arch/x86/crypto/crc32c-intel_glue.c | 7 ++++---
arch/x86/crypto/crct10dif-pclmul_glue.c | 7 ++++---
arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +++++----
arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +++--
arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +++--
arch/x86/crypto/poly1305_glue.c | 4 ++--
arch/x86/crypto/sha1_ssse3_glue.c | 7 ++++---
arch/x86/crypto/sha256_ssse3_glue.c | 7 ++++---
arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++-----
12 files changed, 44 insertions(+), 36 deletions(-)
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 1466d59384ac..21c246799aa5 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -30,8 +30,8 @@
#include <crypto/gcm.h>
#include <crypto/xts.h>
#include <asm/cpu_device_id.h>
-#include <asm/fpu/api.h>
#include <asm/crypto/aes.h>
+#include <asm/simd.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/simd.h>
@@ -332,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
return -EINVAL;
}
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
err = crypto_aes_expand_key(ctx, in_key, key_len);
else {
kernel_fpu_begin();
@@ -353,7 +353,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
{
struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
crypto_aes_encrypt_x86(ctx, dst, src);
else {
kernel_fpu_begin();
@@ -366,7 +366,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
{
struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
crypto_aes_decrypt_x86(ctx, dst, src);
else {
kernel_fpu_begin();
diff --git a/arch/x86/crypto/chacha_glue.c b/arch/x86/crypto/chacha_glue.c
index 45c1c4143176..4967ad620775 100644
--- a/arch/x86/crypto/chacha_glue.c
+++ b/arch/x86/crypto/chacha_glue.c
@@ -12,10 +12,10 @@
#include <crypto/algapi.h>
#include <crypto/chacha.h>
+#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <linux/kernel.h>
#include <linux/module.h>
-#include <asm/fpu/api.h>
#include <asm/simd.h>
#define CHACHA_STATE_ALIGN 16
@@ -170,7 +170,7 @@ static int chacha_simd(struct skcipher_request *req)
struct skcipher_walk walk;
int err;
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_chacha_crypt(req);
err = skcipher_walk_virt(&walk, req, true);
@@ -193,7 +193,7 @@ static int xchacha_simd(struct skcipher_request *req)
u8 real_iv[16];
int err;
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_xchacha_crypt(req);
err = skcipher_walk_virt(&walk, req, true);
diff --git a/arch/x86/crypto/crc32-pclmul_glue.c b/arch/x86/crypto/crc32-pclmul_glue.c
index c8d9cdacbf10..cb4ab6645106 100644
--- a/arch/x86/crypto/crc32-pclmul_glue.c
+++ b/arch/x86/crypto/crc32-pclmul_glue.c
@@ -32,10 +32,11 @@
#include <linux/kernel.h>
#include <linux/crc32.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <asm/cpufeatures.h>
#include <asm/cpu_device_id.h>
-#include <asm/fpu/api.h>
+#include <asm/simd.h>
#define CHKSUM_BLOCK_SIZE 1
#define CHKSUM_DIGEST_SIZE 4
@@ -54,7 +55,7 @@ static u32 __attribute__((pure))
unsigned int iremainder;
unsigned int prealign;
- if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !irq_fpu_usable())
+ if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !crypto_simd_usable())
return crc32_le(crc, p, len);
if ((long)p & SCALE_F_MASK) {
diff --git a/arch/x86/crypto/crc32c-intel_glue.c b/arch/x86/crypto/crc32c-intel_glue.c
index 5773e1161072..a58fe217c856 100644
--- a/arch/x86/crypto/crc32c-intel_glue.c
+++ b/arch/x86/crypto/crc32c-intel_glue.c
@@ -29,10 +29,11 @@
#include <linux/string.h>
#include <linux/kernel.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <asm/cpufeatures.h>
#include <asm/cpu_device_id.h>
-#include <asm/fpu/internal.h>
+#include <asm/simd.h>
#define CHKSUM_BLOCK_SIZE 1
#define CHKSUM_DIGEST_SIZE 4
@@ -177,7 +178,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
* use faster PCL version if datasize is large enough to
* overcome kernel fpu state save/restore overhead
*/
- if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
+ if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) {
kernel_fpu_begin();
*crcp = crc_pcl(data, len, *crcp);
kernel_fpu_end();
@@ -189,7 +190,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
static int __crc32c_pcl_intel_finup(u32 *crcp, const u8 *data, unsigned int len,
u8 *out)
{
- if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
+ if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) {
kernel_fpu_begin();
*(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp));
kernel_fpu_end();
diff --git a/arch/x86/crypto/crct10dif-pclmul_glue.c b/arch/x86/crypto/crct10dif-pclmul_glue.c
index 0e785c0b2354..64f17d2d8b67 100644
--- a/arch/x86/crypto/crct10dif-pclmul_glue.c
+++ b/arch/x86/crypto/crct10dif-pclmul_glue.c
@@ -26,12 +26,13 @@
#include <linux/module.h>
#include <linux/crc-t10dif.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/init.h>
#include <linux/string.h>
#include <linux/kernel.h>
-#include <asm/fpu/api.h>
#include <asm/cpufeatures.h>
#include <asm/cpu_device_id.h>
+#include <asm/simd.h>
asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);
@@ -53,7 +54,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data,
{
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
- if (length >= 16 && irq_fpu_usable()) {
+ if (length >= 16 && crypto_simd_usable()) {
kernel_fpu_begin();
ctx->crc = crc_t10dif_pcl(ctx->crc, data, length);
kernel_fpu_end();
@@ -73,7 +74,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)
static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len,
u8 *out)
{
- if (len >= 16 && irq_fpu_usable()) {
+ if (len >= 16 && crypto_simd_usable()) {
kernel_fpu_begin();
*(__u16 *)out = crc_t10dif_pcl(*crcp, data, len);
kernel_fpu_end();
diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c
index 3582ae885ee1..4099a0ae17dd 100644
--- a/arch/x86/crypto/ghash-clmulni-intel_glue.c
+++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c
@@ -19,8 +19,9 @@
#include <crypto/cryptd.h>
#include <crypto/gf128mul.h>
#include <crypto/internal/hash.h>
-#include <asm/fpu/api.h>
+#include <crypto/internal/simd.h>
#include <asm/cpu_device_id.h>
+#include <asm/simd.h>
#define GHASH_BLOCK_SIZE 16
#define GHASH_DIGEST_SIZE 16
@@ -182,7 +183,7 @@ static int ghash_async_update(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
@@ -200,7 +201,7 @@ static int ghash_async_final(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
@@ -241,7 +242,7 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
diff --git a/arch/x86/crypto/nhpoly1305-avx2-glue.c b/arch/x86/crypto/nhpoly1305-avx2-glue.c
index 20d815ea4b6a..f7567cbd35b6 100644
--- a/arch/x86/crypto/nhpoly1305-avx2-glue.c
+++ b/arch/x86/crypto/nhpoly1305-avx2-glue.c
@@ -7,9 +7,10 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/nhpoly1305.h>
#include <linux/module.h>
-#include <asm/fpu/api.h>
+#include <asm/simd.h>
asmlinkage void nh_avx2(const u32 *key, const u8 *message, size_t message_len,
u8 hash[NH_HASH_BYTES]);
@@ -24,7 +25,7 @@ static void _nh_avx2(const u32 *key, const u8 *message, size_t message_len,
static int nhpoly1305_avx2_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
- if (srclen < 64 || !irq_fpu_usable())
+ if (srclen < 64 || !crypto_simd_usable())
return crypto_nhpoly1305_update(desc, src, srclen);
do {
diff --git a/arch/x86/crypto/nhpoly1305-sse2-glue.c b/arch/x86/crypto/nhpoly1305-sse2-glue.c
index ed68d164ce14..a661ede3b5cf 100644
--- a/arch/x86/crypto/nhpoly1305-sse2-glue.c
+++ b/arch/x86/crypto/nhpoly1305-sse2-glue.c
@@ -7,9 +7,10 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/nhpoly1305.h>
#include <linux/module.h>
-#include <asm/fpu/api.h>
+#include <asm/simd.h>
asmlinkage void nh_sse2(const u32 *key, const u8 *message, size_t message_len,
u8 hash[NH_HASH_BYTES]);
@@ -24,7 +25,7 @@ static void _nh_sse2(const u32 *key, const u8 *message, size_t message_len,
static int nhpoly1305_sse2_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
- if (srclen < 64 || !irq_fpu_usable())
+ if (srclen < 64 || !crypto_simd_usable())
return crypto_nhpoly1305_update(desc, src, srclen);
do {
diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c
index 88cc01506c84..6eb65b237b3c 100644
--- a/arch/x86/crypto/poly1305_glue.c
+++ b/arch/x86/crypto/poly1305_glue.c
@@ -11,11 +11,11 @@
#include <crypto/algapi.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/poly1305.h>
#include <linux/crypto.h>
#include <linux/kernel.h>
#include <linux/module.h>
-#include <asm/fpu/api.h>
#include <asm/simd.h>
struct poly1305_simd_desc_ctx {
@@ -126,7 +126,7 @@ static int poly1305_simd_update(struct shash_desc *desc,
unsigned int bytes;
/* kernel_fpu_begin/end is costly, use fallback for small updates */
- if (srclen <= 288 || !may_use_simd())
+ if (srclen <= 288 || !crypto_simd_usable())
return crypto_poly1305_update(desc, src, srclen);
kernel_fpu_begin();
diff --git a/arch/x86/crypto/sha1_ssse3_glue.c b/arch/x86/crypto/sha1_ssse3_glue.c
index 7391c7de72c7..42f177afc33a 100644
--- a/arch/x86/crypto/sha1_ssse3_glue.c
+++ b/arch/x86/crypto/sha1_ssse3_glue.c
@@ -22,6 +22,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/mm.h>
@@ -29,7 +30,7 @@
#include <linux/types.h>
#include <crypto/sha.h>
#include <crypto/sha1_base.h>
-#include <asm/fpu/api.h>
+#include <asm/simd.h>
typedef void (sha1_transform_fn)(u32 *digest, const char *data,
unsigned int rounds);
@@ -39,7 +40,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,
{
struct sha1_state *sctx = shash_desc_ctx(desc);
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
return crypto_sha1_update(desc, data, len);
@@ -57,7 +58,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,
static int sha1_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out, sha1_transform_fn *sha1_xform)
{
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
return crypto_sha1_finup(desc, data, len, out);
kernel_fpu_begin();
diff --git a/arch/x86/crypto/sha256_ssse3_glue.c b/arch/x86/crypto/sha256_ssse3_glue.c
index 773a873d2b28..73867da3cbee 100644
--- a/arch/x86/crypto/sha256_ssse3_glue.c
+++ b/arch/x86/crypto/sha256_ssse3_glue.c
@@ -30,6 +30,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/mm.h>
@@ -37,8 +38,8 @@
#include <linux/types.h>
#include <crypto/sha.h>
#include <crypto/sha256_base.h>
-#include <asm/fpu/api.h>
#include <linux/string.h>
+#include <asm/simd.h>
asmlinkage void sha256_transform_ssse3(u32 *digest, const char *data,
u64 rounds);
@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
{
struct sha256_state *sctx = shash_desc_ctx(desc);
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
return crypto_sha256_update(desc, data, len);
@@ -67,7 +68,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
static int sha256_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out, sha256_transform_fn *sha256_xform)
{
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
return crypto_sha256_finup(desc, data, len, out);
kernel_fpu_begin();
diff --git a/arch/x86/crypto/sha512_ssse3_glue.c b/arch/x86/crypto/sha512_ssse3_glue.c
index f1b811b60ba6..458356a3f124 100644
--- a/arch/x86/crypto/sha512_ssse3_glue.c
+++ b/arch/x86/crypto/sha512_ssse3_glue.c
@@ -28,16 +28,16 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/cryptohash.h>
+#include <linux/string.h>
#include <linux/types.h>
#include <crypto/sha.h>
#include <crypto/sha512_base.h>
-#include <asm/fpu/api.h>
-
-#include <linux/string.h>
+#include <asm/simd.h>
asmlinkage void sha512_transform_ssse3(u64 *digest, const char *data,
u64 rounds);
@@ -49,7 +49,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,
{
struct sha512_state *sctx = shash_desc_ctx(desc);
- if (!irq_fpu_usable() ||
+ if (!crypto_simd_usable() ||
(sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE)
return crypto_sha512_update(desc, data, len);
@@ -67,7 +67,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,
static int sha512_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out, sha512_transform_fn *sha512_xform)
{
- if (!irq_fpu_usable())
+ if (!crypto_simd_usable())
return crypto_sha512_finup(desc, data, len, out);
kernel_fpu_begin();
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable()
2019-03-13 5:12 ` [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable() Eric Biggers
@ 2019-03-13 10:32 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:32 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> Replace all calls to irq_fpu_usable() in the x86 crypto code with
> crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> arch/x86/crypto/aesni-intel_glue.c | 8 ++++----
> arch/x86/crypto/chacha_glue.c | 6 +++---
> arch/x86/crypto/crc32-pclmul_glue.c | 5 +++--
> arch/x86/crypto/crc32c-intel_glue.c | 7 ++++---
> arch/x86/crypto/crct10dif-pclmul_glue.c | 7 ++++---
> arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +++++----
> arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +++--
> arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +++--
> arch/x86/crypto/poly1305_glue.c | 4 ++--
> arch/x86/crypto/sha1_ssse3_glue.c | 7 ++++---
> arch/x86/crypto/sha256_ssse3_glue.c | 7 ++++---
> arch/x86/crypto/sha512_ssse3_glue.c | 10 +++++-----
> 12 files changed, 44 insertions(+), 36 deletions(-)
>
> diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
> index 1466d59384ac..21c246799aa5 100644
> --- a/arch/x86/crypto/aesni-intel_glue.c
> +++ b/arch/x86/crypto/aesni-intel_glue.c
> @@ -30,8 +30,8 @@
> #include <crypto/gcm.h>
> #include <crypto/xts.h>
> #include <asm/cpu_device_id.h>
> -#include <asm/fpu/api.h>
> #include <asm/crypto/aes.h>
> +#include <asm/simd.h>
> #include <crypto/scatterwalk.h>
> #include <crypto/internal/aead.h>
> #include <crypto/internal/simd.h>
> @@ -332,7 +332,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
> return -EINVAL;
> }
>
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> err = crypto_aes_expand_key(ctx, in_key, key_len);
> else {
> kernel_fpu_begin();
> @@ -353,7 +353,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
> {
> struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
>
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> crypto_aes_encrypt_x86(ctx, dst, src);
> else {
> kernel_fpu_begin();
> @@ -366,7 +366,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
> {
> struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
>
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> crypto_aes_decrypt_x86(ctx, dst, src);
> else {
> kernel_fpu_begin();
> diff --git a/arch/x86/crypto/chacha_glue.c b/arch/x86/crypto/chacha_glue.c
> index 45c1c4143176..4967ad620775 100644
> --- a/arch/x86/crypto/chacha_glue.c
> +++ b/arch/x86/crypto/chacha_glue.c
> @@ -12,10 +12,10 @@
>
> #include <crypto/algapi.h>
> #include <crypto/chacha.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/internal/skcipher.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
> -#include <asm/fpu/api.h>
> #include <asm/simd.h>
>
> #define CHACHA_STATE_ALIGN 16
> @@ -170,7 +170,7 @@ static int chacha_simd(struct skcipher_request *req)
> struct skcipher_walk walk;
> int err;
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_chacha_crypt(req);
>
> err = skcipher_walk_virt(&walk, req, true);
> @@ -193,7 +193,7 @@ static int xchacha_simd(struct skcipher_request *req)
> u8 real_iv[16];
> int err;
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !irq_fpu_usable())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_xchacha_crypt(req);
>
> err = skcipher_walk_virt(&walk, req, true);
> diff --git a/arch/x86/crypto/crc32-pclmul_glue.c b/arch/x86/crypto/crc32-pclmul_glue.c
> index c8d9cdacbf10..cb4ab6645106 100644
> --- a/arch/x86/crypto/crc32-pclmul_glue.c
> +++ b/arch/x86/crypto/crc32-pclmul_glue.c
> @@ -32,10 +32,11 @@
> #include <linux/kernel.h>
> #include <linux/crc32.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
>
> #include <asm/cpufeatures.h>
> #include <asm/cpu_device_id.h>
> -#include <asm/fpu/api.h>
> +#include <asm/simd.h>
>
> #define CHKSUM_BLOCK_SIZE 1
> #define CHKSUM_DIGEST_SIZE 4
> @@ -54,7 +55,7 @@ static u32 __attribute__((pure))
> unsigned int iremainder;
> unsigned int prealign;
>
> - if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !irq_fpu_usable())
> + if (len < PCLMUL_MIN_LEN + SCALE_F_MASK || !crypto_simd_usable())
> return crc32_le(crc, p, len);
>
> if ((long)p & SCALE_F_MASK) {
> diff --git a/arch/x86/crypto/crc32c-intel_glue.c b/arch/x86/crypto/crc32c-intel_glue.c
> index 5773e1161072..a58fe217c856 100644
> --- a/arch/x86/crypto/crc32c-intel_glue.c
> +++ b/arch/x86/crypto/crc32c-intel_glue.c
> @@ -29,10 +29,11 @@
> #include <linux/string.h>
> #include <linux/kernel.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
>
> #include <asm/cpufeatures.h>
> #include <asm/cpu_device_id.h>
> -#include <asm/fpu/internal.h>
> +#include <asm/simd.h>
>
> #define CHKSUM_BLOCK_SIZE 1
> #define CHKSUM_DIGEST_SIZE 4
> @@ -177,7 +178,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
> * use faster PCL version if datasize is large enough to
> * overcome kernel fpu state save/restore overhead
> */
> - if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
> + if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) {
> kernel_fpu_begin();
> *crcp = crc_pcl(data, len, *crcp);
> kernel_fpu_end();
> @@ -189,7 +190,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
> static int __crc32c_pcl_intel_finup(u32 *crcp, const u8 *data, unsigned int len,
> u8 *out)
> {
> - if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
> + if (len >= CRC32C_PCL_BREAKEVEN && crypto_simd_usable()) {
> kernel_fpu_begin();
> *(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp));
> kernel_fpu_end();
> diff --git a/arch/x86/crypto/crct10dif-pclmul_glue.c b/arch/x86/crypto/crct10dif-pclmul_glue.c
> index 0e785c0b2354..64f17d2d8b67 100644
> --- a/arch/x86/crypto/crct10dif-pclmul_glue.c
> +++ b/arch/x86/crypto/crct10dif-pclmul_glue.c
> @@ -26,12 +26,13 @@
> #include <linux/module.h>
> #include <linux/crc-t10dif.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/init.h>
> #include <linux/string.h>
> #include <linux/kernel.h>
> -#include <asm/fpu/api.h>
> #include <asm/cpufeatures.h>
> #include <asm/cpu_device_id.h>
> +#include <asm/simd.h>
>
> asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);
>
> @@ -53,7 +54,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data,
> {
> struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
>
> - if (length >= 16 && irq_fpu_usable()) {
> + if (length >= 16 && crypto_simd_usable()) {
> kernel_fpu_begin();
> ctx->crc = crc_t10dif_pcl(ctx->crc, data, length);
> kernel_fpu_end();
> @@ -73,7 +74,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)
> static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len,
> u8 *out)
> {
> - if (len >= 16 && irq_fpu_usable()) {
> + if (len >= 16 && crypto_simd_usable()) {
> kernel_fpu_begin();
> *(__u16 *)out = crc_t10dif_pcl(*crcp, data, len);
> kernel_fpu_end();
> diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c
> index 3582ae885ee1..4099a0ae17dd 100644
> --- a/arch/x86/crypto/ghash-clmulni-intel_glue.c
> +++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c
> @@ -19,8 +19,9 @@
> #include <crypto/cryptd.h>
> #include <crypto/gf128mul.h>
> #include <crypto/internal/hash.h>
> -#include <asm/fpu/api.h>
> +#include <crypto/internal/simd.h>
> #include <asm/cpu_device_id.h>
> +#include <asm/simd.h>
>
> #define GHASH_BLOCK_SIZE 16
> #define GHASH_DIGEST_SIZE 16
> @@ -182,7 +183,7 @@ static int ghash_async_update(struct ahash_request *req)
> struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> @@ -200,7 +201,7 @@ static int ghash_async_final(struct ahash_request *req)
> struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> @@ -241,7 +242,7 @@ static int ghash_async_digest(struct ahash_request *req)
> struct ahash_request *cryptd_req = ahash_request_ctx(req);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> diff --git a/arch/x86/crypto/nhpoly1305-avx2-glue.c b/arch/x86/crypto/nhpoly1305-avx2-glue.c
> index 20d815ea4b6a..f7567cbd35b6 100644
> --- a/arch/x86/crypto/nhpoly1305-avx2-glue.c
> +++ b/arch/x86/crypto/nhpoly1305-avx2-glue.c
> @@ -7,9 +7,10 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/nhpoly1305.h>
> #include <linux/module.h>
> -#include <asm/fpu/api.h>
> +#include <asm/simd.h>
>
> asmlinkage void nh_avx2(const u32 *key, const u8 *message, size_t message_len,
> u8 hash[NH_HASH_BYTES]);
> @@ -24,7 +25,7 @@ static void _nh_avx2(const u32 *key, const u8 *message, size_t message_len,
> static int nhpoly1305_avx2_update(struct shash_desc *desc,
> const u8 *src, unsigned int srclen)
> {
> - if (srclen < 64 || !irq_fpu_usable())
> + if (srclen < 64 || !crypto_simd_usable())
> return crypto_nhpoly1305_update(desc, src, srclen);
>
> do {
> diff --git a/arch/x86/crypto/nhpoly1305-sse2-glue.c b/arch/x86/crypto/nhpoly1305-sse2-glue.c
> index ed68d164ce14..a661ede3b5cf 100644
> --- a/arch/x86/crypto/nhpoly1305-sse2-glue.c
> +++ b/arch/x86/crypto/nhpoly1305-sse2-glue.c
> @@ -7,9 +7,10 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/nhpoly1305.h>
> #include <linux/module.h>
> -#include <asm/fpu/api.h>
> +#include <asm/simd.h>
>
> asmlinkage void nh_sse2(const u32 *key, const u8 *message, size_t message_len,
> u8 hash[NH_HASH_BYTES]);
> @@ -24,7 +25,7 @@ static void _nh_sse2(const u32 *key, const u8 *message, size_t message_len,
> static int nhpoly1305_sse2_update(struct shash_desc *desc,
> const u8 *src, unsigned int srclen)
> {
> - if (srclen < 64 || !irq_fpu_usable())
> + if (srclen < 64 || !crypto_simd_usable())
> return crypto_nhpoly1305_update(desc, src, srclen);
>
> do {
> diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c
> index 88cc01506c84..6eb65b237b3c 100644
> --- a/arch/x86/crypto/poly1305_glue.c
> +++ b/arch/x86/crypto/poly1305_glue.c
> @@ -11,11 +11,11 @@
>
> #include <crypto/algapi.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/poly1305.h>
> #include <linux/crypto.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
> -#include <asm/fpu/api.h>
> #include <asm/simd.h>
>
> struct poly1305_simd_desc_ctx {
> @@ -126,7 +126,7 @@ static int poly1305_simd_update(struct shash_desc *desc,
> unsigned int bytes;
>
> /* kernel_fpu_begin/end is costly, use fallback for small updates */
> - if (srclen <= 288 || !may_use_simd())
> + if (srclen <= 288 || !crypto_simd_usable())
> return crypto_poly1305_update(desc, src, srclen);
>
> kernel_fpu_begin();
> diff --git a/arch/x86/crypto/sha1_ssse3_glue.c b/arch/x86/crypto/sha1_ssse3_glue.c
> index 7391c7de72c7..42f177afc33a 100644
> --- a/arch/x86/crypto/sha1_ssse3_glue.c
> +++ b/arch/x86/crypto/sha1_ssse3_glue.c
> @@ -22,6 +22,7 @@
> #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/init.h>
> #include <linux/module.h>
> #include <linux/mm.h>
> @@ -29,7 +30,7 @@
> #include <linux/types.h>
> #include <crypto/sha.h>
> #include <crypto/sha1_base.h>
> -#include <asm/fpu/api.h>
> +#include <asm/simd.h>
>
> typedef void (sha1_transform_fn)(u32 *digest, const char *data,
> unsigned int rounds);
> @@ -39,7 +40,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha1_state *sctx = shash_desc_ctx(desc);
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
> return crypto_sha1_update(desc, data, len);
>
> @@ -57,7 +58,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data,
> static int sha1_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out, sha1_transform_fn *sha1_xform)
> {
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> return crypto_sha1_finup(desc, data, len, out);
>
> kernel_fpu_begin();
> diff --git a/arch/x86/crypto/sha256_ssse3_glue.c b/arch/x86/crypto/sha256_ssse3_glue.c
> index 773a873d2b28..73867da3cbee 100644
> --- a/arch/x86/crypto/sha256_ssse3_glue.c
> +++ b/arch/x86/crypto/sha256_ssse3_glue.c
> @@ -30,6 +30,7 @@
> #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/init.h>
> #include <linux/module.h>
> #include <linux/mm.h>
> @@ -37,8 +38,8 @@
> #include <linux/types.h>
> #include <crypto/sha.h>
> #include <crypto/sha256_base.h>
> -#include <asm/fpu/api.h>
> #include <linux/string.h>
> +#include <asm/simd.h>
>
> asmlinkage void sha256_transform_ssse3(u32 *digest, const char *data,
> u64 rounds);
> @@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha256_state *sctx = shash_desc_ctx(desc);
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
> return crypto_sha256_update(desc, data, len);
>
> @@ -67,7 +68,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
> static int sha256_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out, sha256_transform_fn *sha256_xform)
> {
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> return crypto_sha256_finup(desc, data, len, out);
>
> kernel_fpu_begin();
> diff --git a/arch/x86/crypto/sha512_ssse3_glue.c b/arch/x86/crypto/sha512_ssse3_glue.c
> index f1b811b60ba6..458356a3f124 100644
> --- a/arch/x86/crypto/sha512_ssse3_glue.c
> +++ b/arch/x86/crypto/sha512_ssse3_glue.c
> @@ -28,16 +28,16 @@
> #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/init.h>
> #include <linux/module.h>
> #include <linux/mm.h>
> #include <linux/cryptohash.h>
> +#include <linux/string.h>
> #include <linux/types.h>
> #include <crypto/sha.h>
> #include <crypto/sha512_base.h>
> -#include <asm/fpu/api.h>
> -
> -#include <linux/string.h>
> +#include <asm/simd.h>
>
> asmlinkage void sha512_transform_ssse3(u64 *digest, const char *data,
> u64 rounds);
> @@ -49,7 +49,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha512_state *sctx = shash_desc_ctx(desc);
>
> - if (!irq_fpu_usable() ||
> + if (!crypto_simd_usable() ||
> (sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE)
> return crypto_sha512_update(desc, data, len);
>
> @@ -67,7 +67,7 @@ static int sha512_update(struct shash_desc *desc, const u8 *data,
> static int sha512_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out, sha512_transform_fn *sha512_xform)
> {
> - if (!irq_fpu_usable())
> + if (!crypto_simd_usable())
> return crypto_sha512_finup(desc, data, len, out);
>
> kernel_fpu_begin();
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 5/8] crypto: arm - convert to use crypto_simd_usable()
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (3 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 4/8] crypto: x86 - convert to use crypto_simd_usable() Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:33 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 6/8] crypto: arm64 " Eric Biggers
` (4 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
Replace all calls to may_use_simd() in the arm crypto code with
crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm/crypto/chacha-neon-glue.c | 5 +++--
arch/arm/crypto/crc32-ce-glue.c | 5 +++--
arch/arm/crypto/crct10dif-ce-glue.c | 3 ++-
arch/arm/crypto/ghash-ce-glue.c | 7 ++++---
arch/arm/crypto/nhpoly1305-neon-glue.c | 3 ++-
arch/arm/crypto/sha1-ce-glue.c | 5 +++--
arch/arm/crypto/sha1_neon_glue.c | 5 +++--
arch/arm/crypto/sha2-ce-glue.c | 5 +++--
arch/arm/crypto/sha256_neon_glue.c | 5 +++--
arch/arm/crypto/sha512-neon-glue.c | 5 +++--
10 files changed, 29 insertions(+), 19 deletions(-)
diff --git a/arch/arm/crypto/chacha-neon-glue.c b/arch/arm/crypto/chacha-neon-glue.c
index 9d6fda81986d..48a89537b828 100644
--- a/arch/arm/crypto/chacha-neon-glue.c
+++ b/arch/arm/crypto/chacha-neon-glue.c
@@ -21,6 +21,7 @@
#include <crypto/algapi.h>
#include <crypto/chacha.h>
+#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <linux/kernel.h>
#include <linux/module.h>
@@ -93,7 +94,7 @@ static int chacha_neon(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_chacha_crypt(req);
return chacha_neon_stream_xor(req, ctx, req->iv);
@@ -107,7 +108,7 @@ static int xchacha_neon(struct skcipher_request *req)
u32 state[16];
u8 real_iv[16];
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_xchacha_crypt(req);
crypto_chacha_init(state, ctx, req->iv);
diff --git a/arch/arm/crypto/crc32-ce-glue.c b/arch/arm/crypto/crc32-ce-glue.c
index cd9e93b46c2d..e712c2a7d387 100644
--- a/arch/arm/crypto/crc32-ce-glue.c
+++ b/arch/arm/crypto/crc32-ce-glue.c
@@ -16,6 +16,7 @@
#include <linux/string.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <asm/hwcap.h>
#include <asm/neon.h>
@@ -113,7 +114,7 @@ static int crc32_pmull_update(struct shash_desc *desc, const u8 *data,
u32 *crc = shash_desc_ctx(desc);
unsigned int l;
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
if ((u32)data % SCALE_F) {
l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F));
@@ -147,7 +148,7 @@ static int crc32c_pmull_update(struct shash_desc *desc, const u8 *data,
u32 *crc = shash_desc_ctx(desc);
unsigned int l;
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
if ((u32)data % SCALE_F) {
l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F));
diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c
index 3d6b800b8396..3b24f2872592 100644
--- a/arch/arm/crypto/crct10dif-ce-glue.c
+++ b/arch/arm/crypto/crct10dif-ce-glue.c
@@ -15,6 +15,7 @@
#include <linux/string.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <asm/neon.h>
#include <asm/simd.h>
@@ -36,7 +37,7 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data,
{
u16 *crc = shash_desc_ctx(desc);
- if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
+ if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c
index b7d30b6cf49c..60123e9ea9d8 100644
--- a/arch/arm/crypto/ghash-ce-glue.c
+++ b/arch/arm/crypto/ghash-ce-glue.c
@@ -14,6 +14,7 @@
#include <asm/unaligned.h>
#include <crypto/cryptd.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/gf128mul.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
@@ -196,7 +197,7 @@ static int ghash_async_update(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
@@ -214,7 +215,7 @@ static int ghash_async_final(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
@@ -232,7 +233,7 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
diff --git a/arch/arm/crypto/nhpoly1305-neon-glue.c b/arch/arm/crypto/nhpoly1305-neon-glue.c
index 49aae87cb2bc..ae5aefc44a4d 100644
--- a/arch/arm/crypto/nhpoly1305-neon-glue.c
+++ b/arch/arm/crypto/nhpoly1305-neon-glue.c
@@ -9,6 +9,7 @@
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/nhpoly1305.h>
#include <linux/module.h>
@@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
static int nhpoly1305_neon_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
- if (srclen < 64 || !may_use_simd())
+ if (srclen < 64 || !crypto_simd_usable())
return crypto_nhpoly1305_update(desc, src, srclen);
do {
diff --git a/arch/arm/crypto/sha1-ce-glue.c b/arch/arm/crypto/sha1-ce-glue.c
index b732522e20f8..4c6c6900853c 100644
--- a/arch/arm/crypto/sha1-ce-glue.c
+++ b/arch/arm/crypto/sha1-ce-glue.c
@@ -9,6 +9,7 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha1_base.h>
#include <linux/cpufeature.h>
@@ -33,7 +34,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
{
struct sha1_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
return sha1_update_arm(desc, data, len);
@@ -47,7 +48,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha1_finup_arm(desc, data, len, out);
kernel_neon_begin();
diff --git a/arch/arm/crypto/sha1_neon_glue.c b/arch/arm/crypto/sha1_neon_glue.c
index d15e0ea2c95e..d6c95c213d42 100644
--- a/arch/arm/crypto/sha1_neon_glue.c
+++ b/arch/arm/crypto/sha1_neon_glue.c
@@ -19,6 +19,7 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/mm.h>
@@ -39,7 +40,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,
{
struct sha1_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
return sha1_update_arm(desc, data, len);
@@ -54,7 +55,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,
static int sha1_neon_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha1_finup_arm(desc, data, len, out);
kernel_neon_begin();
diff --git a/arch/arm/crypto/sha2-ce-glue.c b/arch/arm/crypto/sha2-ce-glue.c
index 1211a5c129fc..a47a9d4b663e 100644
--- a/arch/arm/crypto/sha2-ce-glue.c
+++ b/arch/arm/crypto/sha2-ce-glue.c
@@ -9,6 +9,7 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha256_base.h>
#include <linux/cpufeature.h>
@@ -34,7 +35,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,
{
struct sha256_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
return crypto_sha256_arm_update(desc, data, len);
@@ -49,7 +50,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,
static int sha2_ce_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha256_arm_finup(desc, data, len, out);
kernel_neon_begin();
diff --git a/arch/arm/crypto/sha256_neon_glue.c b/arch/arm/crypto/sha256_neon_glue.c
index 1d82c6cd31a4..f3f6b1624fc3 100644
--- a/arch/arm/crypto/sha256_neon_glue.c
+++ b/arch/arm/crypto/sha256_neon_glue.c
@@ -15,6 +15,7 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <linux/cryptohash.h>
#include <linux/types.h>
#include <linux/string.h>
@@ -34,7 +35,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
{
struct sha256_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
return crypto_sha256_arm_update(desc, data, len);
@@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
static int sha256_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha256_arm_finup(desc, data, len, out);
kernel_neon_begin();
diff --git a/arch/arm/crypto/sha512-neon-glue.c b/arch/arm/crypto/sha512-neon-glue.c
index 8a5642b41fd6..d33ab59c26c0 100644
--- a/arch/arm/crypto/sha512-neon-glue.c
+++ b/arch/arm/crypto/sha512-neon-glue.c
@@ -9,6 +9,7 @@
*/
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha512_base.h>
#include <linux/crypto.h>
@@ -30,7 +31,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,
{
struct sha512_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE)
return sha512_arm_update(desc, data, len);
@@ -45,7 +46,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,
static int sha512_neon_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha512_arm_finup(desc, data, len, out);
kernel_neon_begin();
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 5/8] crypto: arm - convert to use crypto_simd_usable()
2019-03-13 5:12 ` [PATCH 5/8] crypto: arm " Eric Biggers
@ 2019-03-13 10:33 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:33 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> Replace all calls to may_use_simd() in the arm crypto code with
> crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> arch/arm/crypto/chacha-neon-glue.c | 5 +++--
> arch/arm/crypto/crc32-ce-glue.c | 5 +++--
> arch/arm/crypto/crct10dif-ce-glue.c | 3 ++-
> arch/arm/crypto/ghash-ce-glue.c | 7 ++++---
> arch/arm/crypto/nhpoly1305-neon-glue.c | 3 ++-
> arch/arm/crypto/sha1-ce-glue.c | 5 +++--
> arch/arm/crypto/sha1_neon_glue.c | 5 +++--
> arch/arm/crypto/sha2-ce-glue.c | 5 +++--
> arch/arm/crypto/sha256_neon_glue.c | 5 +++--
> arch/arm/crypto/sha512-neon-glue.c | 5 +++--
> 10 files changed, 29 insertions(+), 19 deletions(-)
>
> diff --git a/arch/arm/crypto/chacha-neon-glue.c b/arch/arm/crypto/chacha-neon-glue.c
> index 9d6fda81986d..48a89537b828 100644
> --- a/arch/arm/crypto/chacha-neon-glue.c
> +++ b/arch/arm/crypto/chacha-neon-glue.c
> @@ -21,6 +21,7 @@
>
> #include <crypto/algapi.h>
> #include <crypto/chacha.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/internal/skcipher.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
> @@ -93,7 +94,7 @@ static int chacha_neon(struct skcipher_request *req)
> struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_chacha_crypt(req);
>
> return chacha_neon_stream_xor(req, ctx, req->iv);
> @@ -107,7 +108,7 @@ static int xchacha_neon(struct skcipher_request *req)
> u32 state[16];
> u8 real_iv[16];
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_xchacha_crypt(req);
>
> crypto_chacha_init(state, ctx, req->iv);
> diff --git a/arch/arm/crypto/crc32-ce-glue.c b/arch/arm/crypto/crc32-ce-glue.c
> index cd9e93b46c2d..e712c2a7d387 100644
> --- a/arch/arm/crypto/crc32-ce-glue.c
> +++ b/arch/arm/crypto/crc32-ce-glue.c
> @@ -16,6 +16,7 @@
> #include <linux/string.h>
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
>
> #include <asm/hwcap.h>
> #include <asm/neon.h>
> @@ -113,7 +114,7 @@ static int crc32_pmull_update(struct shash_desc *desc, const u8 *data,
> u32 *crc = shash_desc_ctx(desc);
> unsigned int l;
>
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> if ((u32)data % SCALE_F) {
> l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F));
>
> @@ -147,7 +148,7 @@ static int crc32c_pmull_update(struct shash_desc *desc, const u8 *data,
> u32 *crc = shash_desc_ctx(desc);
> unsigned int l;
>
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> if ((u32)data % SCALE_F) {
> l = min_t(u32, length, SCALE_F - ((u32)data % SCALE_F));
>
> diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c
> index 3d6b800b8396..3b24f2872592 100644
> --- a/arch/arm/crypto/crct10dif-ce-glue.c
> +++ b/arch/arm/crypto/crct10dif-ce-glue.c
> @@ -15,6 +15,7 @@
> #include <linux/string.h>
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
>
> #include <asm/neon.h>
> #include <asm/simd.h>
> @@ -36,7 +37,7 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data,
> {
> u16 *crc = shash_desc_ctx(desc);
>
> - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
> + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
> kernel_neon_begin();
> *crc = crc_t10dif_pmull(*crc, data, length);
> kernel_neon_end();
> diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c
> index b7d30b6cf49c..60123e9ea9d8 100644
> --- a/arch/arm/crypto/ghash-ce-glue.c
> +++ b/arch/arm/crypto/ghash-ce-glue.c
> @@ -14,6 +14,7 @@
> #include <asm/unaligned.h>
> #include <crypto/cryptd.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/gf128mul.h>
> #include <linux/cpufeature.h>
> #include <linux/crypto.h>
> @@ -196,7 +197,7 @@ static int ghash_async_update(struct ahash_request *req)
> struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> @@ -214,7 +215,7 @@ static int ghash_async_final(struct ahash_request *req)
> struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> @@ -232,7 +233,7 @@ static int ghash_async_digest(struct ahash_request *req)
> struct ahash_request *cryptd_req = ahash_request_ctx(req);
> struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
> memcpy(cryptd_req, req, sizeof(*req));
> ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
> diff --git a/arch/arm/crypto/nhpoly1305-neon-glue.c b/arch/arm/crypto/nhpoly1305-neon-glue.c
> index 49aae87cb2bc..ae5aefc44a4d 100644
> --- a/arch/arm/crypto/nhpoly1305-neon-glue.c
> +++ b/arch/arm/crypto/nhpoly1305-neon-glue.c
> @@ -9,6 +9,7 @@
> #include <asm/neon.h>
> #include <asm/simd.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/nhpoly1305.h>
> #include <linux/module.h>
>
> @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
> static int nhpoly1305_neon_update(struct shash_desc *desc,
> const u8 *src, unsigned int srclen)
> {
> - if (srclen < 64 || !may_use_simd())
> + if (srclen < 64 || !crypto_simd_usable())
> return crypto_nhpoly1305_update(desc, src, srclen);
>
> do {
> diff --git a/arch/arm/crypto/sha1-ce-glue.c b/arch/arm/crypto/sha1-ce-glue.c
> index b732522e20f8..4c6c6900853c 100644
> --- a/arch/arm/crypto/sha1-ce-glue.c
> +++ b/arch/arm/crypto/sha1-ce-glue.c
> @@ -9,6 +9,7 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha1_base.h>
> #include <linux/cpufeature.h>
> @@ -33,7 +34,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha1_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
> return sha1_update_arm(desc, data, len);
>
> @@ -47,7 +48,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
> static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha1_finup_arm(desc, data, len, out);
>
> kernel_neon_begin();
> diff --git a/arch/arm/crypto/sha1_neon_glue.c b/arch/arm/crypto/sha1_neon_glue.c
> index d15e0ea2c95e..d6c95c213d42 100644
> --- a/arch/arm/crypto/sha1_neon_glue.c
> +++ b/arch/arm/crypto/sha1_neon_glue.c
> @@ -19,6 +19,7 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/init.h>
> #include <linux/module.h>
> #include <linux/mm.h>
> @@ -39,7 +40,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha1_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA1_BLOCK_SIZE) + len < SHA1_BLOCK_SIZE)
> return sha1_update_arm(desc, data, len);
>
> @@ -54,7 +55,7 @@ static int sha1_neon_update(struct shash_desc *desc, const u8 *data,
> static int sha1_neon_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha1_finup_arm(desc, data, len, out);
>
> kernel_neon_begin();
> diff --git a/arch/arm/crypto/sha2-ce-glue.c b/arch/arm/crypto/sha2-ce-glue.c
> index 1211a5c129fc..a47a9d4b663e 100644
> --- a/arch/arm/crypto/sha2-ce-glue.c
> +++ b/arch/arm/crypto/sha2-ce-glue.c
> @@ -9,6 +9,7 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha256_base.h>
> #include <linux/cpufeature.h>
> @@ -34,7 +35,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha256_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
> return crypto_sha256_arm_update(desc, data, len);
>
> @@ -49,7 +50,7 @@ static int sha2_ce_update(struct shash_desc *desc, const u8 *data,
> static int sha2_ce_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha256_arm_finup(desc, data, len, out);
>
> kernel_neon_begin();
> diff --git a/arch/arm/crypto/sha256_neon_glue.c b/arch/arm/crypto/sha256_neon_glue.c
> index 1d82c6cd31a4..f3f6b1624fc3 100644
> --- a/arch/arm/crypto/sha256_neon_glue.c
> +++ b/arch/arm/crypto/sha256_neon_glue.c
> @@ -15,6 +15,7 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <linux/cryptohash.h>
> #include <linux/types.h>
> #include <linux/string.h>
> @@ -34,7 +35,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha256_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (sctx->count % SHA256_BLOCK_SIZE) + len < SHA256_BLOCK_SIZE)
> return crypto_sha256_arm_update(desc, data, len);
>
> @@ -49,7 +50,7 @@ static int sha256_update(struct shash_desc *desc, const u8 *data,
> static int sha256_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha256_arm_finup(desc, data, len, out);
>
> kernel_neon_begin();
> diff --git a/arch/arm/crypto/sha512-neon-glue.c b/arch/arm/crypto/sha512-neon-glue.c
> index 8a5642b41fd6..d33ab59c26c0 100644
> --- a/arch/arm/crypto/sha512-neon-glue.c
> +++ b/arch/arm/crypto/sha512-neon-glue.c
> @@ -9,6 +9,7 @@
> */
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha512_base.h>
> #include <linux/crypto.h>
> @@ -30,7 +31,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha512_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (sctx->count[0] % SHA512_BLOCK_SIZE) + len < SHA512_BLOCK_SIZE)
> return sha512_arm_update(desc, data, len);
>
> @@ -45,7 +46,7 @@ static int sha512_neon_update(struct shash_desc *desc, const u8 *data,
> static int sha512_neon_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha512_arm_finup(desc, data, len, out);
>
> kernel_neon_begin();
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 6/8] crypto: arm64 - convert to use crypto_simd_usable()
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (4 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 5/8] crypto: arm " Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:33 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 7/8] crypto: simd " Eric Biggers
` (3 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
Replace all calls to may_use_simd() in the arm64 crypto code with
crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm64/crypto/aes-ce-ccm-glue.c | 7 ++++---
arch/arm64/crypto/aes-ce-glue.c | 5 +++--
arch/arm64/crypto/aes-glue.c | 4 ++--
arch/arm64/crypto/aes-neonbs-glue.c | 2 +-
arch/arm64/crypto/chacha-neon-glue.c | 5 +++--
arch/arm64/crypto/crct10dif-ce-glue.c | 5 +++--
arch/arm64/crypto/ghash-ce-glue.c | 7 ++++---
arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 ++-
arch/arm64/crypto/sha1-ce-glue.c | 7 ++++---
arch/arm64/crypto/sha2-ce-glue.c | 7 ++++---
arch/arm64/crypto/sha256-glue.c | 5 +++--
arch/arm64/crypto/sha3-ce-glue.c | 5 +++--
arch/arm64/crypto/sha512-ce-glue.c | 7 ++++---
arch/arm64/crypto/sm3-ce-glue.c | 7 ++++---
arch/arm64/crypto/sm4-ce-glue.c | 5 +++--
15 files changed, 47 insertions(+), 34 deletions(-)
diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c
index 5fc6f51908fd..9dc4110a2e61 100644
--- a/arch/arm64/crypto/aes-ce-ccm-glue.c
+++ b/arch/arm64/crypto/aes-ce-ccm-glue.c
@@ -14,6 +14,7 @@
#include <crypto/aes.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
+#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <linux/module.h>
@@ -109,7 +110,7 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen)
static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
u32 abytes, u32 *macp)
{
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
kernel_neon_begin();
ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc,
num_rounds(key));
@@ -255,7 +256,7 @@ static int ccm_encrypt(struct aead_request *req)
err = skcipher_walk_aead_encrypt(&walk, req, false);
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
while (walk.nbytes) {
u32 tail = walk.nbytes % AES_BLOCK_SIZE;
@@ -313,7 +314,7 @@ static int ccm_decrypt(struct aead_request *req)
err = skcipher_walk_aead_decrypt(&walk, req, false);
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
while (walk.nbytes) {
u32 tail = walk.nbytes % AES_BLOCK_SIZE;
diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c
index e6b3227bbf57..3213843fcb46 100644
--- a/arch/arm64/crypto/aes-ce-glue.c
+++ b/arch/arm64/crypto/aes-ce-glue.c
@@ -12,6 +12,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/aes.h>
+#include <crypto/internal/simd.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
#include <linux/module.h>
@@ -52,7 +53,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[])
{
struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
__aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx));
return;
}
@@ -66,7 +67,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[])
{
struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
__aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx));
return;
}
diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c
index 1e676625ef33..692cb75f2ca2 100644
--- a/arch/arm64/crypto/aes-glue.c
+++ b/arch/arm64/crypto/aes-glue.c
@@ -405,7 +405,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return aes_ctr_encrypt_fallback(ctx, req);
return ctr_encrypt(req);
@@ -642,7 +642,7 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks,
{
int rounds = 6 + ctx->key_length / 4;
- if (may_use_simd()) {
+ if (crypto_simd_usable()) {
kernel_neon_begin();
aes_mac_update(in, ctx->key_enc, rounds, blocks, dg, enc_before,
enc_after);
diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c
index e7a95a566462..4737b6c6c5cf 100644
--- a/arch/arm64/crypto/aes-neonbs-glue.c
+++ b/arch/arm64/crypto/aes-neonbs-glue.c
@@ -288,7 +288,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return aes_ctr_encrypt_fallback(&ctx->fallback, req);
return ctr_encrypt(req);
diff --git a/arch/arm64/crypto/chacha-neon-glue.c b/arch/arm64/crypto/chacha-neon-glue.c
index bece1d85bd81..3a26a98a7e17 100644
--- a/arch/arm64/crypto/chacha-neon-glue.c
+++ b/arch/arm64/crypto/chacha-neon-glue.c
@@ -21,6 +21,7 @@
#include <crypto/algapi.h>
#include <crypto/chacha.h>
+#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <linux/kernel.h>
#include <linux/module.h>
@@ -90,7 +91,7 @@ static int chacha_neon(struct skcipher_request *req)
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_chacha_crypt(req);
return chacha_neon_stream_xor(req, ctx, req->iv);
@@ -104,7 +105,7 @@ static int xchacha_neon(struct skcipher_request *req)
u32 state[16];
u8 real_iv[16];
- if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
+ if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
return crypto_xchacha_crypt(req);
crypto_chacha_init(state, ctx, req->iv);
diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c
index dd325829ee44..64e92ab70269 100644
--- a/arch/arm64/crypto/crct10dif-ce-glue.c
+++ b/arch/arm64/crypto/crct10dif-ce-glue.c
@@ -16,6 +16,7 @@
#include <linux/string.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <asm/neon.h>
#include <asm/simd.h>
@@ -38,7 +39,7 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data,
{
u16 *crc = shash_desc_ctx(desc);
- if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
+ if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull_p8(*crc, data, length);
kernel_neon_end();
@@ -54,7 +55,7 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data,
{
u16 *crc = shash_desc_ctx(desc);
- if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
+ if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull_p64(*crc, data, length);
kernel_neon_end();
diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c
index 089b09286da7..fcd458c83bc1 100644
--- a/arch/arm64/crypto/ghash-ce-glue.c
+++ b/arch/arm64/crypto/ghash-ce-glue.c
@@ -17,6 +17,7 @@
#include <crypto/gf128mul.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <crypto/scatterwalk.h>
#include <linux/cpufeature.h>
@@ -89,7 +90,7 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,
struct ghash_key const *k,
const char *head))
{
- if (likely(may_use_simd())) {
+ if (likely(crypto_simd_usable())) {
kernel_neon_begin();
simd_update(blocks, dg, src, key, head);
kernel_neon_end();
@@ -441,7 +442,7 @@ static int gcm_encrypt(struct aead_request *req)
err = skcipher_walk_aead_encrypt(&walk, req, false);
- if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) {
+ if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) {
u32 const *rk = NULL;
kernel_neon_begin();
@@ -565,7 +566,7 @@ static int gcm_decrypt(struct aead_request *req)
err = skcipher_walk_aead_decrypt(&walk, req, false);
- if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) {
+ if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) {
u32 const *rk = NULL;
kernel_neon_begin();
diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/nhpoly1305-neon-glue.c
index 22cc32ac9448..d15e872fa3f5 100644
--- a/arch/arm64/crypto/nhpoly1305-neon-glue.c
+++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c
@@ -9,6 +9,7 @@
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/nhpoly1305.h>
#include <linux/module.h>
@@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
static int nhpoly1305_neon_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
- if (srclen < 64 || !may_use_simd())
+ if (srclen < 64 || !crypto_simd_usable())
return crypto_nhpoly1305_update(desc, src, srclen);
do {
diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c
index 17fac2889f56..eaa7a8258f1c 100644
--- a/arch/arm64/crypto/sha1-ce-glue.c
+++ b/arch/arm64/crypto/sha1-ce-glue.c
@@ -12,6 +12,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha1_base.h>
#include <linux/cpufeature.h>
@@ -38,7 +39,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
{
struct sha1_ce_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha1_update(desc, data, len);
sctx->finalize = 0;
@@ -56,7 +57,7 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,
struct sha1_ce_state *sctx = shash_desc_ctx(desc);
bool finalize = !sctx->sst.count && !(len % SHA1_BLOCK_SIZE);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha1_finup(desc, data, len, out);
/*
@@ -78,7 +79,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out)
{
struct sha1_ce_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha1_finup(desc, NULL, 0, out);
sctx->finalize = 0;
diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c
index 261f5195cab7..a725997e55f2 100644
--- a/arch/arm64/crypto/sha2-ce-glue.c
+++ b/arch/arm64/crypto/sha2-ce-glue.c
@@ -12,6 +12,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha256_base.h>
#include <linux/cpufeature.h>
@@ -42,7 +43,7 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data,
{
struct sha256_ce_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
@@ -61,7 +62,7 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,
struct sha256_ce_state *sctx = shash_desc_ctx(desc);
bool finalize = !sctx->sst.count && !(len % SHA256_BLOCK_SIZE);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
if (len)
sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
@@ -90,7 +91,7 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out)
{
struct sha256_ce_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
sha256_base_do_finalize(desc,
(sha256_block_fn *)sha256_block_data_order);
return sha256_base_finish(desc, out);
diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c
index 4aedeaefd61f..54586e0be9fd 100644
--- a/arch/arm64/crypto/sha256-glue.c
+++ b/arch/arm64/crypto/sha256-glue.c
@@ -14,6 +14,7 @@
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha256_base.h>
#include <linux/cryptohash.h>
@@ -89,7 +90,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
{
struct sha256_state *sctx = shash_desc_ctx(desc);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
@@ -119,7 +120,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
static int sha256_finup_neon(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
if (len)
sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-glue.c
index a336feac0f59..9a4bbfc45f40 100644
--- a/arch/arm64/crypto/sha3-ce-glue.c
+++ b/arch/arm64/crypto/sha3-ce-glue.c
@@ -14,6 +14,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha3.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
@@ -32,7 +33,7 @@ static int sha3_update(struct shash_desc *desc, const u8 *data,
struct sha3_state *sctx = shash_desc_ctx(desc);
unsigned int digest_size = crypto_shash_digestsize(desc->tfm);
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha3_update(desc, data, len);
if ((sctx->partial + len) >= sctx->rsiz) {
@@ -76,7 +77,7 @@ static int sha3_final(struct shash_desc *desc, u8 *out)
__le64 *digest = (__le64 *)out;
int i;
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sha3_final(desc, out);
sctx->buf[sctx->partial++] = 0x06;
diff --git a/arch/arm64/crypto/sha512-ce-glue.c b/arch/arm64/crypto/sha512-ce-glue.c
index f2c5f28c622a..2369540040aa 100644
--- a/arch/arm64/crypto/sha512-ce-glue.c
+++ b/arch/arm64/crypto/sha512-ce-glue.c
@@ -13,6 +13,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sha.h>
#include <crypto/sha512_base.h>
#include <linux/cpufeature.h>
@@ -31,7 +32,7 @@ asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks);
static int sha512_ce_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return sha512_base_do_update(desc, data, len,
(sha512_block_fn *)sha512_block_data_order);
@@ -46,7 +47,7 @@ static int sha512_ce_update(struct shash_desc *desc, const u8 *data,
static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
if (len)
sha512_base_do_update(desc, data, len,
(sha512_block_fn *)sha512_block_data_order);
@@ -65,7 +66,7 @@ static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,
static int sha512_ce_final(struct shash_desc *desc, u8 *out)
{
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
sha512_base_do_finalize(desc,
(sha512_block_fn *)sha512_block_data_order);
return sha512_base_finish(desc, out);
diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glue.c
index 88938a20d9b2..5d15533799a2 100644
--- a/arch/arm64/crypto/sm3-ce-glue.c
+++ b/arch/arm64/crypto/sm3-ce-glue.c
@@ -12,6 +12,7 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/cpufeature.h>
@@ -28,7 +29,7 @@ asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src,
static int sm3_ce_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sm3_update(desc, data, len);
kernel_neon_begin();
@@ -40,7 +41,7 @@ static int sm3_ce_update(struct shash_desc *desc, const u8 *data,
static int sm3_ce_final(struct shash_desc *desc, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sm3_finup(desc, NULL, 0, out);
kernel_neon_begin();
@@ -53,7 +54,7 @@ static int sm3_ce_final(struct shash_desc *desc, u8 *out)
static int sm3_ce_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
- if (!may_use_simd())
+ if (!crypto_simd_usable())
return crypto_sm3_finup(desc, data, len, out);
kernel_neon_begin();
diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glue.c
index 0c4fc223f225..2754c875d39c 100644
--- a/arch/arm64/crypto/sm4-ce-glue.c
+++ b/arch/arm64/crypto/sm4-ce-glue.c
@@ -3,6 +3,7 @@
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/sm4.h>
+#include <crypto/internal/simd.h>
#include <linux/module.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
@@ -20,7 +21,7 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
{
const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
crypto_sm4_encrypt(tfm, out, in);
} else {
kernel_neon_begin();
@@ -33,7 +34,7 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
{
const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm);
- if (!may_use_simd()) {
+ if (!crypto_simd_usable()) {
crypto_sm4_decrypt(tfm, out, in);
} else {
kernel_neon_begin();
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 6/8] crypto: arm64 - convert to use crypto_simd_usable()
2019-03-13 5:12 ` [PATCH 6/8] crypto: arm64 " Eric Biggers
@ 2019-03-13 10:33 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:33 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> Replace all calls to may_use_simd() in the arm64 crypto code with
> crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> arch/arm64/crypto/aes-ce-ccm-glue.c | 7 ++++---
> arch/arm64/crypto/aes-ce-glue.c | 5 +++--
> arch/arm64/crypto/aes-glue.c | 4 ++--
> arch/arm64/crypto/aes-neonbs-glue.c | 2 +-
> arch/arm64/crypto/chacha-neon-glue.c | 5 +++--
> arch/arm64/crypto/crct10dif-ce-glue.c | 5 +++--
> arch/arm64/crypto/ghash-ce-glue.c | 7 ++++---
> arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 ++-
> arch/arm64/crypto/sha1-ce-glue.c | 7 ++++---
> arch/arm64/crypto/sha2-ce-glue.c | 7 ++++---
> arch/arm64/crypto/sha256-glue.c | 5 +++--
> arch/arm64/crypto/sha3-ce-glue.c | 5 +++--
> arch/arm64/crypto/sha512-ce-glue.c | 7 ++++---
> arch/arm64/crypto/sm3-ce-glue.c | 7 ++++---
> arch/arm64/crypto/sm4-ce-glue.c | 5 +++--
> 15 files changed, 47 insertions(+), 34 deletions(-)
>
> diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c
> index 5fc6f51908fd..9dc4110a2e61 100644
> --- a/arch/arm64/crypto/aes-ce-ccm-glue.c
> +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c
> @@ -14,6 +14,7 @@
> #include <crypto/aes.h>
> #include <crypto/scatterwalk.h>
> #include <crypto/internal/aead.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/internal/skcipher.h>
> #include <linux/module.h>
>
> @@ -109,7 +110,7 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen)
> static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
> u32 abytes, u32 *macp)
> {
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> kernel_neon_begin();
> ce_aes_ccm_auth_data(mac, in, abytes, macp, key->key_enc,
> num_rounds(key));
> @@ -255,7 +256,7 @@ static int ccm_encrypt(struct aead_request *req)
>
> err = skcipher_walk_aead_encrypt(&walk, req, false);
>
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> while (walk.nbytes) {
> u32 tail = walk.nbytes % AES_BLOCK_SIZE;
>
> @@ -313,7 +314,7 @@ static int ccm_decrypt(struct aead_request *req)
>
> err = skcipher_walk_aead_decrypt(&walk, req, false);
>
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> while (walk.nbytes) {
> u32 tail = walk.nbytes % AES_BLOCK_SIZE;
>
> diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glue.c
> index e6b3227bbf57..3213843fcb46 100644
> --- a/arch/arm64/crypto/aes-ce-glue.c
> +++ b/arch/arm64/crypto/aes-ce-glue.c
> @@ -12,6 +12,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/aes.h>
> +#include <crypto/internal/simd.h>
> #include <linux/cpufeature.h>
> #include <linux/crypto.h>
> #include <linux/module.h>
> @@ -52,7 +53,7 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[])
> {
> struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> __aes_arm64_encrypt(ctx->key_enc, dst, src, num_rounds(ctx));
> return;
> }
> @@ -66,7 +67,7 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[])
> {
> struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> __aes_arm64_decrypt(ctx->key_dec, dst, src, num_rounds(ctx));
> return;
> }
> diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c
> index 1e676625ef33..692cb75f2ca2 100644
> --- a/arch/arm64/crypto/aes-glue.c
> +++ b/arch/arm64/crypto/aes-glue.c
> @@ -405,7 +405,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)
> struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return aes_ctr_encrypt_fallback(ctx, req);
>
> return ctr_encrypt(req);
> @@ -642,7 +642,7 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks,
> {
> int rounds = 6 + ctx->key_length / 4;
>
> - if (may_use_simd()) {
> + if (crypto_simd_usable()) {
> kernel_neon_begin();
> aes_mac_update(in, ctx->key_enc, rounds, blocks, dg, enc_before,
> enc_after);
> diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c
> index e7a95a566462..4737b6c6c5cf 100644
> --- a/arch/arm64/crypto/aes-neonbs-glue.c
> +++ b/arch/arm64/crypto/aes-neonbs-glue.c
> @@ -288,7 +288,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req)
> struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> struct aesbs_ctr_ctx *ctx = crypto_skcipher_ctx(tfm);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return aes_ctr_encrypt_fallback(&ctx->fallback, req);
>
> return ctr_encrypt(req);
> diff --git a/arch/arm64/crypto/chacha-neon-glue.c b/arch/arm64/crypto/chacha-neon-glue.c
> index bece1d85bd81..3a26a98a7e17 100644
> --- a/arch/arm64/crypto/chacha-neon-glue.c
> +++ b/arch/arm64/crypto/chacha-neon-glue.c
> @@ -21,6 +21,7 @@
>
> #include <crypto/algapi.h>
> #include <crypto/chacha.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/internal/skcipher.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
> @@ -90,7 +91,7 @@ static int chacha_neon(struct skcipher_request *req)
> struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_chacha_crypt(req);
>
> return chacha_neon_stream_xor(req, ctx, req->iv);
> @@ -104,7 +105,7 @@ static int xchacha_neon(struct skcipher_request *req)
> u32 state[16];
> u8 real_iv[16];
>
> - if (req->cryptlen <= CHACHA_BLOCK_SIZE || !may_use_simd())
> + if (req->cryptlen <= CHACHA_BLOCK_SIZE || !crypto_simd_usable())
> return crypto_xchacha_crypt(req);
>
> crypto_chacha_init(state, ctx, req->iv);
> diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c
> index dd325829ee44..64e92ab70269 100644
> --- a/arch/arm64/crypto/crct10dif-ce-glue.c
> +++ b/arch/arm64/crypto/crct10dif-ce-glue.c
> @@ -16,6 +16,7 @@
> #include <linux/string.h>
>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
>
> #include <asm/neon.h>
> #include <asm/simd.h>
> @@ -38,7 +39,7 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data,
> {
> u16 *crc = shash_desc_ctx(desc);
>
> - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
> + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
> kernel_neon_begin();
> *crc = crc_t10dif_pmull_p8(*crc, data, length);
> kernel_neon_end();
> @@ -54,7 +55,7 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data,
> {
> u16 *crc = shash_desc_ctx(desc);
>
> - if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
> + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && crypto_simd_usable()) {
> kernel_neon_begin();
> *crc = crc_t10dif_pmull_p64(*crc, data, length);
> kernel_neon_end();
> diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c
> index 089b09286da7..fcd458c83bc1 100644
> --- a/arch/arm64/crypto/ghash-ce-glue.c
> +++ b/arch/arm64/crypto/ghash-ce-glue.c
> @@ -17,6 +17,7 @@
> #include <crypto/gf128mul.h>
> #include <crypto/internal/aead.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/internal/skcipher.h>
> #include <crypto/scatterwalk.h>
> #include <linux/cpufeature.h>
> @@ -89,7 +90,7 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,
> struct ghash_key const *k,
> const char *head))
> {
> - if (likely(may_use_simd())) {
> + if (likely(crypto_simd_usable())) {
> kernel_neon_begin();
> simd_update(blocks, dg, src, key, head);
> kernel_neon_end();
> @@ -441,7 +442,7 @@ static int gcm_encrypt(struct aead_request *req)
>
> err = skcipher_walk_aead_encrypt(&walk, req, false);
>
> - if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) {
> + if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) {
> u32 const *rk = NULL;
>
> kernel_neon_begin();
> @@ -565,7 +566,7 @@ static int gcm_decrypt(struct aead_request *req)
>
> err = skcipher_walk_aead_decrypt(&walk, req, false);
>
> - if (likely(may_use_simd() && walk.total >= 2 * AES_BLOCK_SIZE)) {
> + if (likely(crypto_simd_usable() && walk.total >= 2 * AES_BLOCK_SIZE)) {
> u32 const *rk = NULL;
>
> kernel_neon_begin();
> diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/nhpoly1305-neon-glue.c
> index 22cc32ac9448..d15e872fa3f5 100644
> --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c
> +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c
> @@ -9,6 +9,7 @@
> #include <asm/neon.h>
> #include <asm/simd.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/nhpoly1305.h>
> #include <linux/module.h>
>
> @@ -25,7 +26,7 @@ static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
> static int nhpoly1305_neon_update(struct shash_desc *desc,
> const u8 *src, unsigned int srclen)
> {
> - if (srclen < 64 || !may_use_simd())
> + if (srclen < 64 || !crypto_simd_usable())
> return crypto_nhpoly1305_update(desc, src, srclen);
>
> do {
> diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c
> index 17fac2889f56..eaa7a8258f1c 100644
> --- a/arch/arm64/crypto/sha1-ce-glue.c
> +++ b/arch/arm64/crypto/sha1-ce-glue.c
> @@ -12,6 +12,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha1_base.h>
> #include <linux/cpufeature.h>
> @@ -38,7 +39,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha1_ce_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha1_update(desc, data, len);
>
> sctx->finalize = 0;
> @@ -56,7 +57,7 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,
> struct sha1_ce_state *sctx = shash_desc_ctx(desc);
> bool finalize = !sctx->sst.count && !(len % SHA1_BLOCK_SIZE);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha1_finup(desc, data, len, out);
>
> /*
> @@ -78,7 +79,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out)
> {
> struct sha1_ce_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha1_finup(desc, NULL, 0, out);
>
> sctx->finalize = 0;
> diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c
> index 261f5195cab7..a725997e55f2 100644
> --- a/arch/arm64/crypto/sha2-ce-glue.c
> +++ b/arch/arm64/crypto/sha2-ce-glue.c
> @@ -12,6 +12,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha256_base.h>
> #include <linux/cpufeature.h>
> @@ -42,7 +43,7 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data,
> {
> struct sha256_ce_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha256_base_do_update(desc, data, len,
> (sha256_block_fn *)sha256_block_data_order);
>
> @@ -61,7 +62,7 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,
> struct sha256_ce_state *sctx = shash_desc_ctx(desc);
> bool finalize = !sctx->sst.count && !(len % SHA256_BLOCK_SIZE);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> if (len)
> sha256_base_do_update(desc, data, len,
> (sha256_block_fn *)sha256_block_data_order);
> @@ -90,7 +91,7 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out)
> {
> struct sha256_ce_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> sha256_base_do_finalize(desc,
> (sha256_block_fn *)sha256_block_data_order);
> return sha256_base_finish(desc, out);
> diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c
> index 4aedeaefd61f..54586e0be9fd 100644
> --- a/arch/arm64/crypto/sha256-glue.c
> +++ b/arch/arm64/crypto/sha256-glue.c
> @@ -14,6 +14,7 @@
> #include <asm/neon.h>
> #include <asm/simd.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha256_base.h>
> #include <linux/cryptohash.h>
> @@ -89,7 +90,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
> {
> struct sha256_state *sctx = shash_desc_ctx(desc);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha256_base_do_update(desc, data, len,
> (sha256_block_fn *)sha256_block_data_order);
>
> @@ -119,7 +120,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
> static int sha256_finup_neon(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> if (len)
> sha256_base_do_update(desc, data, len,
> (sha256_block_fn *)sha256_block_data_order);
> diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-glue.c
> index a336feac0f59..9a4bbfc45f40 100644
> --- a/arch/arm64/crypto/sha3-ce-glue.c
> +++ b/arch/arm64/crypto/sha3-ce-glue.c
> @@ -14,6 +14,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha3.h>
> #include <linux/cpufeature.h>
> #include <linux/crypto.h>
> @@ -32,7 +33,7 @@ static int sha3_update(struct shash_desc *desc, const u8 *data,
> struct sha3_state *sctx = shash_desc_ctx(desc);
> unsigned int digest_size = crypto_shash_digestsize(desc->tfm);
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha3_update(desc, data, len);
>
> if ((sctx->partial + len) >= sctx->rsiz) {
> @@ -76,7 +77,7 @@ static int sha3_final(struct shash_desc *desc, u8 *out)
> __le64 *digest = (__le64 *)out;
> int i;
>
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sha3_final(desc, out);
>
> sctx->buf[sctx->partial++] = 0x06;
> diff --git a/arch/arm64/crypto/sha512-ce-glue.c b/arch/arm64/crypto/sha512-ce-glue.c
> index f2c5f28c622a..2369540040aa 100644
> --- a/arch/arm64/crypto/sha512-ce-glue.c
> +++ b/arch/arm64/crypto/sha512-ce-glue.c
> @@ -13,6 +13,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sha.h>
> #include <crypto/sha512_base.h>
> #include <linux/cpufeature.h>
> @@ -31,7 +32,7 @@ asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks);
> static int sha512_ce_update(struct shash_desc *desc, const u8 *data,
> unsigned int len)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return sha512_base_do_update(desc, data, len,
> (sha512_block_fn *)sha512_block_data_order);
>
> @@ -46,7 +47,7 @@ static int sha512_ce_update(struct shash_desc *desc, const u8 *data,
> static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> if (len)
> sha512_base_do_update(desc, data, len,
> (sha512_block_fn *)sha512_block_data_order);
> @@ -65,7 +66,7 @@ static int sha512_ce_finup(struct shash_desc *desc, const u8 *data,
>
> static int sha512_ce_final(struct shash_desc *desc, u8 *out)
> {
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> sha512_base_do_finalize(desc,
> (sha512_block_fn *)sha512_block_data_order);
> return sha512_base_finish(desc, out);
> diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glue.c
> index 88938a20d9b2..5d15533799a2 100644
> --- a/arch/arm64/crypto/sm3-ce-glue.c
> +++ b/arch/arm64/crypto/sm3-ce-glue.c
> @@ -12,6 +12,7 @@
> #include <asm/simd.h>
> #include <asm/unaligned.h>
> #include <crypto/internal/hash.h>
> +#include <crypto/internal/simd.h>
> #include <crypto/sm3.h>
> #include <crypto/sm3_base.h>
> #include <linux/cpufeature.h>
> @@ -28,7 +29,7 @@ asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src,
> static int sm3_ce_update(struct shash_desc *desc, const u8 *data,
> unsigned int len)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sm3_update(desc, data, len);
>
> kernel_neon_begin();
> @@ -40,7 +41,7 @@ static int sm3_ce_update(struct shash_desc *desc, const u8 *data,
>
> static int sm3_ce_final(struct shash_desc *desc, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sm3_finup(desc, NULL, 0, out);
>
> kernel_neon_begin();
> @@ -53,7 +54,7 @@ static int sm3_ce_final(struct shash_desc *desc, u8 *out)
> static int sm3_ce_finup(struct shash_desc *desc, const u8 *data,
> unsigned int len, u8 *out)
> {
> - if (!may_use_simd())
> + if (!crypto_simd_usable())
> return crypto_sm3_finup(desc, data, len, out);
>
> kernel_neon_begin();
> diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glue.c
> index 0c4fc223f225..2754c875d39c 100644
> --- a/arch/arm64/crypto/sm4-ce-glue.c
> +++ b/arch/arm64/crypto/sm4-ce-glue.c
> @@ -3,6 +3,7 @@
> #include <asm/neon.h>
> #include <asm/simd.h>
> #include <crypto/sm4.h>
> +#include <crypto/internal/simd.h>
> #include <linux/module.h>
> #include <linux/cpufeature.h>
> #include <linux/crypto.h>
> @@ -20,7 +21,7 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
> {
> const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> crypto_sm4_encrypt(tfm, out, in);
> } else {
> kernel_neon_begin();
> @@ -33,7 +34,7 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
> {
> const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm);
>
> - if (!may_use_simd()) {
> + if (!crypto_simd_usable()) {
> crypto_sm4_decrypt(tfm, out, in);
> } else {
> kernel_neon_begin();
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 7/8] crypto: simd - convert to use crypto_simd_usable()
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (5 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 6/8] crypto: arm64 " Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:34 ` Ard Biesheuvel
2019-03-13 5:12 ` [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code Eric Biggers
` (2 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
Replace all calls to may_use_simd() in the shared SIMD helpers with
crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/simd.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/crypto/simd.c b/crypto/simd.c
index 7d62686d3a3f..3e3b1d1a6b1f 100644
--- a/crypto/simd.c
+++ b/crypto/simd.c
@@ -85,7 +85,7 @@ static int simd_skcipher_encrypt(struct skcipher_request *req)
subreq = skcipher_request_ctx(req);
*subreq = *req;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm)))
child = &ctx->cryptd_tfm->base;
else
@@ -106,7 +106,7 @@ static int simd_skcipher_decrypt(struct skcipher_request *req)
subreq = skcipher_request_ctx(req);
*subreq = *req;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm)))
child = &ctx->cryptd_tfm->base;
else
@@ -336,7 +336,7 @@ static int simd_aead_encrypt(struct aead_request *req)
subreq = aead_request_ctx(req);
*subreq = *req;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm)))
child = &ctx->cryptd_tfm->base;
else
@@ -357,7 +357,7 @@ static int simd_aead_decrypt(struct aead_request *req)
subreq = aead_request_ctx(req);
*subreq = *req;
- if (!may_use_simd() ||
+ if (!crypto_simd_usable() ||
(in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm)))
child = &ctx->cryptd_tfm->base;
else
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 7/8] crypto: simd - convert to use crypto_simd_usable()
2019-03-13 5:12 ` [PATCH 7/8] crypto: simd " Eric Biggers
@ 2019-03-13 10:34 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:34 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> Replace all calls to may_use_simd() in the shared SIMD helpers with
> crypto_simd_usable(), in order to allow testing the no-SIMD code paths.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> crypto/simd.c | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/crypto/simd.c b/crypto/simd.c
> index 7d62686d3a3f..3e3b1d1a6b1f 100644
> --- a/crypto/simd.c
> +++ b/crypto/simd.c
> @@ -85,7 +85,7 @@ static int simd_skcipher_encrypt(struct skcipher_request *req)
> subreq = skcipher_request_ctx(req);
> *subreq = *req;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm)))
> child = &ctx->cryptd_tfm->base;
> else
> @@ -106,7 +106,7 @@ static int simd_skcipher_decrypt(struct skcipher_request *req)
> subreq = skcipher_request_ctx(req);
> *subreq = *req;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_skcipher_queued(ctx->cryptd_tfm)))
> child = &ctx->cryptd_tfm->base;
> else
> @@ -336,7 +336,7 @@ static int simd_aead_encrypt(struct aead_request *req)
> subreq = aead_request_ctx(req);
> *subreq = *req;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm)))
> child = &ctx->cryptd_tfm->base;
> else
> @@ -357,7 +357,7 @@ static int simd_aead_decrypt(struct aead_request *req)
> subreq = aead_request_ctx(req);
> *subreq = *req;
>
> - if (!may_use_simd() ||
> + if (!crypto_simd_usable() ||
> (in_atomic() && cryptd_aead_queued(ctx->cryptd_tfm)))
> child = &ctx->cryptd_tfm->base;
> else
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (6 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 7/8] crypto: simd " Eric Biggers
@ 2019-03-13 5:12 ` Eric Biggers
2019-03-13 10:35 ` Ard Biesheuvel
2019-03-13 10:50 ` [PATCH 0/8] crypto: " Ard Biesheuvel
2019-03-22 13:03 ` Herbert Xu
9 siblings, 1 reply; 19+ messages in thread
From: Eric Biggers @ 2019-03-13 5:12 UTC (permalink / raw)
To: linux-crypto, Herbert Xu; +Cc: Ard Biesheuvel, linux-arm-kernel, x86
From: Eric Biggers <ebiggers@google.com>
All crypto API algorithms are supposed to support the case where they
are called in a context where SIMD instructions are unusable, e.g. IRQ
context on some architectures. However, this isn't tested for by the
self-tests, causing bugs to go undetected.
Now that all algorithms have been converted to use crypto_simd_usable(),
update the self-tests to test the no-SIMD case. First, a bool
testvec_config::nosimd is added. When set, the crypto operation is
executed with preemption disabled and with crypto_simd_usable() mocked
out to return false on the current CPU.
A bool test_sg_division::nosimd is also added. For hash algorithms it's
honored by the corresponding ->update(). By setting just a subset of
these bools, the case where some ->update()s are done in SIMD context
and some are done in no-SIMD context is also tested.
These bools are then randomly set by generate_random_testvec_config().
For now, all no-SIMD testing is limited to the extra crypto self-tests,
because it might be a bit too invasive for the regular self-tests.
But this could be changed later.
This has already found bugs in the arm64 AES-GCM and ChaCha algorithms.
This would have found some past bugs as well.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/testmgr.c | 116 +++++++++++++++++++++++++++++++++++++----------
1 file changed, 92 insertions(+), 24 deletions(-)
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 52417dde811f..2c2ddebb48d3 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -234,12 +234,14 @@ enum finalization_type {
* @offset
* @flush_type: for hashes, whether an update() should be done now vs.
* continuing to accumulate data
+ * @nosimd: if doing the pending update(), do it with SIMD disabled?
*/
struct test_sg_division {
unsigned int proportion_of_total;
unsigned int offset;
bool offset_relative_to_alignmask;
enum flush_type flush_type;
+ bool nosimd;
};
/**
@@ -259,6 +261,7 @@ struct test_sg_division {
* @iv_offset_relative_to_alignmask: if true, add the algorithm's alignmask to
* the @iv_offset
* @finalization_type: what finalization function to use for hashes
+ * @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP.
*/
struct testvec_config {
const char *name;
@@ -269,6 +272,7 @@ struct testvec_config {
unsigned int iv_offset;
bool iv_offset_relative_to_alignmask;
enum finalization_type finalization_type;
+ bool nosimd;
};
#define TESTVEC_CONFIG_NAMELEN 192
@@ -420,8 +424,11 @@ static unsigned int count_test_sg_divisions(const struct test_sg_division *divs)
return ndivs;
}
+#define SGDIVS_HAVE_FLUSHES BIT(0)
+#define SGDIVS_HAVE_NOSIMD BIT(1)
+
static bool valid_sg_divisions(const struct test_sg_division *divs,
- unsigned int count, bool *any_flushes_ret)
+ unsigned int count, int *flags_ret)
{
unsigned int total = 0;
unsigned int i;
@@ -432,7 +439,9 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,
return false;
total += divs[i].proportion_of_total;
if (divs[i].flush_type != FLUSH_TYPE_NONE)
- *any_flushes_ret = true;
+ *flags_ret |= SGDIVS_HAVE_FLUSHES;
+ if (divs[i].nosimd)
+ *flags_ret |= SGDIVS_HAVE_NOSIMD;
}
return total == TEST_SG_TOTAL &&
memchr_inv(&divs[i], 0, (count - i) * sizeof(divs[0])) == NULL;
@@ -445,19 +454,18 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,
*/
static bool valid_testvec_config(const struct testvec_config *cfg)
{
- bool any_flushes = false;
+ int flags = 0;
if (cfg->name == NULL)
return false;
if (!valid_sg_divisions(cfg->src_divs, ARRAY_SIZE(cfg->src_divs),
- &any_flushes))
+ &flags))
return false;
if (cfg->dst_divs[0].proportion_of_total) {
if (!valid_sg_divisions(cfg->dst_divs,
- ARRAY_SIZE(cfg->dst_divs),
- &any_flushes))
+ ARRAY_SIZE(cfg->dst_divs), &flags))
return false;
} else {
if (memchr_inv(cfg->dst_divs, 0, sizeof(cfg->dst_divs)))
@@ -470,7 +478,12 @@ static bool valid_testvec_config(const struct testvec_config *cfg)
MAX_ALGAPI_ALIGNMASK + 1)
return false;
- if (any_flushes && cfg->finalization_type == FINALIZATION_TYPE_DIGEST)
+ if ((flags & (SGDIVS_HAVE_FLUSHES | SGDIVS_HAVE_NOSIMD)) &&
+ cfg->finalization_type == FINALIZATION_TYPE_DIGEST)
+ return false;
+
+ if ((cfg->nosimd || (flags & SGDIVS_HAVE_NOSIMD)) &&
+ (cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP))
return false;
return true;
@@ -731,13 +744,14 @@ static int build_cipher_test_sglists(struct cipher_test_sglists *tsgls,
#ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS
static char *generate_random_sgl_divisions(struct test_sg_division *divs,
size_t max_divs, char *p, char *end,
- bool gen_flushes)
+ bool gen_flushes, u32 req_flags)
{
struct test_sg_division *div = divs;
unsigned int remaining = TEST_SG_TOTAL;
do {
unsigned int this_len;
+ const char *flushtype_str;
if (div == &divs[max_divs - 1] || prandom_u32() % 2 == 0)
this_len = remaining;
@@ -766,11 +780,31 @@ static char *generate_random_sgl_divisions(struct test_sg_division *divs,
}
}
+ if (div->flush_type != FLUSH_TYPE_NONE &&
+ !(req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) &&
+ prandom_u32() % 2 == 0)
+ div->nosimd = true;
+
+ switch (div->flush_type) {
+ case FLUSH_TYPE_FLUSH:
+ if (div->nosimd)
+ flushtype_str = "<flush,nosimd>";
+ else
+ flushtype_str = "<flush>";
+ break;
+ case FLUSH_TYPE_REIMPORT:
+ if (div->nosimd)
+ flushtype_str = "<reimport,nosimd>";
+ else
+ flushtype_str = "<reimport>";
+ break;
+ default:
+ flushtype_str = "";
+ break;
+ }
+
BUILD_BUG_ON(TEST_SG_TOTAL != 10000); /* for "%u.%u%%" */
- p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s",
- div->flush_type == FLUSH_TYPE_NONE ? "" :
- div->flush_type == FLUSH_TYPE_FLUSH ?
- "<flush> " : "<reimport> ",
+ p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", flushtype_str,
this_len / 100, this_len % 100,
div->offset_relative_to_alignmask ?
"alignmask" : "",
@@ -820,18 +854,26 @@ static void generate_random_testvec_config(struct testvec_config *cfg,
break;
}
+ if (!(cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) &&
+ prandom_u32() % 2 == 0) {
+ cfg->nosimd = true;
+ p += scnprintf(p, end - p, " nosimd");
+ }
+
p += scnprintf(p, end - p, " src_divs=[");
p = generate_random_sgl_divisions(cfg->src_divs,
ARRAY_SIZE(cfg->src_divs), p, end,
(cfg->finalization_type !=
- FINALIZATION_TYPE_DIGEST));
+ FINALIZATION_TYPE_DIGEST),
+ cfg->req_flags);
p += scnprintf(p, end - p, "]");
if (!cfg->inplace && prandom_u32() % 2 == 0) {
p += scnprintf(p, end - p, " dst_divs=[");
p = generate_random_sgl_divisions(cfg->dst_divs,
ARRAY_SIZE(cfg->dst_divs),
- p, end, false);
+ p, end, false,
+ cfg->req_flags);
p += scnprintf(p, end - p, "]");
}
@@ -864,6 +906,23 @@ static void crypto_reenable_simd_for_test(void)
}
#endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
+static int do_ahash_op(int (*op)(struct ahash_request *req),
+ struct ahash_request *req,
+ struct crypto_wait *wait, bool nosimd)
+{
+ int err;
+
+ if (nosimd)
+ crypto_disable_simd_for_test();
+
+ err = op(req);
+
+ if (nosimd)
+ crypto_reenable_simd_for_test();
+
+ return crypto_wait_req(err, wait);
+}
+
static int check_nonfinal_hash_op(const char *op, int err,
u8 *result, unsigned int digestsize,
const char *driver, unsigned int vec_num,
@@ -938,7 +997,7 @@ static int test_hash_vec_cfg(const char *driver,
ahash_request_set_callback(req, req_flags, crypto_req_done,
&wait);
ahash_request_set_crypt(req, tsgl->sgl, result, vec->psize);
- err = crypto_wait_req(crypto_ahash_digest(req), &wait);
+ err = do_ahash_op(crypto_ahash_digest, req, &wait, cfg->nosimd);
if (err) {
pr_err("alg: hash: %s digest() failed with err %d on test vector %u, cfg=\"%s\"\n",
driver, err, vec_num, cfg->name);
@@ -951,7 +1010,7 @@ static int test_hash_vec_cfg(const char *driver,
ahash_request_set_callback(req, req_flags, crypto_req_done, &wait);
ahash_request_set_crypt(req, NULL, result, 0);
- err = crypto_wait_req(crypto_ahash_init(req), &wait);
+ err = do_ahash_op(crypto_ahash_init, req, &wait, cfg->nosimd);
err = check_nonfinal_hash_op("init", err, result, digestsize,
driver, vec_num, cfg);
if (err)
@@ -967,7 +1026,8 @@ static int test_hash_vec_cfg(const char *driver,
crypto_req_done, &wait);
ahash_request_set_crypt(req, pending_sgl, result,
pending_len);
- err = crypto_wait_req(crypto_ahash_update(req), &wait);
+ err = do_ahash_op(crypto_ahash_update, req, &wait,
+ divs[i]->nosimd);
err = check_nonfinal_hash_op("update", err,
result, digestsize,
driver, vec_num, cfg);
@@ -1010,12 +1070,12 @@ static int test_hash_vec_cfg(const char *driver,
ahash_request_set_crypt(req, pending_sgl, result, pending_len);
if (cfg->finalization_type == FINALIZATION_TYPE_FINAL) {
/* finish with update() and final() */
- err = crypto_wait_req(crypto_ahash_update(req), &wait);
+ err = do_ahash_op(crypto_ahash_update, req, &wait, cfg->nosimd);
err = check_nonfinal_hash_op("update", err, result, digestsize,
driver, vec_num, cfg);
if (err)
return err;
- err = crypto_wait_req(crypto_ahash_final(req), &wait);
+ err = do_ahash_op(crypto_ahash_final, req, &wait, cfg->nosimd);
if (err) {
pr_err("alg: hash: %s final() failed with err %d on test vector %u, cfg=\"%s\"\n",
driver, err, vec_num, cfg->name);
@@ -1023,7 +1083,7 @@ static int test_hash_vec_cfg(const char *driver,
}
} else {
/* finish with finup() */
- err = crypto_wait_req(crypto_ahash_finup(req), &wait);
+ err = do_ahash_op(crypto_ahash_finup, req, &wait, cfg->nosimd);
if (err) {
pr_err("alg: hash: %s finup() failed with err %d on test vector %u, cfg=\"%s\"\n",
driver, err, vec_num, cfg->name);
@@ -1259,8 +1319,12 @@ static int test_aead_vec_cfg(const char *driver, int enc,
aead_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr,
enc ? vec->plen : vec->clen, iv);
aead_request_set_ad(req, vec->alen);
- err = crypto_wait_req(enc ? crypto_aead_encrypt(req) :
- crypto_aead_decrypt(req), &wait);
+ if (cfg->nosimd)
+ crypto_disable_simd_for_test();
+ err = enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req);
+ if (cfg->nosimd)
+ crypto_reenable_simd_for_test();
+ err = crypto_wait_req(err, &wait);
if (err) {
if (err == -EBADMSG && vec->novrfy)
return 0;
@@ -1594,8 +1658,12 @@ static int test_skcipher_vec_cfg(const char *driver, int enc,
skcipher_request_set_callback(req, req_flags, crypto_req_done, &wait);
skcipher_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr,
vec->len, iv);
- err = crypto_wait_req(enc ? crypto_skcipher_encrypt(req) :
- crypto_skcipher_decrypt(req), &wait);
+ if (cfg->nosimd)
+ crypto_disable_simd_for_test();
+ err = enc ? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req);
+ if (cfg->nosimd)
+ crypto_reenable_simd_for_test();
+ err = crypto_wait_req(err, &wait);
if (err) {
pr_err("alg: skcipher: %s %s failed with err %d on test vector %u, cfg=\"%s\"\n",
driver, op, err, vec_num, cfg->name);
--
2.21.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code
2019-03-13 5:12 ` [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code Eric Biggers
@ 2019-03-13 10:35 ` Ard Biesheuvel
0 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:35 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> From: Eric Biggers <ebiggers@google.com>
>
> All crypto API algorithms are supposed to support the case where they
> are called in a context where SIMD instructions are unusable, e.g. IRQ
> context on some architectures. However, this isn't tested for by the
> self-tests, causing bugs to go undetected.
>
> Now that all algorithms have been converted to use crypto_simd_usable(),
> update the self-tests to test the no-SIMD case. First, a bool
> testvec_config::nosimd is added. When set, the crypto operation is
> executed with preemption disabled and with crypto_simd_usable() mocked
> out to return false on the current CPU.
>
> A bool test_sg_division::nosimd is also added. For hash algorithms it's
> honored by the corresponding ->update(). By setting just a subset of
> these bools, the case where some ->update()s are done in SIMD context
> and some are done in no-SIMD context is also tested.
>
> These bools are then randomly set by generate_random_testvec_config().
>
> For now, all no-SIMD testing is limited to the extra crypto self-tests,
> because it might be a bit too invasive for the regular self-tests.
> But this could be changed later.
>
> This has already found bugs in the arm64 AES-GCM and ChaCha algorithms.
> This would have found some past bugs as well.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> crypto/testmgr.c | 116 +++++++++++++++++++++++++++++++++++++----------
> 1 file changed, 92 insertions(+), 24 deletions(-)
>
> diff --git a/crypto/testmgr.c b/crypto/testmgr.c
> index 52417dde811f..2c2ddebb48d3 100644
> --- a/crypto/testmgr.c
> +++ b/crypto/testmgr.c
> @@ -234,12 +234,14 @@ enum finalization_type {
> * @offset
> * @flush_type: for hashes, whether an update() should be done now vs.
> * continuing to accumulate data
> + * @nosimd: if doing the pending update(), do it with SIMD disabled?
> */
> struct test_sg_division {
> unsigned int proportion_of_total;
> unsigned int offset;
> bool offset_relative_to_alignmask;
> enum flush_type flush_type;
> + bool nosimd;
> };
>
> /**
> @@ -259,6 +261,7 @@ struct test_sg_division {
> * @iv_offset_relative_to_alignmask: if true, add the algorithm's alignmask to
> * the @iv_offset
> * @finalization_type: what finalization function to use for hashes
> + * @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP.
> */
> struct testvec_config {
> const char *name;
> @@ -269,6 +272,7 @@ struct testvec_config {
> unsigned int iv_offset;
> bool iv_offset_relative_to_alignmask;
> enum finalization_type finalization_type;
> + bool nosimd;
> };
>
> #define TESTVEC_CONFIG_NAMELEN 192
> @@ -420,8 +424,11 @@ static unsigned int count_test_sg_divisions(const struct test_sg_division *divs)
> return ndivs;
> }
>
> +#define SGDIVS_HAVE_FLUSHES BIT(0)
> +#define SGDIVS_HAVE_NOSIMD BIT(1)
> +
> static bool valid_sg_divisions(const struct test_sg_division *divs,
> - unsigned int count, bool *any_flushes_ret)
> + unsigned int count, int *flags_ret)
> {
> unsigned int total = 0;
> unsigned int i;
> @@ -432,7 +439,9 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,
> return false;
> total += divs[i].proportion_of_total;
> if (divs[i].flush_type != FLUSH_TYPE_NONE)
> - *any_flushes_ret = true;
> + *flags_ret |= SGDIVS_HAVE_FLUSHES;
> + if (divs[i].nosimd)
> + *flags_ret |= SGDIVS_HAVE_NOSIMD;
> }
> return total == TEST_SG_TOTAL &&
> memchr_inv(&divs[i], 0, (count - i) * sizeof(divs[0])) == NULL;
> @@ -445,19 +454,18 @@ static bool valid_sg_divisions(const struct test_sg_division *divs,
> */
> static bool valid_testvec_config(const struct testvec_config *cfg)
> {
> - bool any_flushes = false;
> + int flags = 0;
>
> if (cfg->name == NULL)
> return false;
>
> if (!valid_sg_divisions(cfg->src_divs, ARRAY_SIZE(cfg->src_divs),
> - &any_flushes))
> + &flags))
> return false;
>
> if (cfg->dst_divs[0].proportion_of_total) {
> if (!valid_sg_divisions(cfg->dst_divs,
> - ARRAY_SIZE(cfg->dst_divs),
> - &any_flushes))
> + ARRAY_SIZE(cfg->dst_divs), &flags))
> return false;
> } else {
> if (memchr_inv(cfg->dst_divs, 0, sizeof(cfg->dst_divs)))
> @@ -470,7 +478,12 @@ static bool valid_testvec_config(const struct testvec_config *cfg)
> MAX_ALGAPI_ALIGNMASK + 1)
> return false;
>
> - if (any_flushes && cfg->finalization_type == FINALIZATION_TYPE_DIGEST)
> + if ((flags & (SGDIVS_HAVE_FLUSHES | SGDIVS_HAVE_NOSIMD)) &&
> + cfg->finalization_type == FINALIZATION_TYPE_DIGEST)
> + return false;
> +
> + if ((cfg->nosimd || (flags & SGDIVS_HAVE_NOSIMD)) &&
> + (cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP))
> return false;
>
> return true;
> @@ -731,13 +744,14 @@ static int build_cipher_test_sglists(struct cipher_test_sglists *tsgls,
> #ifdef CONFIG_CRYPTO_MANAGER_EXTRA_TESTS
> static char *generate_random_sgl_divisions(struct test_sg_division *divs,
> size_t max_divs, char *p, char *end,
> - bool gen_flushes)
> + bool gen_flushes, u32 req_flags)
> {
> struct test_sg_division *div = divs;
> unsigned int remaining = TEST_SG_TOTAL;
>
> do {
> unsigned int this_len;
> + const char *flushtype_str;
>
> if (div == &divs[max_divs - 1] || prandom_u32() % 2 == 0)
> this_len = remaining;
> @@ -766,11 +780,31 @@ static char *generate_random_sgl_divisions(struct test_sg_division *divs,
> }
> }
>
> + if (div->flush_type != FLUSH_TYPE_NONE &&
> + !(req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) &&
> + prandom_u32() % 2 == 0)
> + div->nosimd = true;
> +
> + switch (div->flush_type) {
> + case FLUSH_TYPE_FLUSH:
> + if (div->nosimd)
> + flushtype_str = "<flush,nosimd>";
> + else
> + flushtype_str = "<flush>";
> + break;
> + case FLUSH_TYPE_REIMPORT:
> + if (div->nosimd)
> + flushtype_str = "<reimport,nosimd>";
> + else
> + flushtype_str = "<reimport>";
> + break;
> + default:
> + flushtype_str = "";
> + break;
> + }
> +
> BUILD_BUG_ON(TEST_SG_TOTAL != 10000); /* for "%u.%u%%" */
> - p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s",
> - div->flush_type == FLUSH_TYPE_NONE ? "" :
> - div->flush_type == FLUSH_TYPE_FLUSH ?
> - "<flush> " : "<reimport> ",
> + p += scnprintf(p, end - p, "%s%u.%u%%@%s+%u%s", flushtype_str,
> this_len / 100, this_len % 100,
> div->offset_relative_to_alignmask ?
> "alignmask" : "",
> @@ -820,18 +854,26 @@ static void generate_random_testvec_config(struct testvec_config *cfg,
> break;
> }
>
> + if (!(cfg->req_flags & CRYPTO_TFM_REQ_MAY_SLEEP) &&
> + prandom_u32() % 2 == 0) {
> + cfg->nosimd = true;
> + p += scnprintf(p, end - p, " nosimd");
> + }
> +
> p += scnprintf(p, end - p, " src_divs=[");
> p = generate_random_sgl_divisions(cfg->src_divs,
> ARRAY_SIZE(cfg->src_divs), p, end,
> (cfg->finalization_type !=
> - FINALIZATION_TYPE_DIGEST));
> + FINALIZATION_TYPE_DIGEST),
> + cfg->req_flags);
> p += scnprintf(p, end - p, "]");
>
> if (!cfg->inplace && prandom_u32() % 2 == 0) {
> p += scnprintf(p, end - p, " dst_divs=[");
> p = generate_random_sgl_divisions(cfg->dst_divs,
> ARRAY_SIZE(cfg->dst_divs),
> - p, end, false);
> + p, end, false,
> + cfg->req_flags);
> p += scnprintf(p, end - p, "]");
> }
>
> @@ -864,6 +906,23 @@ static void crypto_reenable_simd_for_test(void)
> }
> #endif /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */
>
> +static int do_ahash_op(int (*op)(struct ahash_request *req),
> + struct ahash_request *req,
> + struct crypto_wait *wait, bool nosimd)
> +{
> + int err;
> +
> + if (nosimd)
> + crypto_disable_simd_for_test();
> +
> + err = op(req);
> +
> + if (nosimd)
> + crypto_reenable_simd_for_test();
> +
> + return crypto_wait_req(err, wait);
> +}
> +
> static int check_nonfinal_hash_op(const char *op, int err,
> u8 *result, unsigned int digestsize,
> const char *driver, unsigned int vec_num,
> @@ -938,7 +997,7 @@ static int test_hash_vec_cfg(const char *driver,
> ahash_request_set_callback(req, req_flags, crypto_req_done,
> &wait);
> ahash_request_set_crypt(req, tsgl->sgl, result, vec->psize);
> - err = crypto_wait_req(crypto_ahash_digest(req), &wait);
> + err = do_ahash_op(crypto_ahash_digest, req, &wait, cfg->nosimd);
> if (err) {
> pr_err("alg: hash: %s digest() failed with err %d on test vector %u, cfg=\"%s\"\n",
> driver, err, vec_num, cfg->name);
> @@ -951,7 +1010,7 @@ static int test_hash_vec_cfg(const char *driver,
>
> ahash_request_set_callback(req, req_flags, crypto_req_done, &wait);
> ahash_request_set_crypt(req, NULL, result, 0);
> - err = crypto_wait_req(crypto_ahash_init(req), &wait);
> + err = do_ahash_op(crypto_ahash_init, req, &wait, cfg->nosimd);
> err = check_nonfinal_hash_op("init", err, result, digestsize,
> driver, vec_num, cfg);
> if (err)
> @@ -967,7 +1026,8 @@ static int test_hash_vec_cfg(const char *driver,
> crypto_req_done, &wait);
> ahash_request_set_crypt(req, pending_sgl, result,
> pending_len);
> - err = crypto_wait_req(crypto_ahash_update(req), &wait);
> + err = do_ahash_op(crypto_ahash_update, req, &wait,
> + divs[i]->nosimd);
> err = check_nonfinal_hash_op("update", err,
> result, digestsize,
> driver, vec_num, cfg);
> @@ -1010,12 +1070,12 @@ static int test_hash_vec_cfg(const char *driver,
> ahash_request_set_crypt(req, pending_sgl, result, pending_len);
> if (cfg->finalization_type == FINALIZATION_TYPE_FINAL) {
> /* finish with update() and final() */
> - err = crypto_wait_req(crypto_ahash_update(req), &wait);
> + err = do_ahash_op(crypto_ahash_update, req, &wait, cfg->nosimd);
> err = check_nonfinal_hash_op("update", err, result, digestsize,
> driver, vec_num, cfg);
> if (err)
> return err;
> - err = crypto_wait_req(crypto_ahash_final(req), &wait);
> + err = do_ahash_op(crypto_ahash_final, req, &wait, cfg->nosimd);
> if (err) {
> pr_err("alg: hash: %s final() failed with err %d on test vector %u, cfg=\"%s\"\n",
> driver, err, vec_num, cfg->name);
> @@ -1023,7 +1083,7 @@ static int test_hash_vec_cfg(const char *driver,
> }
> } else {
> /* finish with finup() */
> - err = crypto_wait_req(crypto_ahash_finup(req), &wait);
> + err = do_ahash_op(crypto_ahash_finup, req, &wait, cfg->nosimd);
> if (err) {
> pr_err("alg: hash: %s finup() failed with err %d on test vector %u, cfg=\"%s\"\n",
> driver, err, vec_num, cfg->name);
> @@ -1259,8 +1319,12 @@ static int test_aead_vec_cfg(const char *driver, int enc,
> aead_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr,
> enc ? vec->plen : vec->clen, iv);
> aead_request_set_ad(req, vec->alen);
> - err = crypto_wait_req(enc ? crypto_aead_encrypt(req) :
> - crypto_aead_decrypt(req), &wait);
> + if (cfg->nosimd)
> + crypto_disable_simd_for_test();
> + err = enc ? crypto_aead_encrypt(req) : crypto_aead_decrypt(req);
> + if (cfg->nosimd)
> + crypto_reenable_simd_for_test();
> + err = crypto_wait_req(err, &wait);
> if (err) {
> if (err == -EBADMSG && vec->novrfy)
> return 0;
> @@ -1594,8 +1658,12 @@ static int test_skcipher_vec_cfg(const char *driver, int enc,
> skcipher_request_set_callback(req, req_flags, crypto_req_done, &wait);
> skcipher_request_set_crypt(req, tsgls->src.sgl_ptr, tsgls->dst.sgl_ptr,
> vec->len, iv);
> - err = crypto_wait_req(enc ? crypto_skcipher_encrypt(req) :
> - crypto_skcipher_decrypt(req), &wait);
> + if (cfg->nosimd)
> + crypto_disable_simd_for_test();
> + err = enc ? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req);
> + if (cfg->nosimd)
> + crypto_reenable_simd_for_test();
> + err = crypto_wait_req(err, &wait);
> if (err) {
> pr_err("alg: skcipher: %s %s failed with err %d on test vector %u, cfg=\"%s\"\n",
> driver, op, err, vec_num, cfg->name);
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/8] crypto: test the !may_use_simd() fallback code
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (7 preceding siblings ...)
2019-03-13 5:12 ` [PATCH 8/8] crypto: testmgr - test the !may_use_simd() fallback code Eric Biggers
@ 2019-03-13 10:50 ` Ard Biesheuvel
2019-03-22 13:03 ` Herbert Xu
9 siblings, 0 replies; 19+ messages in thread
From: Ard Biesheuvel @ 2019-03-13 10:50 UTC (permalink / raw)
To: Eric Biggers
Cc: open list:HARDWARE RANDOM NUMBER GENERATOR CORE, Herbert Xu,
linux-arm-kernel, the arch/x86 maintainers
On Wed, 13 Mar 2019 at 06:15, Eric Biggers <ebiggers@kernel.org> wrote:
>
> All crypto API algorithms are supposed to support the case where they
> are called in a context where SIMD instructions are unusable, e.g. IRQ
> context on some architectures. However, this isn't tested for by the
> self-tests, causing bugs to go undetected.
>
> This patch series therefore updates the self-tests to test the no-SIMD
> code. It works by converting all may_use_simd() checks to a new macro
> crypto_simd_usable(), which also returns false when the self-tests have
> disabled SIMD in crypto algorithms on the current CPU for test purposes.
>
> For now, all no-SIMD testing is limited to the extra crypto self-tests,
> because it might be a bit too invasive for the regular self-tests.
> But this could be changed later.
>
> This already found bugs in the arm64 implementations of AES-GCM and
> ChaCha. These are fixed by the first two patches. Following this, the
> tests pass on x86, arm, and arm64.
>
Thanks a lot for doing this Eric.
I pushed your branch to kernelci again, let's see if anything else
falls out (although not as likely, given that this shouldn't affect
h/w accelerators)
> This patch series is based on top of my other pending patch series
> "crypto: add SIMD helpers for AEADs". It can also be found in git at:
>
> URL: https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
> Branch: crypto-nosimd-tests
>
> Eric Biggers (8):
> crypto: chacha-generic - fix use as arm64 no-NEON fallback
> crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
> crypto: simd,testmgr - introduce crypto_simd_usable()
> crypto: x86 - convert to use crypto_simd_usable()
> crypto: arm - convert to use crypto_simd_usable()
> crypto: arm64 - convert to use crypto_simd_usable()
> crypto: simd - convert to use crypto_simd_usable()
> crypto: testmgr - test the !may_use_simd() fallback code
>
> arch/arm/crypto/chacha-neon-glue.c | 5 +-
> arch/arm/crypto/crc32-ce-glue.c | 5 +-
> arch/arm/crypto/crct10dif-ce-glue.c | 3 +-
> arch/arm/crypto/ghash-ce-glue.c | 7 +-
> arch/arm/crypto/nhpoly1305-neon-glue.c | 3 +-
> arch/arm/crypto/sha1-ce-glue.c | 5 +-
> arch/arm/crypto/sha1_neon_glue.c | 5 +-
> arch/arm/crypto/sha2-ce-glue.c | 5 +-
> arch/arm/crypto/sha256_neon_glue.c | 5 +-
> arch/arm/crypto/sha512-neon-glue.c | 5 +-
> arch/arm64/crypto/aes-ce-ccm-glue.c | 7 +-
> arch/arm64/crypto/aes-ce-glue.c | 5 +-
> arch/arm64/crypto/aes-glue.c | 4 +-
> arch/arm64/crypto/aes-neonbs-glue.c | 2 +-
> arch/arm64/crypto/chacha-neon-glue.c | 5 +-
> arch/arm64/crypto/crct10dif-ce-glue.c | 5 +-
> arch/arm64/crypto/ghash-ce-glue.c | 17 ++-
> arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 +-
> arch/arm64/crypto/sha1-ce-glue.c | 7 +-
> arch/arm64/crypto/sha2-ce-glue.c | 7 +-
> arch/arm64/crypto/sha256-glue.c | 5 +-
> arch/arm64/crypto/sha3-ce-glue.c | 5 +-
> arch/arm64/crypto/sha512-ce-glue.c | 7 +-
> arch/arm64/crypto/sm3-ce-glue.c | 7 +-
> arch/arm64/crypto/sm4-ce-glue.c | 5 +-
> arch/x86/crypto/aesni-intel_glue.c | 8 +-
> arch/x86/crypto/chacha_glue.c | 6 +-
> arch/x86/crypto/crc32-pclmul_glue.c | 5 +-
> arch/x86/crypto/crc32c-intel_glue.c | 7 +-
> arch/x86/crypto/crct10dif-pclmul_glue.c | 7 +-
> arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +-
> arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +-
> arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +-
> arch/x86/crypto/poly1305_glue.c | 4 +-
> arch/x86/crypto/sha1_ssse3_glue.c | 7 +-
> arch/x86/crypto/sha256_ssse3_glue.c | 7 +-
> arch/x86/crypto/sha512_ssse3_glue.c | 10 +-
> crypto/chacha_generic.c | 2 +-
> crypto/simd.c | 8 +-
> crypto/testmgr.c | 142 +++++++++++++++++----
> include/crypto/internal/simd.h | 24 ++++
> 41 files changed, 272 insertions(+), 123 deletions(-)
>
> --
> 2.21.0
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 0/8] crypto: test the !may_use_simd() fallback code
2019-03-13 5:12 [PATCH 0/8] crypto: test the !may_use_simd() fallback code Eric Biggers
` (8 preceding siblings ...)
2019-03-13 10:50 ` [PATCH 0/8] crypto: " Ard Biesheuvel
@ 2019-03-22 13:03 ` Herbert Xu
9 siblings, 0 replies; 19+ messages in thread
From: Herbert Xu @ 2019-03-22 13:03 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, Ard Biesheuvel, linux-arm-kernel, x86
On Tue, Mar 12, 2019 at 10:12:44PM -0700, Eric Biggers wrote:
> All crypto API algorithms are supposed to support the case where they
> are called in a context where SIMD instructions are unusable, e.g. IRQ
> context on some architectures. However, this isn't tested for by the
> self-tests, causing bugs to go undetected.
>
> This patch series therefore updates the self-tests to test the no-SIMD
> code. It works by converting all may_use_simd() checks to a new macro
> crypto_simd_usable(), which also returns false when the self-tests have
> disabled SIMD in crypto algorithms on the current CPU for test purposes.
>
> For now, all no-SIMD testing is limited to the extra crypto self-tests,
> because it might be a bit too invasive for the regular self-tests.
> But this could be changed later.
>
> This already found bugs in the arm64 implementations of AES-GCM and
> ChaCha. These are fixed by the first two patches. Following this, the
> tests pass on x86, arm, and arm64.
>
> This patch series is based on top of my other pending patch series
> "crypto: add SIMD helpers for AEADs". It can also be found in git at:
>
> URL: https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
> Branch: crypto-nosimd-tests
>
> Eric Biggers (8):
> crypto: chacha-generic - fix use as arm64 no-NEON fallback
> crypto: arm64/gcm-aes-ce - fix no-NEON fallback code
> crypto: simd,testmgr - introduce crypto_simd_usable()
> crypto: x86 - convert to use crypto_simd_usable()
> crypto: arm - convert to use crypto_simd_usable()
> crypto: arm64 - convert to use crypto_simd_usable()
> crypto: simd - convert to use crypto_simd_usable()
> crypto: testmgr - test the !may_use_simd() fallback code
>
> arch/arm/crypto/chacha-neon-glue.c | 5 +-
> arch/arm/crypto/crc32-ce-glue.c | 5 +-
> arch/arm/crypto/crct10dif-ce-glue.c | 3 +-
> arch/arm/crypto/ghash-ce-glue.c | 7 +-
> arch/arm/crypto/nhpoly1305-neon-glue.c | 3 +-
> arch/arm/crypto/sha1-ce-glue.c | 5 +-
> arch/arm/crypto/sha1_neon_glue.c | 5 +-
> arch/arm/crypto/sha2-ce-glue.c | 5 +-
> arch/arm/crypto/sha256_neon_glue.c | 5 +-
> arch/arm/crypto/sha512-neon-glue.c | 5 +-
> arch/arm64/crypto/aes-ce-ccm-glue.c | 7 +-
> arch/arm64/crypto/aes-ce-glue.c | 5 +-
> arch/arm64/crypto/aes-glue.c | 4 +-
> arch/arm64/crypto/aes-neonbs-glue.c | 2 +-
> arch/arm64/crypto/chacha-neon-glue.c | 5 +-
> arch/arm64/crypto/crct10dif-ce-glue.c | 5 +-
> arch/arm64/crypto/ghash-ce-glue.c | 17 ++-
> arch/arm64/crypto/nhpoly1305-neon-glue.c | 3 +-
> arch/arm64/crypto/sha1-ce-glue.c | 7 +-
> arch/arm64/crypto/sha2-ce-glue.c | 7 +-
> arch/arm64/crypto/sha256-glue.c | 5 +-
> arch/arm64/crypto/sha3-ce-glue.c | 5 +-
> arch/arm64/crypto/sha512-ce-glue.c | 7 +-
> arch/arm64/crypto/sm3-ce-glue.c | 7 +-
> arch/arm64/crypto/sm4-ce-glue.c | 5 +-
> arch/x86/crypto/aesni-intel_glue.c | 8 +-
> arch/x86/crypto/chacha_glue.c | 6 +-
> arch/x86/crypto/crc32-pclmul_glue.c | 5 +-
> arch/x86/crypto/crc32c-intel_glue.c | 7 +-
> arch/x86/crypto/crct10dif-pclmul_glue.c | 7 +-
> arch/x86/crypto/ghash-clmulni-intel_glue.c | 9 +-
> arch/x86/crypto/nhpoly1305-avx2-glue.c | 5 +-
> arch/x86/crypto/nhpoly1305-sse2-glue.c | 5 +-
> arch/x86/crypto/poly1305_glue.c | 4 +-
> arch/x86/crypto/sha1_ssse3_glue.c | 7 +-
> arch/x86/crypto/sha256_ssse3_glue.c | 7 +-
> arch/x86/crypto/sha512_ssse3_glue.c | 10 +-
> crypto/chacha_generic.c | 2 +-
> crypto/simd.c | 8 +-
> crypto/testmgr.c | 142 +++++++++++++++++----
> include/crypto/internal/simd.h | 24 ++++
> 41 files changed, 272 insertions(+), 123 deletions(-)
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 19+ messages in thread