From: Ard Biesheuvel <ard.biesheuvel@linaro.org>
To: linux-crypto@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org,
Ard Biesheuvel <ard.biesheuvel@linaro.org>,
Herbert Xu <herbert@gondor.apana.org.au>,
David Miller <davem@davemloft.net>,
Greg KH <gregkh@linuxfoundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Jason A . Donenfeld" <Jason@zx2c4.com>,
Samuel Neves <sneves@dei.uc.pt>,
Dan Carpenter <dan.carpenter@oracle.com>,
Arnd Bergmann <arnd@arndb.de>, Eric Biggers <ebiggers@google.com>,
Andy Lutomirski <luto@kernel.org>, Will Deacon <will@kernel.org>,
Marc Zyngier <maz@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Subject: [RFC PATCH 02/18] crypto: x86/poly1305 - implement .update_from_sg method
Date: Wed, 25 Sep 2019 18:12:39 +0200 [thread overview]
Message-ID: <20190925161255.1871-3-ard.biesheuvel@linaro.org> (raw)
In-Reply-To: <20190925161255.1871-1-ard.biesheuvel@linaro.org>
In order to reduce the number of invocations of the RFC7539 template
into the Poly1305 driver, implement the new internal .update_from_sg
method that allows the driver to amortize the cost of FPU preserve/
restore sequences over a larger chunk of input.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/x86/crypto/poly1305_glue.c | 54 ++++++++++++++++----
1 file changed, 43 insertions(+), 11 deletions(-)
diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c
index 4a1c05dce950..f2afaa8e23c2 100644
--- a/arch/x86/crypto/poly1305_glue.c
+++ b/arch/x86/crypto/poly1305_glue.c
@@ -115,18 +115,11 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx,
return srclen;
}
-static int poly1305_simd_update(struct shash_desc *desc,
- const u8 *src, unsigned int srclen)
+static void poly1305_simd_do_update(struct shash_desc *desc,
+ const u8 *src, unsigned int srclen)
{
- struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc);
unsigned int bytes;
- /* kernel_fpu_begin/end is costly, use fallback for small updates */
- if (srclen <= 288 || !crypto_simd_usable())
- return crypto_poly1305_update(desc, src, srclen);
-
- kernel_fpu_begin();
-
if (unlikely(dctx->buflen)) {
bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen);
memcpy(dctx->buf + dctx->buflen, src, bytes);
@@ -147,12 +140,50 @@ static int poly1305_simd_update(struct shash_desc *desc,
srclen = bytes;
}
- kernel_fpu_end();
-
if (unlikely(srclen)) {
dctx->buflen = srclen;
memcpy(dctx->buf, src, srclen);
}
+}
+
+static int poly1305_simd_update(struct shash_desc *desc,
+ const u8 *src, unsigned int srclen)
+{
+ /* kernel_fpu_begin/end is costly, use fallback for small updates */
+ if (srclen <= 288 || !crypto_simd_usable())
+ return crypto_poly1305_update(desc, src, srclen);
+
+ kernel_fpu_begin();
+ poly1305_simd_do_update(desc, src, srclen);
+ kernel_fpu_end();
+
+ return 0;
+}
+
+static int poly1305_simd_update_from_sg(struct shash_desc *desc,
+ struct scatterlist *sg,
+ unsigned int srclen,
+ int flags)
+{
+ bool do_simd = crypto_simd_usable() && srclen > 288;
+ struct crypto_hash_walk walk;
+ int nbytes;
+
+ if (do_simd) {
+ kernel_fpu_begin();
+ flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP;
+ }
+
+ for (nbytes = crypto_shash_walk_sg(desc, sg, srclen, &walk, flags);
+ nbytes > 0;
+ nbytes = crypto_hash_walk_done(&walk, 0)) {
+ if (do_simd)
+ poly1305_simd_do_update(desc, walk.data, nbytes);
+ else
+ crypto_poly1305_update(desc, walk.data, nbytes);
+ }
+ if (do_simd)
+ kernel_fpu_end();
return 0;
}
@@ -161,6 +192,7 @@ static struct shash_alg alg = {
.digestsize = POLY1305_DIGEST_SIZE,
.init = poly1305_simd_init,
.update = poly1305_simd_update,
+ .update_from_sg = poly1305_simd_update_from_sg,
.final = crypto_poly1305_final,
.descsize = sizeof(struct poly1305_simd_desc_ctx),
.base = {
--
2.20.1
next prev parent reply other threads:[~2019-09-25 16:13 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-25 16:12 [RFC PATCH 00/18] crypto: wireguard using the existing crypto API Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 01/18] crypto: shash - add plumbing for operating on scatterlists Ard Biesheuvel
2019-09-25 16:12 ` Ard Biesheuvel [this message]
2019-09-25 16:12 ` [RFC PATCH 03/18] crypto: arm/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 04/18] crypto: arm64/poly1305 " Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 05/18] crypto: chacha - move existing library code into lib/crypto Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 06/18] crypto: rfc7539 - switch to shash for Poly1305 Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 07/18] crypto: rfc7539 - use zero reqsize for sync instantiations without alignmask Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 08/18] crypto: testmgr - add a chacha20poly1305 test case Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 09/18] crypto: poly1305 - move core algorithm into lib/crypto Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 10/18] crypto: poly1305 - add init/update/final library routines Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 11/18] int128: move __uint128_t compiler test to Kconfig Ard Biesheuvel
2019-09-25 21:01 ` Linus Torvalds
2019-09-25 21:19 ` Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 12/18] crypto: BLAKE2s - generic C library implementation and selftest Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 13/18] crypto: Curve25519 - generic C library implementations " Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 14/18] crypto: chacha20poly1305 - import construction and selftest from Zinc Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 15/18] net: WireGuard secure network tunnel Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 16/18] netlink: use new strict length types in policy for 5.2 Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 17/18] wg switch to lib/crypto algos Ard Biesheuvel
2019-09-25 16:12 ` [RFC PATCH 18/18] net: wireguard - switch to crypto API for packet encryption Ard Biesheuvel
2019-09-25 22:15 ` Linus Torvalds
2019-09-25 22:22 ` Linus Torvalds
2019-09-26 9:40 ` Pascal Van Leeuwen
2019-09-26 16:35 ` Linus Torvalds
2019-09-27 0:15 ` Pascal Van Leeuwen
2019-09-27 1:30 ` Linus Torvalds
2019-09-27 2:54 ` Linus Torvalds
2019-09-27 3:53 ` Herbert Xu
2019-09-27 4:37 ` Andy Lutomirski
2019-09-27 4:59 ` Herbert Xu
2019-09-27 4:01 ` Herbert Xu
2019-09-27 4:13 ` Linus Torvalds
2019-09-27 10:44 ` Pascal Van Leeuwen
2019-09-27 11:08 ` Pascal Van Leeuwen
2019-09-27 4:36 ` Andy Lutomirski
2019-09-27 9:58 ` Pascal Van Leeuwen
2019-09-27 10:11 ` Herbert Xu
2019-09-27 16:23 ` Linus Torvalds
2019-09-30 11:14 ` France didn't want GSM encryption Marc Gonzalez
2019-09-30 21:37 ` Linus Torvalds
2019-09-30 20:44 ` [RFC PATCH 18/18] net: wireguard - switch to crypto API for packet encryption Pascal Van Leeuwen
2019-09-27 2:06 ` Linus Torvalds
2019-09-27 10:11 ` Pascal Van Leeuwen
2019-09-26 11:06 ` Ard Biesheuvel
2019-09-26 12:34 ` Ard Biesheuvel
2019-09-26 8:59 ` [RFC PATCH 00/18] crypto: wireguard using the existing crypto API Jason A. Donenfeld
2019-09-26 10:19 ` Pascal Van Leeuwen
2019-09-26 10:59 ` Jason A. Donenfeld
2019-09-26 11:06 ` chapoly acceleration hardware [Was: Re: [RFC PATCH 00/18] crypto: wireguard using the existing crypto API] Jason A. Donenfeld
2019-09-26 11:38 ` Toke Høiland-Jørgensen
2019-09-26 13:52 ` Pascal Van Leeuwen
2019-09-26 23:13 ` Dave Taht
2019-09-27 12:18 ` Pascal Van Leeuwen
2019-09-26 22:47 ` Jakub Kicinski
2019-09-26 12:07 ` [RFC PATCH 00/18] crypto: wireguard using the existing crypto API Ard Biesheuvel
2019-09-26 13:06 ` Pascal Van Leeuwen
2019-09-26 13:15 ` Ard Biesheuvel
2019-09-26 14:03 ` Pascal Van Leeuwen
2019-09-26 14:52 ` Ard Biesheuvel
2019-09-26 15:04 ` Pascal Van Leeuwen
2019-09-26 20:47 ` Jason A. Donenfeld
2019-09-26 21:22 ` Andrew Lunn
2019-09-26 21:36 ` Andy Lutomirski
2019-09-27 7:20 ` Jason A. Donenfeld
2019-10-01 8:56 ` Ard Biesheuvel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190925161255.1871-3-ard.biesheuvel@linaro.org \
--to=ard.biesheuvel@linaro.org \
--cc=Jason@zx2c4.com \
--cc=arnd@arndb.de \
--cc=catalin.marinas@arm.com \
--cc=dan.carpenter@oracle.com \
--cc=davem@davemloft.net \
--cc=ebiggers@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=herbert@gondor.apana.org.au \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-crypto@vger.kernel.org \
--cc=luto@kernel.org \
--cc=maz@kernel.org \
--cc=sneves@dei.uc.pt \
--cc=torvalds@linux-foundation.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).