* [PATCH v2 1/3] crypto: qce - use cryptlen when adding extra sgl
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
@ 2020-02-06 1:20 ` Eneas U de Queiroz
2020-02-06 1:20 ` [PATCH v2 2/3] crypto: qce - use AES fallback for small requests Eneas U de Queiroz
` (4 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 1:20 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
The qce crypto driver appends an extra entry to the dst sgl, to maintain
private state information.
When the gcm driver sends requests to the ctr skcipher, it passes the
authentication tag after the actual crypto payload, but it must not be
touched.
Commit 1336c2221bee ("crypto: qce - save a sg table slot for result
buf") limited the destination sgl to avoid overwriting the
authentication tag but it assumed the tag would be in a separate sgl
entry.
This is not always the case, so it is better to limit the length of the
destination buffer to req->cryptlen before appending the result buf.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/qce/dma.c b/drivers/crypto/qce/dma.c
index 7da893dc00e7..46db5bf366b4 100644
--- a/drivers/crypto/qce/dma.c
+++ b/drivers/crypto/qce/dma.c
@@ -48,9 +48,10 @@ void qce_dma_release(struct qce_dma_data *dma)
struct scatterlist *
qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,
- int max_ents)
+ unsigned int max_len)
{
struct scatterlist *sg = sgt->sgl, *sg_last = NULL;
+ unsigned int new_len;
while (sg) {
if (!sg_page(sg))
@@ -61,13 +62,13 @@ qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,
if (!sg)
return ERR_PTR(-EINVAL);
- while (new_sgl && sg && max_ents) {
- sg_set_page(sg, sg_page(new_sgl), new_sgl->length,
- new_sgl->offset);
+ while (new_sgl && sg && max_len) {
+ new_len = new_sgl->length > max_len ? max_len : new_sgl->length;
+ sg_set_page(sg, sg_page(new_sgl), new_len, new_sgl->offset);
sg_last = sg;
sg = sg_next(sg);
new_sgl = sg_next(new_sgl);
- max_ents--;
+ max_len -= new_len;
}
return sg_last;
diff --git a/drivers/crypto/qce/dma.h b/drivers/crypto/qce/dma.h
index ed25a0d9829e..786402169360 100644
--- a/drivers/crypto/qce/dma.h
+++ b/drivers/crypto/qce/dma.h
@@ -43,6 +43,6 @@ void qce_dma_issue_pending(struct qce_dma_data *dma);
int qce_dma_terminate_all(struct qce_dma_data *dma);
struct scatterlist *
qce_sgtable_add(struct sg_table *sgt, struct scatterlist *sg_add,
- int max_ents);
+ unsigned int max_len);
#endif /* _DMA_H_ */
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index 4217b745f124..63ae75809cb7 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -97,13 +97,14 @@ qce_skcipher_async_req_handle(struct crypto_async_request *async_req)
sg_init_one(&rctx->result_sg, qce->dma.result_buf, QCE_RESULT_BUF_SZ);
- sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, rctx->dst_nents - 1);
+ sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, req->cryptlen);
if (IS_ERR(sg)) {
ret = PTR_ERR(sg);
goto error_free;
}
- sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->result_sg, 1);
+ sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->result_sg,
+ QCE_RESULT_BUF_SZ);
if (IS_ERR(sg)) {
ret = PTR_ERR(sg);
goto error_free;
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v2 2/3] crypto: qce - use AES fallback for small requests
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
2020-02-06 1:20 ` [PATCH v2 1/3] crypto: qce - use cryptlen when adding extra sgl Eneas U de Queiroz
@ 2020-02-06 1:20 ` Eneas U de Queiroz
2020-02-06 1:20 ` [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails Eneas U de Queiroz
` (3 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 1:20 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
Process small blocks using the fallback cipher, as a workaround for an
observed failure (DMA-related, apparently) when computing the GCM ghash
key. This brings a speed gain as well, since it avoids the latency of
using the hardware engine to process small blocks.
Using software for all 16-byte requests would be enough to make GCM
work, but to increase performance, a larger threshold would be better.
Measuring the performance of supported ciphers with openssl speed,
software matches hardware at around 768-1024 bytes.
Considering the 256-bit ciphers, software is 2-3 times faster than qce
at 256-bytes, 30% faster at 512, and about even at 768-bytes. With
128-bit keys, the break-even point would be around 1024-bytes.
This adds the 'aes_sw_max_len' parameter, to set the largest request
length processed by the software fallback. Its default is being set to
512 bytes, a little lower than the break-even point, to balance the cost
in CPU usage.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
crypto: qce - add aes_sw_max_len parameter
This adds the AES fallback threshold as a parameter, so it can be
changed by the user.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index c2767ed54dfe..052d3ff7fb20 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -685,6 +685,29 @@ choice
endchoice
+config CRYPTO_DEV_QCE_SW_MAX_LEN
+ int "Default maximum request size to use software for AES"
+ depends on CRYPTO_DEV_QCE && CRYPTO_DEV_QCE_SKCIPHER
+ default 512
+ help
+ This sets the default maximum request size to perform AES requests
+ using software instead of the crypto engine. It can be changed by
+ setting the aes_sw_max_len parameter.
+
+ Small blocks are processed faster in software than hardware.
+ Considering the 256-bit ciphers, software is 2-3 times faster than
+ qce at 256-bytes, 30% faster at 512, and about even at 768-bytes.
+ With 128-bit keys, the break-even point would be around 1024-bytes.
+
+ The default is set a little lower, to 512 bytes, to balance the
+ cost in CPU usage. The minimum recommended setting is 16-bytes
+ (1 AES block), since AES-GCM will fail if you set it lower.
+ Setting this to zero will send all requests to the hardware.
+
+ Note that 192-bit keys are not supported by the hardware and are
+ always processed by the software fallback, and all DES requests
+ are done by the hardware.
+
config CRYPTO_DEV_QCOM_RNG
tristate "Qualcomm Random Number Generator Driver"
depends on ARCH_QCOM || COMPILE_TEST
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index 63ae75809cb7..a3536495b6b0 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -5,6 +5,7 @@
#include <linux/device.h>
#include <linux/interrupt.h>
+#include <linux/moduleparam.h>
#include <linux/types.h>
#include <crypto/aes.h>
#include <crypto/internal/des.h>
@@ -12,6 +13,13 @@
#include "cipher.h"
+unsigned int aes_sw_max_len = CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN;
+module_param(aes_sw_max_len, uint, 0644);
+MODULE_PARM_DESC(aes_sw_max_len,
+ "Only use hardware for AES requests larger than this "
+ "[0=always use hardware; anything <16 breaks AES-GCM; default="
+ __stringify(CONFIG_CRYPTO_DEV_QCE_SOFT_THRESHOLD)"]");
+
static LIST_HEAD(skcipher_algs);
static void qce_skcipher_done(void *data)
@@ -166,15 +174,10 @@ static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,
switch (IS_XTS(flags) ? keylen >> 1 : keylen) {
case AES_KEYSIZE_128:
case AES_KEYSIZE_256:
+ memcpy(ctx->enc_key, key, keylen);
break;
- default:
- goto fallback;
}
- ctx->enc_keylen = keylen;
- memcpy(ctx->enc_key, key, keylen);
- return 0;
-fallback:
ret = crypto_sync_skcipher_setkey(ctx->fallback, key, keylen);
if (!ret)
ctx->enc_keylen = keylen;
@@ -224,8 +227,9 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
- if (IS_AES(rctx->flags) && keylen != AES_KEYSIZE_128 &&
- keylen != AES_KEYSIZE_256) {
+ if (IS_AES(rctx->flags) &&
+ ((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
+ || req->cryptlen <= aes_sw_max_len)) {
SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
skcipher_request_set_sync_tfm(subreq, ctx->fallback);
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
2020-02-06 1:20 ` [PATCH v2 1/3] crypto: qce - use cryptlen when adding extra sgl Eneas U de Queiroz
2020-02-06 1:20 ` [PATCH v2 2/3] crypto: qce - use AES fallback for small requests Eneas U de Queiroz
@ 2020-02-06 1:20 ` Eneas U de Queiroz
2020-02-06 23:31 ` kbuild test robot
2020-02-06 11:39 ` [PATCH v3 1/3] crypto: qce - use cryptlen when adding extra sgl Eneas U de Queiroz
` (2 subsequent siblings)
5 siblings, 1 reply; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 1:20 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
QCE hangs when presented with an AES-XTS request whose length is larger
than QCE_SECTOR_SIZE (512-bytes), and is not a multiple of it. Let the
fallback cipher handle them.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/qce/common.c b/drivers/crypto/qce/common.c
index 629e7f34dc09..5006e74c40cd 100644
--- a/drivers/crypto/qce/common.c
+++ b/drivers/crypto/qce/common.c
@@ -15,8 +15,6 @@
#include "regs-v5.h"
#include "sha.h"
-#define QCE_SECTOR_SIZE 512
-
static inline u32 qce_read(struct qce_device *qce, u32 offset)
{
return readl(qce->base + offset);
diff --git a/drivers/crypto/qce/common.h b/drivers/crypto/qce/common.h
index 282d4317470d..9f989cba0f1b 100644
--- a/drivers/crypto/qce/common.h
+++ b/drivers/crypto/qce/common.h
@@ -12,6 +12,9 @@
#include <crypto/hash.h>
#include <crypto/internal/skcipher.h>
+/* xts du size */
+#define QCE_SECTOR_SIZE 512
+
/* key size in bytes */
#define QCE_SHA_HMAC_KEY_SIZE 64
#define QCE_MAX_CIPHER_KEY_SIZE AES_KEYSIZE_256
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index a3536495b6b0..b7c0aaddd7d9 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -227,9 +227,13 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
+ /* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and
+ * is not a multiple of it; pass such requests to the fallback */
if (IS_AES(rctx->flags) &&
((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
- || req->cryptlen <= aes_sw_max_len)) {
+ || req->cryptlen <= aes_sw_max_len)
+ || (IS_XTS(rctx->flags) && req->cryptlen > QCE_SECTOR_SIZE &&
+ req->cryptlen % QCE_SECTOR_SIZE)) {
SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
skcipher_request_set_sync_tfm(subreq, ctx->fallback);
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails
2020-02-06 1:20 ` [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails Eneas U de Queiroz
@ 2020-02-06 23:31 ` kbuild test robot
0 siblings, 0 replies; 11+ messages in thread
From: kbuild test robot @ 2020-02-06 23:31 UTC (permalink / raw)
To: Eneas U de Queiroz
Cc: kbuild-all, linux-crypto, Herbert Xu, David S. Miller,
Ard Biesheuvel, Eneas U de Queiroz
[-- Attachment #1: Type: text/plain, Size: 6226 bytes --]
Hi Eneas,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on cryptodev/master]
[also build test WARNING on crypto/master next-20200206]
[cannot apply to sparc-next/master v5.5]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Eneas-U-de-Queiroz/crypto-qce-driver-fixes-for-gcm/20200207-051805
base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master
config: m68k-allmodconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 7.5.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=m68k
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
drivers/crypto/qce/skcipher.c: In function 'qce_skcipher_crypt':
>> drivers/crypto/qce/skcipher.c:232:26: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
if (IS_AES(rctx->flags) &&
vim +232 drivers/crypto/qce/skcipher.c
5feaaae1b549f3 drivers/crypto/qce/ablkcipher.c Herbert Xu 2019-04-11 216
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 217 static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 218 {
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 219 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 220 struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(tfm);
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 221 struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 222 struct qce_alg_template *tmpl = to_cipher_tmpl(tfm);
7de4c2bd196f11 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2019-12-20 223 int keylen;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 224 int ret;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 225
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 226 rctx->flags = tmpl->alg_flags;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 227 rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
7de4c2bd196f11 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2019-12-20 228 keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 229
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 230 /* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 231 * is not a multiple of it; pass such requests to the fallback */
c23a1c2b41c486 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 @232 if (IS_AES(rctx->flags) &&
c23a1c2b41c486 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 233 ((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 234 || req->cryptlen <= aes_sw_max_len)
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 235 || (IS_XTS(rctx->flags) && req->cryptlen > QCE_SECTOR_SIZE &&
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 236 req->cryptlen % QCE_SECTOR_SIZE)) {
d1e4ba83b0286b drivers/crypto/qce/ablkcipher.c Kees Cook 2018-09-18 237 SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 238
d1e4ba83b0286b drivers/crypto/qce/ablkcipher.c Kees Cook 2018-09-18 239 skcipher_request_set_sync_tfm(subreq, ctx->fallback);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 240 skcipher_request_set_callback(subreq, req->base.flags,
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 241 NULL, NULL);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 242 skcipher_request_set_crypt(subreq, req->src, req->dst,
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 243 req->cryptlen, req->iv);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 244 ret = encrypt ? crypto_skcipher_encrypt(subreq) :
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 245 crypto_skcipher_decrypt(subreq);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 246 skcipher_request_zero(subreq);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 247 return ret;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 248 }
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 249
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 250 return tmpl->qce->async_req_enqueue(tmpl->qce, &req->base);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 251 }
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 252
:::::: The code at line 232 was first introduced by commit
:::::: c23a1c2b41c486bff4ad5cf8b0968e3f55907eba crypto: qce - use AES fallback for small requests
:::::: TO: Eneas U de Queiroz <cotequeiroz@gmail.com>
:::::: CC: 0day robot <lkp@intel.com>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 51937 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v3 1/3] crypto: qce - use cryptlen when adding extra sgl
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
` (2 preceding siblings ...)
2020-02-06 1:20 ` [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails Eneas U de Queiroz
@ 2020-02-06 11:39 ` Eneas U de Queiroz
2020-02-06 11:39 ` [PATCH v3 2/3] crypto: qce - use AES fallback for small requests Eneas U de Queiroz
2020-02-06 11:39 ` [PATCH v3 3/3] crypto: qce - handle AES-XTS cases that qce fails Eneas U de Queiroz
5 siblings, 0 replies; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 11:39 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
The qce crypto driver appends an extra entry to the dst sgl, to maintain
private state information.
When the gcm driver sends requests to the ctr skcipher, it passes the
authentication tag after the actual crypto payload, but it must not be
touched.
Commit 1336c2221bee ("crypto: qce - save a sg table slot for result
buf") limited the destination sgl to avoid overwriting the
authentication tag but it assumed the tag would be in a separate sgl
entry.
This is not always the case, so it is better to limit the length of the
destination buffer to req->cryptlen before appending the result buf.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/qce/dma.c b/drivers/crypto/qce/dma.c
index 7da893dc00e7..46db5bf366b4 100644
--- a/drivers/crypto/qce/dma.c
+++ b/drivers/crypto/qce/dma.c
@@ -48,9 +48,10 @@ void qce_dma_release(struct qce_dma_data *dma)
struct scatterlist *
qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,
- int max_ents)
+ unsigned int max_len)
{
struct scatterlist *sg = sgt->sgl, *sg_last = NULL;
+ unsigned int new_len;
while (sg) {
if (!sg_page(sg))
@@ -61,13 +62,13 @@ qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,
if (!sg)
return ERR_PTR(-EINVAL);
- while (new_sgl && sg && max_ents) {
- sg_set_page(sg, sg_page(new_sgl), new_sgl->length,
- new_sgl->offset);
+ while (new_sgl && sg && max_len) {
+ new_len = new_sgl->length > max_len ? max_len : new_sgl->length;
+ sg_set_page(sg, sg_page(new_sgl), new_len, new_sgl->offset);
sg_last = sg;
sg = sg_next(sg);
new_sgl = sg_next(new_sgl);
- max_ents--;
+ max_len -= new_len;
}
return sg_last;
diff --git a/drivers/crypto/qce/dma.h b/drivers/crypto/qce/dma.h
index ed25a0d9829e..786402169360 100644
--- a/drivers/crypto/qce/dma.h
+++ b/drivers/crypto/qce/dma.h
@@ -43,6 +43,6 @@ void qce_dma_issue_pending(struct qce_dma_data *dma);
int qce_dma_terminate_all(struct qce_dma_data *dma);
struct scatterlist *
qce_sgtable_add(struct sg_table *sgt, struct scatterlist *sg_add,
- int max_ents);
+ unsigned int max_len);
#endif /* _DMA_H_ */
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index 4217b745f124..63ae75809cb7 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -97,13 +97,14 @@ qce_skcipher_async_req_handle(struct crypto_async_request *async_req)
sg_init_one(&rctx->result_sg, qce->dma.result_buf, QCE_RESULT_BUF_SZ);
- sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, rctx->dst_nents - 1);
+ sg = qce_sgtable_add(&rctx->dst_tbl, req->dst, req->cryptlen);
if (IS_ERR(sg)) {
ret = PTR_ERR(sg);
goto error_free;
}
- sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->result_sg, 1);
+ sg = qce_sgtable_add(&rctx->dst_tbl, &rctx->result_sg,
+ QCE_RESULT_BUF_SZ);
if (IS_ERR(sg)) {
ret = PTR_ERR(sg);
goto error_free;
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v3 2/3] crypto: qce - use AES fallback for small requests
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
` (3 preceding siblings ...)
2020-02-06 11:39 ` [PATCH v3 1/3] crypto: qce - use cryptlen when adding extra sgl Eneas U de Queiroz
@ 2020-02-06 11:39 ` Eneas U de Queiroz
2020-02-06 11:39 ` [PATCH v3 3/3] crypto: qce - handle AES-XTS cases that qce fails Eneas U de Queiroz
5 siblings, 0 replies; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 11:39 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
Process small blocks using the fallback cipher, as a workaround for an
observed failure (DMA-related, apparently) when computing the GCM ghash
key. This brings a speed gain as well, since it avoids the latency of
using the hardware engine to process small blocks.
Using software for all 16-byte requests would be enough to make GCM
work, but to increase performance, a larger threshold would be better.
Measuring the performance of supported ciphers with openssl speed,
software matches hardware at around 768-1024 bytes.
Considering the 256-bit ciphers, software is 2-3 times faster than qce
at 256-bytes, 30% faster at 512, and about even at 768-bytes. With
128-bit keys, the break-even point would be around 1024-bytes.
This adds the 'aes_sw_max_len' parameter, to set the largest request
length processed by the software fallback. Its default is being set to
512 bytes, a little lower than the break-even point, to balance the cost
in CPU usage.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index c2767ed54dfe..052d3ff7fb20 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -685,6 +685,29 @@ choice
endchoice
+config CRYPTO_DEV_QCE_SW_MAX_LEN
+ int "Default maximum request size to use software for AES"
+ depends on CRYPTO_DEV_QCE && CRYPTO_DEV_QCE_SKCIPHER
+ default 512
+ help
+ This sets the default maximum request size to perform AES requests
+ using software instead of the crypto engine. It can be changed by
+ setting the aes_sw_max_len parameter.
+
+ Small blocks are processed faster in software than hardware.
+ Considering the 256-bit ciphers, software is 2-3 times faster than
+ qce at 256-bytes, 30% faster at 512, and about even at 768-bytes.
+ With 128-bit keys, the break-even point would be around 1024-bytes.
+
+ The default is set a little lower, to 512 bytes, to balance the
+ cost in CPU usage. The minimum recommended setting is 16-bytes
+ (1 AES block), since AES-GCM will fail if you set it lower.
+ Setting this to zero will send all requests to the hardware.
+
+ Note that 192-bit keys are not supported by the hardware and are
+ always processed by the software fallback, and all DES requests
+ are done by the hardware.
+
config CRYPTO_DEV_QCOM_RNG
tristate "Qualcomm Random Number Generator Driver"
depends on ARCH_QCOM || COMPILE_TEST
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index 63ae75809cb7..a3536495b6b0 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -5,6 +5,7 @@
#include <linux/device.h>
#include <linux/interrupt.h>
+#include <linux/moduleparam.h>
#include <linux/types.h>
#include <crypto/aes.h>
#include <crypto/internal/des.h>
@@ -12,6 +13,13 @@
#include "cipher.h"
+unsigned int aes_sw_max_len = CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN;
+module_param(aes_sw_max_len, uint, 0644);
+MODULE_PARM_DESC(aes_sw_max_len,
+ "Only use hardware for AES requests larger than this "
+ "[0=always use hardware; anything <16 breaks AES-GCM; default="
+ __stringify(CONFIG_CRYPTO_DEV_QCE_SOFT_THRESHOLD) "]");
+
static LIST_HEAD(skcipher_algs);
static void qce_skcipher_done(void *data)
@@ -166,15 +174,10 @@ static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,
switch (IS_XTS(flags) ? keylen >> 1 : keylen) {
case AES_KEYSIZE_128:
case AES_KEYSIZE_256:
+ memcpy(ctx->enc_key, key, keylen);
break;
- default:
- goto fallback;
}
- ctx->enc_keylen = keylen;
- memcpy(ctx->enc_key, key, keylen);
- return 0;
-fallback:
ret = crypto_sync_skcipher_setkey(ctx->fallback, key, keylen);
if (!ret)
ctx->enc_keylen = keylen;
@@ -224,8 +227,9 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
- if (IS_AES(rctx->flags) && keylen != AES_KEYSIZE_128 &&
- keylen != AES_KEYSIZE_256) {
+ if (IS_AES(rctx->flags) &&
+ ((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
+ || req->cryptlen <= aes_sw_max_len)) {
SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
skcipher_request_set_sync_tfm(subreq, ctx->fallback);
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v3 3/3] crypto: qce - handle AES-XTS cases that qce fails
2020-02-06 1:20 ` [PATCH v2 0/3] crypto: qce driver fixes for gcm Eneas U de Queiroz
` (4 preceding siblings ...)
2020-02-06 11:39 ` [PATCH v3 2/3] crypto: qce - use AES fallback for small requests Eneas U de Queiroz
@ 2020-02-06 11:39 ` Eneas U de Queiroz
5 siblings, 0 replies; 11+ messages in thread
From: Eneas U de Queiroz @ 2020-02-06 11:39 UTC (permalink / raw)
To: linux-crypto, Herbert Xu, David S. Miller
Cc: Ard Biesheuvel, Eneas U de Queiroz
QCE hangs when presented with an AES-XTS request whose length is larger
than QCE_SECTOR_SIZE (512-bytes), and is not a multiple of it. Let the
fallback cipher handle them.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
diff --git a/drivers/crypto/qce/common.c b/drivers/crypto/qce/common.c
index 629e7f34dc09..5006e74c40cd 100644
--- a/drivers/crypto/qce/common.c
+++ b/drivers/crypto/qce/common.c
@@ -15,8 +15,6 @@
#include "regs-v5.h"
#include "sha.h"
-#define QCE_SECTOR_SIZE 512
-
static inline u32 qce_read(struct qce_device *qce, u32 offset)
{
return readl(qce->base + offset);
diff --git a/drivers/crypto/qce/common.h b/drivers/crypto/qce/common.h
index 282d4317470d..9f989cba0f1b 100644
--- a/drivers/crypto/qce/common.h
+++ b/drivers/crypto/qce/common.h
@@ -12,6 +12,9 @@
#include <crypto/hash.h>
#include <crypto/internal/skcipher.h>
+/* xts du size */
+#define QCE_SECTOR_SIZE 512
+
/* key size in bytes */
#define QCE_SHA_HMAC_KEY_SIZE 64
#define QCE_MAX_CIPHER_KEY_SIZE AES_KEYSIZE_256
diff --git a/drivers/crypto/qce/skcipher.c b/drivers/crypto/qce/skcipher.c
index a3536495b6b0..377714cea23a 100644
--- a/drivers/crypto/qce/skcipher.c
+++ b/drivers/crypto/qce/skcipher.c
@@ -227,9 +227,14 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
+ /* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and
+ * is not a multiple of it; pass such requests to the fallback
+ */
if (IS_AES(rctx->flags) &&
((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
- || req->cryptlen <= aes_sw_max_len)) {
+ || req->cryptlen <= aes_sw_max_len)
+ || (IS_XTS(rctx->flags) && req->cryptlen > QCE_SECTOR_SIZE &&
+ req->cryptlen % QCE_SECTOR_SIZE)) {
SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
skcipher_request_set_sync_tfm(subreq, ctx->fallback);
^ permalink raw reply related [flat|nested] 11+ messages in thread