linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [Patch V5 0/7] crypto: AES CBC multibuffer implementation
@ 2017-04-20 20:50 Megha Dey
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
                   ` (6 more replies)
  0 siblings, 7 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

In this patch series, we introduce AES CBC encryption that is parallelized on
x86_64 cpu with XMM registers. The multi-buffer technique encrypt 8 data
streams in parallel with SIMD instructions. Decryption is handled as in the
existing AESNI Intel CBC implementation which can already parallelize decryption
even for a single data stream.

Please see the multi-buffer whitepaper for details of the technique:
http://www.intel.com/content/www/us/en/communications/communications-ia-multi-buffer-paper.html

It is important that any driver uses this algorithm properly for scenarios
where we have many data streams that can fill up the data lanes most of the
time. It shouldn't be used when only a single data stream is expected mostly.
Otherwise we may incur extra delays when we have frequent gaps in data lanes,
causing us to wait till data come in to fill the data lanes before initiating
encryption.  We may have to wait for flush operations to commence when no new
data come in after some wait time. However we keep this extra delay to a
minimum by opportunistically flushing the unfinished jobs if crypto daemon is
the only active task running on a cpu.

By using this technique, we saw a throughput increase of up to 5.7x under
optimal conditions when we have fully loaded encryption jobs filling up all
the data lanes.

Change Log:

v5
1. Use an async implementation of the inner algorithm instead of sync and use
the latest skcipher interface instead of the older blkcipher interface.
(we have picked up this work after a while)

v4
1. Make the decrypt path also use ablkcpher walk.
http://lkml.iu.edu/hypermail/linux/kernel/1512.0/01807.html

v3
1. Use ablkcipher_walk helpers to walk the scatter gather list
and eliminated needs to modify blkcipher_walk for multibuffer cipher

v2
1. Update cpu feature check to make sure SSE is supported
2. Fix up unloading of aes-cbc-mb module to properly free memory

Megha Dey (7):
  crypto: Multi-buffer encryption infrastructure support
  crypto: AES CBC multi-buffer data structures
  crypto: AES CBC multi-buffer scheduler
  crypto: AES CBC by8 encryption
  crypto: AES CBC multi-buffer glue code
  crypto: AES vectors for AES CBC multibuffer testing
  crypto: AES CBC multi-buffer tcrypt

 arch/x86/crypto/Makefile                           |    1 +
 arch/x86/crypto/aes-cbc-mb/Makefile                |   22 +
 arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S        |  775 ++++++++++
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c            |  737 ++++++++++
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h        |   97 ++
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h        |  132 ++
 arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c       |  146 ++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S     |  271 ++++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S |  223 +++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S     |  417 ++++++
 arch/x86/crypto/aes-cbc-mb/reg_sizes.S             |  126 ++
 crypto/Kconfig                                     |   15 +
 crypto/mcryptd.c                                   |  298 ++++
 crypto/simd.c                                      |  164 +++
 crypto/tcrypt.c                                    |  257 +++-
 crypto/testmgr.c                                   |  707 +++++++++
 crypto/testmgr.h                                   | 1496 ++++++++++++++++++++
 include/crypto/internal/simd.h                     |    3 +
 include/crypto/mcryptd.h                           |   35 +
 19 files changed, 5921 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aes-cbc-mb/Makefile
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/reg_sizes.S

-- 
1.9.1

^ permalink raw reply	[flat|nested] 15+ messages in thread

* [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-21  0:19   ` kbuild test robot
                     ` (2 more replies)
  2017-04-20 20:50 ` [Patch V5 2/7] crypto: AES CBC multi-buffer data structures Megha Dey
                   ` (5 subsequent siblings)
  6 siblings, 3 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

In this patch, the infrastructure needed to support multibuffer
encryption implementation is added:

a) Enhance mcryptd daemon to support skcipher requests.

b) Add multi-buffer simd skcipher helper which presents the
   top-level algorithm as an skcipher.

b) Update configuration to include multi-buffer encryption build
support.

For an introduction to the multi-buffer implementation, please see
http://www.intel.com/content/www/us/en/communications/communications-ia-multi-buffer-paper.html

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 crypto/Kconfig                 |  15 +++
 crypto/mcryptd.c               | 298 +++++++++++++++++++++++++++++++++++++++++
 crypto/simd.c                  | 164 +++++++++++++++++++++++
 include/crypto/internal/simd.h |   3 +
 include/crypto/mcryptd.h       |  35 +++++
 5 files changed, 515 insertions(+)

diff --git a/crypto/Kconfig b/crypto/Kconfig
index aac4bc9..d172459 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1008,6 +1008,21 @@ config CRYPTO_AES_NI_INTEL
 	  ECB, CBC, LRW, PCBC, XTS. The 64 bit version has additional
 	  acceleration for CTR.
 
+config CRYPTO_AES_CBC_MB
+        tristate "AES CBC algorithm (x86_64 Multi-Buffer, Experimental)"
+        depends on X86 && 64BIT
+        select CRYPTO_SIMD
+        select CRYPTO_MCRYPTD
+        help
+          AES CBC encryption implemented using multi-buffer technique.
+          This algorithm computes on multiple data lanes concurrently with
+          SIMD instructions for better throughput. It should only be used
+          when we expect many concurrent crypto requests to keep all the
+          data lanes filled to realize the performance benefit. If the data
+          lanes are unfilled, a flush operation will be initiated after some
+          delay to process the exisiting crypto jobs, adding some extra
+          latency to low load case.
+
 config CRYPTO_AES_SPARC64
 	tristate "AES cipher algorithms (SPARC64)"
 	depends on SPARC64
diff --git a/crypto/mcryptd.c b/crypto/mcryptd.c
index 4e64726..4e4830e 100644
--- a/crypto/mcryptd.c
+++ b/crypto/mcryptd.c
@@ -273,6 +273,266 @@ static inline bool mcryptd_check_internal(struct rtattr **tb, u32 *type,
 		return false;
 }
 
+static int mcryptd_enqueue_skcipher_request(struct mcryptd_queue *queue,
+				struct crypto_async_request *request,
+				struct mcryptd_skcipher_request_ctx *rctx)
+{
+	int cpu, err;
+	struct mcryptd_cpu_queue *cpu_queue;
+
+	cpu = get_cpu();
+	cpu_queue = this_cpu_ptr(queue->cpu_queue);
+	rctx->tag.cpu = cpu;
+
+	err = crypto_enqueue_request(&cpu_queue->queue, request);
+	pr_debug("enqueue request: cpu %d cpu_queue %p request %p\n",
+		cpu, cpu_queue, request);
+	queue_work_on(cpu, kcrypto_wq, &cpu_queue->work);
+	put_cpu();
+
+	return err;
+}
+
+static int mcryptd_skcipher_setkey(struct crypto_skcipher *parent,
+				const u8 *key, unsigned int keylen)
+{
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(parent);
+	struct crypto_skcipher *child = ctx->child;
+	int err;
+
+	crypto_skcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+	crypto_skcipher_set_flags(child, crypto_skcipher_get_flags(parent) &
+						CRYPTO_TFM_REQ_MASK);
+	err = crypto_skcipher_setkey(child, key, keylen);
+	crypto_skcipher_set_flags(parent, crypto_skcipher_get_flags(child) &
+						CRYPTO_TFM_RES_MASK);
+	return err;
+}
+
+static void mcryptd_skcipher_complete(struct skcipher_request *req, int err)
+{
+	struct mcryptd_skcipher_request_ctx *rctx = skcipher_request_ctx(req);
+
+	local_bh_disable();
+	rctx->complete(&req->base, err);
+	local_bh_enable();
+}
+
+static void mcryptd_skcipher_encrypt(struct crypto_async_request *base,
+								int err)
+{
+	struct skcipher_request *req = skcipher_request_cast(base);
+	struct mcryptd_skcipher_request_ctx *rctx = skcipher_request_ctx(req);
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct crypto_skcipher *child = ctx->child;
+	struct skcipher_request subreq;
+
+	if (unlikely(err == -EINPROGRESS))
+		goto out;
+
+	/* set up the skcipher request to work on */
+	skcipher_request_set_tfm(&subreq, child);
+	skcipher_request_set_callback(&subreq,
+					CRYPTO_TFM_REQ_MAY_SLEEP, 0, 0);
+	skcipher_request_set_crypt(&subreq, req->src, req->dst,
+					req->cryptlen, req->iv);
+
+	/*
+	 * pass addr of descriptor stored in the request context
+	 * so that the callee can get to the request context
+	 */
+	rctx->desc = subreq;
+	err = crypto_skcipher_encrypt(&rctx->desc);
+
+	if (err) {
+		req->base.complete = rctx->complete;
+		goto out;
+	}
+	return;
+
+out:
+	mcryptd_skcipher_complete(req, err);
+}
+
+static void mcryptd_skcipher_decrypt(struct crypto_async_request *base,
+								int err)
+{
+	struct skcipher_request *req = skcipher_request_cast(base);
+	struct mcryptd_skcipher_request_ctx *rctx = skcipher_request_ctx(req);
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct crypto_skcipher *child = ctx->child;
+	struct skcipher_request subreq;
+
+	if (unlikely(err == -EINPROGRESS))
+		goto out;
+
+	/* set up the skcipher request to work on */
+	skcipher_request_set_tfm(&subreq, child);
+	skcipher_request_set_callback(&subreq,
+				CRYPTO_TFM_REQ_MAY_SLEEP, NULL, NULL);
+	skcipher_request_set_crypt(&subreq, req->src, req->dst,
+						req->cryptlen, req->iv);
+
+	/*
+	 * pass addr of descriptor stored in the request context
+	 * so that the callee can get to the request context
+	 */
+	rctx->desc = subreq;
+	err = crypto_skcipher_decrypt(&rctx->desc);
+
+	if (err) {
+	req->base.complete = rctx->complete;
+	goto out;
+	}
+	return;
+
+out:
+	mcryptd_skcipher_complete(req, err);
+}
+
+static int mcryptd_skcipher_enqueue(struct skcipher_request *req,
+					crypto_completion_t complete)
+{
+	struct mcryptd_skcipher_request_ctx *rctx =
+					skcipher_request_ctx(req);
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	struct mcryptd_queue *queue;
+
+	queue = mcryptd_get_queue(crypto_skcipher_tfm(tfm));
+	rctx->complete = req->base.complete;
+	req->base.complete = complete;
+
+	return mcryptd_enqueue_skcipher_request(queue, &req->base, rctx);
+}
+
+static int mcryptd_skcipher_encrypt_enqueue(struct skcipher_request *req)
+{
+	return mcryptd_skcipher_enqueue(req, mcryptd_skcipher_encrypt);
+}
+
+static int mcryptd_skcipher_decrypt_enqueue(struct skcipher_request *req)
+{
+	return mcryptd_skcipher_enqueue(req, mcryptd_skcipher_decrypt);
+}
+
+static int mcryptd_skcipher_init_tfm(struct crypto_skcipher *tfm)
+{
+	struct skcipher_instance *inst = skcipher_alg_instance(tfm);
+	struct mskcipherd_instance_ctx *ictx = skcipher_instance_ctx(inst);
+	struct crypto_skcipher_spawn *spawn = &ictx->spawn;
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+	struct crypto_skcipher *cipher;
+
+	cipher = crypto_spawn_skcipher(spawn);
+	if (IS_ERR(cipher))
+		return PTR_ERR(cipher);
+
+	ctx->child = cipher;
+	crypto_skcipher_set_reqsize(tfm,
+			sizeof(struct mcryptd_skcipher_request_ctx));
+	return 0;
+}
+
+static void mcryptd_skcipher_exit_tfm(struct crypto_skcipher *tfm)
+{
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
+
+	crypto_free_skcipher(ctx->child);
+}
+
+static void mcryptd_skcipher_free(struct skcipher_instance *inst)
+{
+	struct mskcipherd_instance_ctx *ctx = skcipher_instance_ctx(inst);
+
+	crypto_drop_skcipher(&ctx->spawn);
+}
+
+static int mcryptd_init_instance(struct crypto_instance *inst,
+					struct crypto_alg *alg)
+{
+	if (snprintf(inst->alg.cra_driver_name, CRYPTO_MAX_ALG_NAME,
+		"mcryptd(%s)",
+			alg->cra_driver_name) >= CRYPTO_MAX_ALG_NAME)
+		return -ENAMETOOLONG;
+
+	memcpy(inst->alg.cra_name, alg->cra_name, CRYPTO_MAX_ALG_NAME);
+	inst->alg.cra_priority = alg->cra_priority + 50;
+	inst->alg.cra_blocksize = alg->cra_blocksize;
+	inst->alg.cra_alignmask = alg->cra_alignmask;
+
+	return 0;
+}
+
+static int mcryptd_create_skcipher(struct crypto_template *tmpl,
+				   struct rtattr **tb,
+				   struct mcryptd_queue *queue)
+{
+	struct mskcipherd_instance_ctx *ctx;
+	struct skcipher_instance *inst;
+	struct skcipher_alg *alg;
+	const char *name;
+	u32 type;
+	u32 mask;
+	int err;
+
+	type = 0;
+	mask = CRYPTO_ALG_ASYNC;
+
+	mcryptd_check_internal(tb, &type, &mask);
+
+	name = crypto_attr_alg_name(tb[1]);
+	if (IS_ERR(name))
+		return PTR_ERR(name);
+
+	inst = kzalloc(sizeof(*inst) + sizeof(*ctx), GFP_KERNEL);
+	if (!inst)
+		return -ENOMEM;
+
+	ctx = skcipher_instance_ctx(inst);
+	ctx->queue = queue;
+
+	crypto_set_skcipher_spawn(&ctx->spawn, skcipher_crypto_instance(inst));
+	err = crypto_grab_skcipher(&ctx->spawn, name, type, mask);
+
+	if (err)
+		goto out_free_inst;
+
+	alg = crypto_spawn_skcipher_alg(&ctx->spawn);
+	err = mcryptd_init_instance(skcipher_crypto_instance(inst), &alg->base);
+	if (err)
+		goto out_drop_skcipher;
+
+	inst->alg.base.cra_flags = CRYPTO_ALG_ASYNC |
+				(alg->base.cra_flags & CRYPTO_ALG_INTERNAL);
+
+	inst->alg.ivsize = crypto_skcipher_alg_ivsize(alg);
+	inst->alg.chunksize = crypto_skcipher_alg_chunksize(alg);
+	inst->alg.min_keysize = crypto_skcipher_alg_min_keysize(alg);
+	inst->alg.max_keysize = crypto_skcipher_alg_max_keysize(alg);
+
+	inst->alg.base.cra_ctxsize = sizeof(struct mcryptd_skcipher_ctx);
+
+	inst->alg.init = mcryptd_skcipher_init_tfm;
+	inst->alg.exit = mcryptd_skcipher_exit_tfm;
+
+	inst->alg.setkey = mcryptd_skcipher_setkey;
+	inst->alg.encrypt = mcryptd_skcipher_encrypt_enqueue;
+	inst->alg.decrypt = mcryptd_skcipher_decrypt_enqueue;
+
+	inst->free = mcryptd_skcipher_free;
+
+	err = skcipher_register_instance(tmpl, inst);
+	if (err) {
+out_drop_skcipher:
+	crypto_drop_skcipher(&ctx->spawn);
+out_free_inst:
+	kfree(inst);
+	}
+	return err;
+}
+
 static int mcryptd_hash_init_tfm(struct crypto_tfm *tfm)
 {
 	struct crypto_instance *inst = crypto_tfm_alg_instance(tfm);
@@ -564,6 +824,8 @@ static int mcryptd_create(struct crypto_template *tmpl, struct rtattr **tb)
 		return PTR_ERR(algt);
 
 	switch (algt->type & algt->mask & CRYPTO_ALG_TYPE_MASK) {
+	case CRYPTO_ALG_TYPE_BLKCIPHER:
+		return mcryptd_create_skcipher(tmpl, tb, &mqueue);
 	case CRYPTO_ALG_TYPE_DIGEST:
 		return mcryptd_create_hash(tmpl, tb, &mqueue);
 	break;
@@ -595,6 +857,42 @@ static void mcryptd_free(struct crypto_instance *inst)
 	.module = THIS_MODULE,
 };
 
+struct mcryptd_skcipher *mcryptd_alloc_skcipher(const char *alg_name,
+							u32 type, u32 mask)
+{
+	char cryptd_alg_name[CRYPTO_MAX_ALG_NAME];
+	struct crypto_skcipher *tfm;
+
+	if (snprintf(cryptd_alg_name, CRYPTO_MAX_ALG_NAME,
+		"mcryptd(%s)", alg_name) >= CRYPTO_MAX_ALG_NAME)
+		return ERR_PTR(-EINVAL);
+	tfm = crypto_alloc_skcipher(cryptd_alg_name, type, mask);
+	if (IS_ERR(tfm))
+		return ERR_CAST(tfm);
+	if (tfm->base.__crt_alg->cra_module != THIS_MODULE) {
+		crypto_free_skcipher(tfm);
+		return ERR_PTR(-EINVAL);
+	}
+
+	return container_of(tfm, struct mcryptd_skcipher, base);
+}
+EXPORT_SYMBOL_GPL(mcryptd_alloc_skcipher);
+
+struct crypto_skcipher *mcryptd_skcipher_child(
+			struct mcryptd_skcipher *tfm)
+{
+	struct mcryptd_skcipher_ctx *ctx = crypto_skcipher_ctx(&tfm->base);
+
+	return ctx->child;
+}
+EXPORT_SYMBOL_GPL(mcryptd_skcipher_child);
+
+void mcryptd_free_skcipher(struct mcryptd_skcipher *tfm)
+{
+	crypto_free_skcipher(&tfm->base);
+}
+EXPORT_SYMBOL_GPL(mcryptd_free_skcipher);
+
 struct mcryptd_ahash *mcryptd_alloc_ahash(const char *alg_name,
 					u32 type, u32 mask)
 {
diff --git a/crypto/simd.c b/crypto/simd.c
index 8820337..6757d44 100644
--- a/crypto/simd.c
+++ b/crypto/simd.c
@@ -28,11 +28,14 @@
 #include <crypto/cryptd.h>
 #include <crypto/internal/simd.h>
 #include <crypto/internal/skcipher.h>
+#include <crypto/mcryptd.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/preempt.h>
 #include <asm/simd.h>
 
+static struct mcryptd_alg_state cbc_mb_alg_state;
+
 struct simd_skcipher_alg {
 	const char *ialg_name;
 	struct skcipher_alg alg;
@@ -42,6 +45,10 @@ struct simd_skcipher_ctx {
 	struct cryptd_skcipher *cryptd_tfm;
 };
 
+struct simd_skcipher_ctx_mb {
+	struct mcryptd_skcipher *mcryptd_tfm;
+};
+
 static int simd_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
 				unsigned int key_len)
 {
@@ -203,6 +210,163 @@ struct simd_skcipher_alg *simd_skcipher_create_compat(const char *algname,
 }
 EXPORT_SYMBOL_GPL(simd_skcipher_create_compat);
 
+static int simd_skcipher_setkey_mb(struct crypto_skcipher *tfm, const u8 *key,
+				unsigned int key_len)
+{
+	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
+	struct crypto_skcipher *child = &ctx->mcryptd_tfm->base;
+	int err;
+
+	crypto_skcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+	crypto_skcipher_set_flags(child, crypto_skcipher_get_flags(tfm) &
+					CRYPTO_TFM_REQ_MASK);
+	err = crypto_skcipher_setkey(child, key, key_len);
+	crypto_skcipher_set_flags(tfm, crypto_skcipher_get_flags(child) &
+					CRYPTO_TFM_RES_MASK);
+	return err;
+}
+
+static int simd_skcipher_decrypt_mb(struct skcipher_request *req)
+{
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
+	struct skcipher_request *subreq;
+	struct crypto_skcipher *child;
+
+	subreq = skcipher_request_ctx(req);
+	*subreq = *req;
+
+	child = &ctx->mcryptd_tfm->base;
+
+	skcipher_request_set_tfm(subreq, child);
+
+	return crypto_skcipher_decrypt(subreq);
+}
+
+static int simd_skcipher_encrypt_mb(struct skcipher_request *req)
+{
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
+	struct skcipher_request *subreq;
+	struct crypto_skcipher *child;
+
+	subreq = skcipher_request_ctx(req);
+	*subreq = *req;
+
+	child = &ctx->mcryptd_tfm->base;
+
+	skcipher_request_set_tfm(subreq, child);
+
+	return crypto_skcipher_encrypt(subreq);
+}
+
+static void simd_skcipher_exit_mb(struct crypto_skcipher *tfm)
+{
+	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
+
+	mcryptd_free_skcipher(ctx->mcryptd_tfm);
+}
+
+static int simd_skcipher_init_mb(struct crypto_skcipher *tfm)
+{
+	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
+	struct mcryptd_skcipher *mcryptd_tfm;
+	struct simd_skcipher_alg *salg;
+	struct skcipher_alg *alg;
+	unsigned reqsize;
+	struct mcryptd_skcipher_ctx *mctx;
+
+	alg = crypto_skcipher_alg(tfm);
+	salg = container_of(alg, struct simd_skcipher_alg, alg);
+
+	mcryptd_tfm = mcryptd_alloc_skcipher(salg->ialg_name,
+					   CRYPTO_ALG_INTERNAL,
+					   CRYPTO_ALG_INTERNAL);
+	if (IS_ERR(mcryptd_tfm))
+		return PTR_ERR(mcryptd_tfm);
+
+	mctx = crypto_skcipher_ctx(&mcryptd_tfm->base);
+
+	mctx->alg_state = &cbc_mb_alg_state;
+	ctx->mcryptd_tfm = mcryptd_tfm;
+
+	reqsize = sizeof(struct skcipher_request);
+	reqsize += crypto_skcipher_reqsize(&mcryptd_tfm->base);
+
+	crypto_skcipher_set_reqsize(tfm, reqsize);
+
+	return 0;
+}
+
+struct simd_skcipher_alg *simd_skcipher_create_compat_mb(const char *algname,
+							const char *drvname,
+							const char *basename)
+{
+	struct simd_skcipher_alg *salg;
+	struct crypto_skcipher *tfm;
+	struct skcipher_alg *ialg;
+	struct skcipher_alg *alg;
+	int err;
+
+	tfm = crypto_alloc_skcipher(basename, CRYPTO_ALG_INTERNAL,
+				    CRYPTO_ALG_INTERNAL | CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return ERR_CAST(tfm);
+
+	ialg = crypto_skcipher_alg(tfm);
+
+	salg = kzalloc(sizeof(*salg), GFP_KERNEL);
+	if (!salg) {
+		salg = ERR_PTR(-ENOMEM);
+		goto out_put_tfm;
+	}
+
+	salg->ialg_name = basename;
+	alg = &salg->alg;
+
+	err = -ENAMETOOLONG;
+	if (snprintf(alg->base.cra_name, CRYPTO_MAX_ALG_NAME, "%s", algname) >=
+	    CRYPTO_MAX_ALG_NAME)
+		goto out_free_salg;
+
+	if (snprintf(alg->base.cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+		     drvname) >= CRYPTO_MAX_ALG_NAME)
+		goto out_free_salg;
+
+	alg->base.cra_flags = CRYPTO_ALG_ASYNC;
+	alg->base.cra_priority = ialg->base.cra_priority;
+	alg->base.cra_blocksize = ialg->base.cra_blocksize;
+	alg->base.cra_alignmask = ialg->base.cra_alignmask;
+	alg->base.cra_module = ialg->base.cra_module;
+	alg->base.cra_ctxsize = sizeof(struct simd_skcipher_ctx_mb);
+
+	alg->ivsize = ialg->ivsize;
+	alg->chunksize = ialg->chunksize;
+	alg->min_keysize = ialg->min_keysize;
+	alg->max_keysize = ialg->max_keysize;
+
+	alg->init = simd_skcipher_init_mb;
+	alg->exit = simd_skcipher_exit_mb;
+
+	alg->setkey = simd_skcipher_setkey_mb;
+	alg->encrypt = simd_skcipher_encrypt_mb;
+	alg->decrypt = simd_skcipher_decrypt_mb;
+
+	err = crypto_register_skcipher(alg);
+	if (err)
+		goto out_free_salg;
+
+out_put_tfm:
+	crypto_free_skcipher(tfm);
+	return salg;
+
+out_free_salg:
+	kfree(salg);
+	salg = ERR_PTR(err);
+	goto out_put_tfm;
+}
+EXPORT_SYMBOL_GPL(simd_skcipher_create_compat_mb);
+
 struct simd_skcipher_alg *simd_skcipher_create(const char *algname,
 					       const char *basename)
 {
diff --git a/include/crypto/internal/simd.h b/include/crypto/internal/simd.h
index 4295099..2c518a6 100644
--- a/include/crypto/internal/simd.h
+++ b/include/crypto/internal/simd.h
@@ -10,6 +10,9 @@
 struct simd_skcipher_alg *simd_skcipher_create_compat(const char *algname,
 						      const char *drvname,
 						      const char *basename);
+struct simd_skcipher_alg *simd_skcipher_create_compat_mb(const char *algname,
+							const char *drvname,
+							const char *basename);
 struct simd_skcipher_alg *simd_skcipher_create(const char *algname,
 					       const char *basename);
 void simd_skcipher_free(struct simd_skcipher_alg *alg);
diff --git a/include/crypto/mcryptd.h b/include/crypto/mcryptd.h
index 4a53c0d..5b17ce6 100644
--- a/include/crypto/mcryptd.h
+++ b/include/crypto/mcryptd.h
@@ -13,6 +13,28 @@
 #include <linux/crypto.h>
 #include <linux/kernel.h>
 #include <crypto/hash.h>
+#include <crypto/b128ops.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/internal/hash.h>
+
+struct mskcipherd_instance_ctx {
+	struct crypto_skcipher_spawn spawn;
+	struct mcryptd_queue *queue;
+};
+
+struct mcryptd_skcipher_ctx {
+	struct crypto_skcipher *child;
+	struct mcryptd_alg_state *alg_state;
+};
+
+struct mcryptd_skcipher {
+	struct crypto_skcipher base;
+};
+
+struct mcryptd_skcipher *mcryptd_alloc_skcipher(const char *alg_name,
+					      u32 type, u32 mask);
+struct crypto_skcipher *mcryptd_skcipher_child(struct mcryptd_skcipher *tfm);
+void mcryptd_free_skcipher(struct mcryptd_skcipher *tfm);
 
 struct mcryptd_ahash {
 	struct crypto_ahash base;
@@ -62,6 +84,19 @@ struct mcryptd_hash_request_ctx {
 	struct ahash_request areq;
 };
 
+struct mcryptd_skcipher_request_ctx {
+	struct list_head waiter;
+	crypto_completion_t complete;
+	struct mcryptd_tag tag;
+	struct skcipher_walk walk;
+	u8 flag;
+	int nbytes;
+	int error;
+	struct skcipher_request desc;
+	void *job;
+	u128 seq_iv;
+};
+
 struct mcryptd_ahash *mcryptd_alloc_ahash(const char *alg_name,
 					u32 type, u32 mask);
 struct crypto_ahash *mcryptd_ahash_child(struct mcryptd_ahash *tfm);
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 2/7] crypto: AES CBC multi-buffer data structures
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-20 20:50 ` [Patch V5 3/7] crypto: AES CBC multi-buffer scheduler Megha Dey
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

This patch introduces the data structures and prototypes of functions
needed for doing AES CBC encryption using multi-buffer. Included are
the structures of the multi-buffer AES CBC job, job scheduler in C and
data structure defines in x86 assembly code.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h    |  97 +++++++++
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h    | 132 ++++++++++++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S | 271 +++++++++++++++++++++++++
 arch/x86/crypto/aes-cbc-mb/reg_sizes.S         | 126 ++++++++++++
 4 files changed, 626 insertions(+)
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/reg_sizes.S

diff --git a/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h
new file mode 100644
index 0000000..024586b
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_ctx.h
@@ -0,0 +1,97 @@
+/*
+ *	Header file for multi buffer AES CBC algorithm manager
+ *	that deals with 8 buffers at a time
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#ifndef __AES_CBC_MB_CTX_H
+#define __AES_CBC_MB_CTX_H
+
+
+#include <linux/types.h>
+
+#include "aes_cbc_mb_mgr.h"
+
+#define CBC_ENCRYPT	0x01
+#define CBC_DECRYPT	0x02
+#define CBC_START	0x04
+#define CBC_DONE	0x08
+
+#define CBC_CTX_STS_IDLE       0x00
+#define CBC_CTX_STS_PROCESSING 0x01
+#define CBC_CTX_STS_LAST       0x02
+#define CBC_CTX_STS_COMPLETE   0x04
+
+enum cbc_ctx_error {
+	CBC_CTX_ERROR_NONE               =  0,
+	CBC_CTX_ERROR_INVALID_FLAGS      = -1,
+	CBC_CTX_ERROR_ALREADY_PROCESSING = -2,
+	CBC_CTX_ERROR_ALREADY_COMPLETED  = -3,
+};
+
+#define cbc_ctx_init(ctx, n_bytes, op) \
+	do { \
+		(ctx)->flag = (op) | CBC_START; \
+		(ctx)->nbytes = (n_bytes); \
+	} while (0)
+
+/* AESNI routines to perform cbc decrypt and key expansion */
+
+asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
+			      const u8 *in, unsigned int len, u8 *iv);
+asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+			     unsigned int key_len);
+
+#endif /* __AES_CBC_MB_CTX_H */
diff --git a/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h
new file mode 100644
index 0000000..788180e
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb_mgr.h
@@ -0,0 +1,132 @@
+/*
+ *	Header file for multi buffer AES CBC algorithm manager
+ *	that deals with 8 buffers at a time
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#ifndef __AES_CBC_MB_MGR_H
+#define __AES_CBC_MB_MGR_H
+
+
+#include <linux/types.h>
+#include <linux/printk.h>
+#include <crypto/aes.h>
+#include <crypto/b128ops.h>
+
+#define MAX_AES_JOBS		128
+
+enum job_sts {
+	STS_UNKNOWN = 0,
+	STS_BEING_PROCESSED = 1,
+	STS_COMPLETED = 2,
+	STS_INTERNAL_ERROR = 3,
+	STS_ERROR = 4
+};
+
+/* AES CBC multi buffer in order job structure */
+
+struct job_aes_cbc {
+	u8	*plaintext;	/* pointer to plaintext */
+	u8	*ciphertext;	/* pointer to ciphertext */
+	u128	iv;		/* initialization vector */
+	u128	*keys;		/* pointer to keys */
+	u32	len;		/* length in bytes, must be multiple of 16 */
+	enum	job_sts status;	/* status enumeration */
+	void	*user_data;	/* pointer to user data */
+	u32	key_len;	/* key length */
+};
+
+struct aes_cbc_args_x8 {
+	u8	*arg_in[8];	/* array of 8 pointers to in text */
+	u8	*arg_out[8];	/* array of 8 pointers to out text */
+	u128	*arg_keys[8];	/* array of 8 pointers to keys */
+	u128	arg_iv[8] __aligned(16);	/* array of 8 128-bit IVs */
+};
+
+struct aes_cbc_mb_mgr_inorder_x8 {
+	struct aes_cbc_args_x8 args;
+	u16 lens[8] __aligned(16);
+	u64 unused_lanes; /* each nibble is index (0...7) of unused lanes */
+	/* nibble 8 is set to F as a flag */
+	struct job_aes_cbc *job_in_lane[8];
+	/* In-order components */
+	u32 earliest_job; /* byte offset, -1 if none */
+	u32 next_job;      /* byte offset */
+	struct job_aes_cbc jobs[MAX_AES_JOBS];
+};
+
+/* define AES CBC multi buffer manager function proto */
+struct job_aes_cbc *aes_cbc_submit_job_inorder_128x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+struct job_aes_cbc *aes_cbc_submit_job_inorder_192x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+struct job_aes_cbc *aes_cbc_submit_job_inorder_256x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+struct job_aes_cbc *aes_cbc_flush_job_inorder_x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+struct job_aes_cbc *aes_cbc_get_next_job_inorder_x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+struct job_aes_cbc *aes_cbc_get_completed_job_inorder_x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+void aes_cbc_init_mb_mgr_inorder_x8(
+	struct aes_cbc_mb_mgr_inorder_x8 *state);
+void aes_cbc_submit_job_ooo_x8(struct aes_cbc_mb_mgr_inorder_x8 *state,
+		struct job_aes_cbc *job);
+void aes_cbc_flush_job_ooo_x8(struct aes_cbc_mb_mgr_inorder_x8 *state);
+void aes_cbc_flush_job_ooo_128x8(struct aes_cbc_mb_mgr_inorder_x8 *state);
+void aes_cbc_flush_job_ooo_192x8(struct aes_cbc_mb_mgr_inorder_x8 *state);
+void aes_cbc_flush_job_ooo_256x8(struct aes_cbc_mb_mgr_inorder_x8 *state);
+
+#endif /* __AES_CBC_MB_MGR_H */
diff --git a/arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S b/arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S
new file mode 100644
index 0000000..6892893
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/mb_mgr_datastruct.S
@@ -0,0 +1,271 @@
+/*
+ *	Header file: Data structure: AES CBC multibuffer optimization
+ *				(x86_64)
+ *
+ * This is the generic header file defining data stuctures for
+ * AES multi-buffer CBC implementation. This file is to be included
+ * by the AES CBC multibuffer scheduler.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+/* This is the AES CBC BY8 implementation */
+#define	NBY	8
+
+#define AES_KEYSIZE_128 16
+#define AES_KEYSIZE_192 24
+#define AES_KEYSIZE_256 32
+
+#ifndef _AES_CBC_MB_MGR_DATASTRUCT_ASM_
+#define _AES_CBC_MB_MGR_DATASTRUCT_ASM_
+
+/* Macros for defining data structures */
+
+/* Usage example */
+
+/*
+ * Alternate "struct-like" syntax:
+ *	STRUCT job_aes2
+ *	RES_Q	.plaintext,	1
+ *	RES_Q	.ciphertext,	1
+ *	RES_DQ	.IV,		1
+ *	RES_B	.nested,	_JOB_AES_SIZE, _JOB_AES_ALIGN
+ *	RES_U	.union,		size1, align1, \
+ *				size2, align2, \
+ *				...
+ *	ENDSTRUCT
+ *	; Following only needed if nesting
+ *	%assign job_aes2_size	_FIELD_OFFSET
+ *	%assign job_aes2_align	_STRUCT_ALIGN
+ *
+ * RES_* macros take a name, a count and an optional alignment.
+ * The count in terms of the base size of the macro, and the
+ * default alignment is the base size.
+ * The macros are:
+ * Macro    Base size
+ * RES_B	    1
+ * RES_W	    2
+ * RES_D     4
+ * RES_Q     8
+ * RES_DQ   16
+ * RES_Y    32
+ * RES_Z    64
+ *
+ * RES_U defines a union. It's arguments are a name and two or more
+ * pairs of "size, alignment"
+ *
+ * The two assigns are only needed if this structure is being nested
+ * within another. Even if the assigns are not done, one can still use
+ * STRUCT_NAME_size as the size of the structure.
+ *
+ * Note that for nesting, you still need to assign to STRUCT_NAME_size.
+ *
+ * The differences between this and using "struct" directly are that each
+ * type is implicitly aligned to its natural length (although this can be
+ * over-ridden with an explicit third parameter), and that the structure
+ * is padded at the end to its overall alignment.
+ *
+ */
+
+/* START_FIELDS */
+
+.macro START_FIELDS
+_FIELD_OFFSET	=	0
+_STRUCT_ALIGN	=	0
+.endm
+
+/* FIELD name size align */
+.macro FIELD name, size, align
+
+	_FIELD_OFFSET = (_FIELD_OFFSET + (\align) - 1) & (~ ((\align)-1))
+	\name = _FIELD_OFFSET
+	_FIELD_OFFSET = _FIELD_OFFSET + (\size)
+	.if (\align > _STRUCT_ALIGN)
+		_STRUCT_ALIGN = \align
+	.endif
+.endm
+
+/* END_FIELDS */
+
+.macro END_FIELDS
+	_FIELD_OFFSET = \
+	(_FIELD_OFFSET + _STRUCT_ALIGN-1) & (~ (_STRUCT_ALIGN-1))
+.endm
+
+.macro STRUCT p1
+	START_FIELDS
+	.struct \p1
+.endm
+
+.macro ENDSTRUCT
+	tmp = _FIELD_OFFSET
+	END_FIELDS
+	tmp = (_FIELD_OFFSET - %%tmp)
+	.if (tmp > 0)
+		.lcomm	tmp
+.endif
+.endstruc
+.endm
+
+/* RES_int name size align */
+.macro RES_int p1, p2, p3
+	name = \p1
+	size = \p2
+	align = \p3
+
+	_FIELD_OFFSET = (_FIELD_OFFSET + (align) - 1) & (~ ((align)-1))
+	.align align
+	.lcomm name size
+	_FIELD_OFFSET = _FIELD_OFFSET + (size)
+	.if (align > _STRUCT_ALIGN)
+		_STRUCT_ALIGN = align
+	.endif
+.endm
+
+/* macro RES_B name, size [, align] */
+.macro RES_B _name, _size, _align=1
+	RES_int _name, _size, _align
+.endm
+
+/* macro RES_W name, size [, align] */
+.macro RES_W _name, _size, _align=2
+	RES_int _name, 2*(_size), _align
+.endm
+
+/* macro RES_D name, size [, align] */
+.macro RES_D _name, _size, _align=4
+	RES_int _name, 4*(_size), _align
+.endm
+
+/* macro RES_Q name, size [, align] */
+.macro RES_Q _name, _size, _align=8
+	RES_int _name, 8*(_size), _align
+.endm
+
+/* macro RES_DQ name, size [, align] */
+.macro RES_DQ _name, _size, _align=16
+	RES_int _name, 16*(_size), _align
+.endm
+
+/* macro RES_Y name, size [, align] */
+.macro RES_Y _name, _size, _align=32
+	RES_int _name, 32*(_size), _align
+.endm
+
+/*; macro RES_Z name, size [, align] */
+.macro RES_Z _name, _size, _align=64
+	RES_int _name, 64*(_size), _align
+.endm
+#endif /* _AES_CBC_MB_MGR_DATASTRUCT_ASM_ */
+
+/* Define JOB_AES structure */
+
+START_FIELDS	/* JOB_AES */
+/*	name		size	align	*/
+FIELD	_plaintext,	8,	8	/* pointer to plaintext */
+FIELD	_ciphertext,	8,	8	/* pointer to ciphertext */
+FIELD	_IV,		16,	8	/* IV */
+FIELD	_keys,		8,	8	/* pointer to keys */
+FIELD	_len,		4,	4	/* length in bytes */
+FIELD	_status,	4,	4	/* status enumeration */
+FIELD	_user_data,	8,	8	/* pointer to user data */
+FIELD	_key_len,	4,	8	/* key length */
+
+END_FIELDS
+
+_JOB_AES_size	=	_FIELD_OFFSET
+_JOB_AES_align	=	_STRUCT_ALIGN
+
+START_FIELDS	/* AES_ARGS_XX */
+/*	name		size	align	*/
+FIELD	_arg_in,	8*NBY, 8	/* [] of NBY pointers to in text */
+FIELD	_arg_out,	8*NBY, 8	/* [] of NBY pointers to out text */
+FIELD	_arg_keys,	8*NBY, 8	/* [] of NBY pointers to keys */
+FIELD	_arg_IV,	16*NBY, 16	/* [] of NBY 128-bit IV's */
+
+END_FIELDS
+
+_AES_ARGS_SIZE	=	_FIELD_OFFSET
+_AES_ARGS_ALIGN	=	_STRUCT_ALIGN
+
+START_FIELDS	/* MB_MGR_AES_INORDER_XX */
+/*	name		size	align				*/
+FIELD	_args,		_AES_ARGS_SIZE, _AES_ARGS_ALIGN
+FIELD	_lens,		16,	16	/* lengths */
+FIELD	_unused_lanes,	8,	8
+FIELD	_job_in_lane,	8*NBY,	8	/* [] of NBY pointers to jobs */
+
+FIELD	_earliest_job,	4,	4	/* -1 if none */
+FIELD	_next_job,	4,	4
+FIELD	_jobs,		_JOB_AES_size,	_JOB_AES_align	/*> 8, start of array */
+
+END_FIELDS
+
+/* size is not accurate because the arrays are not represented */
+
+_MB_MGR_AES_INORDER_align	=	_STRUCT_ALIGN
+
+_args_in	=	_args+_arg_in
+_args_out	=	_args+_arg_out
+_args_keys	=	_args+_arg_keys
+_args_IV	=	_args+_arg_IV
+
+
+#define STS_UNKNOWN		0
+#define STS_BEING_PROCESSED	1
+#define STS_COMPLETED		2
+#define STS_INTERNAL_ERROR	3
+#define STS_ERROR		4
+
+#define MAX_AES_JOBS		128
diff --git a/arch/x86/crypto/aes-cbc-mb/reg_sizes.S b/arch/x86/crypto/aes-cbc-mb/reg_sizes.S
new file mode 100644
index 0000000..eeff13c
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/reg_sizes.S
@@ -0,0 +1,126 @@
+/*
+ *	Header file AES CBC multibuffer SSE optimization (x86_64)
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+/* define d and w variants for registers */
+
+#define	raxd	eax
+#define raxw	ax
+#define raxb	al
+
+#define	rbxd	ebx
+#define rbxw	bx
+#define rbxb	bl
+
+#define	rcxd	ecx
+#define rcxw	cx
+#define rcxb	cl
+
+#define	rdxd	edx
+#define rdxw	dx
+#define rdxb	dl
+
+#define	rsid	esi
+#define rsiw	si
+#define rsib	sil
+
+#define	rdid	edi
+#define rdiw	di
+#define rdib	dil
+
+#define	rbpd	ebp
+#define rbpw	bp
+#define rbpb	bpl
+
+#define ymm0x	%xmm0
+#define ymm1x	%xmm1
+#define ymm2x	%xmm2
+#define ymm3x	%xmm3
+#define ymm4x	%xmm4
+#define ymm5x	%xmm5
+#define ymm6x	%xmm6
+#define ymm7x	%xmm7
+#define ymm8x	%xmm8
+#define ymm9x	%xmm9
+#define ymm10x	%xmm10
+#define ymm11x	%xmm11
+#define ymm12x	%xmm12
+#define ymm13x	%xmm13
+#define ymm14x	%xmm14
+#define ymm15x	%xmm15
+
+#define CONCAT(a,b)	a##b
+#define DWORD(reg)	CONCAT(reg, d)
+#define WORD(reg)	CONCAT(reg, w)
+#define BYTE(reg)	CONCAT(reg, b)
+
+#define XWORD(reg)	CONCAT(reg,x)
+
+/* common macros */
+
+/* Generate a label to go to */
+.macro LABEL prefix, num
+\prefix\num\():
+.endm
+
+/*
+ * cond_jump ins, name, suffix
+ * ins - conditional jump instruction to execute
+ * name,suffix - concatenate to form the label to go to
+ */
+.macro cond_jump ins, name, suffix
+	\ins	\name\suffix
+.endm
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 3/7] crypto: AES CBC multi-buffer scheduler
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
  2017-04-20 20:50 ` [Patch V5 2/7] crypto: AES CBC multi-buffer data structures Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-20 20:50 ` [Patch V5 4/7] crypto: AES CBC by8 encryption Megha Dey
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

This patch implements in-order scheduler for encrypting multiple buffers
in parallel supporting AES CBC encryption with key sizes of
128, 192 and 256 bits. It uses 8 data lanes by taking advantage of the
SIMD instructions with XMM registers.

The multibuffer manager and scheduler is mostly written in assembly and
the initialization support is written C. The AES CBC multibuffer crypto
driver support interfaces with the multibuffer manager and scheduler
to support AES CBC encryption in parallel. The scheduler supports
job submissions, job flushing and and job retrievals after completion.

The basic flow of usage of the CBC multibuffer scheduler is as follows:

- The caller allocates an aes_cbc_mb_mgr_inorder_x8 object
and initializes it once by calling aes_cbc_init_mb_mgr_inorder_x8().

- The aes_cbc_mb_mgr_inorder_x8 structure has an array of JOB_AES
objects. Allocation and scheduling of JOB_AES objects are managed
by the multibuffer scheduler support routines. The caller allocates
a JOB_AES using aes_cbc_get_next_job_inorder_x8().

- The returned JOB_AES must be filled in with parameters for CBC
encryption (eg: plaintext buffer, ciphertext buffer, key, iv, etc) and
submitted to the manager object using aes_cbc_submit_job_inorder_xx().

- If the oldest JOB_AES is completed during a call to
aes_cbc_submit_job_inorder_x8(), it is returned. Otherwise,
NULL is returned.

- A call to aes_cbc_flush_job_inorder_x8() always returns the
oldest job, unless the multibuffer manager is empty of jobs.

- A call to aes_cbc_get_completed_job_inorder_x8() returns
a completed job. This routine is useful to process completed
jobs instead of waiting for the flusher to engage.

- When a job is returned from submit or flush, the caller extracts
the useful data and returns it to the multibuffer manager implicitly
by the next call to aes_cbc_get_next_job_xx().

Jobs are always returned from submit or flush routines in the order they
were submitted (hence "inorder").A job allocated using
aes_cbc_get_next_job_inorder_x8() must be filled in and submitted before
another call. A job returned by aes_cbc_submit_job_inorder_x8() or
aes_cbc_flush_job_inorder_x8() is 'deallocated' upon the next call to
get a job structure. Calls to get_next_job() cannot fail. If all jobs
are
allocated after a call to get_next_job(), the subsequent call to submit
always returns the oldest job in a completed state.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c       | 146 ++++++++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S | 223 +++++++++++
 arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S     | 417 +++++++++++++++++++++
 3 files changed, 786 insertions(+)
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S
 create mode 100644 arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S

diff --git a/arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c b/arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c
new file mode 100644
index 0000000..2a2ce6c
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/aes_mb_mgr_init.c
@@ -0,0 +1,146 @@
+/*
+ * Initialization code for multi buffer AES CBC algorithm
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "aes_cbc_mb_mgr.h"
+
+void aes_cbc_init_mb_mgr_inorder_x8(struct aes_cbc_mb_mgr_inorder_x8 *state)
+{
+	/* Init "out of order" components */
+	state->unused_lanes = 0xF76543210;
+	state->job_in_lane[0] = NULL;
+	state->job_in_lane[1] = NULL;
+	state->job_in_lane[2] = NULL;
+	state->job_in_lane[3] = NULL;
+	state->job_in_lane[4] = NULL;
+	state->job_in_lane[5] = NULL;
+	state->job_in_lane[6] = NULL;
+	state->job_in_lane[7] = NULL;
+
+	/* Init "in order" components */
+	state->next_job = 0;
+	state->earliest_job = -1;
+
+}
+
+#define JOBS(offset) ((struct job_aes_cbc *)(((u64)state->jobs)+offset))
+
+struct job_aes_cbc *
+aes_cbc_get_next_job_inorder_x8(struct aes_cbc_mb_mgr_inorder_x8 *state)
+{
+	return JOBS(state->next_job);
+}
+
+struct job_aes_cbc *
+aes_cbc_flush_job_inorder_x8(struct aes_cbc_mb_mgr_inorder_x8 *state)
+{
+	struct job_aes_cbc *job;
+
+	/* checking earliest_job < 0 fails and the code walks over bogus */
+	if (state->earliest_job == -1)
+		return NULL; /* empty */
+
+	job = JOBS(state->earliest_job);
+	while (job->status != STS_COMPLETED) {
+		switch (job->key_len) {
+		case AES_KEYSIZE_128:
+			aes_cbc_flush_job_ooo_128x8(state);
+			break;
+		case AES_KEYSIZE_192:
+			aes_cbc_flush_job_ooo_192x8(state);
+			break;
+		case AES_KEYSIZE_256:
+			aes_cbc_flush_job_ooo_256x8(state);
+			break;
+		default:
+			break;
+		}
+	}
+
+	/* advance earliest job */
+	state->earliest_job += sizeof(struct job_aes_cbc);
+	if (state->earliest_job == MAX_AES_JOBS * sizeof(struct job_aes_cbc))
+		state->earliest_job = 0;
+
+	if (state->earliest_job == state->next_job)
+		state->earliest_job = -1;
+
+	return job;
+}
+
+struct job_aes_cbc *
+aes_cbc_get_completed_job_inorder_x8(struct aes_cbc_mb_mgr_inorder_x8 *state)
+{
+	struct job_aes_cbc *job;
+
+	if (state->earliest_job == -1)
+		return NULL; /* empty */
+
+	job = JOBS(state->earliest_job);
+	if (job->status != STS_COMPLETED)
+		return NULL;
+
+	state->earliest_job += sizeof(struct job_aes_cbc);
+
+	if (state->earliest_job == MAX_AES_JOBS * sizeof(struct job_aes_cbc))
+		state->earliest_job = 0;
+	if (state->earliest_job == state->next_job)
+		state->earliest_job = -1;
+
+	/* we have a completed job */
+	return job;
+}
diff --git a/arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S b/arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S
new file mode 100644
index 0000000..5108875
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/mb_mgr_inorder_x8_asm.S
@@ -0,0 +1,223 @@
+/*
+ *	AES CBC by8 multibuffer inorder scheduler optimization (x86_64)
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#include <linux/linkage.h>
+#include "mb_mgr_datastruct.S"
+#include "reg_sizes.S"
+
+#define JUMP
+
+#define arg1	%rdi
+#define arg2	%rsi
+#define state	arg1
+
+/* virtual registers used by submit_job_aes_inorder_x8 */
+#define next_job	%rdx
+#define earliest_job	%rcx
+#define zero		%r8
+#define returned_job	%rax	/* register that returns a value from func */
+
+.extern aes_cbc_submit_job_ooo_128x8
+.extern aes_cbc_submit_job_ooo_192x8
+.extern aes_cbc_submit_job_ooo_256x8
+.extern aes_cbc_flush_job_ooo_128x8
+.extern aes_cbc_flush_job_ooo_192x8
+.extern aes_cbc_flush_job_ooo_256x8
+.extern aes_cbc_flush_job_ooo_x8
+
+/*
+ * struct job_aes* aes_cbc_submit_job_inorder_x8(
+ *		struct aes_cbc_mb_mgr_inorder_x8 *state)
+ */
+
+.macro aes_cbc_submit_job_inorder_x8 key_len
+
+	sub	$8, %rsp	/* align stack for next calls */
+
+	mov	_next_job(state), DWORD(next_job)
+	lea	_jobs(state, next_job), arg2
+
+	.if \key_len == AES_KEYSIZE_128
+		call	aes_cbc_submit_job_ooo_128x8
+	.elseif \key_len == AES_KEYSIZE_192
+		call	aes_cbc_submit_job_ooo_192x8
+	.elseif \key_len == AES_KEYSIZE_256
+		call	aes_cbc_submit_job_ooo_256x8
+	.endif
+
+	mov	_earliest_job(state), DWORD(earliest_job)
+	cmp	$0, DWORD(earliest_job)
+	jl	.Lstate_was_empty\key_len
+
+	/* we have a valid earliest_job */
+
+	/* advance next_job */
+	mov	_next_job(state), DWORD(next_job)
+	add	$_JOB_AES_size, next_job
+#ifdef JUMP
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), next_job
+	jne	.Lskip1\key_len
+	xor	next_job, next_job
+.Lskip1\key_len:
+#else
+	xor	zero,zero
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), next_job
+	cmove	zero, next_job
+#endif
+	mov	DWORD(next_job), _next_job(state)
+
+	lea	_jobs(state, earliest_job), returned_job
+	cmp	next_job, earliest_job
+	je	.Lfull\key_len
+
+	/* not full */
+	cmpl	$STS_COMPLETED, _status(returned_job)
+	jne	.Lreturn_null\key_len
+
+	/* advance earliest_job */
+	add	$_JOB_AES_size, earliest_job
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), earliest_job
+#ifdef JUMP
+	jne	.Lskip2\key_len
+	xor	earliest_job, earliest_job
+.Lskip2\key_len:
+#else
+	cmove	zero, earliest_job
+#endif
+
+	add	$8, %rsp
+	mov	DWORD(earliest_job), _earliest_job(state)
+	ret
+
+.Lreturn_null\key_len:
+	add	$8, %rsp
+	xor	returned_job, returned_job
+	ret
+
+.Lfull\key_len:
+	cmpl	$STS_COMPLETED, _status(returned_job)
+	je	.Lcompleted\key_len
+	mov	earliest_job, (%rsp)
+.Lflush_loop\key_len:
+	.if \key_len == AES_KEYSIZE_128
+		call	aes_cbc_flush_job_ooo_128x8
+	.elseif \key_len == AES_KEYSIZE_192
+		call	aes_cbc_flush_job_ooo_192x8
+	.elseif \key_len == AES_KEYSIZE_256
+		call	aes_cbc_flush_job_ooo_256x8
+	.endif
+	/* state is still valid */
+	mov	(%rsp), earliest_job
+	cmpl	$STS_COMPLETED, _status(returned_job)
+	jne	.Lflush_loop\key_len
+	xor	zero,zero
+.Lcompleted\key_len:
+	/* advance earliest_job */
+	add	$_JOB_AES_size, earliest_job
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), earliest_job
+#ifdef JUMP
+	jne	.Lskip3\key_len
+	xor	earliest_job, earliest_job
+.Lskip3\key_len:
+#else
+	cmove	zero, earliest_job
+#endif
+
+	add	$8, %rsp
+	mov	DWORD(earliest_job), _earliest_job(state)
+	ret
+
+.Lstate_was_empty\key_len:
+	mov	_next_job(state), DWORD(next_job)
+	mov	DWORD(next_job), _earliest_job(state)
+
+	/* advance next_job */
+	add	$_JOB_AES_size, next_job
+#ifdef JUMP
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), next_job
+	jne	.Lskip4\key_len
+	xor	next_job, next_job
+.Lskip4\key_len:
+#else
+	xor	zero,zero
+	cmp	$(MAX_AES_JOBS * _JOB_AES_size), next_job
+	cmove	zero, next_job
+#endif
+	mov	DWORD(next_job), _next_job(state)
+
+	add	$8, %rsp
+	xor	returned_job, returned_job
+	ret
+.endm
+
+ENTRY(aes_cbc_submit_job_inorder_128x8)
+
+	aes_cbc_submit_job_inorder_x8 AES_KEYSIZE_128
+
+ENDPROC(aes_cbc_submit_job_inorder_128x8)
+
+ENTRY(aes_cbc_submit_job_inorder_192x8)
+
+	aes_cbc_submit_job_inorder_x8 AES_KEYSIZE_192
+
+ENDPROC(aes_cbc_submit_job_inorder_192x8)
+
+ENTRY(aes_cbc_submit_job_inorder_256x8)
+
+	aes_cbc_submit_job_inorder_x8 AES_KEYSIZE_256
+
+ENDPROC(aes_cbc_submit_job_inorder_256x8)
diff --git a/arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S b/arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S
new file mode 100644
index 0000000..351f30f
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/mb_mgr_ooo_x8_asm.S
@@ -0,0 +1,417 @@
+/*
+ *	AES CBC by8 multibuffer out-of-order scheduler optimization (x86_64)
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include <linux/linkage.h>
+#include "mb_mgr_datastruct.S"
+#include "reg_sizes.S"
+
+#define arg1	%rdi
+#define arg2	%rsi
+#define state	arg1
+#define job	arg2
+
+/* virtual registers used by aes_cbc_submit_job_ooo_x8 */
+#define unused_lanes	%rax
+#define lane		%rdx
+#define	tmp1		%rcx
+#define	tmp2		%r8
+#define	tmp3		%r9
+#define len		tmp1
+
+#define good_lane	lane
+
+/* virtual registers used by aes_cbc_submit_job_inorder_x8 */
+#define new_job		%rdx
+#define earliest_job	%rcx
+#define minus1		%r8
+#define returned_job	%rax	/* register that returns a value from func */
+
+.section .rodata
+.align 16
+len_masks:
+	.octa 0x0000000000000000000000000000FFFF
+	.octa 0x000000000000000000000000FFFF0000
+	.octa 0x00000000000000000000FFFF00000000
+	.octa 0x0000000000000000FFFF000000000000
+	.octa 0x000000000000FFFF0000000000000000
+	.octa 0x00000000FFFF00000000000000000000
+	.octa 0x0000FFFF000000000000000000000000
+	.octa 0xFFFF0000000000000000000000000000
+
+dupw:
+	.octa 0x01000100010001000100010001000100
+
+one:	.quad  1
+two:	.quad  2
+three:	.quad  3
+four:	.quad  4
+five:	.quad  5
+six:	.quad  6
+seven:	.quad  7
+
+.text
+
+.extern aes_cbc_enc_128_x8
+.extern aes_cbc_enc_192_x8
+.extern aes_cbc_enc_256_x8
+
+/* arg1/state remains intact after call */
+/*
+ * void aes_cbc_submit_job__ooo_128x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state,
+ *	struct job_aes_cbc *job);
+ * void aes_cbc_submit_job__ooo_192x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state,
+ *	struct job_aes_cbc *job);
+ * void aes_cbc_submit_job__ooo_256x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state,
+ *	struct job_aes_cbc *job);
+ */
+
+.global aes_cbc_submit_job_ooo_128x8
+.global aes_cbc_submit_job_ooo_192x8
+.global aes_cbc_submit_job_ooo_256x8
+
+
+.macro aes_cbc_submit_job_ooo_x8 key_len
+
+	mov	_unused_lanes(state), unused_lanes
+	mov	unused_lanes, lane
+	and	$0xF, lane
+	shr	$4, unused_lanes
+
+	/* state->job_in_lane[lane] = job; */
+	mov	job, _job_in_lane(state, lane, 8)
+
+	/*state->lens[lane] = job->len / AES_BLOCK_SIZE; */
+	mov	_len(job), DWORD(len)
+	shr	$4, len
+	mov	WORD(len), _lens(state, lane, 2)
+
+	mov	_plaintext(job), tmp1
+	mov	_ciphertext(job), tmp2
+	mov	_keys(job), tmp3
+	movdqu	_IV(job), %xmm2
+	mov	tmp1, _args_in(state, lane, 8)
+	mov	tmp2, _args_out(state , lane, 8)
+	mov	tmp3, _args_keys(state, lane, 8)
+	shl	$4, lane
+	movdqa	%xmm2, _args_IV(state , lane)
+
+	movl	$STS_BEING_PROCESSED, _status(job)
+
+	mov	unused_lanes, _unused_lanes(state)
+	cmp	$0xF, unused_lanes
+	jne	.Lnot_enough_jobs\key_len
+
+	movdqa	_lens(state), %xmm0
+	phminposuw	%xmm0, %xmm1
+	/*
+	 * xmm1{15:0} = min value
+	 * xmm1{18:16} = index of min
+	 */
+
+	/*
+	 * arg1 = rcx = state = args (and it is not clobbered by routine)
+	 * arg2 = rdx = min len
+	 */
+	movd	%xmm1, arg2
+	and	$0xFFFF, arg2
+
+	/* subtract min len from lengths */
+	pshufb	dupw(%rip), %xmm1	/* duplicate words across all lanes */
+	psubw	%xmm1, %xmm0
+	movdqa	%xmm0, _lens(state)
+
+	/* need to align stack */
+	sub	$8, %rsp
+	.if \key_len == AES_KEYSIZE_128
+		call aes_cbc_enc_128_x8
+	.elseif \key_len == AES_KEYSIZE_192
+		call aes_cbc_enc_192_x8
+	.elseif \key_len == AES_KEYSIZE_256
+		call aes_cbc_enc_256_x8
+	.endif
+	add	$8, %rsp
+	/* arg1/state is still intact */
+
+	/* process completed jobs */
+	movdqa	_lens(state), %xmm0
+	phminposuw	%xmm0, %xmm1
+	/*
+	 * xmm1{15:0} = min value
+	 * xmm1{18:16} = index of min
+	 */
+
+	/*
+	 * at this point at least one len should be 0
+	 * so min value should be 0
+	 * and the index is the index of that lane [0...7]
+	 */
+	lea	len_masks(%rip), tmp3
+	mov	_unused_lanes(state), unused_lanes
+	movd	%xmm1, lane
+.Lcontinue_loop\key_len:
+	/* assert((lane & 0xFFFF) == 0) */
+	shr	$16, lane	/* lane is now index */
+	mov	_job_in_lane(state, lane, 8), job
+	movl	$STS_COMPLETED, _status(job)
+	movq	$0, _job_in_lane(state ,lane, 8)
+	shl	$4, unused_lanes
+	or	lane, unused_lanes
+	shl	$4, lane
+	movdqa	_args_IV(state,lane), %xmm2
+	movdqu	%xmm2, _IV(job)
+	por	(tmp3, lane), %xmm0
+
+	phminposuw	%xmm0, %xmm1
+	movd	%xmm1, lane
+	/* see if bits 15:0 are zero */
+	test	$0xFFFF, lane
+	jz	.Lcontinue_loop\key_len
+
+	/* done; save registers */
+	mov	unused_lanes, _unused_lanes(state)
+	/* don't need to save xmm0/lens */
+
+.Lnot_enough_jobs\key_len:
+	ret
+.endm
+
+ENTRY(aes_cbc_submit_job_ooo_128x8)
+
+	aes_cbc_submit_job_ooo_x8 AES_KEYSIZE_128
+
+ENDPROC(aes_cbc_submit_job_ooo_128x8)
+
+ENTRY(aes_cbc_submit_job_ooo_192x8)
+
+	aes_cbc_submit_job_ooo_x8 AES_KEYSIZE_192
+
+ENDPROC(aes_cbc_submit_job_ooo_192x8)
+
+ENTRY(aes_cbc_submit_job_ooo_256x8)
+
+	aes_cbc_submit_job_ooo_x8 AES_KEYSIZE_256
+
+ENDPROC(aes_cbc_submit_job_ooo_256x8)
+
+
+/* arg1/state remains intact after call */
+/*
+ * void aes_cbc_flush_job_ooo_128x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state)
+ * void aes_cbc_flush_job_ooo_192x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state)
+ * void aes_cbc_flush_job_ooo_256x8(
+ *	struct aes_cbc_mb_mgr_aes_inorder_x8 *state)
+ */
+.global aes_cbc_flush_job_ooo_128x8
+.global aes_cbc_flush_job_ooo_192x8
+.global aes_cbc_flush_job_ooo_256x8
+
+.macro aes_cbc_flush_job_ooo_x8 key_len
+
+	mov	_unused_lanes(state), unused_lanes
+
+	/* if bit (32+3) is set, then all lanes are empty */
+	bt	$(32+3), unused_lanes
+	jc	.Lreturn\key_len
+
+	/* find a lane with a non-null job */
+	xor	good_lane, good_lane
+	cmpq	$0, (_job_in_lane+8*1)(state)
+	cmovne	one(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*2)(state)
+	cmovne	two(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*3)(state)
+	cmovne	three(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*4)(state)
+	cmovne	four(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*5)(state)
+	cmovne	five(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*6)(state)
+	cmovne	six(%rip), good_lane
+	cmpq	$0, (_job_in_lane+8*7)(state)
+	cmovne	seven(%rip), good_lane
+
+	/* copy good_lane to empty lanes */
+	mov	_args_in(state, good_lane, 8), tmp1
+	mov	_args_out(state, good_lane, 8), tmp2
+	mov	_args_keys(state, good_lane, 8), tmp3
+	shl	$4, good_lane
+	movdqa	_args_IV(state, good_lane), %xmm2
+
+	movdqa	_lens(state), %xmm0
+
+	I = 0
+
+.altmacro
+	.rept 8
+		cmpq	$0, (_job_in_lane + 8*I)(state)
+		.if \key_len == AES_KEYSIZE_128
+			cond_jump jne, .Lskip128_,%I
+		.elseif \key_len == AES_KEYSIZE_192
+			cond_jump jne, .Lskip192_,%I
+		.elseif \key_len == AES_KEYSIZE_256
+			cond_jump jne, .Lskip256_,%I
+		.endif
+		mov	tmp1, (_args_in + 8*I)(state)
+		mov	tmp2, (_args_out + 8*I)(state)
+		mov	tmp3, (_args_keys + 8*I)(state)
+		movdqa	%xmm2, (_args_IV + 16*I)(state)
+		por	(len_masks + 16*I)(%rip), %xmm0
+		.if \key_len == AES_KEYSIZE_128
+			LABEL .Lskip128_,%I
+		.elseif \key_len == AES_KEYSIZE_192
+			LABEL .Lskip192_,%I
+		.elseif \key_len == AES_KEYSIZE_256
+			LABEL .Lskip256_,%I
+		.endif
+		I = (I+1)
+	.endr
+.noaltmacro
+
+	phminposuw	%xmm0, %xmm1
+	/*
+	 * xmm1{15:0} = min value
+	 * xmm1{18:16} = index of min
+	 */
+
+	/*
+	 * arg1 = rcx = state = args (and it is not clobbered by routine)
+	 * arg2 = rdx = min len
+	 */
+	movd	%xmm1, arg2
+	and	$0xFFFF, arg2
+
+	/* subtract min len from lengths */
+	pshufb	dupw(%rip), %xmm1	/* duplicate words across all lanes */
+	psubw	%xmm1, %xmm0
+	movdqa	%xmm0, _lens(state)
+
+	/* need to align stack */
+	sub	$8, %rsp
+	.if \key_len == AES_KEYSIZE_128
+		call aes_cbc_enc_128_x8
+	.elseif \key_len == AES_KEYSIZE_192
+		call aes_cbc_enc_192_x8
+	.elseif \key_len == AES_KEYSIZE_256
+		call aes_cbc_enc_256_x8
+	.endif
+	add	$8, %rsp
+	/* arg1/state is still intact */
+
+	/* process completed jobs */
+	movdqa	_lens(state), %xmm0
+	phminposuw	%xmm0, %xmm1
+	/*
+	 * xmm1{15:0} = min value
+	 * xmm1{18:16} = index of min
+	 */
+
+	/*
+	 * at this point at least one len should be 0, so min value should be 0
+	 * and the index is the index of that lane [0...3]
+	 */
+	lea	len_masks(%rip), tmp3
+	mov	_unused_lanes(state), unused_lanes
+	movd	%xmm1, lane
+.Lcontinue_loop2\key_len:
+	/* assert((lane & 0xFFFF) == 0) */
+	shr	$16, lane	/* lane is now index */
+	mov	_job_in_lane(state, lane, 8), job
+	movl	$STS_COMPLETED, _status(job)
+	movq	$0, _job_in_lane(state, lane, 8)
+	shl	$4, unused_lanes
+	or	lane, unused_lanes
+	shl	$4, lane
+	movdqa	_args_IV(state, lane), %xmm2
+	movdqu	%xmm2, _IV(job)
+	por	(tmp3, lane), %xmm0
+
+	phminposuw	%xmm0, %xmm1
+	movd	%xmm1, lane
+	/* see if bits 15:0 are zero */
+	test	$0xFFFF, lane
+	jz	.Lcontinue_loop2\key_len
+
+	/* done; save registers */
+	mov	unused_lanes, _unused_lanes(state)
+	/* don't need to save xmm0/lens */
+.Lreturn\key_len:
+	ret
+.endm
+
+ENTRY(aes_cbc_flush_job_ooo_128x8)
+
+	aes_cbc_flush_job_ooo_x8 AES_KEYSIZE_128
+
+ENDPROC(aes_cbc_flush_job_ooo_128x8)
+
+ENTRY(aes_cbc_flush_job_ooo_192x8)
+
+	aes_cbc_flush_job_ooo_x8 AES_KEYSIZE_192
+
+ENDPROC(aes_cbc_flush_job_ooo_192x8)
+
+ENTRY(aes_cbc_flush_job_ooo_256x8)
+
+	aes_cbc_flush_job_ooo_x8 AES_KEYSIZE_256
+
+ENDPROC(aes_cbc_flush_job_ooo_256x8)
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 4/7] crypto: AES CBC by8 encryption
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
                   ` (2 preceding siblings ...)
  2017-04-20 20:50 ` [Patch V5 3/7] crypto: AES CBC multi-buffer scheduler Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-20 20:50 ` [Patch V5 5/7] crypto: AES CBC multi-buffer glue code Megha Dey
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

This patch introduces the assembly routine to do a by8 AES CBC encryption
in support of the AES CBC multi-buffer implementation.

It encrypts 8 data streams of the same key size simultaneously.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S | 775 ++++++++++++++++++++++++++++
 1 file changed, 775 insertions(+)
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S

diff --git a/arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S b/arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S
new file mode 100644
index 0000000..2130574
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/aes_cbc_enc_x8.S
@@ -0,0 +1,775 @@
+/*
+ *	AES CBC by8 multibuffer optimization (x86_64)
+ *	This file implements 128/192/256 bit AES CBC encryption
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#include <linux/linkage.h>
+
+/* stack size needs to be an odd multiple of 8 for alignment */
+
+#define AES_KEYSIZE_128        16
+#define AES_KEYSIZE_192        24
+#define AES_KEYSIZE_256        32
+
+#define XMM_SAVE_SIZE	16*10
+#define GPR_SAVE_SIZE	8*9
+#define STACK_SIZE	(XMM_SAVE_SIZE + GPR_SAVE_SIZE)
+
+#define GPR_SAVE_REG	%rsp
+#define GPR_SAVE_AREA	%rsp + XMM_SAVE_SIZE
+#define LEN_AREA_OFFSET	XMM_SAVE_SIZE + 8*8
+#define LEN_AREA_REG	%rsp
+#define LEN_AREA	%rsp + XMM_SAVE_SIZE + 8*8
+
+#define IN_OFFSET	0
+#define OUT_OFFSET	8*8
+#define KEYS_OFFSET	16*8
+#define IV_OFFSET	24*8
+
+
+#define IDX	%rax
+#define TMP	%rbx
+#define ARG	%rdi
+#define LEN	%rsi
+
+#define KEYS0	%r14
+#define KEYS1	%r15
+#define KEYS2	%rbp
+#define KEYS3	%rdx
+#define KEYS4	%rcx
+#define KEYS5	%r8
+#define KEYS6	%r9
+#define KEYS7	%r10
+
+#define IN0	%r11
+#define IN2	%r12
+#define IN4	%r13
+#define IN6	LEN
+
+#define XDATA0	%xmm0
+#define XDATA1	%xmm1
+#define XDATA2	%xmm2
+#define XDATA3	%xmm3
+#define XDATA4	%xmm4
+#define XDATA5	%xmm5
+#define XDATA6	%xmm6
+#define XDATA7	%xmm7
+
+#define XKEY0_3	%xmm8
+#define XKEY1_4	%xmm9
+#define XKEY2_5	%xmm10
+#define XKEY3_6	%xmm11
+#define XKEY4_7	%xmm12
+#define XKEY5_8	%xmm13
+#define XKEY6_9	%xmm14
+#define XTMP	%xmm15
+
+#define	MOVDQ movdqu /* assume buffers not aligned */
+#define CONCAT(a, b)	a##b
+#define INPUT_REG_SUFX	1	/* IN */
+#define XDATA_REG_SUFX	2	/* XDAT */
+#define KEY_REG_SUFX	3	/* KEY */
+#define XMM_REG_SUFX	4	/* XMM */
+
+/*
+ * To avoid positional parameter errors while compiling
+ * three registers need to be passed
+ */
+.text
+
+.macro pxor2 x, y, z
+	MOVDQ	(\x,\y), XTMP
+	pxor	XTMP, \z
+.endm
+
+.macro inreg n
+	.if (\n == 0)
+		reg_IN = IN0
+	.elseif (\n == 2)
+		reg_IN = IN2
+	.elseif (\n == 4)
+		reg_IN = IN4
+	.elseif (\n == 6)
+		reg_IN = IN6
+	.else
+		error "inreg: incorrect register number"
+	.endif
+.endm
+.macro xdatareg n
+	.if (\n == 0)
+		reg_XDAT = XDATA0
+	.elseif (\n == 1)
+		reg_XDAT = XDATA1
+	.elseif (\n == 2)
+		reg_XDAT = XDATA2
+	.elseif (\n == 3)
+		reg_XDAT = XDATA3
+	.elseif (\n == 4)
+		reg_XDAT = XDATA4
+	.elseif (\n == 5)
+		reg_XDAT = XDATA5
+	.elseif (\n == 6)
+		reg_XDAT = XDATA6
+	.elseif (\n == 7)
+		reg_XDAT = XDATA7
+	.endif
+.endm
+.macro xkeyreg n
+	.if (\n == 0)
+		reg_KEY = KEYS0
+	.elseif (\n == 1)
+		reg_KEY = KEYS1
+	.elseif (\n == 2)
+		reg_KEY = KEYS2
+	.elseif (\n == 3)
+		reg_KEY = KEYS3
+	.elseif (\n == 4)
+		reg_KEY = KEYS4
+	.elseif (\n == 5)
+		reg_KEY = KEYS5
+	.elseif (\n == 6)
+		reg_KEY = KEYS6
+	.elseif (\n == 7)
+		reg_KEY = KEYS7
+	.endif
+.endm
+.macro xmmreg n
+	.if (\n >= 0) && (\n < 16)
+		/* Valid register number */
+		reg_XMM = %xmm\n
+	.else
+		error "xmmreg: incorrect register number"
+	.endif
+.endm
+
+/*
+ * suffix - register suffix
+ * set up the register name using the loop index I
+ */
+.macro define_reg suffix
+.altmacro
+	.if (\suffix == INPUT_REG_SUFX)
+		inreg  %I
+	.elseif (\suffix == XDATA_REG_SUFX)
+		xdatareg  %I
+	.elseif (\suffix == KEY_REG_SUFX)
+		xkeyreg  %I
+	.elseif (\suffix == XMM_REG_SUFX)
+		xmmreg  %I
+	.else
+		error "define_reg: unknown register suffix"
+	.endif
+.noaltmacro
+.endm
+
+/*
+ * aes_cbc_enc_x8 key_len
+ * macro to encode data for 128bit, 192bit and 256bit keys
+ */
+
+.macro aes_cbc_enc_x8 key_len
+
+	sub	$STACK_SIZE, %rsp
+
+	mov	%rbx, (XMM_SAVE_SIZE + 8*0)(GPR_SAVE_REG)
+	mov	%rbp, (XMM_SAVE_SIZE + 8*3)(GPR_SAVE_REG)
+	mov	%r12, (XMM_SAVE_SIZE + 8*4)(GPR_SAVE_REG)
+	mov	%r13, (XMM_SAVE_SIZE + 8*5)(GPR_SAVE_REG)
+	mov	%r14, (XMM_SAVE_SIZE + 8*6)(GPR_SAVE_REG)
+	mov	%r15, (XMM_SAVE_SIZE + 8*7)(GPR_SAVE_REG)
+
+	mov	$16, IDX
+	shl	$4, LEN	/* LEN = LEN * 16 */
+	/* LEN is now in terms of bytes */
+	mov	LEN, (LEN_AREA_OFFSET)(LEN_AREA_REG)
+
+	/* Run through storing arguments in IN0,2,4,6 */
+	I = 0
+	.rept 4
+		define_reg	INPUT_REG_SUFX
+		mov	(IN_OFFSET + 8*I)(ARG), reg_IN
+		I = (I + 2)
+	.endr
+
+	/* load 1 .. 8 blocks of plain text into XDATA0..XDATA7 */
+	I = 0
+	.rept 4
+		mov		(IN_OFFSET + 8*(I+1))(ARG), TMP
+		define_reg	INPUT_REG_SUFX
+		define_reg	XDATA_REG_SUFX
+		/* load first block of plain text */
+		MOVDQ		(reg_IN), reg_XDAT
+		I = (I + 1)
+		define_reg	XDATA_REG_SUFX
+		/* load next block of plain text */
+		MOVDQ		(TMP), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* Run through XDATA0 .. XDATA7 to perform plaintext XOR IV */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		pxor	(IV_OFFSET + 16*I)(ARG), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	I = 0
+	.rept 8
+		define_reg	KEY_REG_SUFX
+		mov	(KEYS_OFFSET + 8*I)(ARG), reg_KEY
+		I = (I + 1)
+	.endr
+
+	I = 0
+	/* 0..7 ARK */
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		pxor	16*0(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	I = 0
+		/* 1. ENC */
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*1)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	movdqa		16*3(KEYS0), XKEY0_3	/* load round 3 key */
+
+	I = 0
+		/* 2. ENC */
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*2)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	movdqa		16*4(KEYS1), XKEY1_4	/* load round 4 key */
+
+		/* 3. ENC */
+	aesenc		XKEY0_3, XDATA0
+	I = 1
+	.rept 7
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*3)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/*
+	 * FIXME:
+	 * why can't we reorder encrypt DATA0..DATA7 and load 5th round?
+	 */
+	aesenc		(16*4)(KEYS0), XDATA0		/* 4. ENC */
+	movdqa		16*5(KEYS2), XKEY2_5		/* load round 5 key */
+	aesenc		XKEY1_4, XDATA1			/* 4. ENC */
+
+	I = 2
+	.rept 6
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*4)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	aesenc		(16*5)(KEYS0), XDATA0		/* 5. ENC */
+	aesenc		(16*5)(KEYS1), XDATA1		/* 5. ENC */
+	movdqa		16*6(KEYS3), XKEY3_6		/* load round 6 key */
+	aesenc		XKEY2_5, XDATA2			/* 5. ENC */
+	I = 3
+	.rept 5
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*5)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	aesenc		(16*6)(KEYS0), XDATA0		/* 6. ENC */
+	aesenc		(16*6)(KEYS1), XDATA1		/* 6. ENC */
+	aesenc		(16*6)(KEYS2), XDATA2		/* 6. ENC */
+	movdqa		16*7(KEYS4), XKEY4_7		/* load round 7 key */
+	aesenc		XKEY3_6, XDATA3			/* 6. ENC */
+
+	I = 4
+	.rept 4
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*6)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	I = 0
+	.rept 4
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*7)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	movdqa		16*8(KEYS5), XKEY5_8		/* load round 8 key */
+	aesenc		XKEY4_7, XDATA4			/* 7. ENC */
+	I = 5
+	.rept 3
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*7)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	I = 0
+	.rept 5
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*8)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	movdqa		16*9(KEYS6), XKEY6_9		/* load round 9 key */
+	aesenc		XKEY5_8, XDATA5			/* 8. ENC */
+	aesenc		16*8(KEYS6), XDATA6		/* 8. ENC */
+	aesenc		16*8(KEYS7), XDATA7		/* 8. ENC */
+
+	I = 0
+	.rept 6
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	(16*9)(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	mov		(OUT_OFFSET + 8*0)(ARG), TMP
+	aesenc		XKEY6_9, XDATA6			/* 9. ENC */
+	aesenc		16*9(KEYS7), XDATA7		/* 9. ENC */
+
+		/* 10. ENC (last for 128bit keys) */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		.if (\key_len == AES_KEYSIZE_128)
+			aesenclast	(16*10)(reg_KEY), reg_XDAT
+		.else
+			aesenc	(16*10)(reg_KEY), reg_XDAT
+		.endif
+		I = (I + 1)
+	.endr
+
+	.if (\key_len != AES_KEYSIZE_128)
+		/* 11. ENC */
+		I = 0
+		.rept 8
+			define_reg	XDATA_REG_SUFX
+			define_reg	KEY_REG_SUFX
+			aesenc	(16*11)(reg_KEY), reg_XDAT
+			I = (I + 1)
+		.endr
+
+		/* 12. ENC (last for 192bit key) */
+		I = 0
+		.rept 8
+			define_reg	XDATA_REG_SUFX
+			define_reg	KEY_REG_SUFX
+			.if (\key_len == AES_KEYSIZE_192)
+				aesenclast	(16*12)(reg_KEY), reg_XDAT
+			.else
+				aesenc		(16*12)(reg_KEY), reg_XDAT
+			.endif
+			I = (I + 1)
+		.endr
+
+		/* for 256bit, two more rounds */
+		.if \key_len == AES_KEYSIZE_256
+
+			/* 13. ENC */
+			I = 0
+			.rept 8
+				define_reg	XDATA_REG_SUFX
+				define_reg	KEY_REG_SUFX
+				aesenc	(16*13)(reg_KEY), reg_XDAT
+				I = (I + 1)
+			.endr
+
+			/* 14. ENC last encode for 256bit key */
+			I = 0
+			.rept 8
+				define_reg	XDATA_REG_SUFX
+				define_reg	KEY_REG_SUFX
+				aesenclast	(16*14)(reg_KEY), reg_XDAT
+				I = (I + 1)
+			.endr
+		.endif
+
+	.endif
+
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		MOVDQ	reg_XDAT, (TMP)	/* write back ciphertext */
+		I = (I + 1)
+		.if (I < 8)
+			mov	(OUT_OFFSET + 8*I)(ARG), TMP
+		.endif
+	.endr
+
+	cmp		IDX, LEN_AREA_OFFSET(LEN_AREA_REG)
+	je		.Ldone\key_len
+
+.Lmain_loop\key_len:
+	mov		(IN_OFFSET + 8*1)(ARG), TMP
+	pxor2		IN0, IDX, XDATA0	/* next block of plain text */
+	pxor2		TMP, IDX, XDATA1	/* next block of plain text */
+
+	mov		(IN_OFFSET + 8*3)(ARG), TMP
+	pxor2		IN2, IDX, XDATA2	/* next block of plain text */
+	pxor2		TMP, IDX, XDATA3	/* next block of plain text */
+
+	mov		(IN_OFFSET + 8*5)(ARG), TMP
+	pxor2		IN4, IDX, XDATA4	/* next block of plain text */
+	pxor2		TMP, IDX, XDATA5	/* next block of plain text */
+
+	mov		(IN_OFFSET + 8*7)(ARG), TMP
+	pxor2		IN6, IDX, XDATA6	/* next block of plain text */
+	pxor2		TMP, IDX, XDATA7	/* next block of plain text */
+
+	/* 0. ARK */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		pxor	16*0(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 1. ENC */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*1(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 2. ENC */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*2(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 3. ENC */
+	aesenc		XKEY0_3, XDATA0		/* 3. ENC */
+	I = 1
+	.rept 7
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*3(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 4. ENC */
+	aesenc		16*4(KEYS0), XDATA0	/* 4. ENC */
+	aesenc		XKEY1_4, XDATA1		/* 4. ENC */
+	I = 2
+	.rept 6
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*4(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 5. ENC */
+	aesenc		16*5(KEYS0), XDATA0	/* 5. ENC */
+	aesenc		16*5(KEYS1), XDATA1	/* 5. ENC */
+	aesenc		XKEY2_5, XDATA2		/* 5. ENC */
+
+	I = 3
+	.rept 5
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*5(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 6. ENC */
+	I = 0
+	.rept 3
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*6(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	aesenc		XKEY3_6, XDATA3		/* 6. ENC */
+	I = 4
+	.rept 4
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*6(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 7. ENC */
+	I = 0
+	.rept 4
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*7(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	aesenc		XKEY4_7, XDATA4		/* 7. ENC */
+	I = 5
+	.rept 3
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*7(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+
+	/* 8. ENC */
+	I = 0
+	.rept 5
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*8(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	aesenc		XKEY5_8, XDATA5		/* 8. ENC */
+	aesenc		16*8(KEYS6), XDATA6	/* 8. ENC */
+	aesenc		16*8(KEYS7), XDATA7	/* 8. ENC */
+
+	/* 9. ENC */
+	I = 0
+	.rept 6
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		aesenc	16*9(reg_KEY), reg_XDAT
+		I = (I + 1)
+	.endr
+	mov		(OUT_OFFSET + 8*0)(ARG), TMP
+	aesenc		XKEY6_9, XDATA6		/* 9. ENC */
+	aesenc		16*9(KEYS7), XDATA7	/* 9. ENC */
+
+	/* 10. ENC (last for 128 bit key) */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		define_reg	KEY_REG_SUFX
+		.if (\key_len == AES_KEYSIZE_128)
+			aesenclast	16*10(reg_KEY), reg_XDAT
+		.else
+			aesenc	16*10(reg_KEY), reg_XDAT
+		.endif
+		I = (I + 1)
+	.endr
+
+	.if (\key_len != AES_KEYSIZE_128)
+		/* 11. ENC */
+		I = 0
+		.rept 8
+			define_reg	XDATA_REG_SUFX
+			define_reg	KEY_REG_SUFX
+			aesenc	16*11(reg_KEY), reg_XDAT
+			I = (I + 1)
+		.endr
+
+		/* 12. last ENC for 192bit key */
+		I = 0
+		.rept 8
+			define_reg	XDATA_REG_SUFX
+			define_reg	KEY_REG_SUFX
+			.if (\key_len == AES_KEYSIZE_192)
+				aesenclast	16*12(reg_KEY), reg_XDAT
+			.else
+				aesenc		16*12(reg_KEY), reg_XDAT
+			.endif
+			I = (I + 1)
+		.endr
+
+		.if \key_len == AES_KEYSIZE_256
+			/* for 256bit, two more rounds */
+			/* 13. ENC */
+			I = 0
+			.rept 8
+				define_reg	XDATA_REG_SUFX
+				define_reg	KEY_REG_SUFX
+				aesenc	16*13(reg_KEY), reg_XDAT
+				I = (I + 1)
+			.endr
+
+			/* 14. last ENC for 256bit key */
+			I = 0
+			.rept 8
+				define_reg	XDATA_REG_SUFX
+				define_reg	KEY_REG_SUFX
+				aesenclast	16*14(reg_KEY), reg_XDAT
+				I = (I + 1)
+			.endr
+		.endif
+	.endif
+
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		/* write back cipher text */
+		MOVDQ		reg_XDAT, (TMP , IDX)
+		I = (I + 1)
+		.if (I < 8)
+			mov		(OUT_OFFSET + 8*I)(ARG), TMP
+		.endif
+	.endr
+
+	add	$16, IDX
+	cmp	IDX, LEN_AREA_OFFSET(LEN_AREA_REG)
+	jne	.Lmain_loop\key_len
+
+.Ldone\key_len:
+	/* update IV */
+	I = 0
+	.rept 8
+		define_reg	XDATA_REG_SUFX
+		movdqa	reg_XDAT, (IV_OFFSET + 16*I)(ARG)
+		I = (I + 1)
+	.endr
+
+	/* update IN and OUT */
+	movd	LEN_AREA_OFFSET(LEN_AREA_REG), %xmm0
+	pshufd	$0x44, %xmm0, %xmm0
+
+	I = 1
+	.rept 4
+		define_reg	XMM_REG_SUFX
+		movdqa	(IN_OFFSET + 16*(I-1))(ARG), reg_XMM
+		I = (I + 1)
+	.endr
+
+	paddq	%xmm0, %xmm1
+	paddq	%xmm0, %xmm2
+	paddq	%xmm0, %xmm3
+	paddq	%xmm0, %xmm4
+
+	I = 5
+	.rept 4
+		define_reg	XMM_REG_SUFX
+		movdqa	(OUT_OFFSET + 16*(I-5))(ARG), reg_XMM
+		I = (I + 1)
+	.endr
+
+	I = 1
+	.rept 4
+		define_reg	XMM_REG_SUFX
+		movdqa	reg_XMM, (IN_OFFSET + 16*(I-1))(ARG)
+		I = (I + 1)
+	.endr
+
+	paddq	%xmm0, %xmm5
+	paddq	%xmm0, %xmm6
+	paddq	%xmm0, %xmm7
+	paddq	%xmm0, %xmm8
+
+	I = 5
+	.rept 4
+		define_reg	XMM_REG_SUFX
+		movdqa	reg_XMM, (OUT_OFFSET + 16*(I-5))(ARG)
+		I = (I + 1)
+	.endr
+
+	mov	(XMM_SAVE_SIZE + 8*0)(GPR_SAVE_REG), %rbx
+	mov	(XMM_SAVE_SIZE + 8*3)(GPR_SAVE_REG), %rbp
+	mov	(XMM_SAVE_SIZE + 8*4)(GPR_SAVE_REG), %r12
+	mov	(XMM_SAVE_SIZE + 8*5)(GPR_SAVE_REG), %r13
+	mov	(XMM_SAVE_SIZE + 8*6)(GPR_SAVE_REG), %r14
+	mov	(XMM_SAVE_SIZE + 8*7)(GPR_SAVE_REG), %r15
+
+	add	$STACK_SIZE, %rsp
+
+	ret
+.endm
+
+/*
+ * AES CBC encryption routine supporting 128/192/256 bit keys
+ *
+ * void aes_cbc_enc_128_x8(struct aes_cbc_args_x8 *args, u64 len);
+ * arg 1: rcx : addr of AES_ARGS_x8 structure
+ * arg 2: rdx : len (in units of 16-byte blocks)
+ * void aes_cbc_enc_192_x8(struct aes_cbc_args_x8 *args, u64 len);
+ * arg 1: rcx : addr of aes_cbc_args_x8 structure
+ * arg 2: rdx : len (in units of 16-byte blocks)
+ * void aes_cbc_enc_256_x8(struct aes_cbc_args_x8 *args, u64 len);
+ * arg 1: rcx : addr of aes_cbc_args_x8 structure
+ * arg 2: rdx : len (in units of 16-byte blocks)
+ */
+
+ENTRY(aes_cbc_enc_128_x8)
+
+	aes_cbc_enc_x8 AES_KEYSIZE_128
+
+ENDPROC(aes_cbc_enc_128_x8)
+
+ENTRY(aes_cbc_enc_192_x8)
+
+	aes_cbc_enc_x8 AES_KEYSIZE_192
+
+ENDPROC(aes_cbc_enc_192_x8)
+
+ENTRY(aes_cbc_enc_256_x8)
+
+	aes_cbc_enc_x8 AES_KEYSIZE_256
+
+ENDPROC(aes_cbc_enc_256_x8)
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 5/7] crypto: AES CBC multi-buffer glue code
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
                   ` (3 preceding siblings ...)
  2017-04-20 20:50 ` [Patch V5 4/7] crypto: AES CBC by8 encryption Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-20 20:50 ` [Patch V5 6/7] crypto: AES vectors for AES CBC multibuffer testing Megha Dey
  2017-04-20 20:50 ` [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt Megha Dey
  6 siblings, 0 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

This patch introduces the multi-buffer job manager which is responsible
for submitting scatter-gather buffers from several AES CBC jobs
to the multi-buffer algorithm. The glue code interfaces with the
underlying algorithm that handles 8 data streams of AES CBC encryption
in parallel. AES key expansion and CBC decryption requests are performed
in a manner similar to the existing AESNI Intel glue driver.

The outline of the algorithm for AES CBC encryption requests is
sketched below:

Any driver requesting the crypto service will place an async crypto
request on the workqueue.  The multi-buffer crypto daemon will pull an
AES CBC encryption request from work queue and put each request in an
empty data lane for multi-buffer crypto computation.  When all the empty
lanes are filled, computation will commence on the jobs in parallel and
the job with the shortest remaining buffer will get completed and be
returned. To prevent prolonged stall, when no new jobs arrive, we will
flush workqueue of jobs after a maximum allowable delay has elapsed.

To accommodate the fragmented nature of scatter-gather, we will keep
submitting the next scatter-buffer fragment for a job for multi-buffer
computation until a job is completed and no more buffer fragments
remain.
At that time we will pull a new job to fill the now empty data slot.
We check with the multibuffer scheduler to see if there are other
completed jobs to prevent extraneous delay in returning any completed
jobs.

This multi-buffer algorithm should be used for cases where we get at
least 8 streams of crypto jobs submitted at a reasonably high rate.
For low crypto job submission rate and low number of data streams, this
algorithm will not be beneficial. The reason is at low rate, we do not
fill out the data lanes before flushing the jobs instead of processing
them with all the data lanes full.  We will miss the benefit of parallel
computation, and adding delay to the processing of the crypto job at the
same time.  Some tuning of the maximum latency parameter may be needed
to get the best performance.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 arch/x86/crypto/Makefile                |   1 +
 arch/x86/crypto/aes-cbc-mb/Makefile     |  22 +
 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c | 737 ++++++++++++++++++++++++++++++++
 3 files changed, 760 insertions(+)
 create mode 100644 arch/x86/crypto/aes-cbc-mb/Makefile
 create mode 100644 arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c

diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 34b3fa2..cc556a7 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -33,6 +33,7 @@ obj-$(CONFIG_CRYPTO_CRC32_PCLMUL) += crc32-pclmul.o
 obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o
 obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
 obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o
+obj-$(CONFIG_CRYPTO_AES_CBC_MB) += aes-cbc-mb/
 obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o
 
 # These modules require assembler to support AVX.
diff --git a/arch/x86/crypto/aes-cbc-mb/Makefile b/arch/x86/crypto/aes-cbc-mb/Makefile
new file mode 100644
index 0000000..b642bd8
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/Makefile
@@ -0,0 +1,22 @@
+#
+# Arch-specific CryptoAPI modules.
+#
+
+avx_supported := $(call as-instr,vpxor %xmm0$(comma)%xmm0$(comma)%xmm0,yes,no)
+
+# we need decryption and key expansion routine symbols
+# if either AESNI_NI_INTEL or AES_CBC_MB is a module
+
+ifeq ($(CONFIG_CRYPTO_AES_NI_INTEL),m)
+	dec_support := ../aesni-intel_asm.o
+endif
+ifeq ($(CONFIG_CRYPTO_AES_CBC_MB),m)
+	dec_support := ../aesni-intel_asm.o
+endif
+
+ifeq ($(avx_supported),yes)
+	obj-$(CONFIG_CRYPTO_AES_CBC_MB) += aes-cbc-mb.o
+	aes-cbc-mb-y := $(dec_support) aes_cbc_mb.o aes_mb_mgr_init.o \
+			mb_mgr_inorder_x8_asm.o mb_mgr_ooo_x8_asm.o \
+			aes_cbc_enc_x8.o
+endif
diff --git a/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c
new file mode 100644
index 0000000..0d30383
--- /dev/null
+++ b/arch/x86/crypto/aes-cbc-mb/aes_cbc_mb.c
@@ -0,0 +1,737 @@
+/*
+ *	Multi buffer AES CBC algorithm glue code
+ *
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * James Guilford <james.guilford@intel.com>
+ * Sean Gulley <sean.m.gulley@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ * Megha Dey <megha.dey@linux.intel.com>
+ */
+
+#define pr_fmt(fmt)	KBUILD_MODNAME ": " fmt
+#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_EXTRA)
+#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
+#include <linux/hardirq.h>
+#include <linux/types.h>
+#include <linux/crypto.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/internal/hash.h>
+#include <crypto/mcryptd.h>
+#include <crypto/crypto_wq.h>
+#include <crypto/ctr.h>
+#include <crypto/b128ops.h>
+#include <crypto/lrw.h>
+#include <crypto/xts.h>
+#include <asm/cpu_device_id.h>
+#include <asm/fpu/api.h>
+#include <asm/crypto/aes.h>
+#include <crypto/ablk_helper.h>
+#include <crypto/scatterwalk.h>
+#include <crypto/internal/aead.h>
+#include <linux/workqueue.h>
+#include <linux/spinlock.h>
+#include <crypto/internal/simd.h>
+#include <crypto/internal/skcipher.h>
+#ifdef CONFIG_X86_64
+#include <asm/crypto/glue_helper.h>
+#endif
+#include <asm/simd.h>
+
+#include "aes_cbc_mb_ctx.h"
+
+#define AESNI_ALIGN	(16)
+#define AES_BLOCK_MASK	(~(AES_BLOCK_SIZE-1))
+#define FLUSH_INTERVAL 500 /* in usec */
+
+static struct mcryptd_alg_state cbc_mb_alg_state;
+
+struct aes_cbc_mb_ctx {
+	struct mcryptd_skcipher *mcryptd_tfm;
+};
+
+static inline struct aes_cbc_mb_mgr_inorder_x8
+	*get_key_mgr(void *mgr, u32 key_len)
+{
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+
+	key_mgr = (struct aes_cbc_mb_mgr_inorder_x8 *) mgr;
+	/* valid keysize is guranteed to be one of 128/192/256 */
+	switch (key_len) {
+	case AES_KEYSIZE_256:
+		return key_mgr+2;
+	case AES_KEYSIZE_192:
+		return key_mgr+1;
+	case AES_KEYSIZE_128:
+	default:
+		return key_mgr;
+	}
+}
+
+/* support code from arch/x86/crypto/aesni-intel_glue.c */
+static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx)
+{
+	unsigned long addr = (unsigned long)raw_ctx;
+	unsigned long align = AESNI_ALIGN;
+	struct crypto_aes_ctx *ret_ctx;
+
+	if (align <= crypto_tfm_ctx_alignment())
+		align = 1;
+	ret_ctx = (struct crypto_aes_ctx *)ALIGN(addr, align);
+	return ret_ctx;
+}
+
+static struct job_aes_cbc *aes_cbc_job_mgr_submit(
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr, u32 key_len)
+{
+	/* valid keysize is guranteed to be one of 128/192/256 */
+	switch (key_len) {
+	case AES_KEYSIZE_256:
+		return aes_cbc_submit_job_inorder_256x8(key_mgr);
+	case AES_KEYSIZE_192:
+		return aes_cbc_submit_job_inorder_192x8(key_mgr);
+	case AES_KEYSIZE_128:
+	default:
+		return aes_cbc_submit_job_inorder_128x8(key_mgr);
+	}
+}
+
+static inline struct skcipher_request *cast_mcryptd_ctx_to_req(
+	struct mcryptd_skcipher_request_ctx *ctx)
+{
+	return container_of((void *) ctx, struct skcipher_request, __ctx);
+}
+
+/*
+ * Interface functions to the synchronous algorithm with acces
+ * to the underlying multibuffer AES CBC implementation
+ */
+
+/* Map the error in request context appropriately */
+
+static struct mcryptd_skcipher_request_ctx *process_job_sts(
+		struct job_aes_cbc *job)
+{
+	struct mcryptd_skcipher_request_ctx *ret_rctx;
+
+	ret_rctx = (struct mcryptd_skcipher_request_ctx *)job->user_data;
+
+	switch (job->status) {
+	default:
+	case STS_COMPLETED:
+		ret_rctx->error = CBC_CTX_ERROR_NONE;
+		break;
+	case STS_BEING_PROCESSED:
+		ret_rctx->error = -EINPROGRESS;
+		break;
+	case STS_INTERNAL_ERROR:
+	case STS_ERROR:
+	case STS_UNKNOWN:
+		/* mark it done with error */
+		ret_rctx->flag = CBC_DONE;
+		ret_rctx->error = -EIO;
+		break;
+	}
+	return ret_rctx;
+}
+
+static struct mcryptd_skcipher_request_ctx
+	*aes_cbc_ctx_mgr_flush(struct aes_cbc_mb_mgr_inorder_x8 *key_mgr)
+{
+	struct job_aes_cbc *job;
+
+	job = aes_cbc_flush_job_inorder_x8(key_mgr);
+	if (job)
+		return process_job_sts(job);
+	return NULL;
+}
+
+static struct mcryptd_skcipher_request_ctx *aes_cbc_ctx_mgr_submit(
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr,
+	struct mcryptd_skcipher_request_ctx *rctx
+	)
+{
+	struct crypto_aes_ctx *mb_key_ctx;
+	struct job_aes_cbc *job;
+	unsigned long src_paddr;
+	unsigned long dst_paddr;
+
+	mb_key_ctx = aes_ctx(crypto_tfm_ctx(rctx->desc.base.tfm));
+
+	/* get job, fill the details and submit */
+	job = aes_cbc_get_next_job_inorder_x8(key_mgr);
+
+	src_paddr = (page_to_phys(rctx->walk.src.phys.page) +
+						rctx->walk.src.phys.offset);
+	dst_paddr = (page_to_phys(rctx->walk.dst.phys.page) +
+						rctx->walk.dst.phys.offset);
+	job->plaintext = phys_to_virt(src_paddr);
+	job->ciphertext = phys_to_virt(dst_paddr);
+	if (rctx->flag & CBC_START) {
+		/* fresh sequence, copy iv from walk buffer initially */
+		memcpy(&job->iv, rctx->walk.iv, AES_BLOCK_SIZE);
+		rctx->flag &= ~CBC_START;
+	} else {
+		/* For a multi-part sequence, set up the updated IV */
+		job->iv = rctx->seq_iv;
+	}
+
+	job->keys = (u128 *)mb_key_ctx->key_enc;
+	/* set up updated length from the walk buffers */
+	job->len = rctx->walk.nbytes & AES_BLOCK_MASK;
+	/* stow away the req_ctx so we can later check */
+	job->user_data = (void *)rctx;
+	job->key_len = mb_key_ctx->key_length;
+	rctx->job = job;
+	rctx->error = CBC_CTX_ERROR_NONE;
+	job = aes_cbc_job_mgr_submit(key_mgr, mb_key_ctx->key_length);
+	if (job) {
+		/* we already have the request context stashed in job */
+		return process_job_sts(job);
+	}
+	return NULL;
+}
+
+static int cbc_encrypt_finish(struct mcryptd_skcipher_request_ctx **ret_rctx,
+			      struct mcryptd_alg_cstate *cstate,
+			      bool flush)
+{
+	struct mcryptd_skcipher_request_ctx *rctx = *ret_rctx;
+	struct job_aes_cbc *job;
+	int err = 0;
+	unsigned int nbytes;
+	struct crypto_aes_ctx *mb_key_ctx;
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+	struct skcipher_request *req;
+
+
+	mb_key_ctx = aes_ctx(crypto_tfm_ctx(rctx->desc.base.tfm));
+	key_mgr = get_key_mgr(cstate->mgr, mb_key_ctx->key_length);
+
+	/*
+	 * Some low-level mb job is done. Keep going till done.
+	 * This loop may process multiple multi part requests
+	 */
+	while (!(rctx->flag & CBC_DONE)) {
+		/* update bytes and check for more work */
+		nbytes = rctx->walk.nbytes & (AES_BLOCK_SIZE - 1);
+		req = cast_mcryptd_ctx_to_req(rctx);
+		err = skcipher_walk_done(&rctx->walk, nbytes);
+		if (err) {
+			/* done with error */
+			rctx->flag = CBC_DONE;
+			rctx->error = err;
+			goto out;
+		}
+		nbytes = rctx->walk.nbytes;
+		if (!nbytes) {
+			/* done with successful encryption */
+			rctx->flag = CBC_DONE;
+			goto out;
+		}
+		/*
+		 * This is a multi-part job and there is more work to do.
+		 * From the completed job, copy the running sequence of IV
+		 * and start the next one in sequence.
+		 */
+		job = (struct job_aes_cbc *)rctx->job;
+		rctx->seq_iv = job->iv;	/* copy the running sequence of iv */
+		kernel_fpu_begin();
+		rctx = aes_cbc_ctx_mgr_submit(key_mgr, rctx);
+		if (!rctx) {
+			/* multi part job submitted, no completed job. */
+			if (flush)
+				rctx = aes_cbc_ctx_mgr_flush(key_mgr);
+		}
+		kernel_fpu_end();
+		if (!rctx) {
+			/* no completions yet to process further */
+			break;
+		}
+		/* some job finished when we submitted multi part job. */
+		if (rctx->error) {
+			/*
+			 * some request completed with error
+			 * bail out of chain processing
+			 */
+			err = rctx->error;
+			break;
+		}
+		/* we have a valid request context to process further */
+	}
+	/* encrypted text is expected to be in out buffer already */
+out:
+	/* We came out multi-part processing for some request */
+	*ret_rctx = rctx;
+	return err;
+}
+
+/* notify the caller of progress ; request still stays in queue */
+
+static void notify_callback(struct mcryptd_skcipher_request_ctx *rctx,
+			    struct mcryptd_alg_cstate *cstate,
+			    int err)
+{
+	struct skcipher_request *req = cast_mcryptd_ctx_to_req(rctx);
+
+	if (irqs_disabled())
+		rctx->complete(&req->base, err);
+	else {
+		local_bh_disable();
+		rctx->complete(&req->base, err);
+		local_bh_enable();
+	}
+}
+
+/* A request that completed is dequeued and the caller is notified */
+
+static void completion_callback(struct mcryptd_skcipher_request_ctx *rctx,
+			    struct mcryptd_alg_cstate *cstate,
+			    int err)
+{
+	struct skcipher_request *req = cast_mcryptd_ctx_to_req(rctx);
+
+       /* remove from work list and invoke completion callback */
+	spin_lock(&cstate->work_lock);
+	list_del(&rctx->waiter);
+	spin_unlock(&cstate->work_lock);
+
+	if (irqs_disabled())
+		rctx->complete(&req->base, err);
+	else {
+		local_bh_disable();
+		rctx->complete(&req->base, err);
+		local_bh_enable();
+	}
+}
+
+/* complete a blkcipher request and process any further completions */
+
+static void cbc_complete_job(struct mcryptd_skcipher_request_ctx *rctx,
+			    struct mcryptd_alg_cstate *cstate,
+			    int err)
+{
+	struct job_aes_cbc *job;
+	int ret;
+	struct mcryptd_skcipher_request_ctx *sctx;
+	struct crypto_aes_ctx *mb_key_ctx;
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+	struct skcipher_request *req;
+
+	req = cast_mcryptd_ctx_to_req(rctx);
+	skcipher_walk_complete(&rctx->walk, err);
+	completion_callback(rctx, cstate, err);
+
+	mb_key_ctx = aes_ctx(crypto_tfm_ctx(rctx->desc.base.tfm));
+	key_mgr = get_key_mgr(cstate->mgr, mb_key_ctx->key_length);
+
+	/* check for more completed jobs and process */
+	while ((job = aes_cbc_get_completed_job_inorder_x8(key_mgr)) != NULL) {
+		sctx = process_job_sts(job);
+		if (WARN_ON(sctx == NULL))
+			return;
+		ret = sctx->error;
+		if (!ret) {
+			/* further process it */
+			ret = cbc_encrypt_finish(&sctx, cstate, false);
+		}
+		if (sctx) {
+			req = cast_mcryptd_ctx_to_req(sctx);
+			skcipher_walk_complete(&sctx->walk, err);
+			completion_callback(sctx, cstate, ret);
+		}
+	}
+}
+
+/* Add request to the waiter list. It stays in queue until completion */
+
+static void cbc_mb_add_list(struct mcryptd_skcipher_request_ctx *rctx,
+				struct mcryptd_alg_cstate *cstate)
+{
+	unsigned long next_flush;
+	unsigned long delay = usecs_to_jiffies(FLUSH_INTERVAL);
+
+	/* initialize tag */
+	rctx->tag.arrival = jiffies;    /* tag the arrival time */
+	rctx->tag.seq_num = cstate->next_seq_num++;
+	next_flush = rctx->tag.arrival + delay;
+	rctx->tag.expire = next_flush;
+
+	spin_lock(&cstate->work_lock);
+	list_add_tail(&rctx->waiter, &cstate->work_list);
+	spin_unlock(&cstate->work_lock);
+
+	mcryptd_arm_flusher(cstate, delay);
+}
+
+static int mb_aes_cbc_encrypt(struct skcipher_request *desc)
+{
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(desc);
+	struct mcryptd_skcipher_request_ctx *rctx =
+		container_of(desc, struct mcryptd_skcipher_request_ctx, desc);
+	struct mcryptd_skcipher_request_ctx *ret_rctx;
+	struct mcryptd_alg_cstate *cstate =
+				this_cpu_ptr(cbc_mb_alg_state.alg_cstate);
+	int err;
+	int ret = 0;
+	struct crypto_aes_ctx *mb_key_ctx;
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+	struct skcipher_request *req;
+	unsigned int nbytes;
+
+	mb_key_ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+	key_mgr = get_key_mgr(cstate->mgr, mb_key_ctx->key_length);
+
+	/* sanity check */
+	if (rctx->tag.cpu != smp_processor_id()) {
+		/* job not on list yet */
+		pr_err("mcryptd error: cpu clash\n");
+		notify_callback(rctx, cstate, -EINVAL);
+		return 0;
+	}
+
+	/* a new job, initialize the cbc context and add to worklist */
+	cbc_ctx_init(rctx, nbytes, CBC_ENCRYPT);
+	cbc_mb_add_list(rctx, cstate);
+
+	req = cast_mcryptd_ctx_to_req(rctx);
+
+	err = skcipher_walk_async(&rctx->walk, req);
+	if (err || !rctx->walk.nbytes) {
+		/* terminate this request */
+		skcipher_walk_complete(&rctx->walk, err);
+		completion_callback(rctx, cstate, (!err) ? -EINVAL : err);
+		return 0;
+	}
+	/* submit job */
+	kernel_fpu_begin();
+	ret_rctx = aes_cbc_ctx_mgr_submit(key_mgr, rctx);
+	kernel_fpu_end();
+
+	if (!ret_rctx) {
+		/* we submitted a job, but none completed */
+		/* just notify the caller */
+		notify_callback(rctx, cstate, -EINPROGRESS);
+		return 0;
+	}
+	/* some job completed */
+	if (ret_rctx->error) {
+		/* some job finished with error */
+		cbc_complete_job(ret_rctx, cstate, ret_rctx->error);
+		return 0;
+	}
+	/* some job finished without error, process it */
+	ret = cbc_encrypt_finish(&ret_rctx, cstate, false);
+	if (!ret_rctx) {
+		/* No completed job yet, notify caller */
+		notify_callback(rctx, cstate, -EINPROGRESS);
+		return 0;
+	}
+
+	/* complete the job */
+	cbc_complete_job(ret_rctx, cstate, ret);
+	return 0;
+}
+
+static int mb_aes_cbc_decrypt(struct skcipher_request *desc)
+{
+	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(desc);
+	struct crypto_aes_ctx *aesni_ctx;
+	struct mcryptd_skcipher_request_ctx *rctx =
+		container_of(desc, struct mcryptd_skcipher_request_ctx, desc);
+	struct skcipher_request *req;
+	bool is_mcryptd_req;
+	unsigned long src_paddr;
+	unsigned long dst_paddr;
+	unsigned int nbytes;
+	int err;
+
+	/* note here whether it is mcryptd req */
+	is_mcryptd_req = desc->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP;
+	req = cast_mcryptd_ctx_to_req(rctx);
+	aesni_ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+
+	err = skcipher_walk_async(&rctx->walk, req);
+	if (err || !rctx->walk.nbytes)
+		goto done1;
+
+	kernel_fpu_begin();
+	while ((nbytes = rctx->walk.nbytes)) {
+		src_paddr = (page_to_phys(rctx->walk.src.phys.page) +
+						rctx->walk.src.phys.offset);
+		dst_paddr = (page_to_phys(rctx->walk.dst.phys.page) +
+						rctx->walk.dst.phys.offset);
+		aesni_cbc_dec(aesni_ctx, phys_to_virt(dst_paddr),
+					phys_to_virt(src_paddr),
+					rctx->walk.nbytes & AES_BLOCK_MASK,
+					rctx->walk.iv);
+		nbytes &= AES_BLOCK_SIZE - 1;
+		err = skcipher_walk_done(&rctx->walk, nbytes);
+		if (err)
+			goto done2;
+	}
+done2:
+	kernel_fpu_end();
+done1:
+	skcipher_walk_complete(&rctx->walk, err);
+	if (!is_mcryptd_req) {
+		/* synchronous request */
+		return err;
+	}
+	/* from mcryptd, we need to callback */
+	if (irqs_disabled())
+		rctx->complete(&req->base, err);
+	else {
+		local_bh_disable();
+		rctx->complete(&req->base, err);
+		local_bh_enable();
+	}
+	return 0;
+}
+
+/* use the same common code in aesni to expand key */
+
+static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
+			      const u8 *in_key, unsigned int key_len)
+{
+	struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx);
+	u32 *flags = &tfm->crt_flags;
+	int err;
+
+	if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 &&
+	    key_len != AES_KEYSIZE_256) {
+		*flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
+		return -EINVAL;
+	}
+
+	if (!irq_fpu_usable())
+		err = crypto_aes_expand_key(ctx, in_key, key_len);
+	else {
+		kernel_fpu_begin();
+		err = aesni_set_key(ctx, in_key, key_len);
+		kernel_fpu_end();
+	}
+
+	return err;
+}
+
+static int aes_set_key(struct crypto_skcipher *tfm, const u8 *in_key,
+		       unsigned int key_len)
+{
+	return aes_set_key_common(crypto_skcipher_tfm(tfm),
+				crypto_skcipher_ctx(tfm), in_key, key_len);
+}
+
+/*
+ * CRYPTO_ALG_ASYNC flag is passed to indicate we have an ablk
+ * scatter-gather walk.
+ */
+static struct skcipher_alg aes_cbc_mb_alg = {
+	.base = {
+		.cra_name		= "__cbc(aes)",
+		.cra_driver_name	= "__cbc-aes-aesni-mb",
+		.cra_priority		= 500,
+		.cra_flags		= CRYPTO_ALG_INTERNAL,
+		.cra_blocksize		= AES_BLOCK_SIZE,
+		.cra_ctxsize		= CRYPTO_AES_CTX_SIZE,
+		.cra_module		= THIS_MODULE,
+	},
+	.min_keysize	= AES_MIN_KEY_SIZE,
+	.max_keysize	= AES_MAX_KEY_SIZE,
+	.ivsize		= AES_BLOCK_SIZE,
+	.setkey		= aes_set_key,
+	.encrypt	= mb_aes_cbc_encrypt,
+	.decrypt	= mb_aes_cbc_decrypt
+};
+
+/*
+ * When there are no new jobs arriving, the multibuffer queue may stall.
+ * To prevent prolonged stall, the flusher can be invoked to alleviate
+ * the following conditions:
+ * a) There are partially completed multi-part crypto jobs after a
+ * maximum allowable delay
+ * b) We have exhausted crypto jobs in queue, and the cpu
+ * does not have other tasks and cpu will become idle otherwise.
+ */
+unsigned long cbc_mb_flusher(struct mcryptd_alg_cstate *cstate)
+{
+	struct mcryptd_skcipher_request_ctx *rctx;
+	unsigned long cur_time;
+	unsigned long next_flush = 0;
+
+	struct crypto_aes_ctx *mb_key_ctx;
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+
+
+	cur_time = jiffies;
+
+	while (!list_empty(&cstate->work_list)) {
+		rctx = list_entry(cstate->work_list.next,
+				struct mcryptd_skcipher_request_ctx, waiter);
+		if time_before(cur_time, rctx->tag.expire)
+			break;
+
+		mb_key_ctx = aes_ctx(crypto_tfm_ctx(rctx->desc.base.tfm));
+		key_mgr = get_key_mgr(cstate->mgr, mb_key_ctx->key_length);
+
+		kernel_fpu_begin();
+		rctx = aes_cbc_ctx_mgr_flush(key_mgr);
+		kernel_fpu_end();
+		if (!rctx) {
+			pr_err("cbc_mb_flusher: nothing got flushed\n");
+			break;
+		}
+		cbc_encrypt_finish(&rctx, cstate, true);
+		if (rctx)
+			cbc_complete_job(rctx, cstate, rctx->error);
+	}
+
+	if (!list_empty(&cstate->work_list)) {
+		rctx = list_entry(cstate->work_list.next,
+				struct mcryptd_skcipher_request_ctx, waiter);
+		/* get the blkcipher context and then flush time */
+		next_flush = rctx->tag.expire;
+		mcryptd_arm_flusher(cstate, get_delay(next_flush));
+	}
+	return next_flush;
+}
+struct simd_skcipher_alg *aes_cbc_mb_simd_skciphers;
+
+static int __init aes_cbc_mb_mod_init(void)
+{
+	struct simd_skcipher_alg *simd;
+	const char *basename;
+	const char *algname;
+	const char *drvname;
+
+	int cpu, i;
+	int err;
+	struct mcryptd_alg_cstate *cpu_state;
+	struct aes_cbc_mb_mgr_inorder_x8 *key_mgr;
+
+	/* check for dependent cpu features */
+	if (!boot_cpu_has(X86_FEATURE_AES)) {
+		pr_err("aes_cbc_mb_mod_init: no aes support\n");
+		err = -ENODEV;
+		goto err1;
+	}
+
+	if (!boot_cpu_has(X86_FEATURE_XMM)) {
+		pr_err("aes_cbc_mb_mod_init: no xmm support\n");
+		err = -ENODEV;
+		goto err1;
+	}
+
+	/* initialize multibuffer structures */
+
+	cbc_mb_alg_state.alg_cstate = alloc_percpu(struct mcryptd_alg_cstate);
+	if (!cbc_mb_alg_state.alg_cstate) {
+		pr_err("aes_cbc_mb_mod_init: insufficient memory\n");
+		err = -ENOMEM;
+		goto err1;
+	}
+
+	for_each_possible_cpu(cpu) {
+		cpu_state = per_cpu_ptr(cbc_mb_alg_state.alg_cstate, cpu);
+		cpu_state->next_flush = 0;
+		cpu_state->next_seq_num = 0;
+		cpu_state->flusher_engaged = false;
+		INIT_DELAYED_WORK(&cpu_state->flush, mcryptd_flusher);
+		cpu_state->cpu = cpu;
+		cpu_state->alg_state = &cbc_mb_alg_state;
+		cpu_state->mgr =
+			(struct aes_cbc_mb_mgr_inorder_x8 *)
+			kzalloc(3 * sizeof(struct aes_cbc_mb_mgr_inorder_x8),
+				GFP_KERNEL);
+		if (!cpu_state->mgr) {
+			err = -ENOMEM;
+			goto err2;
+		}
+		key_mgr = (struct aes_cbc_mb_mgr_inorder_x8 *) cpu_state->mgr;
+		/* initialize manager state for 128, 192 and 256 bit keys */
+		for (i = 0; i < 3; ++i) {
+			aes_cbc_init_mb_mgr_inorder_x8(key_mgr);
+			++key_mgr;
+		}
+		INIT_LIST_HEAD(&cpu_state->work_list);
+		spin_lock_init(&cpu_state->work_lock);
+	}
+	cbc_mb_alg_state.flusher = &cbc_mb_flusher;
+
+	/* register the synchronous mb algo */
+	err = crypto_register_skcipher(&aes_cbc_mb_alg);
+	if (err)
+		goto err3;
+
+	algname = aes_cbc_mb_alg.base.cra_name + 2;
+	drvname = aes_cbc_mb_alg.base.cra_driver_name + 2;
+	basename = aes_cbc_mb_alg.base.cra_driver_name;
+
+	simd = simd_skcipher_create_compat_mb(algname, drvname, basename);
+	err = PTR_ERR(simd);
+
+	if (IS_ERR(simd))
+		goto unregister_simds;
+
+	aes_cbc_mb_simd_skciphers = simd;
+
+	pr_info("x86 CBC multibuffer crypto module initialized successfully\n");
+	return 0; /* module init success */
+
+	/* error in algo registration */
+unregister_simds:
+	simd_skcipher_free(aes_cbc_mb_simd_skciphers);
+err3:
+	for_each_possible_cpu(cpu) {
+		cpu_state = per_cpu_ptr(cbc_mb_alg_state.alg_cstate, cpu);
+		kfree(cpu_state->mgr);
+	}
+err2:
+	free_percpu(cbc_mb_alg_state.alg_cstate);
+err1:
+	return err;
+}
+
+static void __exit aes_cbc_mb_mod_fini(void)
+{
+	int cpu;
+	struct mcryptd_alg_cstate *cpu_state;
+
+	simd_skcipher_free(aes_cbc_mb_simd_skciphers);
+	crypto_unregister_skcipher(&aes_cbc_mb_alg);
+
+	for_each_possible_cpu(cpu) {
+		cpu_state = per_cpu_ptr(cbc_mb_alg_state.alg_cstate, cpu);
+		kfree(cpu_state->mgr);
+	}
+	free_percpu(cbc_mb_alg_state.alg_cstate);
+}
+
+module_init(aes_cbc_mb_mod_init);
+module_exit(aes_cbc_mb_mod_fini);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("AES CBC Algorithm, multi buffer accelerated");
+MODULE_AUTHOR("Tim Chen <tim.c.chen@linux.intel.com");
+
+MODULE_ALIAS("aes-cbc-mb");
+MODULE_ALIAS_CRYPTO("cbc-aes-aesni-mb");
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 6/7] crypto: AES vectors for AES CBC multibuffer testing
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
                   ` (4 preceding siblings ...)
  2017-04-20 20:50 ` [Patch V5 5/7] crypto: AES CBC multi-buffer glue code Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-20 20:50 ` [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt Megha Dey
  6 siblings, 0 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

For more robust testing of AES CBC multibuffer support, additional
test vectors have been added to the AES CBC encrypt/decrypt
test case.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 crypto/testmgr.h | 1456 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 1456 insertions(+)

diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index 15c043f..85679b3 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -34413,4 +34413,1460 @@ struct comp_testvec {
 	},
 };
 
+#ifdef CONFIG_CRYPTO_AES_CBC_MB
+static struct cipher_testvec aes_cbc_enc_tv_template_rnddata_klen16[] = {
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc1\x62\x66\x62\xb8\x65\x28\xfa\x5f\x36\xd3\x09\xb1\x2c\xa1\xa3",
+	.input =
+"\x4f\x6c\x63\xa5\xd0\x19\x08\x4e\xd4\x58\x33\xf6\x2b\xeb\x26\xb9",
+	.ilen = 16,
+	.result =
+"\xa0\x35\xb0\x33\xc0\x2e\xe5\xbb\xbc\xe6\x01\x9e\xf4\x67\x11\x14",
+	.rlen = 16,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x78\x6c\x27\xd6\xb2\xdc\xbe\x7b\xab\xc2\x43\xd7\x81\x0c\xe5\x20",
+	.input =
+"\x9a\x00\x4e\x5a\xb3\x51\x68\xaa\xdb\x6e\xe5\xa4\x7f\x23\x6e\x4d"
+"\x1e\x72\x5e\xad\x64\xc9\x96\x23\xf8\xae\xef\xf6\x7b\x7d\xd6\xf0",
+	.ilen = 32,
+	.result =
+"\x5a\xc0\x04\xc6\x53\xef\x3b\x69\xb1\x41\xc7\x85\xeb\x69\x82\xd0"
+"\xed\x09\xbb\xec\xb2\x8d\x5c\xc9\x61\x81\x5c\xf6\x99\x49\xa0\x4d",
+	.rlen = 32,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc9\x05\x4c\x35\x96\x77\xd3\x3c\x3d\x97\x7c\x82\xf5\x58\x71\xf1",
+	.input =
+"\xa9\x5b\x03\xec\xec\x73\xed\xcb\x5c\x4c\xd2\x40\xb6\x9b\x49\x31"
+"\x5d\xf2\x23\xb3\x11\x98\xeb\x89\xab\x3e\x3a\xdd\xaa\xfd\xd1\xde"
+"\xab\x73\x59\x86\x1a\x59\x32\xb2\x55\x46\x4a\x80\xa4\xcc\xa8\xd9",
+	.ilen = 48,
+	.result =
+"\xdb\x05\x69\xe1\x33\x8b\x0b\x3d\x33\x12\x0d\xef\x94\x0f\xa3\xb3"
+"\xd7\x0a\x53\x7b\x98\x53\xc6\xc2\xa3\xd4\x7a\x30\x1a\xed\x45\xcc"
+"\x47\x38\xc1\x75\x0b\x3c\xd4\x8d\xa8\xf9\xd3\x71\xb8\x22\xa6\xae",
+	.rlen = 48,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x6c\xb4\x84\x61\x1e\x39\x4b\x22\x37\xaa\x7b\x78\xc0\x71\x20\x60",
+	.input =
+"\x05\x43\x76\x1e\xc6\x68\x43\x52\x5f\x43\x39\xbf\x93\x38\x38\x83"
+"\x38\x1d\x3c\xb5\xc8\xab\xe4\xd0\x7f\x1a\xac\xca\xcc\x16\xea\x75"
+"\x30\x75\x40\xe8\x61\x07\xc6\x04\x55\x2b\xf3\x29\xc3\x37\x83\x42"
+"\xe0\x21\xfb\xb4\x5d\x93\xbb\x87\x01\x3e\xa6\x9d\x3b\x0a\x5a\x37",
+	.ilen = 64,
+	.result =
+"\x83\x9f\xa0\xac\x14\x14\x88\x68\x7f\x9a\x5f\x98\x91\x71\xa8\xce"
+"\x28\xfb\x5e\xb1\x49\xe7\x63\x39\x12\x62\x00\x3e\x5c\x63\x2b\x12"
+"\x3d\xff\xd5\x0a\x43\x28\x52\x68\x78\x62\xc7\xa4\xbb\xca\x5d\x5e"
+"\xe3\xd5\x23\xb3\xe7\x22\xae\xf3\xd0\xd9\x00\x14\x0c\x46\x67\x17",
+	.rlen = 64,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xf9\xe8\xab\xe2\xf9\x28\xeb\x05\x10\xc4\x97\x37\x76\xe4\xe0\xd9",
+	.input =
+"\xab\x99\xe8\x2a\x18\x50\xdc\x80\x1f\x38\xb9\x01\x34\xd4\x59\x60"
+"\x4e\x1c\x21\x71\x22\x06\xbe\x5f\x71\x07\x3b\x13\xe7\x05\xca\xa5"
+"\x7b\x23\xb5\xaa\xc6\xdb\xe3\x17\xa9\x9f\xe1\xbc\xd5\x1b\xe6\xf5"
+"\xfa\x43\xdd\x80\x50\xc8\x8a\x32\x2f\x65\x25\xa4\xeb\xd1\x74\x02"
+"\x07\xc1\x04\x94\x6b\x34\xa1\x74\x62\xb2\x8d\x60\xf5\x7e\xda\x1a"
+"\x0f\xf5\x21\xe1\xd7\x88\xc8\x26\xd7\x49\xb2\x4a\x84\x2c\x00\x3b"
+"\x96\xde\x4e\xa7\x57\x27\xa0\xa4\x3a\xff\x69\x19\xf7\xec\xeb\x62"
+"\xff\x5a\x82\x0d\x25\x5e\x3c\x63\xb3\x6d\xc4\xb9\xe3\xc9\x3a\xc2",
+	.ilen = 128,
+	.result =
+"\xec\xd5\x2f\x6a\xfd\x61\xf2\x37\x19\x6f\x55\x31\xd7\x2c\x14\x4d"
+"\xc1\xb4\xbb\x7d\xa9\x1a\xe6\x85\x8c\x2f\xbf\x7e\x66\x21\xf8\x17"
+"\x9e\x09\x1b\x2a\x11\xbf\xdf\x7d\xdf\xf5\xfb\x0a\x16\x79\xe2\x43"
+"\x5c\x3b\x3e\x84\x35\xfd\x92\x9e\xe0\x31\x50\x1d\x62\xd6\x22\x99"
+"\x5f\x25\xb3\xe8\xdf\xb0\xc0\xab\xd9\xdb\xac\x4b\x9c\xe2\x89\xc6"
+"\x49\x7f\x5f\xee\xcb\xf6\x25\x10\x9f\x32\x58\x85\x45\x50\x74\x8a"
+"\x55\xce\x86\x44\xda\xe4\x93\x58\x4d\xd3\x73\x76\x40\xf6\x92\x8b"
+"\x99\xc1\x2b\xf9\x18\xd0\xfa\xd0\xa6\x84\x03\xf5\xd4\xcb\xfa\xe7",
+	.rlen = 128,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x58\x1e\x1a\x65\x16\x25\xaa\x55\x97\xcd\xeb\x4c\xd6\xb3\x9c\x2b",
+	.input =
+"\xef\x85\x0b\xe5\x02\xd5\xce\xcc\xad\x2d\x5e\xec\x1e\x01\x8c\x28"
+"\xf0\x2c\x23\x10\xaa\x84\xf0\x61\xe2\x56\x29\x21\x9f\x09\xaf\x9d"
+"\x7d\xfc\x60\x16\x4c\x67\xdd\xdf\x74\x35\x49\x81\xca\x68\xb6\xc7"
+"\x31\x9f\x49\x29\x96\x01\xb9\x3c\xfb\xa3\x00\x04\x05\xd8\xe6\xa2"
+"\x3f\x0c\xee\xfc\xd6\x88\x7a\x2d\xd6\x32\x27\x15\xe3\x53\xa0\x52"
+"\x1d\x96\x5a\x95\x09\x0c\x5f\x07\xc8\xdf\xab\xc7\x78\x57\x6a\x49"
+"\x09\x88\x54\x2e\x80\x27\xb0\x8a\x40\xb8\x9e\x7a\x22\x85\x8d\xaa"
+"\x95\x48\x45\xf5\xfd\x6f\x4c\x69\xe3\x38\xa6\x05\x69\xf0\xba\xb5"
+"\xd5\x9a\x9f\x77\x98\x23\xef\x98\x1f\xf3\xfe\x53\x23\xf6\xc6\x74"
+"\x6a\x2f\x1b\x34\x75\xd0\x51\x0c\x88\x10\xf9\x80\x19\xaf\x4f\xf1"
+"\xb1\xf3\xc0\x0e\x3a\x7d\x63\x3e\xbd\xb9\xe9\x3c\x69\x56\x0d\xb9"
+"\x8d\x69\xea\xb9\xa7\x39\x4c\x5d\xb8\x06\xa3\x1b\x66\x66\x14\x80"
+"\xe1\x8f\xf3\x65\x0c\xd5\x39\xe4\xed\xb9\x1f\x88\x74\x49\xd7\x4f"
+"\xc1\x4b\x3d\xea\x5d\xa2\x44\xd6\xad\x5d\x8d\xd1\xf7\x56\x9c\x9e"
+"\xda\x52\x56\x51\x00\x14\x1b\xb4\x00\x6b\x83\x4f\x41\x0b\xba\xaa"
+"\x11\xe6\xee\x23\xf7\x85\xa9\xb9\xd8\xe3\xbd\xbb\x7b\x83\x5f\xf8",
+	.ilen = 256,
+	.result =
+"\xc8\x8e\xbf\x95\x57\xa8\xcd\x47\xbc\x32\xee\x76\x97\xee\x02\x12"
+"\x11\x36\x81\xaa\x5b\xd9\xb3\x14\x80\xf3\xab\x62\x9b\x7f\x99\x98"
+"\x3b\x46\xd6\xfb\x68\xc8\xce\x1d\xa5\x47\x79\x6a\xdf\x7c\xda\x01"
+"\x44\x01\xfc\xed\xab\x2a\x51\xae\x2f\x72\x60\xed\x61\xc5\x23\x1d"
+"\xc7\xb5\x3c\xb7\x0b\x29\x62\xd6\x77\x8c\xea\x51\x0c\x39\x90\xe7"
+"\x99\x8c\x5d\xb7\x16\xf3\xc6\xea\xe0\xff\xc3\xd7\xc8\x1a\x7d\xde"
+"\x4d\x25\xaa\x0b\x90\x0d\x49\xd7\x98\x44\x4b\x75\x46\x01\x30\xa3"
+"\xdc\x47\xd9\x66\xc7\x7a\xcb\x4a\x33\x69\x60\x5d\x96\x73\x31\xf1"
+"\xce\xdc\xa9\x15\xb5\xae\x08\x2b\x08\x4a\xbc\x9b\x68\x1e\x49\xe4"
+"\x6e\x11\xe8\x61\x37\x58\x66\x69\x67\x97\x65\x1d\xd4\x62\x7c\x29"
+"\x10\xba\x8f\x2f\x0f\x23\x3d\x72\xb1\xcf\x01\xbc\x73\x10\xd8\xde"
+"\x21\xe6\xfc\xce\x3b\x3e\x19\xdc\xc2\xa7\x87\x62\x33\x88\xb4\x37"
+"\x1f\xfc\x1a\x2b\xef\x14\x24\x4a\xb5\x86\x55\x45\xf8\xc4\xcd\xaa"
+"\x0d\x8a\x5a\xdc\xfd\x7b\x41\xd7\xa6\x8f\x05\x25\x4a\x61\xcb\xa7"
+"\x14\x84\x21\xfc\xa6\x4b\x0f\xaa\x7d\xc6\xa2\x04\x04\xff\x39\xfc"
+"\x27\x8d\x7a\xce\x94\x31\x7c\xb4\xd5\x90\xbd\xb6\xdb\x6a\x55\xd9",
+	.rlen = 256,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xed\xa3\x2d\xa6\x9f\x5e\x38\x6f\xbb\xf9\xb3\x32\xae\x73\x05\x87",
+	.input =
+"\xf6\x24\x50\x2d\xa4\xfb\x09\x41\x95\xcd\x25\x13\xc0\xdc\x26\x0b"
+"\x20\x66\x70\x79\xc5\x58\xde\x63\xba\x37\x13\xb2\x0a\x40\x58\xef"
+"\x37\xcb\x04\x38\x10\x6a\x90\x97\x14\xd2\x71\x03\xa4\xa3\x6a\x59"
+"\x15\x6e\x5d\x45\xc9\xcc\xa9\x47\x8c\x0f\x1d\x6c\x62\x06\x90\xba"
+"\xef\x1c\x23\x4d\xc4\xa0\xa5\x56\x49\x19\xa9\xb1\x2a\xdd\x26\x00"
+"\x6e\xed\xd8\x4d\xd4\x3f\x68\x99\x24\xe2\xfe\x99\xb4\xe6\xf8\x3f"
+"\x60\xef\x97\x5f\x87\xa6\xde\x82\xc4\x11\xf6\x91\x7f\xd4\xa6\xa8"
+"\xee\x97\x41\x43\x14\xd2\x6e\x8d\x72\x30\x83\x5b\x67\x01\x38\xa2"
+"\xca\x93\xf4\x1e\x80\x2e\x8f\x7e\xc3\x78\xf0\xc1\x68\xb4\xf9\x1f"
+"\x15\x3c\x5c\x8b\xa1\xb5\x2f\x0c\xbf\xf7\x21\x74\xdb\x57\x98\x85"
+"\xe9\x89\x16\x20\x8b\x7c\x71\xef\x46\xc0\x78\x04\x23\x3b\x58\x24"
+"\x51\xa1\xa6\xfc\x6e\x9e\x29\x95\x55\x4c\x05\x1c\xc5\x9a\x59\x7e"
+"\x40\x4d\xe8\x81\x76\x41\x6f\x15\xde\x68\xce\x2d\x42\x03\x74\x73"
+"\xd3\x9a\x9c\xac\xa5\x05\x03\x7e\x53\x6e\x53\xa2\x57\x36\xee\x0c"
+"\x05\xde\xa9\x61\x55\xb9\x9d\x7d\x18\x18\xaa\x20\xde\xb8\x43\xd7"
+"\x30\x8e\x20\xc0\x78\xbe\xec\x24\xcf\xd7\x66\xb7\x5a\x1f\x5a\x81"
+"\xec\x19\x48\xc3\xa7\x62\xbf\x83\xbb\xbd\xf4\x51\xec\xb5\xec\x90"
+"\x05\xe1\xa9\xbf\x4d\x9b\x30\xf1\xb9\xa6\x49\xe9\xad\x65\x0d\x08"
+"\x1f\x3f\x81\xa5\x40\x4f\x3d\x42\xd8\x68\x29\xe3\x6c\xcc\x4d\x20"
+"\x7e\xb9\x0c\x33\x1f\x20\xd2\xaf\x39\xd6\xb4\x20\x06\xd0\xc3\x54"
+"\xcd\x96\x84\x88\x13\xc0\x09\x57\x18\x90\xad\xec\x18\xab\x72\x0b"
+"\xb4\x4c\x0a\x65\x67\x2a\x96\x2c\x98\x58\x6f\xdf\xc0\xe4\x51\x7c"
+"\xc8\x66\x1d\x21\x91\x1f\xab\xac\xed\x86\x38\x70\x54\x6f\x0c\xbf"
+"\x1a\xea\x9b\x33\xf4\x7c\x99\x0c\x0a\xdf\x39\x25\x78\x3b\x8d\x9c"
+"\x46\xc0\x07\x08\xfa\xeb\x19\x12\xf8\xc1\xf7\x18\x13\xbd\x7f\xd1"
+"\xa4\x3c\x7e\x03\xbd\xcf\xa1\xf3\x37\x4a\x4d\xc3\xaa\x23\xed\x58"
+"\xca\x68\x35\x91\x3e\x23\x09\xb8\xf3\x8d\xc3\x1b\x23\xe8\x1c\xda"
+"\x41\x90\xa2\x4b\x48\xb5\x7c\xa0\x8d\xaf\x66\x5e\xad\x7f\x06\xa2"
+"\x62\x32\x40\x69\x41\xb1\x2f\x6c\x0e\xf9\xd1\x48\xbd\xfc\x44\x0f"
+"\x65\x5e\xa1\x38\x83\xea\xfe\x42\x53\x9a\x2a\x85\xea\x92\xf6\x29"
+"\xbf\xb5\x78\x1e\x8d\x03\x6b\x09\xaf\x94\x4b\x39\x20\xc1\x17\x20"
+"\x95\x42\xfe\x72\x02\x10\x61\x21\x0f\x23\xcb\x33\x35\x52\x57\x9e",
+	.ilen = 512,
+	.result =
+"\x25\x3d\xad\x25\x4f\xb4\x50\x55\xbf\xc1\x66\xe3\x52\x22\x01\x10"
+"\xde\xed\x83\xc0\x18\x49\xda\xa4\xdb\xf1\x2f\x73\x90\x6f\xf2\x4f"
+"\x9b\xa2\x32\x2b\x6f\xc7\x80\xc8\x47\xbd\xf3\x24\x8a\xcd\x9b\x8d"
+"\x00\x33\xd1\x6a\xf2\x5f\xf2\xc7\xd8\x7c\x3a\x84\x1c\x12\x3c\x3e"
+"\xe0\x58\xb7\xc9\xf8\x73\x9e\x98\x2f\x8f\x03\x38\xe2\xc2\xb9\xae"
+"\xb6\xc6\xef\x78\xd0\xfa\xbf\x81\xcc\xf7\xb3\x82\x5b\x80\xb9\x0b"
+"\x57\xe3\x33\xa6\xfc\x3c\xd1\x78\xc7\x61\xc5\x5a\xe9\x01\xf5\xf7"
+"\x87\x0f\xa4\xe7\x90\xdf\xd5\x9f\x79\xc5\x5c\x1a\x2c\x29\x8e\x79"
+"\x10\xbc\xb2\xc6\x89\x9d\x95\x65\xa8\x25\xb3\x20\x97\xcc\xdf\x62"
+"\x2f\x9c\x85\x36\xe6\x34\xcc\xc0\xee\x7e\x10\xf6\x07\x57\xed\x2e"
+"\x60\x7e\x5e\xa0\x8e\x4c\xec\xe8\x73\xa3\x55\x4d\x7f\x6d\xff\x8c"
+"\x7a\x8c\x62\x3b\x10\x22\x75\xc0\x0b\x4a\x99\x83\x4d\x09\x80\x36"
+"\x41\x33\x19\x53\x9b\x51\xa6\x92\x82\xd8\x97\xe7\x98\x42\x36\x0d"
+"\x93\xb2\xf4\xbf\x96\xc7\x71\xfb\xc1\xf7\xf0\x94\xa3\x88\x28\xfa"
+"\x7c\xef\x3b\x1c\x77\x72\x23\x9b\xaf\x8c\x6a\xf8\x2b\xb2\xd4\xb9"
+"\xeb\x7f\x9f\xa5\x02\x50\x08\x47\x52\x6c\xaf\xe7\x73\x71\x85\x72"
+"\x49\x6b\xc8\x47\x88\xa7\xd8\xc2\x16\xbf\x3c\xe9\x22\x21\xeb\x54"
+"\xd1\xcd\x43\x18\x08\x8f\xa1\xcf\x1c\x2b\xa7\xfd\x65\x4a\x9d\x12"
+"\x0d\xdb\xd5\xf6\x1a\x97\x64\x83\x3c\x5a\x04\xa8\x15\x9d\x61\xd3"
+"\x43\x2a\x56\x35\xed\x08\xb7\x41\xc6\x49\xba\x02\x14\x59\xab\xca"
+"\x84\x1f\xfb\x67\x3a\x00\xe5\x41\xb8\xd1\x6e\x5c\x9d\x6f\xf2\x76"
+"\x3e\x21\x5d\x34\x5c\x78\x0d\x41\x5a\x4f\x62\x69\x1a\x76\x42\xee"
+"\x84\x6b\x1d\x47\x42\xeb\xb2\x11\x8f\x08\xb8\xc8\xea\xf4\x0d\xf7"
+"\x5d\x51\x4c\x4b\xed\x2d\x1b\x48\x30\x38\x38\x58\x0d\xe3\x2d\x80"
+"\xd9\xfb\xed\xe0\xc4\x55\xfe\x4f\x3f\xcf\x55\x57\x08\xaa\xa8\xa2"
+"\xa5\x5a\xe4\xff\x19\xf2\xae\x29\x74\xb9\x40\xea\xf4\x4d\x58\xac"
+"\x9f\x48\xea\x0f\xe0\xb0\xae\x72\x9f\xd8\x34\x95\x59\x01\x20\x7c"
+"\x98\x5d\xe6\x9f\x37\x23\x52\x8d\xa0\x62\x2b\x3a\x9c\x2e\x31\xe7"
+"\xd5\x75\xcc\x4c\x62\x2f\xa4\x3e\x2e\xb9\xe6\xe1\x4b\x69\xb4\x62"
+"\x31\x03\xfc\x08\xfd\xba\x87\xb9\x79\x3a\x68\x19\x65\x49\x2e\x2c"
+"\x65\x5f\xd8\x60\x07\xf4\x73\x8d\xdf\x37\x7e\x00\x88\xaf\x23\x48"
+"\x8b\xad\x74\x9c\x0b\xa3\x3a\x1a\x4b\xa0\x27\x6f\x04\x8d\xd9\x38",
+	.rlen = 512,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xfd\xd6\xe8\x2f\xfe\xd4\xfe\x42\x23\x4b\x7c\x09\x8c\xde\x4f\x4b",
+	.input =
+"\xff\x7f\xb0\x11\x72\x5a\x91\x4a\xb5\x2d\xb0\x41\x3a\x96\x0d\xa1"
+"\xd9\xbe\x60\x09\x24\x51\x73\xb8\x00\xf0\x48\x1f\x6b\x96\x5b\xe7"
+"\x4d\x47\x88\xc7\xef\x4b\xb4\x33\xa1\x2b\xbe\xdd\x46\x4f\x27\x11"
+"\x8b\x30\x9c\xba\x2c\x7a\xf3\xdb\x48\x54\xbd\xfe\x24\x2f\x83\x91"
+"\x5c\x63\xb9\x12\xd9\xd9\xb9\x71\xcf\x28\x7e\xf8\xe0\xb8\x12\xf7"
+"\x63\xad\xde\x49\xd5\x4d\xa7\x13\x32\xee\x71\x13\x56\x4d\x10\xd5"
+"\x2c\x1d\x8e\x94\x0d\x37\x3d\x7e\x9c\xb4\xeb\xe5\x6f\x12\x30\x7f"
+"\xc3\xa0\xf3\x49\xac\xa6\xab\x1b\xec\xd4\x6c\x95\x2a\x57\xe0\xfa"
+"\x89\x00\x61\xe9\xea\x21\x9a\x2f\x71\xd7\xdb\x11\x52\xb6\x32\x91"
+"\xed\xa3\xdf\xa5\x46\xc1\x50\x5b\xab\x15\x43\x7f\x7d\x82\x34\xf2"
+"\xfa\x6e\x84\xaf\x40\x20\xde\x1f\x90\x39\xab\xdc\xe8\xf3\xf9\x65"
+"\xbc\xdc\xd3\x5c\xcf\xe2\x1b\x43\x08\x68\xd8\x0d\xfb\xc2\x7f\x31"
+"\x91\xb5\x66\x2a\xea\x43\x08\x6d\xa6\xb4\xd3\x0e\x78\x3c\xf1\x6c"
+"\x4d\x27\x47\x7d\x92\x42\xb1\x62\x82\x9f\x13\xdf\x51\xc3\x6b\xec"
+"\x83\x53\xd6\x89\x75\xac\x62\x9a\x89\x7d\xf9\x82\x66\xbe\x93\x6f"
+"\x71\x7d\x01\x79\xec\x10\x10\x50\xe9\x6c\x76\xc6\x7a\xfa\xbb\x69"
+"\x46\x09\x1a\x68\x2f\x07\x28\xf4\xd0\xb6\xb4\x82\xf5\x3a\x90\xdc"
+"\x61\x03\xd9\x8e\xa5\x13\xfd\xdd\xe0\x65\x03\xfb\x78\x6b\x4e\xae"
+"\x7f\x30\xe2\x9e\x39\xb1\x3a\x39\xda\x21\x80\x2c\x09\xdd\xe8\xa2"
+"\x8c\x4a\x2c\x40\x24\x39\xf0\x3f\x7f\x51\x6a\x48\xea\x7b\x68\x3d"
+"\xad\x56\xed\xbe\x86\x0a\x9a\xe6\x9f\x18\x95\x26\x14\x57\x5b\x71"
+"\x9e\x8d\x45\x0d\xad\x23\xb4\x37\xa5\x59\x66\x8c\x13\x8e\x5e\xeb"
+"\xbf\x4a\x0d\x72\xc9\x4a\xcf\x42\xbd\x28\x1f\x91\xad\x55\x81\x78"
+"\x48\xf3\xed\xab\x2b\x6d\x61\xc7\x08\x2c\x07\xcb\x17\xf8\xf1\x7c"
+"\x39\xc8\x44\x63\x3a\x2a\x55\xbe\xe1\xb5\x12\x61\x0a\x4c\x32\x83"
+"\x9a\xa0\xf8\x93\x8c\xfa\x45\x92\x4e\xad\x48\xd9\x84\xe8\x0d\x7a"
+"\xca\xad\xbf\xd2\x5a\x1d\x58\x67\x57\x68\xca\x2f\x40\xa5\x1c\x38"
+"\x2a\xde\xa7\x57\x87\x4f\x11\x97\x3e\x11\xe7\x58\x54\xbd\x06\x48"
+"\xf7\x60\x45\x5b\x9d\x08\x5a\xef\xf9\x28\xa5\xf5\x48\x5c\x9c\xa0"
+"\x96\x76\x56\x51\x40\xec\xbe\xdb\x6e\xba\x4b\xb0\xa2\xe9\x55\xe6"
+"\xb7\x7e\x8a\x06\x3b\xeb\x17\xeb\xe6\xd9\xf6\xb2\xa1\x8c\x9e\xcc"
+"\xf3\x89\xd5\x78\x29\x1f\x74\x60\xe2\x61\x72\x78\x05\x52\x23\x07"
+"\x2a\x46\x85\x3c\xcf\x12\x9a\x9d\x3d\xf0\x93\x0e\xd2\x22\x63\x07"
+"\x01\x8b\x96\x73\xb5\x26\x29\xf5\x4f\x90\xf9\x37\x55\x76\x15\x02"
+"\xe8\x4c\x56\x3e\xf1\x14\xaf\x34\x0d\xa8\xde\xee\x0e\x13\xfa\xb8"
+"\xe4\xb7\x6d\x71\x37\xdb\x1e\x42\xdd\xca\xec\xe1\x99\xf9\xc7\x18"
+"\x16\xb0\x41\xd0\xfe\x9a\xa6\xa0\x7a\x5e\x5d\x0a\x96\x4c\x52\x44"
+"\x9a\x29\x69\x09\xa2\x0e\x5a\x1e\xc2\xb3\x5e\xca\x25\xc0\xe1\xa9"
+"\xd1\x41\x7f\x82\xaf\x1f\xf4\x3c\xf8\x3d\x65\xae\xf0\xa2\x1a\x8f"
+"\x41\xdb\x01\x11\x4c\x01\xcb\x24\xb3\xec\xbb\xf3\xe5\x1b\x53\xf0"
+"\x7a\x81\x01\x61\xa2\x8e\xa4\xd0\xaa\x8f\xa1\x71\xc1\x15\x15\xda"
+"\xf3\x7b\x32\x87\xa6\xb7\x7f\x2b\xac\x2b\x28\xfc\xe4\x1a\x94\xab"
+"\x19\xc9\x13\x72\x33\xfa\x42\xec\x6f\x3f\xe1\xe0\xc7\x23\x4b\x17"
+"\xeb\x89\xd3\x1f\x49\xe1\x49\x56\xee\xe3\x82\x46\x43\x00\x80\xbc"
+"\xa3\xfe\x31\xbc\xc9\xcd\x61\x5b\x7a\xf9\xf7\xb7\x48\x98\xbf\xdc"
+"\x79\xca\x71\x3b\xb0\xda\x08\x1e\x25\x97\x83\xd7\x21\x2c\xaa\xc0"
+"\x5c\xfd\x7f\xc4\x30\xd8\x7b\x59\x35\x49\x62\x0f\x4c\x03\x02\xe5"
+"\x73\x63\x61\x0b\x69\x2f\x7d\xb3\x99\xc9\x6b\x0a\x29\x9b\xda\xbe"
+"\x98\xdc\x2c\x29\x28\x9a\x75\x2e\xf1\x11\xd3\x71\x5b\x20\x45\x5b"
+"\xb7\x5e\xc1\xd1\xcc\x4e\x5a\x0d\xa5\x70\xa6\x56\xb8\x80\x8c\x97"
+"\x9d\x65\x8d\xec\xa0\x15\x45\xe6\x04\xd8\x3b\x6b\x36\x3f\x71\x58"
+"\x9e\x7a\x9c\xd2\x44\x86\xbf\x89\xa6\x80\x5d\x5e\x99\xc9\x7e\x56"
+"\x76\x17\x02\x98\x5b\xbb\xa0\xe5\xe5\x10\x25\x3e\x82\xc7\xe0\x91"
+"\x77\x39\x50\x9c\x3d\x2a\x91\x03\x13\x6d\x6d\xd3\xc6\x68\xd3\xa0"
+"\x88\xbc\x24\x5d\xf1\x26\x19\xf4\xb0\x74\x51\x93\x17\xcf\x67\x6c"
+"\x72\x30\xed\x39\xfe\x59\x54\x88\x84\x70\x56\x11\xaf\x41\x66\xa5"
+"\xf9\xf0\x95\xdb\x80\xb8\xae\x2f\xb7\xc3\x65\x72\xd2\xec\xaf\x5f"
+"\xf9\x30\x1e\x5b\x45\x7f\x38\xd5\x03\x02\x60\xaa\xf9\xb7\xd9\xfc"
+"\xa2\x5c\x46\x3e\x9c\xe6\xd6\x8e\x95\x54\xbf\xd8\xe6\xe4\x4b\xc0"
+"\x4c\xa1\x4c\x2c\xb3\xc4\x9f\xef\xeb\x39\x70\x77\xac\xf9\x1f\xb6"
+"\x06\xa2\x53\x7d\x18\xc8\xf8\xda\x8e\x82\x97\x4f\xdd\xd5\x19\x2f"
+"\xa2\x70\x4a\xbd\x5a\x15\x70\xb6\x55\x04\x14\xba\x0a\x04\xdc\x8e"
+"\xaf\xf2\x52\xd5\x90\x4c\x30\xd3\x29\x53\x1c\x66\x37\x5f\x8e\xfc"
+"\x45\x83\xd9\xac\x75\x9e\x0f\x66\x51\xc0\x8a\xc5\x34\x25\x9e\x3b",
+	.ilen = 1024,
+	.result =
+"\xa8\x47\xa1\x1d\xcb\xa3\x88\xae\x42\xab\x6d\xf2\x82\xc2\xed\xd5"
+"\x66\x42\x09\x85\x28\x7d\x49\x6f\x37\xdc\xff\x1c\x7e\x33\xc9\xcd"
+"\x6e\xe9\x33\x36\x01\x62\x1d\x67\x77\x6a\x97\xbf\xb1\xdc\x2f\x98"
+"\x2c\xdb\xac\x44\x9d\xed\x31\x7d\x2d\x41\x4b\xd1\x66\x40\x62\x74"
+"\xdc\x00\xd0\x05\xdc\x54\x4c\x63\xeb\xd9\x42\xe1\xdf\xc4\xde\xdd"
+"\xb6\xb8\x93\xfd\x25\x39\x2d\x7f\x85\xf8\x15\xc3\xbc\xbf\x0b\x95"
+"\x11\xef\x57\x0d\x15\x49\x07\xce\x42\xb0\x50\xe1\x07\xb4\x81\x71"
+"\x35\x71\x4b\x66\x89\x7f\x94\x13\x3e\x57\x43\xc3\x36\x28\xcd\xdd"
+"\xc9\x06\x68\xf8\xf3\x09\x3d\x86\x12\x52\x06\xa9\xe9\x83\x2d\x8f"
+"\x90\xfa\x42\xfe\x79\x3f\x68\x4c\x7b\xfa\x94\xa7\xf7\x16\xc7\x41"
+"\x09\xae\xe2\x82\xb5\x2b\xbc\xca\x65\x65\x2c\x27\x2c\x07\x50\x83"
+"\x2d\xad\x55\xaf\x35\xcc\x6a\xc5\x7c\xd8\xed\x75\x91\x9d\x73\xcb"
+"\x4c\xa5\x8f\xc4\x4f\xda\xa8\xb9\xb6\xa7\xb1\x1a\x75\xb4\x08\xbc"
+"\xb2\x90\x50\xfd\x1f\x05\xa8\x88\x35\x81\xb0\xc9\xac\xbc\x25\x7a"
+"\x95\x33\x02\x2b\x74\xe0\x95\x11\x88\xf7\xc3\x63\xb3\x7b\x09\xd5"
+"\xac\x22\x04\x67\x16\xea\xd6\x37\x38\x8e\xa5\xbd\x62\xa2\x1f\xa5"
+"\x04\x31\x89\xdf\x69\xb1\xde\xe3\x7c\x9d\x7b\x27\xba\x0a\x74\xdc"
+"\x06\x1c\xcd\x6e\x4b\x52\xe7\x6d\x34\x29\x38\xe2\x19\xfc\x0c\xc4"
+"\x78\x03\x1d\x53\x98\x00\x5c\x7a\xec\x23\x5f\x95\xd5\xb3\x16\xde"
+"\xc2\x17\xc2\x0c\x13\x63\x0a\x4b\x7e\x6c\xc7\xbc\x4a\xd0\xae\x29"
+"\xc0\x50\x16\x6f\x01\x2b\xdc\x40\x9f\x91\x8f\xa3\xaf\xd4\x40\xa8"
+"\x2e\x09\x7c\xf4\x3d\x85\xe6\xd9\x3c\x78\x7c\xf1\x6d\xe4\x13\x00"
+"\x98\xf5\xb4\x06\x9f\x90\x0a\x3e\x9f\x51\x0f\xbb\x0f\x13\x07\xc0"
+"\xfd\x26\x53\x24\x24\xf7\x21\x41\xcf\x20\x9d\x77\xe4\xe0\x52\x2a"
+"\x48\xd9\xeb\x65\xce\xf3\x90\x03\x47\x8d\x2b\x77\x54\x46\xda\xff"
+"\x15\x3d\xa5\xd9\x5a\xb6\xd3\xdf\x9c\x91\xc3\xf2\xd2\xdf\xd7\x8c"
+"\x1d\x83\x77\x47\xcd\x74\x23\x44\x04\x06\x8e\x64\x62\x29\xe5\xa0"
+"\xf7\xa7\xc7\xb7\x84\xdb\x9c\x5c\x04\x7f\xca\xb3\x85\x2c\x44\xa6"
+"\x09\x0e\xa3\x2c\x52\x42\x25\x02\x63\x99\xd0\xa5\x27\x61\x64\x4f"
+"\x65\xd7\x31\x56\x24\x97\xb0\x2d\xbb\x0c\xbe\x06\x68\x8a\x2e\xa3"
+"\x0c\xb9\x05\x52\xdb\xbd\x7e\x89\x60\x2e\x28\x76\xba\x5a\x94\xb6"
+"\x94\xc4\xf6\x68\x50\x35\x24\x7b\x2b\x04\x0e\x4c\xf3\x17\x54\xcb"
+"\xcd\x32\x18\x60\xff\xc9\xfe\xe1\x83\xe4\xe6\x9b\x5e\xd8\x21\xbf"
+"\xbf\x69\x01\x3a\x03\xc6\x9f\xe5\xd4\xdf\x01\x20\x8e\xea\x5b\xe1"
+"\xbd\x46\x3c\x3a\x60\x30\xa0\x48\xa0\x07\x82\x27\x4e\x03\xc3\x15"
+"\x98\x1f\xea\x4f\x8c\x90\x4d\xb1\xc5\x90\x40\x59\xda\x5b\x02\x65"
+"\x07\xb9\x64\xe7\x4c\x76\x70\x16\x8a\xc3\xf9\x4f\xed\x25\x47\xaa"
+"\x3b\x49\x8f\xf6\xf0\x71\x94\x34\xda\x29\x0f\x4e\xd4\x95\x3b\xe3"
+"\xef\x99\x3b\x1c\xf7\x09\x5d\xe0\x0d\x03\xe6\x9d\x47\x4c\x8c\xe8"
+"\x26\xb6\x30\x1b\x81\xdc\xa5\x5a\xf1\x04\x18\xf3\xaf\x81\xa2\x7e"
+"\xce\x8b\x33\xfc\xf2\xb1\x5a\x06\xd1\xb9\x59\x73\xd7\xda\x85\xd9"
+"\x30\x73\x98\x4d\x63\x50\x66\x71\x15\x88\x9a\x5d\xd5\x25\x40\x9a"
+"\xe3\x9c\x0b\x4f\xd8\xf5\xbf\xb3\xec\x02\x95\xca\x90\x07\x5d\x99"
+"\x9e\x16\xa2\x18\xa5\xa2\x03\xb1\x16\x6b\x4e\x32\xab\x19\x29\x55"
+"\xcc\xbe\xa8\x7b\xf7\x68\x64\x0e\xc0\x54\x91\x6d\x19\xec\xe9\x8c"
+"\x56\x5e\x71\xa5\x73\x50\x5d\x0d\xd3\xb2\x31\xca\x97\x7b\xf8\x6e"
+"\xfd\xb9\x47\x9b\x17\xf9\x56\x3a\xc6\xb0\x52\x45\x4f\x4a\x13\xe9"
+"\xb7\x64\x02\xdb\xe8\x67\xa3\x9e\xe4\xd9\x49\xc4\xf3\x27\xe3\xb0"
+"\xad\x6e\x51\x65\x14\x4f\xb2\x4b\x8a\xd6\x87\x17\x8c\xe2\x7a\xa1"
+"\x13\xbb\x8c\x7c\x3e\x69\xd2\x29\x06\x36\xf3\x55\x80\xcc\x0e\xa5"
+"\x18\x5a\x5f\xcb\x15\x2e\x7c\x62\xff\x3f\xe7\x7b\xd8\xe4\xa6\x9c"
+"\x4c\x5b\x55\x73\x4a\x0d\x21\x07\xf9\x79\xcb\x17\x51\x06\xf3\xcc"
+"\xfc\x08\x72\x6e\xbc\x04\xe2\x6d\xd8\x52\x1d\x29\x7e\x7a\x06\x8d"
+"\x87\x65\x2e\x2e\x7c\x07\x77\x3a\x35\x4d\x3a\x13\xd3\xf6\xc2\x1f"
+"\x2d\x5d\x14\xa5\x04\xe5\xc5\x7b\xd6\xa9\x70\x4b\x43\x21\x93\xdf"
+"\xe4\xf1\xf8\x75\xf1\x65\x9c\xf8\x0b\x07\x31\xdc\xf2\xba\x06\x91"
+"\xe1\x84\x87\x34\x2d\xdd\xa7\x87\xc0\xc2\x4d\x8d\xe0\x18\x70\xbb"
+"\xe3\x3e\x13\x48\xfc\xf4\x13\x85\xc4\x65\xcf\xe4\x43\x98\x14\x8f"
+"\xf4\x17\x62\x27\x39\xe5\xb6\x45\x76\x61\x78\x0b\x3d\x48\xb3\x41"
+"\xa6\xca\x7c\xed\x52\x19\x99\xea\x73\xc9\xd0\x0b\xeb\xbb\x5a\x69"
+"\x44\x3d\xb2\x81\x25\xb0\x2f\x08\xf0\x8c\x32\xa9\xf0\x79\x3c\x42"
+"\xc3\xdc\x9e\xd1\xec\x93\x49\xc9\x82\x0e\x13\x12\xb3\x8a\x98\x1b"
+"\x35\xe1\x4a\xef\xb4\x73\x28\x1a\x17\x96\xe2\x9a\x50\xc8\xd5\x98"
+"\xec\x96\x6f\x81\x05\x37\xee\x8b\x93\x12\x7c\x41\x26\xd5\x9c\x05",
+	.rlen = 1024,
+},
+/* repeat the above with sg list */
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc1\x62\x66\x62\xb8\x65\x28\xfa\x5f\x36\xd3\x09\xb1\x2c\xa1\xa3",
+	.input =
+"\x4f\x6c\x63\xa5\xd0\x19\x08\x4e\xd4\x58\x33\xf6\x2b\xeb\x26\xb9",
+	.ilen = 16,
+	.result =
+"\xa0\x35\xb0\x33\xc0\x2e\xe5\xbb\xbc\xe6\x01\x9e\xf4\x67\x11\x14",
+	.rlen = 16,
+	.np	= 2,
+	.also_non_np = 1,
+	.tap	= { 8, 8 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x78\x6c\x27\xd6\xb2\xdc\xbe\x7b\xab\xc2\x43\xd7\x81\x0c\xe5\x20",
+	.input =
+"\x9a\x00\x4e\x5a\xb3\x51\x68\xaa\xdb\x6e\xe5\xa4\x7f\x23\x6e\x4d"
+"\x1e\x72\x5e\xad\x64\xc9\x96\x23\xf8\xae\xef\xf6\x7b\x7d\xd6\xf0",
+	.ilen = 32,
+	.result =
+"\x5a\xc0\x04\xc6\x53\xef\x3b\x69\xb1\x41\xc7\x85\xeb\x69\x82\xd0"
+"\xed\x09\xbb\xec\xb2\x8d\x5c\xc9\x61\x81\x5c\xf6\x99\x49\xa0\x4d",
+	.rlen = 32,
+	.np	= 3,
+	.also_non_np = 1,
+	.tap	= { 8, 8, 16 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc9\x05\x4c\x35\x96\x77\xd3\x3c\x3d\x97\x7c\x82\xf5\x58\x71\xf1",
+	.input =
+"\xa9\x5b\x03\xec\xec\x73\xed\xcb\x5c\x4c\xd2\x40\xb6\x9b\x49\x31"
+"\x5d\xf2\x23\xb3\x11\x98\xeb\x89\xab\x3e\x3a\xdd\xaa\xfd\xd1\xde"
+"\xab\x73\x59\x86\x1a\x59\x32\xb2\x55\x46\x4a\x80\xa4\xcc\xa8\xd9",
+	.ilen = 48,
+	.result =
+"\xdb\x05\x69\xe1\x33\x8b\x0b\x3d\x33\x12\x0d\xef\x94\x0f\xa3\xb3"
+"\xd7\x0a\x53\x7b\x98\x53\xc6\xc2\xa3\xd4\x7a\x30\x1a\xed\x45\xcc"
+"\x47\x38\xc1\x75\x0b\x3c\xd4\x8d\xa8\xf9\xd3\x71\xb8\x22\xa6\xae",
+	.rlen = 48,
+	.np	= 4,
+	.also_non_np = 1,
+	.tap	= { 8, 8, 20, 12 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x6c\xb4\x84\x61\x1e\x39\x4b\x22\x37\xaa\x7b\x78\xc0\x71\x20\x60",
+	.input =
+"\x05\x43\x76\x1e\xc6\x68\x43\x52\x5f\x43\x39\xbf\x93\x38\x38\x83"
+"\x38\x1d\x3c\xb5\xc8\xab\xe4\xd0\x7f\x1a\xac\xca\xcc\x16\xea\x75"
+"\x30\x75\x40\xe8\x61\x07\xc6\x04\x55\x2b\xf3\x29\xc3\x37\x83\x42"
+"\xe0\x21\xfb\xb4\x5d\x93\xbb\x87\x01\x3e\xa6\x9d\x3b\x0a\x5a\x37",
+	.ilen = 64,
+	.result =
+"\x83\x9f\xa0\xac\x14\x14\x88\x68\x7f\x9a\x5f\x98\x91\x71\xa8\xce"
+"\x28\xfb\x5e\xb1\x49\xe7\x63\x39\x12\x62\x00\x3e\x5c\x63\x2b\x12"
+"\x3d\xff\xd5\x0a\x43\x28\x52\x68\x78\x62\xc7\xa4\xbb\xca\x5d\x5e"
+"\xe3\xd5\x23\xb3\xe7\x22\xae\xf3\xd0\xd9\x00\x14\x0c\x46\x67\x17",
+	.rlen = 64,
+	.np	= 2,
+	.also_non_np = 1,
+	.tap	= { 32, 32 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xf9\xe8\xab\xe2\xf9\x28\xeb\x05\x10\xc4\x97\x37\x76\xe4\xe0\xd9",
+	.input =
+"\xab\x99\xe8\x2a\x18\x50\xdc\x80\x1f\x38\xb9\x01\x34\xd4\x59\x60"
+"\x4e\x1c\x21\x71\x22\x06\xbe\x5f\x71\x07\x3b\x13\xe7\x05\xca\xa5"
+"\x7b\x23\xb5\xaa\xc6\xdb\xe3\x17\xa9\x9f\xe1\xbc\xd5\x1b\xe6\xf5"
+"\xfa\x43\xdd\x80\x50\xc8\x8a\x32\x2f\x65\x25\xa4\xeb\xd1\x74\x02"
+"\x07\xc1\x04\x94\x6b\x34\xa1\x74\x62\xb2\x8d\x60\xf5\x7e\xda\x1a"
+"\x0f\xf5\x21\xe1\xd7\x88\xc8\x26\xd7\x49\xb2\x4a\x84\x2c\x00\x3b"
+"\x96\xde\x4e\xa7\x57\x27\xa0\xa4\x3a\xff\x69\x19\xf7\xec\xeb\x62"
+"\xff\x5a\x82\x0d\x25\x5e\x3c\x63\xb3\x6d\xc4\xb9\xe3\xc9\x3a\xc2",
+	.ilen = 128,
+	.result =
+"\xec\xd5\x2f\x6a\xfd\x61\xf2\x37\x19\x6f\x55\x31\xd7\x2c\x14\x4d"
+"\xc1\xb4\xbb\x7d\xa9\x1a\xe6\x85\x8c\x2f\xbf\x7e\x66\x21\xf8\x17"
+"\x9e\x09\x1b\x2a\x11\xbf\xdf\x7d\xdf\xf5\xfb\x0a\x16\x79\xe2\x43"
+"\x5c\x3b\x3e\x84\x35\xfd\x92\x9e\xe0\x31\x50\x1d\x62\xd6\x22\x99"
+"\x5f\x25\xb3\xe8\xdf\xb0\xc0\xab\xd9\xdb\xac\x4b\x9c\xe2\x89\xc6"
+"\x49\x7f\x5f\xee\xcb\xf6\x25\x10\x9f\x32\x58\x85\x45\x50\x74\x8a"
+"\x55\xce\x86\x44\xda\xe4\x93\x58\x4d\xd3\x73\x76\x40\xf6\x92\x8b"
+"\x99\xc1\x2b\xf9\x18\xd0\xfa\xd0\xa6\x84\x03\xf5\xd4\xcb\xfa\xe7",
+	.rlen = 128,
+	.np = 3,
+	.also_non_np = 1,
+	.tap = { 64, 16, 48 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x58\x1e\x1a\x65\x16\x25\xaa\x55\x97\xcd\xeb\x4c\xd6\xb3\x9c\x2b",
+	.input =
+"\xef\x85\x0b\xe5\x02\xd5\xce\xcc\xad\x2d\x5e\xec\x1e\x01\x8c\x28"
+"\xf0\x2c\x23\x10\xaa\x84\xf0\x61\xe2\x56\x29\x21\x9f\x09\xaf\x9d"
+"\x7d\xfc\x60\x16\x4c\x67\xdd\xdf\x74\x35\x49\x81\xca\x68\xb6\xc7"
+"\x31\x9f\x49\x29\x96\x01\xb9\x3c\xfb\xa3\x00\x04\x05\xd8\xe6\xa2"
+"\x3f\x0c\xee\xfc\xd6\x88\x7a\x2d\xd6\x32\x27\x15\xe3\x53\xa0\x52"
+"\x1d\x96\x5a\x95\x09\x0c\x5f\x07\xc8\xdf\xab\xc7\x78\x57\x6a\x49"
+"\x09\x88\x54\x2e\x80\x27\xb0\x8a\x40\xb8\x9e\x7a\x22\x85\x8d\xaa"
+"\x95\x48\x45\xf5\xfd\x6f\x4c\x69\xe3\x38\xa6\x05\x69\xf0\xba\xb5"
+"\xd5\x9a\x9f\x77\x98\x23\xef\x98\x1f\xf3\xfe\x53\x23\xf6\xc6\x74"
+"\x6a\x2f\x1b\x34\x75\xd0\x51\x0c\x88\x10\xf9\x80\x19\xaf\x4f\xf1"
+"\xb1\xf3\xc0\x0e\x3a\x7d\x63\x3e\xbd\xb9\xe9\x3c\x69\x56\x0d\xb9"
+"\x8d\x69\xea\xb9\xa7\x39\x4c\x5d\xb8\x06\xa3\x1b\x66\x66\x14\x80"
+"\xe1\x8f\xf3\x65\x0c\xd5\x39\xe4\xed\xb9\x1f\x88\x74\x49\xd7\x4f"
+"\xc1\x4b\x3d\xea\x5d\xa2\x44\xd6\xad\x5d\x8d\xd1\xf7\x56\x9c\x9e"
+"\xda\x52\x56\x51\x00\x14\x1b\xb4\x00\x6b\x83\x4f\x41\x0b\xba\xaa"
+"\x11\xe6\xee\x23\xf7\x85\xa9\xb9\xd8\xe3\xbd\xbb\x7b\x83\x5f\xf8",
+	.ilen = 256,
+	.result =
+"\xc8\x8e\xbf\x95\x57\xa8\xcd\x47\xbc\x32\xee\x76\x97\xee\x02\x12"
+"\x11\x36\x81\xaa\x5b\xd9\xb3\x14\x80\xf3\xab\x62\x9b\x7f\x99\x98"
+"\x3b\x46\xd6\xfb\x68\xc8\xce\x1d\xa5\x47\x79\x6a\xdf\x7c\xda\x01"
+"\x44\x01\xfc\xed\xab\x2a\x51\xae\x2f\x72\x60\xed\x61\xc5\x23\x1d"
+"\xc7\xb5\x3c\xb7\x0b\x29\x62\xd6\x77\x8c\xea\x51\x0c\x39\x90\xe7"
+"\x99\x8c\x5d\xb7\x16\xf3\xc6\xea\xe0\xff\xc3\xd7\xc8\x1a\x7d\xde"
+"\x4d\x25\xaa\x0b\x90\x0d\x49\xd7\x98\x44\x4b\x75\x46\x01\x30\xa3"
+"\xdc\x47\xd9\x66\xc7\x7a\xcb\x4a\x33\x69\x60\x5d\x96\x73\x31\xf1"
+"\xce\xdc\xa9\x15\xb5\xae\x08\x2b\x08\x4a\xbc\x9b\x68\x1e\x49\xe4"
+"\x6e\x11\xe8\x61\x37\x58\x66\x69\x67\x97\x65\x1d\xd4\x62\x7c\x29"
+"\x10\xba\x8f\x2f\x0f\x23\x3d\x72\xb1\xcf\x01\xbc\x73\x10\xd8\xde"
+"\x21\xe6\xfc\xce\x3b\x3e\x19\xdc\xc2\xa7\x87\x62\x33\x88\xb4\x37"
+"\x1f\xfc\x1a\x2b\xef\x14\x24\x4a\xb5\x86\x55\x45\xf8\xc4\xcd\xaa"
+"\x0d\x8a\x5a\xdc\xfd\x7b\x41\xd7\xa6\x8f\x05\x25\x4a\x61\xcb\xa7"
+"\x14\x84\x21\xfc\xa6\x4b\x0f\xaa\x7d\xc6\xa2\x04\x04\xff\x39\xfc"
+"\x27\x8d\x7a\xce\x94\x31\x7c\xb4\xd5\x90\xbd\xb6\xdb\x6a\x55\xd9",
+	.rlen = 256,
+	.np	= 4,
+	.also_non_np = 1,
+	.tap	= { 32, 48, 80, 96 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xed\xa3\x2d\xa6\x9f\x5e\x38\x6f\xbb\xf9\xb3\x32\xae\x73\x05\x87",
+	.input =
+"\xf6\x24\x50\x2d\xa4\xfb\x09\x41\x95\xcd\x25\x13\xc0\xdc\x26\x0b"
+"\x20\x66\x70\x79\xc5\x58\xde\x63\xba\x37\x13\xb2\x0a\x40\x58\xef"
+"\x37\xcb\x04\x38\x10\x6a\x90\x97\x14\xd2\x71\x03\xa4\xa3\x6a\x59"
+"\x15\x6e\x5d\x45\xc9\xcc\xa9\x47\x8c\x0f\x1d\x6c\x62\x06\x90\xba"
+"\xef\x1c\x23\x4d\xc4\xa0\xa5\x56\x49\x19\xa9\xb1\x2a\xdd\x26\x00"
+"\x6e\xed\xd8\x4d\xd4\x3f\x68\x99\x24\xe2\xfe\x99\xb4\xe6\xf8\x3f"
+"\x60\xef\x97\x5f\x87\xa6\xde\x82\xc4\x11\xf6\x91\x7f\xd4\xa6\xa8"
+"\xee\x97\x41\x43\x14\xd2\x6e\x8d\x72\x30\x83\x5b\x67\x01\x38\xa2"
+"\xca\x93\xf4\x1e\x80\x2e\x8f\x7e\xc3\x78\xf0\xc1\x68\xb4\xf9\x1f"
+"\x15\x3c\x5c\x8b\xa1\xb5\x2f\x0c\xbf\xf7\x21\x74\xdb\x57\x98\x85"
+"\xe9\x89\x16\x20\x8b\x7c\x71\xef\x46\xc0\x78\x04\x23\x3b\x58\x24"
+"\x51\xa1\xa6\xfc\x6e\x9e\x29\x95\x55\x4c\x05\x1c\xc5\x9a\x59\x7e"
+"\x40\x4d\xe8\x81\x76\x41\x6f\x15\xde\x68\xce\x2d\x42\x03\x74\x73"
+"\xd3\x9a\x9c\xac\xa5\x05\x03\x7e\x53\x6e\x53\xa2\x57\x36\xee\x0c"
+"\x05\xde\xa9\x61\x55\xb9\x9d\x7d\x18\x18\xaa\x20\xde\xb8\x43\xd7"
+"\x30\x8e\x20\xc0\x78\xbe\xec\x24\xcf\xd7\x66\xb7\x5a\x1f\x5a\x81"
+"\xec\x19\x48\xc3\xa7\x62\xbf\x83\xbb\xbd\xf4\x51\xec\xb5\xec\x90"
+"\x05\xe1\xa9\xbf\x4d\x9b\x30\xf1\xb9\xa6\x49\xe9\xad\x65\x0d\x08"
+"\x1f\x3f\x81\xa5\x40\x4f\x3d\x42\xd8\x68\x29\xe3\x6c\xcc\x4d\x20"
+"\x7e\xb9\x0c\x33\x1f\x20\xd2\xaf\x39\xd6\xb4\x20\x06\xd0\xc3\x54"
+"\xcd\x96\x84\x88\x13\xc0\x09\x57\x18\x90\xad\xec\x18\xab\x72\x0b"
+"\xb4\x4c\x0a\x65\x67\x2a\x96\x2c\x98\x58\x6f\xdf\xc0\xe4\x51\x7c"
+"\xc8\x66\x1d\x21\x91\x1f\xab\xac\xed\x86\x38\x70\x54\x6f\x0c\xbf"
+"\x1a\xea\x9b\x33\xf4\x7c\x99\x0c\x0a\xdf\x39\x25\x78\x3b\x8d\x9c"
+"\x46\xc0\x07\x08\xfa\xeb\x19\x12\xf8\xc1\xf7\x18\x13\xbd\x7f\xd1"
+"\xa4\x3c\x7e\x03\xbd\xcf\xa1\xf3\x37\x4a\x4d\xc3\xaa\x23\xed\x58"
+"\xca\x68\x35\x91\x3e\x23\x09\xb8\xf3\x8d\xc3\x1b\x23\xe8\x1c\xda"
+"\x41\x90\xa2\x4b\x48\xb5\x7c\xa0\x8d\xaf\x66\x5e\xad\x7f\x06\xa2"
+"\x62\x32\x40\x69\x41\xb1\x2f\x6c\x0e\xf9\xd1\x48\xbd\xfc\x44\x0f"
+"\x65\x5e\xa1\x38\x83\xea\xfe\x42\x53\x9a\x2a\x85\xea\x92\xf6\x29"
+"\xbf\xb5\x78\x1e\x8d\x03\x6b\x09\xaf\x94\x4b\x39\x20\xc1\x17\x20"
+"\x95\x42\xfe\x72\x02\x10\x61\x21\x0f\x23\xcb\x33\x35\x52\x57\x9e",
+	.ilen = 512,
+	.result =
+"\x25\x3d\xad\x25\x4f\xb4\x50\x55\xbf\xc1\x66\xe3\x52\x22\x01\x10"
+"\xde\xed\x83\xc0\x18\x49\xda\xa4\xdb\xf1\x2f\x73\x90\x6f\xf2\x4f"
+"\x9b\xa2\x32\x2b\x6f\xc7\x80\xc8\x47\xbd\xf3\x24\x8a\xcd\x9b\x8d"
+"\x00\x33\xd1\x6a\xf2\x5f\xf2\xc7\xd8\x7c\x3a\x84\x1c\x12\x3c\x3e"
+"\xe0\x58\xb7\xc9\xf8\x73\x9e\x98\x2f\x8f\x03\x38\xe2\xc2\xb9\xae"
+"\xb6\xc6\xef\x78\xd0\xfa\xbf\x81\xcc\xf7\xb3\x82\x5b\x80\xb9\x0b"
+"\x57\xe3\x33\xa6\xfc\x3c\xd1\x78\xc7\x61\xc5\x5a\xe9\x01\xf5\xf7"
+"\x87\x0f\xa4\xe7\x90\xdf\xd5\x9f\x79\xc5\x5c\x1a\x2c\x29\x8e\x79"
+"\x10\xbc\xb2\xc6\x89\x9d\x95\x65\xa8\x25\xb3\x20\x97\xcc\xdf\x62"
+"\x2f\x9c\x85\x36\xe6\x34\xcc\xc0\xee\x7e\x10\xf6\x07\x57\xed\x2e"
+"\x60\x7e\x5e\xa0\x8e\x4c\xec\xe8\x73\xa3\x55\x4d\x7f\x6d\xff\x8c"
+"\x7a\x8c\x62\x3b\x10\x22\x75\xc0\x0b\x4a\x99\x83\x4d\x09\x80\x36"
+"\x41\x33\x19\x53\x9b\x51\xa6\x92\x82\xd8\x97\xe7\x98\x42\x36\x0d"
+"\x93\xb2\xf4\xbf\x96\xc7\x71\xfb\xc1\xf7\xf0\x94\xa3\x88\x28\xfa"
+"\x7c\xef\x3b\x1c\x77\x72\x23\x9b\xaf\x8c\x6a\xf8\x2b\xb2\xd4\xb9"
+"\xeb\x7f\x9f\xa5\x02\x50\x08\x47\x52\x6c\xaf\xe7\x73\x71\x85\x72"
+"\x49\x6b\xc8\x47\x88\xa7\xd8\xc2\x16\xbf\x3c\xe9\x22\x21\xeb\x54"
+"\xd1\xcd\x43\x18\x08\x8f\xa1\xcf\x1c\x2b\xa7\xfd\x65\x4a\x9d\x12"
+"\x0d\xdb\xd5\xf6\x1a\x97\x64\x83\x3c\x5a\x04\xa8\x15\x9d\x61\xd3"
+"\x43\x2a\x56\x35\xed\x08\xb7\x41\xc6\x49\xba\x02\x14\x59\xab\xca"
+"\x84\x1f\xfb\x67\x3a\x00\xe5\x41\xb8\xd1\x6e\x5c\x9d\x6f\xf2\x76"
+"\x3e\x21\x5d\x34\x5c\x78\x0d\x41\x5a\x4f\x62\x69\x1a\x76\x42\xee"
+"\x84\x6b\x1d\x47\x42\xeb\xb2\x11\x8f\x08\xb8\xc8\xea\xf4\x0d\xf7"
+"\x5d\x51\x4c\x4b\xed\x2d\x1b\x48\x30\x38\x38\x58\x0d\xe3\x2d\x80"
+"\xd9\xfb\xed\xe0\xc4\x55\xfe\x4f\x3f\xcf\x55\x57\x08\xaa\xa8\xa2"
+"\xa5\x5a\xe4\xff\x19\xf2\xae\x29\x74\xb9\x40\xea\xf4\x4d\x58\xac"
+"\x9f\x48\xea\x0f\xe0\xb0\xae\x72\x9f\xd8\x34\x95\x59\x01\x20\x7c"
+"\x98\x5d\xe6\x9f\x37\x23\x52\x8d\xa0\x62\x2b\x3a\x9c\x2e\x31\xe7"
+"\xd5\x75\xcc\x4c\x62\x2f\xa4\x3e\x2e\xb9\xe6\xe1\x4b\x69\xb4\x62"
+"\x31\x03\xfc\x08\xfd\xba\x87\xb9\x79\x3a\x68\x19\x65\x49\x2e\x2c"
+"\x65\x5f\xd8\x60\x07\xf4\x73\x8d\xdf\x37\x7e\x00\x88\xaf\x23\x48"
+"\x8b\xad\x74\x9c\x0b\xa3\x3a\x1a\x4b\xa0\x27\x6f\x04\x8d\xd9\x38",
+	.rlen = 512,
+	.np	= 5,
+	.also_non_np = 1,
+	.tap	= { 32, 64, 96, 128, 192 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xfd\xd6\xe8\x2f\xfe\xd4\xfe\x42\x23\x4b\x7c\x09\x8c\xde\x4f\x4b",
+	.input =
+"\xff\x7f\xb0\x11\x72\x5a\x91\x4a\xb5\x2d\xb0\x41\x3a\x96\x0d\xa1"
+"\xd9\xbe\x60\x09\x24\x51\x73\xb8\x00\xf0\x48\x1f\x6b\x96\x5b\xe7"
+"\x4d\x47\x88\xc7\xef\x4b\xb4\x33\xa1\x2b\xbe\xdd\x46\x4f\x27\x11"
+"\x8b\x30\x9c\xba\x2c\x7a\xf3\xdb\x48\x54\xbd\xfe\x24\x2f\x83\x91"
+"\x5c\x63\xb9\x12\xd9\xd9\xb9\x71\xcf\x28\x7e\xf8\xe0\xb8\x12\xf7"
+"\x63\xad\xde\x49\xd5\x4d\xa7\x13\x32\xee\x71\x13\x56\x4d\x10\xd5"
+"\x2c\x1d\x8e\x94\x0d\x37\x3d\x7e\x9c\xb4\xeb\xe5\x6f\x12\x30\x7f"
+"\xc3\xa0\xf3\x49\xac\xa6\xab\x1b\xec\xd4\x6c\x95\x2a\x57\xe0\xfa"
+"\x89\x00\x61\xe9\xea\x21\x9a\x2f\x71\xd7\xdb\x11\x52\xb6\x32\x91"
+"\xed\xa3\xdf\xa5\x46\xc1\x50\x5b\xab\x15\x43\x7f\x7d\x82\x34\xf2"
+"\xfa\x6e\x84\xaf\x40\x20\xde\x1f\x90\x39\xab\xdc\xe8\xf3\xf9\x65"
+"\xbc\xdc\xd3\x5c\xcf\xe2\x1b\x43\x08\x68\xd8\x0d\xfb\xc2\x7f\x31"
+"\x91\xb5\x66\x2a\xea\x43\x08\x6d\xa6\xb4\xd3\x0e\x78\x3c\xf1\x6c"
+"\x4d\x27\x47\x7d\x92\x42\xb1\x62\x82\x9f\x13\xdf\x51\xc3\x6b\xec"
+"\x83\x53\xd6\x89\x75\xac\x62\x9a\x89\x7d\xf9\x82\x66\xbe\x93\x6f"
+"\x71\x7d\x01\x79\xec\x10\x10\x50\xe9\x6c\x76\xc6\x7a\xfa\xbb\x69"
+"\x46\x09\x1a\x68\x2f\x07\x28\xf4\xd0\xb6\xb4\x82\xf5\x3a\x90\xdc"
+"\x61\x03\xd9\x8e\xa5\x13\xfd\xdd\xe0\x65\x03\xfb\x78\x6b\x4e\xae"
+"\x7f\x30\xe2\x9e\x39\xb1\x3a\x39\xda\x21\x80\x2c\x09\xdd\xe8\xa2"
+"\x8c\x4a\x2c\x40\x24\x39\xf0\x3f\x7f\x51\x6a\x48\xea\x7b\x68\x3d"
+"\xad\x56\xed\xbe\x86\x0a\x9a\xe6\x9f\x18\x95\x26\x14\x57\x5b\x71"
+"\x9e\x8d\x45\x0d\xad\x23\xb4\x37\xa5\x59\x66\x8c\x13\x8e\x5e\xeb"
+"\xbf\x4a\x0d\x72\xc9\x4a\xcf\x42\xbd\x28\x1f\x91\xad\x55\x81\x78"
+"\x48\xf3\xed\xab\x2b\x6d\x61\xc7\x08\x2c\x07\xcb\x17\xf8\xf1\x7c"
+"\x39\xc8\x44\x63\x3a\x2a\x55\xbe\xe1\xb5\x12\x61\x0a\x4c\x32\x83"
+"\x9a\xa0\xf8\x93\x8c\xfa\x45\x92\x4e\xad\x48\xd9\x84\xe8\x0d\x7a"
+"\xca\xad\xbf\xd2\x5a\x1d\x58\x67\x57\x68\xca\x2f\x40\xa5\x1c\x38"
+"\x2a\xde\xa7\x57\x87\x4f\x11\x97\x3e\x11\xe7\x58\x54\xbd\x06\x48"
+"\xf7\x60\x45\x5b\x9d\x08\x5a\xef\xf9\x28\xa5\xf5\x48\x5c\x9c\xa0"
+"\x96\x76\x56\x51\x40\xec\xbe\xdb\x6e\xba\x4b\xb0\xa2\xe9\x55\xe6"
+"\xb7\x7e\x8a\x06\x3b\xeb\x17\xeb\xe6\xd9\xf6\xb2\xa1\x8c\x9e\xcc"
+"\xf3\x89\xd5\x78\x29\x1f\x74\x60\xe2\x61\x72\x78\x05\x52\x23\x07"
+"\x2a\x46\x85\x3c\xcf\x12\x9a\x9d\x3d\xf0\x93\x0e\xd2\x22\x63\x07"
+"\x01\x8b\x96\x73\xb5\x26\x29\xf5\x4f\x90\xf9\x37\x55\x76\x15\x02"
+"\xe8\x4c\x56\x3e\xf1\x14\xaf\x34\x0d\xa8\xde\xee\x0e\x13\xfa\xb8"
+"\xe4\xb7\x6d\x71\x37\xdb\x1e\x42\xdd\xca\xec\xe1\x99\xf9\xc7\x18"
+"\x16\xb0\x41\xd0\xfe\x9a\xa6\xa0\x7a\x5e\x5d\x0a\x96\x4c\x52\x44"
+"\x9a\x29\x69\x09\xa2\x0e\x5a\x1e\xc2\xb3\x5e\xca\x25\xc0\xe1\xa9"
+"\xd1\x41\x7f\x82\xaf\x1f\xf4\x3c\xf8\x3d\x65\xae\xf0\xa2\x1a\x8f"
+"\x41\xdb\x01\x11\x4c\x01\xcb\x24\xb3\xec\xbb\xf3\xe5\x1b\x53\xf0"
+"\x7a\x81\x01\x61\xa2\x8e\xa4\xd0\xaa\x8f\xa1\x71\xc1\x15\x15\xda"
+"\xf3\x7b\x32\x87\xa6\xb7\x7f\x2b\xac\x2b\x28\xfc\xe4\x1a\x94\xab"
+"\x19\xc9\x13\x72\x33\xfa\x42\xec\x6f\x3f\xe1\xe0\xc7\x23\x4b\x17"
+"\xeb\x89\xd3\x1f\x49\xe1\x49\x56\xee\xe3\x82\x46\x43\x00\x80\xbc"
+"\xa3\xfe\x31\xbc\xc9\xcd\x61\x5b\x7a\xf9\xf7\xb7\x48\x98\xbf\xdc"
+"\x79\xca\x71\x3b\xb0\xda\x08\x1e\x25\x97\x83\xd7\x21\x2c\xaa\xc0"
+"\x5c\xfd\x7f\xc4\x30\xd8\x7b\x59\x35\x49\x62\x0f\x4c\x03\x02\xe5"
+"\x73\x63\x61\x0b\x69\x2f\x7d\xb3\x99\xc9\x6b\x0a\x29\x9b\xda\xbe"
+"\x98\xdc\x2c\x29\x28\x9a\x75\x2e\xf1\x11\xd3\x71\x5b\x20\x45\x5b"
+"\xb7\x5e\xc1\xd1\xcc\x4e\x5a\x0d\xa5\x70\xa6\x56\xb8\x80\x8c\x97"
+"\x9d\x65\x8d\xec\xa0\x15\x45\xe6\x04\xd8\x3b\x6b\x36\x3f\x71\x58"
+"\x9e\x7a\x9c\xd2\x44\x86\xbf\x89\xa6\x80\x5d\x5e\x99\xc9\x7e\x56"
+"\x76\x17\x02\x98\x5b\xbb\xa0\xe5\xe5\x10\x25\x3e\x82\xc7\xe0\x91"
+"\x77\x39\x50\x9c\x3d\x2a\x91\x03\x13\x6d\x6d\xd3\xc6\x68\xd3\xa0"
+"\x88\xbc\x24\x5d\xf1\x26\x19\xf4\xb0\x74\x51\x93\x17\xcf\x67\x6c"
+"\x72\x30\xed\x39\xfe\x59\x54\x88\x84\x70\x56\x11\xaf\x41\x66\xa5"
+"\xf9\xf0\x95\xdb\x80\xb8\xae\x2f\xb7\xc3\x65\x72\xd2\xec\xaf\x5f"
+"\xf9\x30\x1e\x5b\x45\x7f\x38\xd5\x03\x02\x60\xaa\xf9\xb7\xd9\xfc"
+"\xa2\x5c\x46\x3e\x9c\xe6\xd6\x8e\x95\x54\xbf\xd8\xe6\xe4\x4b\xc0"
+"\x4c\xa1\x4c\x2c\xb3\xc4\x9f\xef\xeb\x39\x70\x77\xac\xf9\x1f\xb6"
+"\x06\xa2\x53\x7d\x18\xc8\xf8\xda\x8e\x82\x97\x4f\xdd\xd5\x19\x2f"
+"\xa2\x70\x4a\xbd\x5a\x15\x70\xb6\x55\x04\x14\xba\x0a\x04\xdc\x8e"
+"\xaf\xf2\x52\xd5\x90\x4c\x30\xd3\x29\x53\x1c\x66\x37\x5f\x8e\xfc"
+"\x45\x83\xd9\xac\x75\x9e\x0f\x66\x51\xc0\x8a\xc5\x34\x25\x9e\x3b",
+	.ilen = 1024,
+	.result =
+"\xa8\x47\xa1\x1d\xcb\xa3\x88\xae\x42\xab\x6d\xf2\x82\xc2\xed\xd5"
+"\x66\x42\x09\x85\x28\x7d\x49\x6f\x37\xdc\xff\x1c\x7e\x33\xc9\xcd"
+"\x6e\xe9\x33\x36\x01\x62\x1d\x67\x77\x6a\x97\xbf\xb1\xdc\x2f\x98"
+"\x2c\xdb\xac\x44\x9d\xed\x31\x7d\x2d\x41\x4b\xd1\x66\x40\x62\x74"
+"\xdc\x00\xd0\x05\xdc\x54\x4c\x63\xeb\xd9\x42\xe1\xdf\xc4\xde\xdd"
+"\xb6\xb8\x93\xfd\x25\x39\x2d\x7f\x85\xf8\x15\xc3\xbc\xbf\x0b\x95"
+"\x11\xef\x57\x0d\x15\x49\x07\xce\x42\xb0\x50\xe1\x07\xb4\x81\x71"
+"\x35\x71\x4b\x66\x89\x7f\x94\x13\x3e\x57\x43\xc3\x36\x28\xcd\xdd"
+"\xc9\x06\x68\xf8\xf3\x09\x3d\x86\x12\x52\x06\xa9\xe9\x83\x2d\x8f"
+"\x90\xfa\x42\xfe\x79\x3f\x68\x4c\x7b\xfa\x94\xa7\xf7\x16\xc7\x41"
+"\x09\xae\xe2\x82\xb5\x2b\xbc\xca\x65\x65\x2c\x27\x2c\x07\x50\x83"
+"\x2d\xad\x55\xaf\x35\xcc\x6a\xc5\x7c\xd8\xed\x75\x91\x9d\x73\xcb"
+"\x4c\xa5\x8f\xc4\x4f\xda\xa8\xb9\xb6\xa7\xb1\x1a\x75\xb4\x08\xbc"
+"\xb2\x90\x50\xfd\x1f\x05\xa8\x88\x35\x81\xb0\xc9\xac\xbc\x25\x7a"
+"\x95\x33\x02\x2b\x74\xe0\x95\x11\x88\xf7\xc3\x63\xb3\x7b\x09\xd5"
+"\xac\x22\x04\x67\x16\xea\xd6\x37\x38\x8e\xa5\xbd\x62\xa2\x1f\xa5"
+"\x04\x31\x89\xdf\x69\xb1\xde\xe3\x7c\x9d\x7b\x27\xba\x0a\x74\xdc"
+"\x06\x1c\xcd\x6e\x4b\x52\xe7\x6d\x34\x29\x38\xe2\x19\xfc\x0c\xc4"
+"\x78\x03\x1d\x53\x98\x00\x5c\x7a\xec\x23\x5f\x95\xd5\xb3\x16\xde"
+"\xc2\x17\xc2\x0c\x13\x63\x0a\x4b\x7e\x6c\xc7\xbc\x4a\xd0\xae\x29"
+"\xc0\x50\x16\x6f\x01\x2b\xdc\x40\x9f\x91\x8f\xa3\xaf\xd4\x40\xa8"
+"\x2e\x09\x7c\xf4\x3d\x85\xe6\xd9\x3c\x78\x7c\xf1\x6d\xe4\x13\x00"
+"\x98\xf5\xb4\x06\x9f\x90\x0a\x3e\x9f\x51\x0f\xbb\x0f\x13\x07\xc0"
+"\xfd\x26\x53\x24\x24\xf7\x21\x41\xcf\x20\x9d\x77\xe4\xe0\x52\x2a"
+"\x48\xd9\xeb\x65\xce\xf3\x90\x03\x47\x8d\x2b\x77\x54\x46\xda\xff"
+"\x15\x3d\xa5\xd9\x5a\xb6\xd3\xdf\x9c\x91\xc3\xf2\xd2\xdf\xd7\x8c"
+"\x1d\x83\x77\x47\xcd\x74\x23\x44\x04\x06\x8e\x64\x62\x29\xe5\xa0"
+"\xf7\xa7\xc7\xb7\x84\xdb\x9c\x5c\x04\x7f\xca\xb3\x85\x2c\x44\xa6"
+"\x09\x0e\xa3\x2c\x52\x42\x25\x02\x63\x99\xd0\xa5\x27\x61\x64\x4f"
+"\x65\xd7\x31\x56\x24\x97\xb0\x2d\xbb\x0c\xbe\x06\x68\x8a\x2e\xa3"
+"\x0c\xb9\x05\x52\xdb\xbd\x7e\x89\x60\x2e\x28\x76\xba\x5a\x94\xb6"
+"\x94\xc4\xf6\x68\x50\x35\x24\x7b\x2b\x04\x0e\x4c\xf3\x17\x54\xcb"
+"\xcd\x32\x18\x60\xff\xc9\xfe\xe1\x83\xe4\xe6\x9b\x5e\xd8\x21\xbf"
+"\xbf\x69\x01\x3a\x03\xc6\x9f\xe5\xd4\xdf\x01\x20\x8e\xea\x5b\xe1"
+"\xbd\x46\x3c\x3a\x60\x30\xa0\x48\xa0\x07\x82\x27\x4e\x03\xc3\x15"
+"\x98\x1f\xea\x4f\x8c\x90\x4d\xb1\xc5\x90\x40\x59\xda\x5b\x02\x65"
+"\x07\xb9\x64\xe7\x4c\x76\x70\x16\x8a\xc3\xf9\x4f\xed\x25\x47\xaa"
+"\x3b\x49\x8f\xf6\xf0\x71\x94\x34\xda\x29\x0f\x4e\xd4\x95\x3b\xe3"
+"\xef\x99\x3b\x1c\xf7\x09\x5d\xe0\x0d\x03\xe6\x9d\x47\x4c\x8c\xe8"
+"\x26\xb6\x30\x1b\x81\xdc\xa5\x5a\xf1\x04\x18\xf3\xaf\x81\xa2\x7e"
+"\xce\x8b\x33\xfc\xf2\xb1\x5a\x06\xd1\xb9\x59\x73\xd7\xda\x85\xd9"
+"\x30\x73\x98\x4d\x63\x50\x66\x71\x15\x88\x9a\x5d\xd5\x25\x40\x9a"
+"\xe3\x9c\x0b\x4f\xd8\xf5\xbf\xb3\xec\x02\x95\xca\x90\x07\x5d\x99"
+"\x9e\x16\xa2\x18\xa5\xa2\x03\xb1\x16\x6b\x4e\x32\xab\x19\x29\x55"
+"\xcc\xbe\xa8\x7b\xf7\x68\x64\x0e\xc0\x54\x91\x6d\x19\xec\xe9\x8c"
+"\x56\x5e\x71\xa5\x73\x50\x5d\x0d\xd3\xb2\x31\xca\x97\x7b\xf8\x6e"
+"\xfd\xb9\x47\x9b\x17\xf9\x56\x3a\xc6\xb0\x52\x45\x4f\x4a\x13\xe9"
+"\xb7\x64\x02\xdb\xe8\x67\xa3\x9e\xe4\xd9\x49\xc4\xf3\x27\xe3\xb0"
+"\xad\x6e\x51\x65\x14\x4f\xb2\x4b\x8a\xd6\x87\x17\x8c\xe2\x7a\xa1"
+"\x13\xbb\x8c\x7c\x3e\x69\xd2\x29\x06\x36\xf3\x55\x80\xcc\x0e\xa5"
+"\x18\x5a\x5f\xcb\x15\x2e\x7c\x62\xff\x3f\xe7\x7b\xd8\xe4\xa6\x9c"
+"\x4c\x5b\x55\x73\x4a\x0d\x21\x07\xf9\x79\xcb\x17\x51\x06\xf3\xcc"
+"\xfc\x08\x72\x6e\xbc\x04\xe2\x6d\xd8\x52\x1d\x29\x7e\x7a\x06\x8d"
+"\x87\x65\x2e\x2e\x7c\x07\x77\x3a\x35\x4d\x3a\x13\xd3\xf6\xc2\x1f"
+"\x2d\x5d\x14\xa5\x04\xe5\xc5\x7b\xd6\xa9\x70\x4b\x43\x21\x93\xdf"
+"\xe4\xf1\xf8\x75\xf1\x65\x9c\xf8\x0b\x07\x31\xdc\xf2\xba\x06\x91"
+"\xe1\x84\x87\x34\x2d\xdd\xa7\x87\xc0\xc2\x4d\x8d\xe0\x18\x70\xbb"
+"\xe3\x3e\x13\x48\xfc\xf4\x13\x85\xc4\x65\xcf\xe4\x43\x98\x14\x8f"
+"\xf4\x17\x62\x27\x39\xe5\xb6\x45\x76\x61\x78\x0b\x3d\x48\xb3\x41"
+"\xa6\xca\x7c\xed\x52\x19\x99\xea\x73\xc9\xd0\x0b\xeb\xbb\x5a\x69"
+"\x44\x3d\xb2\x81\x25\xb0\x2f\x08\xf0\x8c\x32\xa9\xf0\x79\x3c\x42"
+"\xc3\xdc\x9e\xd1\xec\x93\x49\xc9\x82\x0e\x13\x12\xb3\x8a\x98\x1b"
+"\x35\xe1\x4a\xef\xb4\x73\x28\x1a\x17\x96\xe2\x9a\x50\xc8\xd5\x98"
+"\xec\x96\x6f\x81\x05\x37\xee\x8b\x93\x12\x7c\x41\x26\xd5\x9c\x05",
+	.rlen = 1024,
+	/* limit to maximum of 8 */
+	.np	= 8,
+	.also_non_np = 1,
+	.tap	= { 32, 64, 96, 128, 192, 32, 64, 96+128+192 },
+},
+};
+
+#define AES_CBC_ENC_TV_TEMPLATE_RNDDATA_KEY16_VEC_COUNT	16
+
+static struct cipher_testvec aes_cbc_dec_tv_template_rnddata_klen16[] = {
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc1\x62\x66\x62\xb8\x65\x28\xfa\x5f\x36\xd3\x09\xb1\x2c\xa1\xa3",
+	.result =
+"\x4f\x6c\x63\xa5\xd0\x19\x08\x4e\xd4\x58\x33\xf6\x2b\xeb\x26\xb9",
+	.ilen = 16,
+	.input =
+"\xa0\x35\xb0\x33\xc0\x2e\xe5\xbb\xbc\xe6\x01\x9e\xf4\x67\x11\x14",
+	.rlen = 16,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x78\x6c\x27\xd6\xb2\xdc\xbe\x7b\xab\xc2\x43\xd7\x81\x0c\xe5\x20",
+	.result =
+"\x9a\x00\x4e\x5a\xb3\x51\x68\xaa\xdb\x6e\xe5\xa4\x7f\x23\x6e\x4d"
+"\x1e\x72\x5e\xad\x64\xc9\x96\x23\xf8\xae\xef\xf6\x7b\x7d\xd6\xf0",
+	.ilen = 32,
+	.input =
+"\x5a\xc0\x04\xc6\x53\xef\x3b\x69\xb1\x41\xc7\x85\xeb\x69\x82\xd0"
+"\xed\x09\xbb\xec\xb2\x8d\x5c\xc9\x61\x81\x5c\xf6\x99\x49\xa0\x4d",
+	.rlen = 32,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc9\x05\x4c\x35\x96\x77\xd3\x3c\x3d\x97\x7c\x82\xf5\x58\x71\xf1",
+	.result =
+"\xa9\x5b\x03\xec\xec\x73\xed\xcb\x5c\x4c\xd2\x40\xb6\x9b\x49\x31"
+"\x5d\xf2\x23\xb3\x11\x98\xeb\x89\xab\x3e\x3a\xdd\xaa\xfd\xd1\xde"
+"\xab\x73\x59\x86\x1a\x59\x32\xb2\x55\x46\x4a\x80\xa4\xcc\xa8\xd9",
+	.ilen = 48,
+	.input =
+"\xdb\x05\x69\xe1\x33\x8b\x0b\x3d\x33\x12\x0d\xef\x94\x0f\xa3\xb3"
+"\xd7\x0a\x53\x7b\x98\x53\xc6\xc2\xa3\xd4\x7a\x30\x1a\xed\x45\xcc"
+"\x47\x38\xc1\x75\x0b\x3c\xd4\x8d\xa8\xf9\xd3\x71\xb8\x22\xa6\xae",
+	.rlen = 48,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x6c\xb4\x84\x61\x1e\x39\x4b\x22\x37\xaa\x7b\x78\xc0\x71\x20\x60",
+	.result =
+"\x05\x43\x76\x1e\xc6\x68\x43\x52\x5f\x43\x39\xbf\x93\x38\x38\x83"
+"\x38\x1d\x3c\xb5\xc8\xab\xe4\xd0\x7f\x1a\xac\xca\xcc\x16\xea\x75"
+"\x30\x75\x40\xe8\x61\x07\xc6\x04\x55\x2b\xf3\x29\xc3\x37\x83\x42"
+"\xe0\x21\xfb\xb4\x5d\x93\xbb\x87\x01\x3e\xa6\x9d\x3b\x0a\x5a\x37",
+	.ilen = 64,
+	.input =
+"\x83\x9f\xa0\xac\x14\x14\x88\x68\x7f\x9a\x5f\x98\x91\x71\xa8\xce"
+"\x28\xfb\x5e\xb1\x49\xe7\x63\x39\x12\x62\x00\x3e\x5c\x63\x2b\x12"
+"\x3d\xff\xd5\x0a\x43\x28\x52\x68\x78\x62\xc7\xa4\xbb\xca\x5d\x5e"
+"\xe3\xd5\x23\xb3\xe7\x22\xae\xf3\xd0\xd9\x00\x14\x0c\x46\x67\x17",
+	.rlen = 64,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xf9\xe8\xab\xe2\xf9\x28\xeb\x05\x10\xc4\x97\x37\x76\xe4\xe0\xd9",
+	.result =
+"\xab\x99\xe8\x2a\x18\x50\xdc\x80\x1f\x38\xb9\x01\x34\xd4\x59\x60"
+"\x4e\x1c\x21\x71\x22\x06\xbe\x5f\x71\x07\x3b\x13\xe7\x05\xca\xa5"
+"\x7b\x23\xb5\xaa\xc6\xdb\xe3\x17\xa9\x9f\xe1\xbc\xd5\x1b\xe6\xf5"
+"\xfa\x43\xdd\x80\x50\xc8\x8a\x32\x2f\x65\x25\xa4\xeb\xd1\x74\x02"
+"\x07\xc1\x04\x94\x6b\x34\xa1\x74\x62\xb2\x8d\x60\xf5\x7e\xda\x1a"
+"\x0f\xf5\x21\xe1\xd7\x88\xc8\x26\xd7\x49\xb2\x4a\x84\x2c\x00\x3b"
+"\x96\xde\x4e\xa7\x57\x27\xa0\xa4\x3a\xff\x69\x19\xf7\xec\xeb\x62"
+"\xff\x5a\x82\x0d\x25\x5e\x3c\x63\xb3\x6d\xc4\xb9\xe3\xc9\x3a\xc2",
+	.ilen = 128,
+	.input =
+"\xec\xd5\x2f\x6a\xfd\x61\xf2\x37\x19\x6f\x55\x31\xd7\x2c\x14\x4d"
+"\xc1\xb4\xbb\x7d\xa9\x1a\xe6\x85\x8c\x2f\xbf\x7e\x66\x21\xf8\x17"
+"\x9e\x09\x1b\x2a\x11\xbf\xdf\x7d\xdf\xf5\xfb\x0a\x16\x79\xe2\x43"
+"\x5c\x3b\x3e\x84\x35\xfd\x92\x9e\xe0\x31\x50\x1d\x62\xd6\x22\x99"
+"\x5f\x25\xb3\xe8\xdf\xb0\xc0\xab\xd9\xdb\xac\x4b\x9c\xe2\x89\xc6"
+"\x49\x7f\x5f\xee\xcb\xf6\x25\x10\x9f\x32\x58\x85\x45\x50\x74\x8a"
+"\x55\xce\x86\x44\xda\xe4\x93\x58\x4d\xd3\x73\x76\x40\xf6\x92\x8b"
+"\x99\xc1\x2b\xf9\x18\xd0\xfa\xd0\xa6\x84\x03\xf5\xd4\xcb\xfa\xe7",
+	.rlen = 128,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x58\x1e\x1a\x65\x16\x25\xaa\x55\x97\xcd\xeb\x4c\xd6\xb3\x9c\x2b",
+	.result =
+"\xef\x85\x0b\xe5\x02\xd5\xce\xcc\xad\x2d\x5e\xec\x1e\x01\x8c\x28"
+"\xf0\x2c\x23\x10\xaa\x84\xf0\x61\xe2\x56\x29\x21\x9f\x09\xaf\x9d"
+"\x7d\xfc\x60\x16\x4c\x67\xdd\xdf\x74\x35\x49\x81\xca\x68\xb6\xc7"
+"\x31\x9f\x49\x29\x96\x01\xb9\x3c\xfb\xa3\x00\x04\x05\xd8\xe6\xa2"
+"\x3f\x0c\xee\xfc\xd6\x88\x7a\x2d\xd6\x32\x27\x15\xe3\x53\xa0\x52"
+"\x1d\x96\x5a\x95\x09\x0c\x5f\x07\xc8\xdf\xab\xc7\x78\x57\x6a\x49"
+"\x09\x88\x54\x2e\x80\x27\xb0\x8a\x40\xb8\x9e\x7a\x22\x85\x8d\xaa"
+"\x95\x48\x45\xf5\xfd\x6f\x4c\x69\xe3\x38\xa6\x05\x69\xf0\xba\xb5"
+"\xd5\x9a\x9f\x77\x98\x23\xef\x98\x1f\xf3\xfe\x53\x23\xf6\xc6\x74"
+"\x6a\x2f\x1b\x34\x75\xd0\x51\x0c\x88\x10\xf9\x80\x19\xaf\x4f\xf1"
+"\xb1\xf3\xc0\x0e\x3a\x7d\x63\x3e\xbd\xb9\xe9\x3c\x69\x56\x0d\xb9"
+"\x8d\x69\xea\xb9\xa7\x39\x4c\x5d\xb8\x06\xa3\x1b\x66\x66\x14\x80"
+"\xe1\x8f\xf3\x65\x0c\xd5\x39\xe4\xed\xb9\x1f\x88\x74\x49\xd7\x4f"
+"\xc1\x4b\x3d\xea\x5d\xa2\x44\xd6\xad\x5d\x8d\xd1\xf7\x56\x9c\x9e"
+"\xda\x52\x56\x51\x00\x14\x1b\xb4\x00\x6b\x83\x4f\x41\x0b\xba\xaa"
+"\x11\xe6\xee\x23\xf7\x85\xa9\xb9\xd8\xe3\xbd\xbb\x7b\x83\x5f\xf8",
+	.ilen = 256,
+	.input =
+"\xc8\x8e\xbf\x95\x57\xa8\xcd\x47\xbc\x32\xee\x76\x97\xee\x02\x12"
+"\x11\x36\x81\xaa\x5b\xd9\xb3\x14\x80\xf3\xab\x62\x9b\x7f\x99\x98"
+"\x3b\x46\xd6\xfb\x68\xc8\xce\x1d\xa5\x47\x79\x6a\xdf\x7c\xda\x01"
+"\x44\x01\xfc\xed\xab\x2a\x51\xae\x2f\x72\x60\xed\x61\xc5\x23\x1d"
+"\xc7\xb5\x3c\xb7\x0b\x29\x62\xd6\x77\x8c\xea\x51\x0c\x39\x90\xe7"
+"\x99\x8c\x5d\xb7\x16\xf3\xc6\xea\xe0\xff\xc3\xd7\xc8\x1a\x7d\xde"
+"\x4d\x25\xaa\x0b\x90\x0d\x49\xd7\x98\x44\x4b\x75\x46\x01\x30\xa3"
+"\xdc\x47\xd9\x66\xc7\x7a\xcb\x4a\x33\x69\x60\x5d\x96\x73\x31\xf1"
+"\xce\xdc\xa9\x15\xb5\xae\x08\x2b\x08\x4a\xbc\x9b\x68\x1e\x49\xe4"
+"\x6e\x11\xe8\x61\x37\x58\x66\x69\x67\x97\x65\x1d\xd4\x62\x7c\x29"
+"\x10\xba\x8f\x2f\x0f\x23\x3d\x72\xb1\xcf\x01\xbc\x73\x10\xd8\xde"
+"\x21\xe6\xfc\xce\x3b\x3e\x19\xdc\xc2\xa7\x87\x62\x33\x88\xb4\x37"
+"\x1f\xfc\x1a\x2b\xef\x14\x24\x4a\xb5\x86\x55\x45\xf8\xc4\xcd\xaa"
+"\x0d\x8a\x5a\xdc\xfd\x7b\x41\xd7\xa6\x8f\x05\x25\x4a\x61\xcb\xa7"
+"\x14\x84\x21\xfc\xa6\x4b\x0f\xaa\x7d\xc6\xa2\x04\x04\xff\x39\xfc"
+"\x27\x8d\x7a\xce\x94\x31\x7c\xb4\xd5\x90\xbd\xb6\xdb\x6a\x55\xd9",
+	.rlen = 256,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xed\xa3\x2d\xa6\x9f\x5e\x38\x6f\xbb\xf9\xb3\x32\xae\x73\x05\x87",
+	.result =
+"\xf6\x24\x50\x2d\xa4\xfb\x09\x41\x95\xcd\x25\x13\xc0\xdc\x26\x0b"
+"\x20\x66\x70\x79\xc5\x58\xde\x63\xba\x37\x13\xb2\x0a\x40\x58\xef"
+"\x37\xcb\x04\x38\x10\x6a\x90\x97\x14\xd2\x71\x03\xa4\xa3\x6a\x59"
+"\x15\x6e\x5d\x45\xc9\xcc\xa9\x47\x8c\x0f\x1d\x6c\x62\x06\x90\xba"
+"\xef\x1c\x23\x4d\xc4\xa0\xa5\x56\x49\x19\xa9\xb1\x2a\xdd\x26\x00"
+"\x6e\xed\xd8\x4d\xd4\x3f\x68\x99\x24\xe2\xfe\x99\xb4\xe6\xf8\x3f"
+"\x60\xef\x97\x5f\x87\xa6\xde\x82\xc4\x11\xf6\x91\x7f\xd4\xa6\xa8"
+"\xee\x97\x41\x43\x14\xd2\x6e\x8d\x72\x30\x83\x5b\x67\x01\x38\xa2"
+"\xca\x93\xf4\x1e\x80\x2e\x8f\x7e\xc3\x78\xf0\xc1\x68\xb4\xf9\x1f"
+"\x15\x3c\x5c\x8b\xa1\xb5\x2f\x0c\xbf\xf7\x21\x74\xdb\x57\x98\x85"
+"\xe9\x89\x16\x20\x8b\x7c\x71\xef\x46\xc0\x78\x04\x23\x3b\x58\x24"
+"\x51\xa1\xa6\xfc\x6e\x9e\x29\x95\x55\x4c\x05\x1c\xc5\x9a\x59\x7e"
+"\x40\x4d\xe8\x81\x76\x41\x6f\x15\xde\x68\xce\x2d\x42\x03\x74\x73"
+"\xd3\x9a\x9c\xac\xa5\x05\x03\x7e\x53\x6e\x53\xa2\x57\x36\xee\x0c"
+"\x05\xde\xa9\x61\x55\xb9\x9d\x7d\x18\x18\xaa\x20\xde\xb8\x43\xd7"
+"\x30\x8e\x20\xc0\x78\xbe\xec\x24\xcf\xd7\x66\xb7\x5a\x1f\x5a\x81"
+"\xec\x19\x48\xc3\xa7\x62\xbf\x83\xbb\xbd\xf4\x51\xec\xb5\xec\x90"
+"\x05\xe1\xa9\xbf\x4d\x9b\x30\xf1\xb9\xa6\x49\xe9\xad\x65\x0d\x08"
+"\x1f\x3f\x81\xa5\x40\x4f\x3d\x42\xd8\x68\x29\xe3\x6c\xcc\x4d\x20"
+"\x7e\xb9\x0c\x33\x1f\x20\xd2\xaf\x39\xd6\xb4\x20\x06\xd0\xc3\x54"
+"\xcd\x96\x84\x88\x13\xc0\x09\x57\x18\x90\xad\xec\x18\xab\x72\x0b"
+"\xb4\x4c\x0a\x65\x67\x2a\x96\x2c\x98\x58\x6f\xdf\xc0\xe4\x51\x7c"
+"\xc8\x66\x1d\x21\x91\x1f\xab\xac\xed\x86\x38\x70\x54\x6f\x0c\xbf"
+"\x1a\xea\x9b\x33\xf4\x7c\x99\x0c\x0a\xdf\x39\x25\x78\x3b\x8d\x9c"
+"\x46\xc0\x07\x08\xfa\xeb\x19\x12\xf8\xc1\xf7\x18\x13\xbd\x7f\xd1"
+"\xa4\x3c\x7e\x03\xbd\xcf\xa1\xf3\x37\x4a\x4d\xc3\xaa\x23\xed\x58"
+"\xca\x68\x35\x91\x3e\x23\x09\xb8\xf3\x8d\xc3\x1b\x23\xe8\x1c\xda"
+"\x41\x90\xa2\x4b\x48\xb5\x7c\xa0\x8d\xaf\x66\x5e\xad\x7f\x06\xa2"
+"\x62\x32\x40\x69\x41\xb1\x2f\x6c\x0e\xf9\xd1\x48\xbd\xfc\x44\x0f"
+"\x65\x5e\xa1\x38\x83\xea\xfe\x42\x53\x9a\x2a\x85\xea\x92\xf6\x29"
+"\xbf\xb5\x78\x1e\x8d\x03\x6b\x09\xaf\x94\x4b\x39\x20\xc1\x17\x20"
+"\x95\x42\xfe\x72\x02\x10\x61\x21\x0f\x23\xcb\x33\x35\x52\x57\x9e",
+	.ilen = 512,
+	.input =
+"\x25\x3d\xad\x25\x4f\xb4\x50\x55\xbf\xc1\x66\xe3\x52\x22\x01\x10"
+"\xde\xed\x83\xc0\x18\x49\xda\xa4\xdb\xf1\x2f\x73\x90\x6f\xf2\x4f"
+"\x9b\xa2\x32\x2b\x6f\xc7\x80\xc8\x47\xbd\xf3\x24\x8a\xcd\x9b\x8d"
+"\x00\x33\xd1\x6a\xf2\x5f\xf2\xc7\xd8\x7c\x3a\x84\x1c\x12\x3c\x3e"
+"\xe0\x58\xb7\xc9\xf8\x73\x9e\x98\x2f\x8f\x03\x38\xe2\xc2\xb9\xae"
+"\xb6\xc6\xef\x78\xd0\xfa\xbf\x81\xcc\xf7\xb3\x82\x5b\x80\xb9\x0b"
+"\x57\xe3\x33\xa6\xfc\x3c\xd1\x78\xc7\x61\xc5\x5a\xe9\x01\xf5\xf7"
+"\x87\x0f\xa4\xe7\x90\xdf\xd5\x9f\x79\xc5\x5c\x1a\x2c\x29\x8e\x79"
+"\x10\xbc\xb2\xc6\x89\x9d\x95\x65\xa8\x25\xb3\x20\x97\xcc\xdf\x62"
+"\x2f\x9c\x85\x36\xe6\x34\xcc\xc0\xee\x7e\x10\xf6\x07\x57\xed\x2e"
+"\x60\x7e\x5e\xa0\x8e\x4c\xec\xe8\x73\xa3\x55\x4d\x7f\x6d\xff\x8c"
+"\x7a\x8c\x62\x3b\x10\x22\x75\xc0\x0b\x4a\x99\x83\x4d\x09\x80\x36"
+"\x41\x33\x19\x53\x9b\x51\xa6\x92\x82\xd8\x97\xe7\x98\x42\x36\x0d"
+"\x93\xb2\xf4\xbf\x96\xc7\x71\xfb\xc1\xf7\xf0\x94\xa3\x88\x28\xfa"
+"\x7c\xef\x3b\x1c\x77\x72\x23\x9b\xaf\x8c\x6a\xf8\x2b\xb2\xd4\xb9"
+"\xeb\x7f\x9f\xa5\x02\x50\x08\x47\x52\x6c\xaf\xe7\x73\x71\x85\x72"
+"\x49\x6b\xc8\x47\x88\xa7\xd8\xc2\x16\xbf\x3c\xe9\x22\x21\xeb\x54"
+"\xd1\xcd\x43\x18\x08\x8f\xa1\xcf\x1c\x2b\xa7\xfd\x65\x4a\x9d\x12"
+"\x0d\xdb\xd5\xf6\x1a\x97\x64\x83\x3c\x5a\x04\xa8\x15\x9d\x61\xd3"
+"\x43\x2a\x56\x35\xed\x08\xb7\x41\xc6\x49\xba\x02\x14\x59\xab\xca"
+"\x84\x1f\xfb\x67\x3a\x00\xe5\x41\xb8\xd1\x6e\x5c\x9d\x6f\xf2\x76"
+"\x3e\x21\x5d\x34\x5c\x78\x0d\x41\x5a\x4f\x62\x69\x1a\x76\x42\xee"
+"\x84\x6b\x1d\x47\x42\xeb\xb2\x11\x8f\x08\xb8\xc8\xea\xf4\x0d\xf7"
+"\x5d\x51\x4c\x4b\xed\x2d\x1b\x48\x30\x38\x38\x58\x0d\xe3\x2d\x80"
+"\xd9\xfb\xed\xe0\xc4\x55\xfe\x4f\x3f\xcf\x55\x57\x08\xaa\xa8\xa2"
+"\xa5\x5a\xe4\xff\x19\xf2\xae\x29\x74\xb9\x40\xea\xf4\x4d\x58\xac"
+"\x9f\x48\xea\x0f\xe0\xb0\xae\x72\x9f\xd8\x34\x95\x59\x01\x20\x7c"
+"\x98\x5d\xe6\x9f\x37\x23\x52\x8d\xa0\x62\x2b\x3a\x9c\x2e\x31\xe7"
+"\xd5\x75\xcc\x4c\x62\x2f\xa4\x3e\x2e\xb9\xe6\xe1\x4b\x69\xb4\x62"
+"\x31\x03\xfc\x08\xfd\xba\x87\xb9\x79\x3a\x68\x19\x65\x49\x2e\x2c"
+"\x65\x5f\xd8\x60\x07\xf4\x73\x8d\xdf\x37\x7e\x00\x88\xaf\x23\x48"
+"\x8b\xad\x74\x9c\x0b\xa3\x3a\x1a\x4b\xa0\x27\x6f\x04\x8d\xd9\x38",
+	.rlen = 512,
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xfd\xd6\xe8\x2f\xfe\xd4\xfe\x42\x23\x4b\x7c\x09\x8c\xde\x4f\x4b",
+	.result =
+"\xff\x7f\xb0\x11\x72\x5a\x91\x4a\xb5\x2d\xb0\x41\x3a\x96\x0d\xa1"
+"\xd9\xbe\x60\x09\x24\x51\x73\xb8\x00\xf0\x48\x1f\x6b\x96\x5b\xe7"
+"\x4d\x47\x88\xc7\xef\x4b\xb4\x33\xa1\x2b\xbe\xdd\x46\x4f\x27\x11"
+"\x8b\x30\x9c\xba\x2c\x7a\xf3\xdb\x48\x54\xbd\xfe\x24\x2f\x83\x91"
+"\x5c\x63\xb9\x12\xd9\xd9\xb9\x71\xcf\x28\x7e\xf8\xe0\xb8\x12\xf7"
+"\x63\xad\xde\x49\xd5\x4d\xa7\x13\x32\xee\x71\x13\x56\x4d\x10\xd5"
+"\x2c\x1d\x8e\x94\x0d\x37\x3d\x7e\x9c\xb4\xeb\xe5\x6f\x12\x30\x7f"
+"\xc3\xa0\xf3\x49\xac\xa6\xab\x1b\xec\xd4\x6c\x95\x2a\x57\xe0\xfa"
+"\x89\x00\x61\xe9\xea\x21\x9a\x2f\x71\xd7\xdb\x11\x52\xb6\x32\x91"
+"\xed\xa3\xdf\xa5\x46\xc1\x50\x5b\xab\x15\x43\x7f\x7d\x82\x34\xf2"
+"\xfa\x6e\x84\xaf\x40\x20\xde\x1f\x90\x39\xab\xdc\xe8\xf3\xf9\x65"
+"\xbc\xdc\xd3\x5c\xcf\xe2\x1b\x43\x08\x68\xd8\x0d\xfb\xc2\x7f\x31"
+"\x91\xb5\x66\x2a\xea\x43\x08\x6d\xa6\xb4\xd3\x0e\x78\x3c\xf1\x6c"
+"\x4d\x27\x47\x7d\x92\x42\xb1\x62\x82\x9f\x13\xdf\x51\xc3\x6b\xec"
+"\x83\x53\xd6\x89\x75\xac\x62\x9a\x89\x7d\xf9\x82\x66\xbe\x93\x6f"
+"\x71\x7d\x01\x79\xec\x10\x10\x50\xe9\x6c\x76\xc6\x7a\xfa\xbb\x69"
+"\x46\x09\x1a\x68\x2f\x07\x28\xf4\xd0\xb6\xb4\x82\xf5\x3a\x90\xdc"
+"\x61\x03\xd9\x8e\xa5\x13\xfd\xdd\xe0\x65\x03\xfb\x78\x6b\x4e\xae"
+"\x7f\x30\xe2\x9e\x39\xb1\x3a\x39\xda\x21\x80\x2c\x09\xdd\xe8\xa2"
+"\x8c\x4a\x2c\x40\x24\x39\xf0\x3f\x7f\x51\x6a\x48\xea\x7b\x68\x3d"
+"\xad\x56\xed\xbe\x86\x0a\x9a\xe6\x9f\x18\x95\x26\x14\x57\x5b\x71"
+"\x9e\x8d\x45\x0d\xad\x23\xb4\x37\xa5\x59\x66\x8c\x13\x8e\x5e\xeb"
+"\xbf\x4a\x0d\x72\xc9\x4a\xcf\x42\xbd\x28\x1f\x91\xad\x55\x81\x78"
+"\x48\xf3\xed\xab\x2b\x6d\x61\xc7\x08\x2c\x07\xcb\x17\xf8\xf1\x7c"
+"\x39\xc8\x44\x63\x3a\x2a\x55\xbe\xe1\xb5\x12\x61\x0a\x4c\x32\x83"
+"\x9a\xa0\xf8\x93\x8c\xfa\x45\x92\x4e\xad\x48\xd9\x84\xe8\x0d\x7a"
+"\xca\xad\xbf\xd2\x5a\x1d\x58\x67\x57\x68\xca\x2f\x40\xa5\x1c\x38"
+"\x2a\xde\xa7\x57\x87\x4f\x11\x97\x3e\x11\xe7\x58\x54\xbd\x06\x48"
+"\xf7\x60\x45\x5b\x9d\x08\x5a\xef\xf9\x28\xa5\xf5\x48\x5c\x9c\xa0"
+"\x96\x76\x56\x51\x40\xec\xbe\xdb\x6e\xba\x4b\xb0\xa2\xe9\x55\xe6"
+"\xb7\x7e\x8a\x06\x3b\xeb\x17\xeb\xe6\xd9\xf6\xb2\xa1\x8c\x9e\xcc"
+"\xf3\x89\xd5\x78\x29\x1f\x74\x60\xe2\x61\x72\x78\x05\x52\x23\x07"
+"\x2a\x46\x85\x3c\xcf\x12\x9a\x9d\x3d\xf0\x93\x0e\xd2\x22\x63\x07"
+"\x01\x8b\x96\x73\xb5\x26\x29\xf5\x4f\x90\xf9\x37\x55\x76\x15\x02"
+"\xe8\x4c\x56\x3e\xf1\x14\xaf\x34\x0d\xa8\xde\xee\x0e\x13\xfa\xb8"
+"\xe4\xb7\x6d\x71\x37\xdb\x1e\x42\xdd\xca\xec\xe1\x99\xf9\xc7\x18"
+"\x16\xb0\x41\xd0\xfe\x9a\xa6\xa0\x7a\x5e\x5d\x0a\x96\x4c\x52\x44"
+"\x9a\x29\x69\x09\xa2\x0e\x5a\x1e\xc2\xb3\x5e\xca\x25\xc0\xe1\xa9"
+"\xd1\x41\x7f\x82\xaf\x1f\xf4\x3c\xf8\x3d\x65\xae\xf0\xa2\x1a\x8f"
+"\x41\xdb\x01\x11\x4c\x01\xcb\x24\xb3\xec\xbb\xf3\xe5\x1b\x53\xf0"
+"\x7a\x81\x01\x61\xa2\x8e\xa4\xd0\xaa\x8f\xa1\x71\xc1\x15\x15\xda"
+"\xf3\x7b\x32\x87\xa6\xb7\x7f\x2b\xac\x2b\x28\xfc\xe4\x1a\x94\xab"
+"\x19\xc9\x13\x72\x33\xfa\x42\xec\x6f\x3f\xe1\xe0\xc7\x23\x4b\x17"
+"\xeb\x89\xd3\x1f\x49\xe1\x49\x56\xee\xe3\x82\x46\x43\x00\x80\xbc"
+"\xa3\xfe\x31\xbc\xc9\xcd\x61\x5b\x7a\xf9\xf7\xb7\x48\x98\xbf\xdc"
+"\x79\xca\x71\x3b\xb0\xda\x08\x1e\x25\x97\x83\xd7\x21\x2c\xaa\xc0"
+"\x5c\xfd\x7f\xc4\x30\xd8\x7b\x59\x35\x49\x62\x0f\x4c\x03\x02\xe5"
+"\x73\x63\x61\x0b\x69\x2f\x7d\xb3\x99\xc9\x6b\x0a\x29\x9b\xda\xbe"
+"\x98\xdc\x2c\x29\x28\x9a\x75\x2e\xf1\x11\xd3\x71\x5b\x20\x45\x5b"
+"\xb7\x5e\xc1\xd1\xcc\x4e\x5a\x0d\xa5\x70\xa6\x56\xb8\x80\x8c\x97"
+"\x9d\x65\x8d\xec\xa0\x15\x45\xe6\x04\xd8\x3b\x6b\x36\x3f\x71\x58"
+"\x9e\x7a\x9c\xd2\x44\x86\xbf\x89\xa6\x80\x5d\x5e\x99\xc9\x7e\x56"
+"\x76\x17\x02\x98\x5b\xbb\xa0\xe5\xe5\x10\x25\x3e\x82\xc7\xe0\x91"
+"\x77\x39\x50\x9c\x3d\x2a\x91\x03\x13\x6d\x6d\xd3\xc6\x68\xd3\xa0"
+"\x88\xbc\x24\x5d\xf1\x26\x19\xf4\xb0\x74\x51\x93\x17\xcf\x67\x6c"
+"\x72\x30\xed\x39\xfe\x59\x54\x88\x84\x70\x56\x11\xaf\x41\x66\xa5"
+"\xf9\xf0\x95\xdb\x80\xb8\xae\x2f\xb7\xc3\x65\x72\xd2\xec\xaf\x5f"
+"\xf9\x30\x1e\x5b\x45\x7f\x38\xd5\x03\x02\x60\xaa\xf9\xb7\xd9\xfc"
+"\xa2\x5c\x46\x3e\x9c\xe6\xd6\x8e\x95\x54\xbf\xd8\xe6\xe4\x4b\xc0"
+"\x4c\xa1\x4c\x2c\xb3\xc4\x9f\xef\xeb\x39\x70\x77\xac\xf9\x1f\xb6"
+"\x06\xa2\x53\x7d\x18\xc8\xf8\xda\x8e\x82\x97\x4f\xdd\xd5\x19\x2f"
+"\xa2\x70\x4a\xbd\x5a\x15\x70\xb6\x55\x04\x14\xba\x0a\x04\xdc\x8e"
+"\xaf\xf2\x52\xd5\x90\x4c\x30\xd3\x29\x53\x1c\x66\x37\x5f\x8e\xfc"
+"\x45\x83\xd9\xac\x75\x9e\x0f\x66\x51\xc0\x8a\xc5\x34\x25\x9e\x3b",
+	.ilen = 1024,
+	.input =
+"\xa8\x47\xa1\x1d\xcb\xa3\x88\xae\x42\xab\x6d\xf2\x82\xc2\xed\xd5"
+"\x66\x42\x09\x85\x28\x7d\x49\x6f\x37\xdc\xff\x1c\x7e\x33\xc9\xcd"
+"\x6e\xe9\x33\x36\x01\x62\x1d\x67\x77\x6a\x97\xbf\xb1\xdc\x2f\x98"
+"\x2c\xdb\xac\x44\x9d\xed\x31\x7d\x2d\x41\x4b\xd1\x66\x40\x62\x74"
+"\xdc\x00\xd0\x05\xdc\x54\x4c\x63\xeb\xd9\x42\xe1\xdf\xc4\xde\xdd"
+"\xb6\xb8\x93\xfd\x25\x39\x2d\x7f\x85\xf8\x15\xc3\xbc\xbf\x0b\x95"
+"\x11\xef\x57\x0d\x15\x49\x07\xce\x42\xb0\x50\xe1\x07\xb4\x81\x71"
+"\x35\x71\x4b\x66\x89\x7f\x94\x13\x3e\x57\x43\xc3\x36\x28\xcd\xdd"
+"\xc9\x06\x68\xf8\xf3\x09\x3d\x86\x12\x52\x06\xa9\xe9\x83\x2d\x8f"
+"\x90\xfa\x42\xfe\x79\x3f\x68\x4c\x7b\xfa\x94\xa7\xf7\x16\xc7\x41"
+"\x09\xae\xe2\x82\xb5\x2b\xbc\xca\x65\x65\x2c\x27\x2c\x07\x50\x83"
+"\x2d\xad\x55\xaf\x35\xcc\x6a\xc5\x7c\xd8\xed\x75\x91\x9d\x73\xcb"
+"\x4c\xa5\x8f\xc4\x4f\xda\xa8\xb9\xb6\xa7\xb1\x1a\x75\xb4\x08\xbc"
+"\xb2\x90\x50\xfd\x1f\x05\xa8\x88\x35\x81\xb0\xc9\xac\xbc\x25\x7a"
+"\x95\x33\x02\x2b\x74\xe0\x95\x11\x88\xf7\xc3\x63\xb3\x7b\x09\xd5"
+"\xac\x22\x04\x67\x16\xea\xd6\x37\x38\x8e\xa5\xbd\x62\xa2\x1f\xa5"
+"\x04\x31\x89\xdf\x69\xb1\xde\xe3\x7c\x9d\x7b\x27\xba\x0a\x74\xdc"
+"\x06\x1c\xcd\x6e\x4b\x52\xe7\x6d\x34\x29\x38\xe2\x19\xfc\x0c\xc4"
+"\x78\x03\x1d\x53\x98\x00\x5c\x7a\xec\x23\x5f\x95\xd5\xb3\x16\xde"
+"\xc2\x17\xc2\x0c\x13\x63\x0a\x4b\x7e\x6c\xc7\xbc\x4a\xd0\xae\x29"
+"\xc0\x50\x16\x6f\x01\x2b\xdc\x40\x9f\x91\x8f\xa3\xaf\xd4\x40\xa8"
+"\x2e\x09\x7c\xf4\x3d\x85\xe6\xd9\x3c\x78\x7c\xf1\x6d\xe4\x13\x00"
+"\x98\xf5\xb4\x06\x9f\x90\x0a\x3e\x9f\x51\x0f\xbb\x0f\x13\x07\xc0"
+"\xfd\x26\x53\x24\x24\xf7\x21\x41\xcf\x20\x9d\x77\xe4\xe0\x52\x2a"
+"\x48\xd9\xeb\x65\xce\xf3\x90\x03\x47\x8d\x2b\x77\x54\x46\xda\xff"
+"\x15\x3d\xa5\xd9\x5a\xb6\xd3\xdf\x9c\x91\xc3\xf2\xd2\xdf\xd7\x8c"
+"\x1d\x83\x77\x47\xcd\x74\x23\x44\x04\x06\x8e\x64\x62\x29\xe5\xa0"
+"\xf7\xa7\xc7\xb7\x84\xdb\x9c\x5c\x04\x7f\xca\xb3\x85\x2c\x44\xa6"
+"\x09\x0e\xa3\x2c\x52\x42\x25\x02\x63\x99\xd0\xa5\x27\x61\x64\x4f"
+"\x65\xd7\x31\x56\x24\x97\xb0\x2d\xbb\x0c\xbe\x06\x68\x8a\x2e\xa3"
+"\x0c\xb9\x05\x52\xdb\xbd\x7e\x89\x60\x2e\x28\x76\xba\x5a\x94\xb6"
+"\x94\xc4\xf6\x68\x50\x35\x24\x7b\x2b\x04\x0e\x4c\xf3\x17\x54\xcb"
+"\xcd\x32\x18\x60\xff\xc9\xfe\xe1\x83\xe4\xe6\x9b\x5e\xd8\x21\xbf"
+"\xbf\x69\x01\x3a\x03\xc6\x9f\xe5\xd4\xdf\x01\x20\x8e\xea\x5b\xe1"
+"\xbd\x46\x3c\x3a\x60\x30\xa0\x48\xa0\x07\x82\x27\x4e\x03\xc3\x15"
+"\x98\x1f\xea\x4f\x8c\x90\x4d\xb1\xc5\x90\x40\x59\xda\x5b\x02\x65"
+"\x07\xb9\x64\xe7\x4c\x76\x70\x16\x8a\xc3\xf9\x4f\xed\x25\x47\xaa"
+"\x3b\x49\x8f\xf6\xf0\x71\x94\x34\xda\x29\x0f\x4e\xd4\x95\x3b\xe3"
+"\xef\x99\x3b\x1c\xf7\x09\x5d\xe0\x0d\x03\xe6\x9d\x47\x4c\x8c\xe8"
+"\x26\xb6\x30\x1b\x81\xdc\xa5\x5a\xf1\x04\x18\xf3\xaf\x81\xa2\x7e"
+"\xce\x8b\x33\xfc\xf2\xb1\x5a\x06\xd1\xb9\x59\x73\xd7\xda\x85\xd9"
+"\x30\x73\x98\x4d\x63\x50\x66\x71\x15\x88\x9a\x5d\xd5\x25\x40\x9a"
+"\xe3\x9c\x0b\x4f\xd8\xf5\xbf\xb3\xec\x02\x95\xca\x90\x07\x5d\x99"
+"\x9e\x16\xa2\x18\xa5\xa2\x03\xb1\x16\x6b\x4e\x32\xab\x19\x29\x55"
+"\xcc\xbe\xa8\x7b\xf7\x68\x64\x0e\xc0\x54\x91\x6d\x19\xec\xe9\x8c"
+"\x56\x5e\x71\xa5\x73\x50\x5d\x0d\xd3\xb2\x31\xca\x97\x7b\xf8\x6e"
+"\xfd\xb9\x47\x9b\x17\xf9\x56\x3a\xc6\xb0\x52\x45\x4f\x4a\x13\xe9"
+"\xb7\x64\x02\xdb\xe8\x67\xa3\x9e\xe4\xd9\x49\xc4\xf3\x27\xe3\xb0"
+"\xad\x6e\x51\x65\x14\x4f\xb2\x4b\x8a\xd6\x87\x17\x8c\xe2\x7a\xa1"
+"\x13\xbb\x8c\x7c\x3e\x69\xd2\x29\x06\x36\xf3\x55\x80\xcc\x0e\xa5"
+"\x18\x5a\x5f\xcb\x15\x2e\x7c\x62\xff\x3f\xe7\x7b\xd8\xe4\xa6\x9c"
+"\x4c\x5b\x55\x73\x4a\x0d\x21\x07\xf9\x79\xcb\x17\x51\x06\xf3\xcc"
+"\xfc\x08\x72\x6e\xbc\x04\xe2\x6d\xd8\x52\x1d\x29\x7e\x7a\x06\x8d"
+"\x87\x65\x2e\x2e\x7c\x07\x77\x3a\x35\x4d\x3a\x13\xd3\xf6\xc2\x1f"
+"\x2d\x5d\x14\xa5\x04\xe5\xc5\x7b\xd6\xa9\x70\x4b\x43\x21\x93\xdf"
+"\xe4\xf1\xf8\x75\xf1\x65\x9c\xf8\x0b\x07\x31\xdc\xf2\xba\x06\x91"
+"\xe1\x84\x87\x34\x2d\xdd\xa7\x87\xc0\xc2\x4d\x8d\xe0\x18\x70\xbb"
+"\xe3\x3e\x13\x48\xfc\xf4\x13\x85\xc4\x65\xcf\xe4\x43\x98\x14\x8f"
+"\xf4\x17\x62\x27\x39\xe5\xb6\x45\x76\x61\x78\x0b\x3d\x48\xb3\x41"
+"\xa6\xca\x7c\xed\x52\x19\x99\xea\x73\xc9\xd0\x0b\xeb\xbb\x5a\x69"
+"\x44\x3d\xb2\x81\x25\xb0\x2f\x08\xf0\x8c\x32\xa9\xf0\x79\x3c\x42"
+"\xc3\xdc\x9e\xd1\xec\x93\x49\xc9\x82\x0e\x13\x12\xb3\x8a\x98\x1b"
+"\x35\xe1\x4a\xef\xb4\x73\x28\x1a\x17\x96\xe2\x9a\x50\xc8\xd5\x98"
+"\xec\x96\x6f\x81\x05\x37\xee\x8b\x93\x12\x7c\x41\x26\xd5\x9c\x05",
+	.rlen = 1024,
+},
+/* repeat the above with sg list */
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc1\x62\x66\x62\xb8\x65\x28\xfa\x5f\x36\xd3\x09\xb1\x2c\xa1\xa3",
+	.result =
+"\x4f\x6c\x63\xa5\xd0\x19\x08\x4e\xd4\x58\x33\xf6\x2b\xeb\x26\xb9",
+	.ilen = 16,
+	.input =
+"\xa0\x35\xb0\x33\xc0\x2e\xe5\xbb\xbc\xe6\x01\x9e\xf4\x67\x11\x14",
+	.rlen = 16,
+	.np	= 2,
+	.also_non_np = 1,
+	.tap	= { 8, 8 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x78\x6c\x27\xd6\xb2\xdc\xbe\x7b\xab\xc2\x43\xd7\x81\x0c\xe5\x20",
+	.result =
+"\x9a\x00\x4e\x5a\xb3\x51\x68\xaa\xdb\x6e\xe5\xa4\x7f\x23\x6e\x4d"
+"\x1e\x72\x5e\xad\x64\xc9\x96\x23\xf8\xae\xef\xf6\x7b\x7d\xd6\xf0",
+	.ilen = 32,
+	.input =
+"\x5a\xc0\x04\xc6\x53\xef\x3b\x69\xb1\x41\xc7\x85\xeb\x69\x82\xd0"
+"\xed\x09\xbb\xec\xb2\x8d\x5c\xc9\x61\x81\x5c\xf6\x99\x49\xa0\x4d",
+	.rlen = 32,
+	.np	= 3,
+	.also_non_np = 1,
+	.tap	= { 8, 8, 16 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xc9\x05\x4c\x35\x96\x77\xd3\x3c\x3d\x97\x7c\x82\xf5\x58\x71\xf1",
+	.result =
+"\xa9\x5b\x03\xec\xec\x73\xed\xcb\x5c\x4c\xd2\x40\xb6\x9b\x49\x31"
+"\x5d\xf2\x23\xb3\x11\x98\xeb\x89\xab\x3e\x3a\xdd\xaa\xfd\xd1\xde"
+"\xab\x73\x59\x86\x1a\x59\x32\xb2\x55\x46\x4a\x80\xa4\xcc\xa8\xd9",
+	.ilen = 48,
+	.input =
+"\xdb\x05\x69\xe1\x33\x8b\x0b\x3d\x33\x12\x0d\xef\x94\x0f\xa3\xb3"
+"\xd7\x0a\x53\x7b\x98\x53\xc6\xc2\xa3\xd4\x7a\x30\x1a\xed\x45\xcc"
+"\x47\x38\xc1\x75\x0b\x3c\xd4\x8d\xa8\xf9\xd3\x71\xb8\x22\xa6\xae",
+	.rlen = 48,
+	.np	= 4,
+	.also_non_np = 1,
+	.tap	= { 8, 8, 20, 12 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x6c\xb4\x84\x61\x1e\x39\x4b\x22\x37\xaa\x7b\x78\xc0\x71\x20\x60",
+	.result =
+"\x05\x43\x76\x1e\xc6\x68\x43\x52\x5f\x43\x39\xbf\x93\x38\x38\x83"
+"\x38\x1d\x3c\xb5\xc8\xab\xe4\xd0\x7f\x1a\xac\xca\xcc\x16\xea\x75"
+"\x30\x75\x40\xe8\x61\x07\xc6\x04\x55\x2b\xf3\x29\xc3\x37\x83\x42"
+"\xe0\x21\xfb\xb4\x5d\x93\xbb\x87\x01\x3e\xa6\x9d\x3b\x0a\x5a\x37",
+	.ilen = 64,
+	.input =
+"\x83\x9f\xa0\xac\x14\x14\x88\x68\x7f\x9a\x5f\x98\x91\x71\xa8\xce"
+"\x28\xfb\x5e\xb1\x49\xe7\x63\x39\x12\x62\x00\x3e\x5c\x63\x2b\x12"
+"\x3d\xff\xd5\x0a\x43\x28\x52\x68\x78\x62\xc7\xa4\xbb\xca\x5d\x5e"
+"\xe3\xd5\x23\xb3\xe7\x22\xae\xf3\xd0\xd9\x00\x14\x0c\x46\x67\x17",
+	.rlen = 64,
+	.np	= 2,
+	.also_non_np = 1,
+	.tap	= { 32, 32 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xf9\xe8\xab\xe2\xf9\x28\xeb\x05\x10\xc4\x97\x37\x76\xe4\xe0\xd9",
+	.result =
+"\xab\x99\xe8\x2a\x18\x50\xdc\x80\x1f\x38\xb9\x01\x34\xd4\x59\x60"
+"\x4e\x1c\x21\x71\x22\x06\xbe\x5f\x71\x07\x3b\x13\xe7\x05\xca\xa5"
+"\x7b\x23\xb5\xaa\xc6\xdb\xe3\x17\xa9\x9f\xe1\xbc\xd5\x1b\xe6\xf5"
+"\xfa\x43\xdd\x80\x50\xc8\x8a\x32\x2f\x65\x25\xa4\xeb\xd1\x74\x02"
+"\x07\xc1\x04\x94\x6b\x34\xa1\x74\x62\xb2\x8d\x60\xf5\x7e\xda\x1a"
+"\x0f\xf5\x21\xe1\xd7\x88\xc8\x26\xd7\x49\xb2\x4a\x84\x2c\x00\x3b"
+"\x96\xde\x4e\xa7\x57\x27\xa0\xa4\x3a\xff\x69\x19\xf7\xec\xeb\x62"
+"\xff\x5a\x82\x0d\x25\x5e\x3c\x63\xb3\x6d\xc4\xb9\xe3\xc9\x3a\xc2",
+	.ilen = 128,
+	.input =
+"\xec\xd5\x2f\x6a\xfd\x61\xf2\x37\x19\x6f\x55\x31\xd7\x2c\x14\x4d"
+"\xc1\xb4\xbb\x7d\xa9\x1a\xe6\x85\x8c\x2f\xbf\x7e\x66\x21\xf8\x17"
+"\x9e\x09\x1b\x2a\x11\xbf\xdf\x7d\xdf\xf5\xfb\x0a\x16\x79\xe2\x43"
+"\x5c\x3b\x3e\x84\x35\xfd\x92\x9e\xe0\x31\x50\x1d\x62\xd6\x22\x99"
+"\x5f\x25\xb3\xe8\xdf\xb0\xc0\xab\xd9\xdb\xac\x4b\x9c\xe2\x89\xc6"
+"\x49\x7f\x5f\xee\xcb\xf6\x25\x10\x9f\x32\x58\x85\x45\x50\x74\x8a"
+"\x55\xce\x86\x44\xda\xe4\x93\x58\x4d\xd3\x73\x76\x40\xf6\x92\x8b"
+"\x99\xc1\x2b\xf9\x18\xd0\xfa\xd0\xa6\x84\x03\xf5\xd4\xcb\xfa\xe7",
+	.rlen = 128,
+	.np = 3,
+	.also_non_np = 1,
+	.tap = { 64, 16, 48 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\x58\x1e\x1a\x65\x16\x25\xaa\x55\x97\xcd\xeb\x4c\xd6\xb3\x9c\x2b",
+	.result =
+"\xef\x85\x0b\xe5\x02\xd5\xce\xcc\xad\x2d\x5e\xec\x1e\x01\x8c\x28"
+"\xf0\x2c\x23\x10\xaa\x84\xf0\x61\xe2\x56\x29\x21\x9f\x09\xaf\x9d"
+"\x7d\xfc\x60\x16\x4c\x67\xdd\xdf\x74\x35\x49\x81\xca\x68\xb6\xc7"
+"\x31\x9f\x49\x29\x96\x01\xb9\x3c\xfb\xa3\x00\x04\x05\xd8\xe6\xa2"
+"\x3f\x0c\xee\xfc\xd6\x88\x7a\x2d\xd6\x32\x27\x15\xe3\x53\xa0\x52"
+"\x1d\x96\x5a\x95\x09\x0c\x5f\x07\xc8\xdf\xab\xc7\x78\x57\x6a\x49"
+"\x09\x88\x54\x2e\x80\x27\xb0\x8a\x40\xb8\x9e\x7a\x22\x85\x8d\xaa"
+"\x95\x48\x45\xf5\xfd\x6f\x4c\x69\xe3\x38\xa6\x05\x69\xf0\xba\xb5"
+"\xd5\x9a\x9f\x77\x98\x23\xef\x98\x1f\xf3\xfe\x53\x23\xf6\xc6\x74"
+"\x6a\x2f\x1b\x34\x75\xd0\x51\x0c\x88\x10\xf9\x80\x19\xaf\x4f\xf1"
+"\xb1\xf3\xc0\x0e\x3a\x7d\x63\x3e\xbd\xb9\xe9\x3c\x69\x56\x0d\xb9"
+"\x8d\x69\xea\xb9\xa7\x39\x4c\x5d\xb8\x06\xa3\x1b\x66\x66\x14\x80"
+"\xe1\x8f\xf3\x65\x0c\xd5\x39\xe4\xed\xb9\x1f\x88\x74\x49\xd7\x4f"
+"\xc1\x4b\x3d\xea\x5d\xa2\x44\xd6\xad\x5d\x8d\xd1\xf7\x56\x9c\x9e"
+"\xda\x52\x56\x51\x00\x14\x1b\xb4\x00\x6b\x83\x4f\x41\x0b\xba\xaa"
+"\x11\xe6\xee\x23\xf7\x85\xa9\xb9\xd8\xe3\xbd\xbb\x7b\x83\x5f\xf8",
+	.ilen = 256,
+	.input =
+"\xc8\x8e\xbf\x95\x57\xa8\xcd\x47\xbc\x32\xee\x76\x97\xee\x02\x12"
+"\x11\x36\x81\xaa\x5b\xd9\xb3\x14\x80\xf3\xab\x62\x9b\x7f\x99\x98"
+"\x3b\x46\xd6\xfb\x68\xc8\xce\x1d\xa5\x47\x79\x6a\xdf\x7c\xda\x01"
+"\x44\x01\xfc\xed\xab\x2a\x51\xae\x2f\x72\x60\xed\x61\xc5\x23\x1d"
+"\xc7\xb5\x3c\xb7\x0b\x29\x62\xd6\x77\x8c\xea\x51\x0c\x39\x90\xe7"
+"\x99\x8c\x5d\xb7\x16\xf3\xc6\xea\xe0\xff\xc3\xd7\xc8\x1a\x7d\xde"
+"\x4d\x25\xaa\x0b\x90\x0d\x49\xd7\x98\x44\x4b\x75\x46\x01\x30\xa3"
+"\xdc\x47\xd9\x66\xc7\x7a\xcb\x4a\x33\x69\x60\x5d\x96\x73\x31\xf1"
+"\xce\xdc\xa9\x15\xb5\xae\x08\x2b\x08\x4a\xbc\x9b\x68\x1e\x49\xe4"
+"\x6e\x11\xe8\x61\x37\x58\x66\x69\x67\x97\x65\x1d\xd4\x62\x7c\x29"
+"\x10\xba\x8f\x2f\x0f\x23\x3d\x72\xb1\xcf\x01\xbc\x73\x10\xd8\xde"
+"\x21\xe6\xfc\xce\x3b\x3e\x19\xdc\xc2\xa7\x87\x62\x33\x88\xb4\x37"
+"\x1f\xfc\x1a\x2b\xef\x14\x24\x4a\xb5\x86\x55\x45\xf8\xc4\xcd\xaa"
+"\x0d\x8a\x5a\xdc\xfd\x7b\x41\xd7\xa6\x8f\x05\x25\x4a\x61\xcb\xa7"
+"\x14\x84\x21\xfc\xa6\x4b\x0f\xaa\x7d\xc6\xa2\x04\x04\xff\x39\xfc"
+"\x27\x8d\x7a\xce\x94\x31\x7c\xb4\xd5\x90\xbd\xb6\xdb\x6a\x55\xd9",
+	.rlen = 256,
+	.np	= 4,
+	.also_non_np = 1,
+	.tap	= { 32, 48, 80, 96 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xed\xa3\x2d\xa6\x9f\x5e\x38\x6f\xbb\xf9\xb3\x32\xae\x73\x05\x87",
+	.result =
+"\xf6\x24\x50\x2d\xa4\xfb\x09\x41\x95\xcd\x25\x13\xc0\xdc\x26\x0b"
+"\x20\x66\x70\x79\xc5\x58\xde\x63\xba\x37\x13\xb2\x0a\x40\x58\xef"
+"\x37\xcb\x04\x38\x10\x6a\x90\x97\x14\xd2\x71\x03\xa4\xa3\x6a\x59"
+"\x15\x6e\x5d\x45\xc9\xcc\xa9\x47\x8c\x0f\x1d\x6c\x62\x06\x90\xba"
+"\xef\x1c\x23\x4d\xc4\xa0\xa5\x56\x49\x19\xa9\xb1\x2a\xdd\x26\x00"
+"\x6e\xed\xd8\x4d\xd4\x3f\x68\x99\x24\xe2\xfe\x99\xb4\xe6\xf8\x3f"
+"\x60\xef\x97\x5f\x87\xa6\xde\x82\xc4\x11\xf6\x91\x7f\xd4\xa6\xa8"
+"\xee\x97\x41\x43\x14\xd2\x6e\x8d\x72\x30\x83\x5b\x67\x01\x38\xa2"
+"\xca\x93\xf4\x1e\x80\x2e\x8f\x7e\xc3\x78\xf0\xc1\x68\xb4\xf9\x1f"
+"\x15\x3c\x5c\x8b\xa1\xb5\x2f\x0c\xbf\xf7\x21\x74\xdb\x57\x98\x85"
+"\xe9\x89\x16\x20\x8b\x7c\x71\xef\x46\xc0\x78\x04\x23\x3b\x58\x24"
+"\x51\xa1\xa6\xfc\x6e\x9e\x29\x95\x55\x4c\x05\x1c\xc5\x9a\x59\x7e"
+"\x40\x4d\xe8\x81\x76\x41\x6f\x15\xde\x68\xce\x2d\x42\x03\x74\x73"
+"\xd3\x9a\x9c\xac\xa5\x05\x03\x7e\x53\x6e\x53\xa2\x57\x36\xee\x0c"
+"\x05\xde\xa9\x61\x55\xb9\x9d\x7d\x18\x18\xaa\x20\xde\xb8\x43\xd7"
+"\x30\x8e\x20\xc0\x78\xbe\xec\x24\xcf\xd7\x66\xb7\x5a\x1f\x5a\x81"
+"\xec\x19\x48\xc3\xa7\x62\xbf\x83\xbb\xbd\xf4\x51\xec\xb5\xec\x90"
+"\x05\xe1\xa9\xbf\x4d\x9b\x30\xf1\xb9\xa6\x49\xe9\xad\x65\x0d\x08"
+"\x1f\x3f\x81\xa5\x40\x4f\x3d\x42\xd8\x68\x29\xe3\x6c\xcc\x4d\x20"
+"\x7e\xb9\x0c\x33\x1f\x20\xd2\xaf\x39\xd6\xb4\x20\x06\xd0\xc3\x54"
+"\xcd\x96\x84\x88\x13\xc0\x09\x57\x18\x90\xad\xec\x18\xab\x72\x0b"
+"\xb4\x4c\x0a\x65\x67\x2a\x96\x2c\x98\x58\x6f\xdf\xc0\xe4\x51\x7c"
+"\xc8\x66\x1d\x21\x91\x1f\xab\xac\xed\x86\x38\x70\x54\x6f\x0c\xbf"
+"\x1a\xea\x9b\x33\xf4\x7c\x99\x0c\x0a\xdf\x39\x25\x78\x3b\x8d\x9c"
+"\x46\xc0\x07\x08\xfa\xeb\x19\x12\xf8\xc1\xf7\x18\x13\xbd\x7f\xd1"
+"\xa4\x3c\x7e\x03\xbd\xcf\xa1\xf3\x37\x4a\x4d\xc3\xaa\x23\xed\x58"
+"\xca\x68\x35\x91\x3e\x23\x09\xb8\xf3\x8d\xc3\x1b\x23\xe8\x1c\xda"
+"\x41\x90\xa2\x4b\x48\xb5\x7c\xa0\x8d\xaf\x66\x5e\xad\x7f\x06\xa2"
+"\x62\x32\x40\x69\x41\xb1\x2f\x6c\x0e\xf9\xd1\x48\xbd\xfc\x44\x0f"
+"\x65\x5e\xa1\x38\x83\xea\xfe\x42\x53\x9a\x2a\x85\xea\x92\xf6\x29"
+"\xbf\xb5\x78\x1e\x8d\x03\x6b\x09\xaf\x94\x4b\x39\x20\xc1\x17\x20"
+"\x95\x42\xfe\x72\x02\x10\x61\x21\x0f\x23\xcb\x33\x35\x52\x57\x9e",
+	.ilen = 512,
+	.input =
+"\x25\x3d\xad\x25\x4f\xb4\x50\x55\xbf\xc1\x66\xe3\x52\x22\x01\x10"
+"\xde\xed\x83\xc0\x18\x49\xda\xa4\xdb\xf1\x2f\x73\x90\x6f\xf2\x4f"
+"\x9b\xa2\x32\x2b\x6f\xc7\x80\xc8\x47\xbd\xf3\x24\x8a\xcd\x9b\x8d"
+"\x00\x33\xd1\x6a\xf2\x5f\xf2\xc7\xd8\x7c\x3a\x84\x1c\x12\x3c\x3e"
+"\xe0\x58\xb7\xc9\xf8\x73\x9e\x98\x2f\x8f\x03\x38\xe2\xc2\xb9\xae"
+"\xb6\xc6\xef\x78\xd0\xfa\xbf\x81\xcc\xf7\xb3\x82\x5b\x80\xb9\x0b"
+"\x57\xe3\x33\xa6\xfc\x3c\xd1\x78\xc7\x61\xc5\x5a\xe9\x01\xf5\xf7"
+"\x87\x0f\xa4\xe7\x90\xdf\xd5\x9f\x79\xc5\x5c\x1a\x2c\x29\x8e\x79"
+"\x10\xbc\xb2\xc6\x89\x9d\x95\x65\xa8\x25\xb3\x20\x97\xcc\xdf\x62"
+"\x2f\x9c\x85\x36\xe6\x34\xcc\xc0\xee\x7e\x10\xf6\x07\x57\xed\x2e"
+"\x60\x7e\x5e\xa0\x8e\x4c\xec\xe8\x73\xa3\x55\x4d\x7f\x6d\xff\x8c"
+"\x7a\x8c\x62\x3b\x10\x22\x75\xc0\x0b\x4a\x99\x83\x4d\x09\x80\x36"
+"\x41\x33\x19\x53\x9b\x51\xa6\x92\x82\xd8\x97\xe7\x98\x42\x36\x0d"
+"\x93\xb2\xf4\xbf\x96\xc7\x71\xfb\xc1\xf7\xf0\x94\xa3\x88\x28\xfa"
+"\x7c\xef\x3b\x1c\x77\x72\x23\x9b\xaf\x8c\x6a\xf8\x2b\xb2\xd4\xb9"
+"\xeb\x7f\x9f\xa5\x02\x50\x08\x47\x52\x6c\xaf\xe7\x73\x71\x85\x72"
+"\x49\x6b\xc8\x47\x88\xa7\xd8\xc2\x16\xbf\x3c\xe9\x22\x21\xeb\x54"
+"\xd1\xcd\x43\x18\x08\x8f\xa1\xcf\x1c\x2b\xa7\xfd\x65\x4a\x9d\x12"
+"\x0d\xdb\xd5\xf6\x1a\x97\x64\x83\x3c\x5a\x04\xa8\x15\x9d\x61\xd3"
+"\x43\x2a\x56\x35\xed\x08\xb7\x41\xc6\x49\xba\x02\x14\x59\xab\xca"
+"\x84\x1f\xfb\x67\x3a\x00\xe5\x41\xb8\xd1\x6e\x5c\x9d\x6f\xf2\x76"
+"\x3e\x21\x5d\x34\x5c\x78\x0d\x41\x5a\x4f\x62\x69\x1a\x76\x42\xee"
+"\x84\x6b\x1d\x47\x42\xeb\xb2\x11\x8f\x08\xb8\xc8\xea\xf4\x0d\xf7"
+"\x5d\x51\x4c\x4b\xed\x2d\x1b\x48\x30\x38\x38\x58\x0d\xe3\x2d\x80"
+"\xd9\xfb\xed\xe0\xc4\x55\xfe\x4f\x3f\xcf\x55\x57\x08\xaa\xa8\xa2"
+"\xa5\x5a\xe4\xff\x19\xf2\xae\x29\x74\xb9\x40\xea\xf4\x4d\x58\xac"
+"\x9f\x48\xea\x0f\xe0\xb0\xae\x72\x9f\xd8\x34\x95\x59\x01\x20\x7c"
+"\x98\x5d\xe6\x9f\x37\x23\x52\x8d\xa0\x62\x2b\x3a\x9c\x2e\x31\xe7"
+"\xd5\x75\xcc\x4c\x62\x2f\xa4\x3e\x2e\xb9\xe6\xe1\x4b\x69\xb4\x62"
+"\x31\x03\xfc\x08\xfd\xba\x87\xb9\x79\x3a\x68\x19\x65\x49\x2e\x2c"
+"\x65\x5f\xd8\x60\x07\xf4\x73\x8d\xdf\x37\x7e\x00\x88\xaf\x23\x48"
+"\x8b\xad\x74\x9c\x0b\xa3\x3a\x1a\x4b\xa0\x27\x6f\x04\x8d\xd9\x38",
+	.rlen = 512,
+	.np	= 5,
+	.also_non_np = 1,
+	.tap	= { 32, 64, 96, 128, 192 },
+},
+{
+	.key =
+"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
+	.klen = 16,
+	.iv =
+"\xfd\xd6\xe8\x2f\xfe\xd4\xfe\x42\x23\x4b\x7c\x09\x8c\xde\x4f\x4b",
+	.result =
+"\xff\x7f\xb0\x11\x72\x5a\x91\x4a\xb5\x2d\xb0\x41\x3a\x96\x0d\xa1"
+"\xd9\xbe\x60\x09\x24\x51\x73\xb8\x00\xf0\x48\x1f\x6b\x96\x5b\xe7"
+"\x4d\x47\x88\xc7\xef\x4b\xb4\x33\xa1\x2b\xbe\xdd\x46\x4f\x27\x11"
+"\x8b\x30\x9c\xba\x2c\x7a\xf3\xdb\x48\x54\xbd\xfe\x24\x2f\x83\x91"
+"\x5c\x63\xb9\x12\xd9\xd9\xb9\x71\xcf\x28\x7e\xf8\xe0\xb8\x12\xf7"
+"\x63\xad\xde\x49\xd5\x4d\xa7\x13\x32\xee\x71\x13\x56\x4d\x10\xd5"
+"\x2c\x1d\x8e\x94\x0d\x37\x3d\x7e\x9c\xb4\xeb\xe5\x6f\x12\x30\x7f"
+"\xc3\xa0\xf3\x49\xac\xa6\xab\x1b\xec\xd4\x6c\x95\x2a\x57\xe0\xfa"
+"\x89\x00\x61\xe9\xea\x21\x9a\x2f\x71\xd7\xdb\x11\x52\xb6\x32\x91"
+"\xed\xa3\xdf\xa5\x46\xc1\x50\x5b\xab\x15\x43\x7f\x7d\x82\x34\xf2"
+"\xfa\x6e\x84\xaf\x40\x20\xde\x1f\x90\x39\xab\xdc\xe8\xf3\xf9\x65"
+"\xbc\xdc\xd3\x5c\xcf\xe2\x1b\x43\x08\x68\xd8\x0d\xfb\xc2\x7f\x31"
+"\x91\xb5\x66\x2a\xea\x43\x08\x6d\xa6\xb4\xd3\x0e\x78\x3c\xf1\x6c"
+"\x4d\x27\x47\x7d\x92\x42\xb1\x62\x82\x9f\x13\xdf\x51\xc3\x6b\xec"
+"\x83\x53\xd6\x89\x75\xac\x62\x9a\x89\x7d\xf9\x82\x66\xbe\x93\x6f"
+"\x71\x7d\x01\x79\xec\x10\x10\x50\xe9\x6c\x76\xc6\x7a\xfa\xbb\x69"
+"\x46\x09\x1a\x68\x2f\x07\x28\xf4\xd0\xb6\xb4\x82\xf5\x3a\x90\xdc"
+"\x61\x03\xd9\x8e\xa5\x13\xfd\xdd\xe0\x65\x03\xfb\x78\x6b\x4e\xae"
+"\x7f\x30\xe2\x9e\x39\xb1\x3a\x39\xda\x21\x80\x2c\x09\xdd\xe8\xa2"
+"\x8c\x4a\x2c\x40\x24\x39\xf0\x3f\x7f\x51\x6a\x48\xea\x7b\x68\x3d"
+"\xad\x56\xed\xbe\x86\x0a\x9a\xe6\x9f\x18\x95\x26\x14\x57\x5b\x71"
+"\x9e\x8d\x45\x0d\xad\x23\xb4\x37\xa5\x59\x66\x8c\x13\x8e\x5e\xeb"
+"\xbf\x4a\x0d\x72\xc9\x4a\xcf\x42\xbd\x28\x1f\x91\xad\x55\x81\x78"
+"\x48\xf3\xed\xab\x2b\x6d\x61\xc7\x08\x2c\x07\xcb\x17\xf8\xf1\x7c"
+"\x39\xc8\x44\x63\x3a\x2a\x55\xbe\xe1\xb5\x12\x61\x0a\x4c\x32\x83"
+"\x9a\xa0\xf8\x93\x8c\xfa\x45\x92\x4e\xad\x48\xd9\x84\xe8\x0d\x7a"
+"\xca\xad\xbf\xd2\x5a\x1d\x58\x67\x57\x68\xca\x2f\x40\xa5\x1c\x38"
+"\x2a\xde\xa7\x57\x87\x4f\x11\x97\x3e\x11\xe7\x58\x54\xbd\x06\x48"
+"\xf7\x60\x45\x5b\x9d\x08\x5a\xef\xf9\x28\xa5\xf5\x48\x5c\x9c\xa0"
+"\x96\x76\x56\x51\x40\xec\xbe\xdb\x6e\xba\x4b\xb0\xa2\xe9\x55\xe6"
+"\xb7\x7e\x8a\x06\x3b\xeb\x17\xeb\xe6\xd9\xf6\xb2\xa1\x8c\x9e\xcc"
+"\xf3\x89\xd5\x78\x29\x1f\x74\x60\xe2\x61\x72\x78\x05\x52\x23\x07"
+"\x2a\x46\x85\x3c\xcf\x12\x9a\x9d\x3d\xf0\x93\x0e\xd2\x22\x63\x07"
+"\x01\x8b\x96\x73\xb5\x26\x29\xf5\x4f\x90\xf9\x37\x55\x76\x15\x02"
+"\xe8\x4c\x56\x3e\xf1\x14\xaf\x34\x0d\xa8\xde\xee\x0e\x13\xfa\xb8"
+"\xe4\xb7\x6d\x71\x37\xdb\x1e\x42\xdd\xca\xec\xe1\x99\xf9\xc7\x18"
+"\x16\xb0\x41\xd0\xfe\x9a\xa6\xa0\x7a\x5e\x5d\x0a\x96\x4c\x52\x44"
+"\x9a\x29\x69\x09\xa2\x0e\x5a\x1e\xc2\xb3\x5e\xca\x25\xc0\xe1\xa9"
+"\xd1\x41\x7f\x82\xaf\x1f\xf4\x3c\xf8\x3d\x65\xae\xf0\xa2\x1a\x8f"
+"\x41\xdb\x01\x11\x4c\x01\xcb\x24\xb3\xec\xbb\xf3\xe5\x1b\x53\xf0"
+"\x7a\x81\x01\x61\xa2\x8e\xa4\xd0\xaa\x8f\xa1\x71\xc1\x15\x15\xda"
+"\xf3\x7b\x32\x87\xa6\xb7\x7f\x2b\xac\x2b\x28\xfc\xe4\x1a\x94\xab"
+"\x19\xc9\x13\x72\x33\xfa\x42\xec\x6f\x3f\xe1\xe0\xc7\x23\x4b\x17"
+"\xeb\x89\xd3\x1f\x49\xe1\x49\x56\xee\xe3\x82\x46\x43\x00\x80\xbc"
+"\xa3\xfe\x31\xbc\xc9\xcd\x61\x5b\x7a\xf9\xf7\xb7\x48\x98\xbf\xdc"
+"\x79\xca\x71\x3b\xb0\xda\x08\x1e\x25\x97\x83\xd7\x21\x2c\xaa\xc0"
+"\x5c\xfd\x7f\xc4\x30\xd8\x7b\x59\x35\x49\x62\x0f\x4c\x03\x02\xe5"
+"\x73\x63\x61\x0b\x69\x2f\x7d\xb3\x99\xc9\x6b\x0a\x29\x9b\xda\xbe"
+"\x98\xdc\x2c\x29\x28\x9a\x75\x2e\xf1\x11\xd3\x71\x5b\x20\x45\x5b"
+"\xb7\x5e\xc1\xd1\xcc\x4e\x5a\x0d\xa5\x70\xa6\x56\xb8\x80\x8c\x97"
+"\x9d\x65\x8d\xec\xa0\x15\x45\xe6\x04\xd8\x3b\x6b\x36\x3f\x71\x58"
+"\x9e\x7a\x9c\xd2\x44\x86\xbf\x89\xa6\x80\x5d\x5e\x99\xc9\x7e\x56"
+"\x76\x17\x02\x98\x5b\xbb\xa0\xe5\xe5\x10\x25\x3e\x82\xc7\xe0\x91"
+"\x77\x39\x50\x9c\x3d\x2a\x91\x03\x13\x6d\x6d\xd3\xc6\x68\xd3\xa0"
+"\x88\xbc\x24\x5d\xf1\x26\x19\xf4\xb0\x74\x51\x93\x17\xcf\x67\x6c"
+"\x72\x30\xed\x39\xfe\x59\x54\x88\x84\x70\x56\x11\xaf\x41\x66\xa5"
+"\xf9\xf0\x95\xdb\x80\xb8\xae\x2f\xb7\xc3\x65\x72\xd2\xec\xaf\x5f"
+"\xf9\x30\x1e\x5b\x45\x7f\x38\xd5\x03\x02\x60\xaa\xf9\xb7\xd9\xfc"
+"\xa2\x5c\x46\x3e\x9c\xe6\xd6\x8e\x95\x54\xbf\xd8\xe6\xe4\x4b\xc0"
+"\x4c\xa1\x4c\x2c\xb3\xc4\x9f\xef\xeb\x39\x70\x77\xac\xf9\x1f\xb6"
+"\x06\xa2\x53\x7d\x18\xc8\xf8\xda\x8e\x82\x97\x4f\xdd\xd5\x19\x2f"
+"\xa2\x70\x4a\xbd\x5a\x15\x70\xb6\x55\x04\x14\xba\x0a\x04\xdc\x8e"
+"\xaf\xf2\x52\xd5\x90\x4c\x30\xd3\x29\x53\x1c\x66\x37\x5f\x8e\xfc"
+"\x45\x83\xd9\xac\x75\x9e\x0f\x66\x51\xc0\x8a\xc5\x34\x25\x9e\x3b",
+	.ilen = 1024,
+	.input =
+"\xa8\x47\xa1\x1d\xcb\xa3\x88\xae\x42\xab\x6d\xf2\x82\xc2\xed\xd5"
+"\x66\x42\x09\x85\x28\x7d\x49\x6f\x37\xdc\xff\x1c\x7e\x33\xc9\xcd"
+"\x6e\xe9\x33\x36\x01\x62\x1d\x67\x77\x6a\x97\xbf\xb1\xdc\x2f\x98"
+"\x2c\xdb\xac\x44\x9d\xed\x31\x7d\x2d\x41\x4b\xd1\x66\x40\x62\x74"
+"\xdc\x00\xd0\x05\xdc\x54\x4c\x63\xeb\xd9\x42\xe1\xdf\xc4\xde\xdd"
+"\xb6\xb8\x93\xfd\x25\x39\x2d\x7f\x85\xf8\x15\xc3\xbc\xbf\x0b\x95"
+"\x11\xef\x57\x0d\x15\x49\x07\xce\x42\xb0\x50\xe1\x07\xb4\x81\x71"
+"\x35\x71\x4b\x66\x89\x7f\x94\x13\x3e\x57\x43\xc3\x36\x28\xcd\xdd"
+"\xc9\x06\x68\xf8\xf3\x09\x3d\x86\x12\x52\x06\xa9\xe9\x83\x2d\x8f"
+"\x90\xfa\x42\xfe\x79\x3f\x68\x4c\x7b\xfa\x94\xa7\xf7\x16\xc7\x41"
+"\x09\xae\xe2\x82\xb5\x2b\xbc\xca\x65\x65\x2c\x27\x2c\x07\x50\x83"
+"\x2d\xad\x55\xaf\x35\xcc\x6a\xc5\x7c\xd8\xed\x75\x91\x9d\x73\xcb"
+"\x4c\xa5\x8f\xc4\x4f\xda\xa8\xb9\xb6\xa7\xb1\x1a\x75\xb4\x08\xbc"
+"\xb2\x90\x50\xfd\x1f\x05\xa8\x88\x35\x81\xb0\xc9\xac\xbc\x25\x7a"
+"\x95\x33\x02\x2b\x74\xe0\x95\x11\x88\xf7\xc3\x63\xb3\x7b\x09\xd5"
+"\xac\x22\x04\x67\x16\xea\xd6\x37\x38\x8e\xa5\xbd\x62\xa2\x1f\xa5"
+"\x04\x31\x89\xdf\x69\xb1\xde\xe3\x7c\x9d\x7b\x27\xba\x0a\x74\xdc"
+"\x06\x1c\xcd\x6e\x4b\x52\xe7\x6d\x34\x29\x38\xe2\x19\xfc\x0c\xc4"
+"\x78\x03\x1d\x53\x98\x00\x5c\x7a\xec\x23\x5f\x95\xd5\xb3\x16\xde"
+"\xc2\x17\xc2\x0c\x13\x63\x0a\x4b\x7e\x6c\xc7\xbc\x4a\xd0\xae\x29"
+"\xc0\x50\x16\x6f\x01\x2b\xdc\x40\x9f\x91\x8f\xa3\xaf\xd4\x40\xa8"
+"\x2e\x09\x7c\xf4\x3d\x85\xe6\xd9\x3c\x78\x7c\xf1\x6d\xe4\x13\x00"
+"\x98\xf5\xb4\x06\x9f\x90\x0a\x3e\x9f\x51\x0f\xbb\x0f\x13\x07\xc0"
+"\xfd\x26\x53\x24\x24\xf7\x21\x41\xcf\x20\x9d\x77\xe4\xe0\x52\x2a"
+"\x48\xd9\xeb\x65\xce\xf3\x90\x03\x47\x8d\x2b\x77\x54\x46\xda\xff"
+"\x15\x3d\xa5\xd9\x5a\xb6\xd3\xdf\x9c\x91\xc3\xf2\xd2\xdf\xd7\x8c"
+"\x1d\x83\x77\x47\xcd\x74\x23\x44\x04\x06\x8e\x64\x62\x29\xe5\xa0"
+"\xf7\xa7\xc7\xb7\x84\xdb\x9c\x5c\x04\x7f\xca\xb3\x85\x2c\x44\xa6"
+"\x09\x0e\xa3\x2c\x52\x42\x25\x02\x63\x99\xd0\xa5\x27\x61\x64\x4f"
+"\x65\xd7\x31\x56\x24\x97\xb0\x2d\xbb\x0c\xbe\x06\x68\x8a\x2e\xa3"
+"\x0c\xb9\x05\x52\xdb\xbd\x7e\x89\x60\x2e\x28\x76\xba\x5a\x94\xb6"
+"\x94\xc4\xf6\x68\x50\x35\x24\x7b\x2b\x04\x0e\x4c\xf3\x17\x54\xcb"
+"\xcd\x32\x18\x60\xff\xc9\xfe\xe1\x83\xe4\xe6\x9b\x5e\xd8\x21\xbf"
+"\xbf\x69\x01\x3a\x03\xc6\x9f\xe5\xd4\xdf\x01\x20\x8e\xea\x5b\xe1"
+"\xbd\x46\x3c\x3a\x60\x30\xa0\x48\xa0\x07\x82\x27\x4e\x03\xc3\x15"
+"\x98\x1f\xea\x4f\x8c\x90\x4d\xb1\xc5\x90\x40\x59\xda\x5b\x02\x65"
+"\x07\xb9\x64\xe7\x4c\x76\x70\x16\x8a\xc3\xf9\x4f\xed\x25\x47\xaa"
+"\x3b\x49\x8f\xf6\xf0\x71\x94\x34\xda\x29\x0f\x4e\xd4\x95\x3b\xe3"
+"\xef\x99\x3b\x1c\xf7\x09\x5d\xe0\x0d\x03\xe6\x9d\x47\x4c\x8c\xe8"
+"\x26\xb6\x30\x1b\x81\xdc\xa5\x5a\xf1\x04\x18\xf3\xaf\x81\xa2\x7e"
+"\xce\x8b\x33\xfc\xf2\xb1\x5a\x06\xd1\xb9\x59\x73\xd7\xda\x85\xd9"
+"\x30\x73\x98\x4d\x63\x50\x66\x71\x15\x88\x9a\x5d\xd5\x25\x40\x9a"
+"\xe3\x9c\x0b\x4f\xd8\xf5\xbf\xb3\xec\x02\x95\xca\x90\x07\x5d\x99"
+"\x9e\x16\xa2\x18\xa5\xa2\x03\xb1\x16\x6b\x4e\x32\xab\x19\x29\x55"
+"\xcc\xbe\xa8\x7b\xf7\x68\x64\x0e\xc0\x54\x91\x6d\x19\xec\xe9\x8c"
+"\x56\x5e\x71\xa5\x73\x50\x5d\x0d\xd3\xb2\x31\xca\x97\x7b\xf8\x6e"
+"\xfd\xb9\x47\x9b\x17\xf9\x56\x3a\xc6\xb0\x52\x45\x4f\x4a\x13\xe9"
+"\xb7\x64\x02\xdb\xe8\x67\xa3\x9e\xe4\xd9\x49\xc4\xf3\x27\xe3\xb0"
+"\xad\x6e\x51\x65\x14\x4f\xb2\x4b\x8a\xd6\x87\x17\x8c\xe2\x7a\xa1"
+"\x13\xbb\x8c\x7c\x3e\x69\xd2\x29\x06\x36\xf3\x55\x80\xcc\x0e\xa5"
+"\x18\x5a\x5f\xcb\x15\x2e\x7c\x62\xff\x3f\xe7\x7b\xd8\xe4\xa6\x9c"
+"\x4c\x5b\x55\x73\x4a\x0d\x21\x07\xf9\x79\xcb\x17\x51\x06\xf3\xcc"
+"\xfc\x08\x72\x6e\xbc\x04\xe2\x6d\xd8\x52\x1d\x29\x7e\x7a\x06\x8d"
+"\x87\x65\x2e\x2e\x7c\x07\x77\x3a\x35\x4d\x3a\x13\xd3\xf6\xc2\x1f"
+"\x2d\x5d\x14\xa5\x04\xe5\xc5\x7b\xd6\xa9\x70\x4b\x43\x21\x93\xdf"
+"\xe4\xf1\xf8\x75\xf1\x65\x9c\xf8\x0b\x07\x31\xdc\xf2\xba\x06\x91"
+"\xe1\x84\x87\x34\x2d\xdd\xa7\x87\xc0\xc2\x4d\x8d\xe0\x18\x70\xbb"
+"\xe3\x3e\x13\x48\xfc\xf4\x13\x85\xc4\x65\xcf\xe4\x43\x98\x14\x8f"
+"\xf4\x17\x62\x27\x39\xe5\xb6\x45\x76\x61\x78\x0b\x3d\x48\xb3\x41"
+"\xa6\xca\x7c\xed\x52\x19\x99\xea\x73\xc9\xd0\x0b\xeb\xbb\x5a\x69"
+"\x44\x3d\xb2\x81\x25\xb0\x2f\x08\xf0\x8c\x32\xa9\xf0\x79\x3c\x42"
+"\xc3\xdc\x9e\xd1\xec\x93\x49\xc9\x82\x0e\x13\x12\xb3\x8a\x98\x1b"
+"\x35\xe1\x4a\xef\xb4\x73\x28\x1a\x17\x96\xe2\x9a\x50\xc8\xd5\x98"
+"\xec\x96\x6f\x81\x05\x37\xee\x8b\x93\x12\x7c\x41\x26\xd5\x9c\x05",
+	.rlen = 1024,
+	/* limit to maximum of 8 */
+	.np	= 8,
+	.also_non_np = 1,
+	.tap	= { 32, 64, 96, 128, 192, 32, 64, 96+128+192 },
+},
+};
+#define AES_CBC_DEC_TV_TEMPLATE_RNDDATA_KEY16_VEC_COUNT	16
+
+#endif /* CONFIG_CRYPTO_AES_CBC_MB */
+
 #endif	/* _CRYPTO_TESTMGR_H */
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt
  2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
                   ` (5 preceding siblings ...)
  2017-04-20 20:50 ` [Patch V5 6/7] crypto: AES vectors for AES CBC multibuffer testing Megha Dey
@ 2017-04-20 20:50 ` Megha Dey
  2017-04-21  0:54   ` kbuild test robot
  2017-04-21  1:56   ` kbuild test robot
  6 siblings, 2 replies; 15+ messages in thread
From: Megha Dey @ 2017-04-20 20:50 UTC (permalink / raw)
  To: herbert
  Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey, Megha Dey

The tcrypt test framework for CBC multi-buffer testing is
laid out in this patch. Tcrypt has been extended to validate
the functionality and performance of AES CBC multi-buffer support.

A new test(mode=600) has been added to test the speed of the multibuffer
case, as multi-buffer encrypt will wait for additional encrypt requests
that never arrive to kick off computation. So we always incur the extra
delay before flush timer expires to trigger the computation in the
(mode=500)
test. We create the new tests that will send out these additional
requests
that can be aggregated and computed in parallel for true throughput
speed test
of the multi-buffer encrypt test case.case.

The enhanced CBC tests create multiple transforms and exercise
the multi-buffer implementation. Crafted requests are sent at once
to the multiple transforms created and the returned responses
are compared with expected results. The test vectors are so chosen
as to exercise the scatter-gather list to the maximum allowable limit
within the framework.

Originally-by: Chandramouli Narayanan <mouli_7982@yahoo.com>
Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
---
 crypto/tcrypt.c  | 257 +++++++++++++++++++-
 crypto/testmgr.c | 707 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 crypto/testmgr.h |  64 ++++-
 3 files changed, 1015 insertions(+), 13 deletions(-)

diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c
index 9a11f3c..1d4416a 100644
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -38,6 +38,7 @@
 #include <linux/jiffies.h>
 #include <linux/timex.h>
 #include <linux/interrupt.h>
+#include <linux/crypto.h>
 #include "tcrypt.h"
 
 /*
@@ -84,7 +85,7 @@ struct tcrypt_result {
 	int err;
 };
 
-static void tcrypt_complete(struct crypto_async_request *req, int err)
+void tcrypt_complete(struct crypto_async_request *req, int err)
 {
 	struct tcrypt_result *res = req->data;
 
@@ -183,6 +184,11 @@ static int test_aead_cycles(struct aead_request *req, int enc, int blen)
 
 #define XBUFSIZE 8
 #define MAX_IVLEN 32
+#define MB_WIDTH 8
+struct scatterlist mb_sg[MB_WIDTH][XBUFSIZE];
+struct skcipher_request *mb_req[MB_WIDTH];
+struct tcrypt_result mb_tresult[MB_WIDTH];
+char *mb_xbuf[MB_WIDTH][XBUFSIZE];
 
 static int testmgr_alloc_buf(char *buf[XBUFSIZE])
 {
@@ -780,6 +786,46 @@ static inline int do_one_acipher_op(struct skcipher_request *req, int ret)
 	return ret;
 }
 
+
+/*
+ * Perform a maximum of MB_WIDTH operations.
+ * Await the results and measure performance.
+ */
+cycles_t mb_start, mb_end;
+static int mb_err[MB_WIDTH];
+
+static inline int do_multi_acipher_op(
+	struct skcipher_request *req[MB_WIDTH], int enc)
+{
+	int i, ret, comp_ret = 0;
+	bool is_async;
+
+	for (i = 0; i < MB_WIDTH; ++i) {
+		ret = enc == ENCRYPT ? crypto_skcipher_encrypt(req[i])
+					: crypto_skcipher_decrypt(req[i]);
+		mb_err[i] = ret;
+		if (ret == -EINPROGRESS || ret == -EBUSY)
+			continue; /* on with next req */
+		/* any other error, bail out */
+		if (ret)
+			return ret;
+	}
+	for (i = 0; i < MB_WIDTH; ++i) {
+		struct tcrypt_result *tr = req[i]->base.data;
+
+		is_async = mb_err[i] == -EINPROGRESS || mb_err[i] == -EBUSY;
+		if (is_async) {
+			wait_for_completion(&tr->completion);
+			reinit_completion(&tr->completion);
+		}
+		comp_ret = tr->err;
+		if (comp_ret)
+			pr_info("multi_acipher_op error\n");
+	}
+
+	return comp_ret;
+}
+
 static int test_acipher_jiffies(struct skcipher_request *req, int enc,
 				int blen, int secs)
 {
@@ -927,6 +973,7 @@ static void test_skcipher_speed(const char *algo, int enc, unsigned int secs,
 			if (ret) {
 				pr_err("setkey() failed flags=%x\n",
 					crypto_skcipher_get_flags(tfm));
+
 				goto out_free_req;
 			}
 
@@ -980,6 +1027,203 @@ static void test_skcipher_speed(const char *algo, int enc, unsigned int secs,
 	crypto_free_skcipher(tfm);
 }
 
+static int test_mb_acipher_jiffies(
+	struct skcipher_request *req[MB_WIDTH], int enc, int blen, int secs)
+{
+	unsigned long start, end;
+	int bcount;
+	int ret;
+
+	/* initiate a maximum of MB_WIDTH operations and measure performance */
+	for (start = jiffies, end = start + secs * HZ, bcount = 0;
+		time_before(jiffies, end); bcount += MB_WIDTH) {
+		ret = do_multi_acipher_op(req, enc);
+		if (ret)
+		return ret;
+	}
+
+	pr_cont("%d operations in %d seconds (%ld bytes)\n",
+		bcount, secs, (long)bcount * blen);
+	return 0;
+}
+
+#define ITR 8
+static int test_mb_acipher_cycles(
+	struct skcipher_request *req[MB_WIDTH], int enc, int blen)
+{
+	cycles_t cycles = 0;
+	int ret = 0;
+	int i;
+
+	/* Warm-up run. */
+	for (i = 0; i < 4; i++) {
+		ret = do_multi_acipher_op(req, enc);
+
+		if (ret)
+			goto out;
+	}
+	/*
+	 * Initiate a maximum of MB_WIDTH operations per loop
+	 * Measure performance over MB_WIDTH iterations
+	 * Let do_multi_acipher_op count the cycles
+	 */
+	for (i = 0; i < ITR; i++) {
+		mb_start = get_cycles();
+		ret = do_multi_acipher_op(req, enc);
+
+		mb_end = get_cycles();
+		cycles += mb_end - mb_start;
+		if (ret)
+			goto out;
+	}
+
+out:
+	if (ret == 0)
+		pr_cont("1 operation in %llu cycles (%d bytes)\n",
+			(cycles + 4) / (ITR*MB_WIDTH), blen);
+
+	return ret;
+}
+
+static void test_mb_acipher_speed(const char *algo, int enc, unsigned int secs,
+				struct cipher_speed_template *template,
+				unsigned int tcount, u8 *keysize)
+{
+	unsigned int ret, i, j, k, iv_len, r;
+	const char *key;
+	char iv[128];
+	struct crypto_skcipher *tfm;
+	const char *e, *driver;
+	u32 *b_size;
+
+	pr_info("test_mb_acipher_speed: test algo %s\n", algo);
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	tfm = crypto_alloc_skcipher(algo, 0, 0);
+
+	if (IS_ERR(tfm)) {
+		pr_err("failed to load transform for %s: %ld\n", algo,
+			PTR_ERR(tfm));
+		return;
+	}
+
+       /* FIXME: do we need to check this? */
+	driver = get_driver_name(crypto_skcipher, tfm);
+	pr_info("\ntesting speed of async %s (%s) %s\n", algo, driver, e);
+
+	/* set up multiple requests for the transform */
+	for (r = 0; r < MB_WIDTH; ++r) {
+		init_completion(&mb_tresult[r].completion);
+		mb_req[r] = skcipher_request_alloc(tfm, GFP_KERNEL);
+		if (!mb_req[r]) {
+			pr_err("tcrypt: skcipher: Failed to allocate request for %s\n",
+					algo);
+			goto out;
+		}
+
+		skcipher_request_set_callback(mb_req[r],
+				CRYPTO_TFM_REQ_MAY_BACKLOG,
+				tcrypt_complete, &mb_tresult[r]);
+	}
+
+	/* loop through different data sizes to encrypt/decrypt */
+	i = 0;
+	do {
+		b_size = block_sizes;
+
+		do {
+			if ((*keysize + *b_size) > TVMEMSIZE * PAGE_SIZE) {
+				pr_err("template (%u) too big for tvmem (%lu)\n",
+				*keysize + *b_size, TVMEMSIZE * PAGE_SIZE);
+				goto out_free_req;
+			}
+
+			pr_info("test %u (%d bit key, %d byte blocks): ", i,
+					*keysize * 8, *b_size);
+
+			memset(tvmem[0], 0xff, PAGE_SIZE);
+
+			/* set key, plain text and IV */
+			key = tvmem[0];
+			for (j = 0; j < tcount; j++) {
+				if (template[j].klen == *keysize) {
+					key = template[j].key;
+					break;
+				}
+			}
+
+			crypto_skcipher_clear_flags(tfm, ~0);
+
+			ret = crypto_skcipher_setkey(tfm, key, *keysize);
+			if (ret) {
+				pr_err("setkey() failed flags=%x keysize=%d\n",
+					crypto_skcipher_get_flags(tfm),
+					*keysize);
+				goto out_free_req;
+			}
+
+			/* set scatter-gather list of data */
+			for (r = 0; r < MB_WIDTH; ++r) {
+				sg_init_table(mb_sg[r], TVMEMSIZE);
+
+				k = *keysize + *b_size;
+				if (k > PAGE_SIZE) {
+					sg_set_buf(mb_sg[r],
+							tvmem[0] + *keysize,
+							PAGE_SIZE - *keysize);
+					k -= PAGE_SIZE;
+					j = 1;
+					while (k > PAGE_SIZE) {
+						sg_set_buf(&mb_sg[r][j],
+							tvmem[j], PAGE_SIZE);
+						memset(tvmem[j], 0xff,
+								PAGE_SIZE);
+						j++;
+						k -= PAGE_SIZE;
+					}
+					sg_set_buf(&mb_sg[r][j], tvmem[j], k);
+					memset(tvmem[j], 0xff, k);
+				} else {
+					sg_set_buf(mb_sg[r],
+						tvmem[0] + *keysize, *b_size);
+				}
+
+				iv_len = crypto_skcipher_ivsize(tfm);
+				if (iv_len)
+					memset(&iv, 0xff, iv_len);
+
+				skcipher_request_set_crypt(mb_req[r],
+						mb_sg[r], mb_sg[r],
+						*b_size, iv);
+			}
+			if (secs)
+				ret = test_mb_acipher_jiffies(mb_req, enc,
+						*b_size, secs);
+			else
+				ret = test_mb_acipher_cycles(mb_req, enc,
+						*b_size);
+
+			if (ret) {
+				pr_err("%s() failed flags=%x\n", e,
+					crypto_skcipher_get_flags(tfm));
+				break;
+			}
+			b_size++;
+			i++;
+		} while (*b_size);
+		keysize++;
+	} while (*keysize);
+
+out_free_req:
+	for (r = 0; r < MB_WIDTH; ++r)
+		skcipher_request_free(mb_req[r]);
+out:
+	crypto_free_skcipher(tfm);
+}
+
 static void test_acipher_speed(const char *algo, int enc, unsigned int secs,
 			       struct cipher_speed_template *template,
 			       unsigned int tcount, u8 *keysize)
@@ -2039,6 +2283,17 @@ static int do_test(const char *alg, u32 type, u32 mask, int m)
 				   speed_template_8_32);
 		break;
 
+	case 600:
+		/* Measure performance of aes-cbc multibuffer support */
+		test_mb_acipher_speed("cbc(aes)", ENCRYPT, sec, NULL, 0,
+			speed_template_16_24_32);
+		test_mb_acipher_speed("cbc(aes)", DECRYPT, sec, NULL, 0,
+			speed_template_16_24_32);
+		break;
+	case 601:
+		ret += tcrypt_test("cbc(aes)");
+		break;
+
 	case 1000:
 		test_available();
 		break;
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index cd075c7..ef5d919 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -1,3 +1,4 @@
+
 /*
  * Algorithm testing framework and tests.
  *
@@ -148,6 +149,23 @@ struct alg_test_desc {
 static const unsigned int IDX[8] = {
 	IDX1, IDX2, IDX3, IDX4, IDX5, IDX6, IDX7, IDX8 };
 
+#ifdef CONFIG_CRYPTO_AES_CBC_MB
+/*
+ * Indexes into the xbuf to simulate cross-page access for multibuffer tests.
+ */
+#define MB_IDX1                32
+#define MB_IDX2                32400
+#define MB_IDX3                4222
+#define MB_IDX4                8193
+#define MB_IDX5                22222
+#define MB_IDX6                17101
+#define MB_IDX7                27333
+#define MB_IDX8                13222
+static unsigned int MB_IDX[8] = {
+	MB_IDX1, MB_IDX2, MB_IDX3, MB_IDX4,
+	MB_IDX5, MB_IDX6, MB_IDX7, MB_IDX8 };
+#endif /* CONFIG_CRYPTO_AES_CBC_MB */
+
 static void hexdump(unsigned char *buf, unsigned int len)
 {
 	print_hex_dump(KERN_CONT, "", DUMP_PREFIX_OFFSET,
@@ -1057,6 +1075,8 @@ static int test_cipher(struct crypto_cipher *tfm, int enc,
 			printk(KERN_ERR "alg: cipher: Test %d failed "
 			       "on %s for %s\n", j, e, algo);
 			hexdump(q, template[i].rlen);
+			printk(KERN_ERR "alg: cipher: Test %d expected on %s for %s\n",
+				j, e, algo);
 			ret = -EINVAL;
 			goto out;
 		}
@@ -1070,6 +1090,7 @@ static int test_cipher(struct crypto_cipher *tfm, int enc,
 	return ret;
 }
 
+
 static int __test_skcipher(struct crypto_skcipher *tfm, int enc,
 			   const struct cipher_testvec *template,
 			   unsigned int tcount,
@@ -1189,6 +1210,8 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc,
 			pr_err("alg: skcipher%s: Test %d failed (invalid result) on %s for %s\n",
 			       d, j, e, algo);
 			hexdump(q, template[i].rlen);
+			pr_err("alg: skcipher%s: Test %d expected %s for %s\n",
+				d, j, e, algo);
 			ret = -EINVAL;
 			goto out;
 		}
@@ -1307,7 +1330,14 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc,
 				pr_err("alg: skcipher%s: Chunk test %d failed on %s at page %u for %s\n",
 				       d, j, e, k, algo);
 				hexdump(q, template[i].tap[k]);
+				pr_err(
+	"alg: skcipher%s: Chunk test %d expected on %s at page %u for %s\n",
+					d, j, e, k, algo);
 				goto out;
+			} else {
+				pr_err(
+	"alg: skcipher%s: Chunk test %d succeeded on %s at page %u for %s klen %d\n",
+					d, j, e, k, algo, template[i].klen);
 			}
 
 			q += template[i].tap[k];
@@ -1335,6 +1365,677 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc,
 	return ret;
 }
 
+#ifdef CONFIG_CRYPTO_AES_CBC_MB
+/*
+ * AES CBC multibuffer implementation can encrypt
+ * a maximum of 8 requests at once. Therefore, the
+ * MAX_REQ should be set >= 8 for better performance.
+ * The MAX_XFM allows multiple transforms created within
+ * the test framework.
+ *
+ * The multibuffer tests set up several requests and
+ * hand them off to the multibuffer driver. Error processing
+ * does not simply bail out. The test code walks through all
+ * requests and awaits their completion. Indiscriminate premature
+ * bailout on error while other requests are in progress will cause
+ * trouble.
+ */
+
+#define MAX_REQ        16
+#define MAX_XFM        MAX_REQ
+static struct skcipher_request *mb_req[MAX_REQ][MAX_REQ];
+static struct scatterlist mb_sg[MAX_REQ][MAX_REQ][8];
+static struct scatterlist mb_sgout[MAX_REQ][MAX_REQ][8];
+static struct tcrypt_result mb_result[MAX_REQ][MAX_REQ];
+static char *mb_xbuf[MAX_REQ][MAX_REQ][XBUFSIZE];
+static char *mb_xoutbuf[MAX_REQ][MAX_REQ][XBUFSIZE];
+static int mb_err[MAX_REQ][MAX_REQ];
+static char ivec[MAX_REQ][MAX_REQ][MAX_IVLEN];
+/* random data for cbc multibuffer tests */
+static struct cipher_test_suite mb_cbc_cipher[] = {
+	{
+		{
+			.vecs = aes_cbc_enc_tv_template_rnddata_klenmix,
+			.count = AES_CBC_ENC_TV_TEMPLATE_RNDDATA_KEY16_VEC_COUNT
+		},
+		{
+			.vecs = aes_cbc_dec_tv_template_rnddata_klen16,
+			.count = AES_CBC_DEC_TV_TEMPLATE_RNDDATA_KEY16_VEC_COUNT
+		}
+	},
+};
+
+/*
+ * Test multibuffer version AES CBC crypto algorithm via multiple transforms.
+ * The test iterates through the test vectors sending MAX_REQ requests with
+ * the same vector and IV.
+ */
+
+/* free buffers allocated for testing multibuffer cbc */
+static void free_mbxbuf(int tidx)
+{
+	int i;
+
+	for (i = 0; i < MAX_REQ; ++i) {
+		if (mb_xbuf[tidx][i])
+			testmgr_free_buf(mb_xbuf[tidx][i]);
+	}
+}
+
+/* free MAX_REQ mb_xout buffers for a given transform */
+static void free_mbxoutbuf(int tidx)
+{
+	int i;
+
+	for (i = 0; i < MAX_REQ; ++i) {
+		if (mb_xoutbuf[tidx][i])
+			testmgr_free_buf(mb_xoutbuf[tidx][i]);
+	}
+}
+
+/* free MAX_REQ requests for a given transform */
+static void free_mbreq(int tidx)
+{
+	int i;
+
+	for (i = 0; i < MAX_REQ; ++i)
+		skcipher_request_free(mb_req[tidx][i]);
+}
+
+/* For a given transform, allocate buffers to test multibuffer cbc */
+static int allocbuf_mb(int tidx, struct crypto_skcipher *tfm,
+		       const bool diff_dst, const char *algo)
+{
+	int r, n, err = 0;
+	char *ybuf[XBUFSIZE];
+
+	for (r = 0; r < MAX_REQ; ++r) {
+		if (testmgr_alloc_buf(ybuf))
+			goto out_nobuf;
+
+	for (n = 0; n < XBUFSIZE; ++n)
+		mb_xbuf[tidx][r][n] = ybuf[n];
+
+	if (diff_dst) {
+		if (testmgr_alloc_buf(ybuf))
+			goto out_nooutbuf;
+		for (n = 0; n < XBUFSIZE; ++n)
+			mb_xoutbuf[tidx][r][n] = ybuf[n];
+	}
+
+	init_completion(&mb_result[tidx][r].completion);
+
+	mb_req[tidx][r] = skcipher_request_alloc(
+				tfm,
+				GFP_KERNEL);
+
+	if (!mb_req[tidx][r]) {
+		err = -ENOMEM;
+		pr_err(
+		"alg: __test_skcipher: Failed to allocate request for %s\n",
+			algo);
+		goto out;
+	}
+	skcipher_request_set_callback(mb_req[tidx][r],
+		CRYPTO_TFM_REQ_MAY_BACKLOG,
+		tcrypt_complete, &mb_result[tidx][r]);
+	}
+	return 0;
+
+out:
+	free_mbreq(tidx);
+	if (diff_dst)
+		free_mbxoutbuf(tidx);
+
+out_nooutbuf:
+	free_mbxbuf(tidx);
+
+out_nobuf:
+	return err;
+}
+
+static void set_mb_input(unsigned int tidx, unsigned int vidx,
+			 const struct cipher_testvec *template,
+			 const int align_offset, bool uniq_vec)
+{
+	void *data;
+	const struct cipher_testvec *tvec;
+
+	tvec = &template[vidx] + tidx;
+	data = mb_xbuf[tidx][0][0];
+	data += align_offset;
+	memcpy(data, tvec->input, tvec->ilen);
+}
+
+static void send_mb_req(int tidx, unsigned int vidx, int enc,
+			const bool diff_dst, const int align_offset,
+			const struct cipher_testvec *template, bool uniq_vec,
+			const char *algo)
+{
+	int ret;
+	void *data;
+	const char *iv;
+	char *d, *e;
+	unsigned short ilen;
+	const struct cipher_testvec *tvec;
+	char *thisiv;
+
+	tvec = &template[vidx] + tidx;
+	iv = tvec->iv;
+	ilen = tvec->ilen;
+	if (diff_dst)
+		d = "-ddst";
+	else
+		d = "";
+
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	memset(&mb_err[tidx][0], 0, MAX_REQ); /* no error to begin with */
+
+	data = mb_xbuf[tidx][0][0];
+	data += align_offset;
+	sg_init_one(&mb_sg[tidx][0][0], data, ilen);
+	if (diff_dst) {
+		data = mb_xoutbuf[tidx][0][0];
+		data += align_offset;
+		sg_init_one(&mb_sgout[tidx][0][0], data, ilen);
+	}
+
+	thisiv = ivec[tidx][0];
+	memcpy(thisiv, iv, MAX_IVLEN);
+	skcipher_request_set_crypt(
+			mb_req[tidx][0],
+			mb_sg[tidx][0],
+			(diff_dst) ? mb_sgout[tidx][0]
+			: mb_sg[tidx][0],
+			ilen, thisiv);
+	ret = enc ?
+		crypto_skcipher_encrypt(
+				mb_req[tidx][0])
+		:
+		crypto_skcipher_decrypt(
+				mb_req[tidx][0]);
+
+	if (ret == -EINPROGRESS || ret == -EBUSY || ret == 0) {
+		/* deal with return status properly */
+		mb_err[tidx][0] = ret;
+	} else if (ret) {
+		unsigned int id;
+
+		mb_err[tidx][0] = ret;
+		id = vidx;
+		/* error */
+		pr_err("skcipher%s: %s failed on test %d for %s: ret=%d\n",
+				d, e, id, algo, -ret);
+		pr_err("skcipher%s: req=%d failed\n",
+				d, tidx);
+	}
+}
+
+static void await_mb_result(int tidx)
+{
+	int ret;
+	struct tcrypt_result *tr = &mb_result[tidx][0];
+
+	if (mb_err[tidx][0]) {
+		if (mb_err[tidx][0] != -EINPROGRESS &&
+				       mb_err[tidx][0] != -EBUSY) {
+			pr_err("skcipher error\n"); /* skip reqs that failed */
+			return;
+		}
+		/* wait on async completions */
+		wait_for_completion(&tr->completion);
+		ret = tr->err;
+		mb_err[tidx][0] = ret;
+		if (!ret) {
+			/* no error, on with next */
+			reinit_completion(&tr->completion);
+		} else {
+			pr_err("skcipher: xfm=%d completion error %d\n",
+						tidx, ret);
+		}
+	}
+	/* no wait on synchronous completions */
+}
+
+static void check_mb_result(int tidx, unsigned int vidx, int enc,
+			const bool diff_dst, const int align_offset,
+			const struct cipher_testvec *template, bool uniq_vec,
+			const char *algo)
+{
+	void *data;
+	char *q, *d, *e;
+	const struct cipher_testvec *tvec;
+
+	tvec = &template[vidx] + tidx;
+	if (diff_dst)
+		d = "-ddst";
+	else
+		d = "";
+
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	/* the request resulted in error, move on */
+	if (mb_err[tidx][0])
+		return;
+
+	if (diff_dst) {
+		data = mb_xoutbuf[tidx][0][0];
+		data += align_offset;
+	} else {
+		data = mb_xbuf[tidx][0][0];
+		data += align_offset;
+	}
+	q = data;
+	if (memcmp(q, tvec->result, tvec->rlen)) {
+		pr_err("skcipher%s: Test %d(%d) failed on %s for %s\n",
+				d, tidx, vidx, e, algo);
+		pr_err("skcipher: xfm=%d result mismatch\n",
+				tidx);
+		pr_err("Expected result for xfm=%d\n", tidx);
+		pr_err("Encountered result for xfm=%d\n",
+				tidx);
+	} else {
+		pr_err("skcipher%s: Test %d(%d) succeeded on %s for %s\n",
+				d, tidx, vidx, e, algo);
+	}
+}
+
+static void check_mb_sg_result(int tidx, unsigned int vidx, int enc,
+			bool diff_dst, const struct cipher_testvec *template,
+			bool uniq_vec, const char *algo)
+{
+	unsigned int k, n;
+	unsigned int temp;
+	char *q, *d, *e;
+	const struct cipher_testvec *tvec;
+	unsigned int cor_pg, cor_bytes;
+	unsigned int id; /* test id */
+
+	tvec = &template[vidx] + tidx;
+	if (diff_dst)
+		d = "-ddst";
+	else
+		d = "";
+
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	temp = 0;
+	id = vidx;
+	if (mb_err[tidx][0])
+		return; /* on with other reqs */
+	for (k = 0; k < tvec->np; k++) {
+		unsigned int pg;
+
+		pg = MB_IDX[k] >> PAGE_SHIFT;
+		if (diff_dst)
+			q = mb_xoutbuf[tidx][0][pg] +
+				offset_in_page(MB_IDX[k]);
+		else
+			q = mb_xbuf[tidx][0][pg] +
+				offset_in_page(MB_IDX[k]);
+
+		cor_bytes = tvec->tap[k];
+		cor_pg = k;
+		if (memcmp(q, tvec->result + temp, tvec->tap[k])) {
+			pr_err(
+				"skcipher%s: chunk test %d failed/corruption %s @pg %u for %s:%u bytes:\n",
+					d, id, e, cor_pg, algo, cor_bytes);
+			return;
+		}
+		pr_err(
+			"skcipher%s: chunk test %d succeeded %s @pg %u for %s:%u bytes:\n",
+				d, id, e, cor_pg, algo, cor_bytes);
+
+		q += tvec->tap[k];
+		for (n = 0; offset_in_page(q + n) && q[n]; n++)
+			;
+		if (n) {
+			cor_bytes = n;
+			cor_pg = k;
+			pr_err(
+				"skcipher%s: chunk test %d result corruption %s @pg %u for %s:%u bytes:\n",
+					d, id, e, cor_pg, algo, cor_bytes);
+				break; /* on with next request */
+			}
+		temp += tvec->tap[k];
+	}
+}
+
+static void send_mb_sg_req(int tidx, unsigned int vidx, int enc,
+			bool diff_dst, const struct cipher_testvec *template,
+			bool uniq_vec, const char *algo)
+{
+	unsigned int k, n;
+	unsigned int temp;
+	int ret;
+	char *q, *d, *e;
+	char *ybuf[XBUFSIZE];
+	const struct cipher_testvec *tvec;
+	char *thisiv;
+
+	tvec = &template[vidx] + tidx;
+	if (diff_dst)
+		d = "-ddst";
+	else
+		d = "";
+
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	memset(&mb_err[tidx][0], 0, MAX_REQ);
+
+	temp = 0;
+	sg_init_table(&mb_sg[tidx][0][0], tvec->np);
+	if (diff_dst)
+		sg_init_table(&mb_sgout[tidx][0][0], tvec->np);
+
+	for (k = 0; k < tvec->np; ++k) {
+		unsigned int pg;
+
+		if (WARN_ON((offset_in_page(MB_IDX[k]) + tvec->tap[k]) >
+						PAGE_SIZE)) {
+			pr_err("skcipher%s: %s Invalid sg for %s\n",
+						d, e, algo);
+			pr_err("offset + tap(%d) > PAGE_SIZE(%lu)\n",
+						tvec->tap[k], PAGE_SIZE);
+			pr_err("req=%d k=%d tap(%d)\n",
+						tidx, k, tvec->tap[k]);
+			break;  /* skip this */
+		}
+
+		for (n = 0; n < XBUFSIZE; ++n)
+			ybuf[n] = mb_xbuf[tidx][0][n];
+		pg = MB_IDX[k] >> PAGE_SHIFT;
+		q = ybuf[pg] + offset_in_page(MB_IDX[k]);
+
+		memcpy(q, tvec->input + temp, tvec->tap[k]);
+
+		if ((offset_in_page(q) + tvec->tap[k]) < PAGE_SIZE)
+			q[tvec->tap[k]] = 0;
+
+		sg_set_buf(&mb_sg[tidx][0][k], q, tvec->tap[k]);
+		if (diff_dst) {
+			unsigned int segs;
+
+			segs = tvec->tap[k];
+			q = mb_xoutbuf[tidx][0][pg] +
+				offset_in_page(MB_IDX[k]);
+
+			sg_set_buf(&mb_sgout[tidx][0][k], q, segs);
+
+			memset(q, 0, tvec->tap[k]);
+			if ((offset_in_page(q) + tvec->tap[k]) <
+					PAGE_SIZE)
+				q[segs] = 0;
+		}
+
+		temp += tvec->tap[k];
+	}
+
+	thisiv = ivec[tidx][0];
+	memcpy(thisiv, tvec->iv, MAX_IVLEN);
+	skcipher_request_set_crypt(
+			mb_req[tidx][0],
+			&mb_sg[tidx][0][0],
+			(diff_dst) ? &mb_sgout[tidx][0][0]
+			: &mb_sg[tidx][0][0],
+			tvec->ilen,
+			thisiv);
+
+	ret = enc ? crypto_skcipher_encrypt(
+			mb_req[tidx][0])
+			: crypto_skcipher_decrypt(
+				mb_req[tidx][0]);
+
+	if (ret == -EBUSY || ret == -EINPROGRESS || ret == 0) {
+		/* deal with return status properly */
+		mb_err[tidx][0] = ret;
+		if (uniq_vec)
+			++tvec;
+	} else if (ret) {
+		mb_err[tidx][0] = ret;
+		pr_err("skcipher%s: xfm=%d failed for %s algo %s\n",
+				d, tidx, e, algo);
+	}
+}
+
+static int __test_mb_skcipher(struct crypto_skcipher *tfm[MAX_REQ],
+			      int enc, const struct cipher_testvec *template,
+			      unsigned int tcount,
+			      const bool diff_dst, const int align_offset)
+{
+	const char *algo;
+	unsigned int i, j;
+	const char *e, *d;
+	int ret = -ENOMEM;
+	bool sent[MAX_REQ];
+
+	/* same algorithm, multiple xfms */
+	algo = crypto_tfm_alg_driver_name(
+			crypto_skcipher_tfm(tfm[0]));
+
+	memset(mb_xbuf, '\0', sizeof(mb_xbuf));
+	memset(mb_xoutbuf, '\0', sizeof(mb_xoutbuf));
+
+	for (i = 0; i < MAX_REQ; ++i) {
+		if (allocbuf_mb(i, tfm[i], diff_dst, algo))
+			goto out_nobuf;
+	}
+
+	if (diff_dst)
+		d = "-ddst";
+	else
+		d = "";
+
+	if (enc == ENCRYPT)
+		e = "encryption";
+	else
+		e = "decryption";
+
+	for (i = 0; i < MAX_REQ; ++i)
+		sent[i] = false;
+
+	/* multi xfm, and send multi requests for all xfms and await results */
+	j = 0;
+	for (i = 0; i < tcount; i++) {
+		if (template[i].np && !template[i].also_non_np)
+			continue;
+		/* ensure that the vector data is within page size */
+		if (template[i].ilen > PAGE_SIZE)
+			continue;
+
+		ret = -EINVAL;
+		if (WARN_ON(align_offset + template[i].ilen > PAGE_SIZE))
+			goto out;
+
+		/* set the data for multiple xfms */
+		set_mb_input(i, 0, template,
+				align_offset, false);
+
+		/*
+		 * Set the key for multiple xfms.
+		 * To proceed with test all xfms should be successful
+		 */
+
+		crypto_skcipher_clear_flags(tfm[i], ~0);
+		if (template[i].wk)
+			crypto_skcipher_set_flags(tfm[i],
+					CRYPTO_TFM_REQ_WEAK_KEY);
+
+		ret = crypto_skcipher_setkey(tfm[i], template[i].key,
+				template[i].klen);
+		if (!ret == template[i].fail) {
+			pr_err(
+				"alg: skcipher%s: setkey failed on test %d for %s: flags=%x\n",
+				d, i, algo,
+				crypto_skcipher_get_flags(tfm[i]));
+			goto out;
+		} else if (ret)
+			break;
+		/* move on to next test if key could not be setup */
+		if (ret)
+			continue;
+
+		j++;
+
+		sent[i] = true;
+		send_mb_req(i, 0, enc, diff_dst, align_offset,
+				template, false, algo);
+
+	}
+
+	/* await results from multiple requests from multiple xfms */
+	for (i = 0; i < tcount; ++i) {
+		if (sent[i])
+			await_mb_result(i);
+	}
+
+	/* check results from multiple requests from multiple xfms */
+	for (i = 0; i < tcount; ++i) {
+		if (sent[i])
+			check_mb_result(i, 0, enc, diff_dst,
+					align_offset, template, false, algo);
+		sent[i] = false;
+	}
+
+	j = 0;
+	for (i = 0; i < tcount; i++) {
+		/* alignment tests are only done with continuous buffers */
+
+		if (align_offset != 0)
+			break;
+		if (!template[i].np)
+			continue;
+
+		j++;
+		/* set the key for multiple transforms */
+		crypto_skcipher_clear_flags(tfm[i], ~0);
+		if (template[i].wk)
+			crypto_skcipher_set_flags(
+					tfm[i],
+					CRYPTO_TFM_REQ_WEAK_KEY);
+		ret = crypto_skcipher_setkey(tfm[i], template[i].key,
+				template[i].klen);
+		if (!ret == template[i].fail) {
+			pr_err(
+				"skcipher%s: setkey failed on chunk test %d xfm=%d for %s: flags=%x\n",
+					d, j, i, algo,
+					crypto_skcipher_get_flags(tfm[i]));
+				goto out;
+		} else if (ret)
+			break;
+		if (ret)
+			continue; /* on to next test */
+
+		/* iterate the test over multiple requests & xfms */
+
+		sent[i] = true;
+		send_mb_sg_req(i, 0, enc, diff_dst,
+				       template, false, algo);
+	}
+
+	/* wait for completion from all xfms */
+	for (i = 0; i < tcount; ++i) {
+		if (sent[i])
+			await_mb_result(i);
+	}
+
+	/* check results from all xfms */
+	for (i = 0; i < tcount; ++i) {
+		if (sent[i])
+			check_mb_sg_result(i, 0, enc, diff_dst,
+					template, false, algo);
+	}
+
+	ret = 0;
+
+out:
+	for (i = 0; i < MAX_REQ; ++i)
+		free_mbreq(i);
+
+	if (diff_dst) {
+		for (i = 0; i < MAX_REQ; ++i)
+			free_mbxoutbuf(i);
+	}
+	for (i = 0; i < MAX_REQ; ++i)
+		free_mbxbuf(i);
+
+out_nobuf:
+	return ret;
+}
+
+static int test_mb_skcipher(struct crypto_skcipher *tfm[MAX_XFM],
+			    int enc, const struct cipher_testvec *template,
+			    unsigned int tcount)
+{
+	int ret;
+
+	/* test 'dst == src' case */
+	ret = __test_mb_skcipher(tfm, enc, template, tcount, false, 0);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int alg_test_mb_skcipher(const struct alg_test_desc *desc,
+				const char *driver, u32 type, u32 mask)
+{
+	struct crypto_skcipher *tfm[MAX_REQ];
+	int err = 0;
+	int i;
+
+	/* create multiple transforms to test AES CBC */
+	for (i = 0; i < MAX_REQ; i++) {
+		tfm[i] = crypto_alloc_skcipher(driver,
+					type | CRYPTO_ALG_INTERNAL, mask);
+		if (IS_ERR(tfm[i])) {
+			printk(KERN_ERR "alg: skcipher: Failed to load transform for %s: %ld\n",
+					driver, PTR_ERR(tfm[i]));
+			return PTR_ERR(tfm[i]);
+		}
+	}
+
+	for (i = 0; i < ARRAY_SIZE(mb_cbc_cipher); ++i) {
+		err = test_mb_skcipher(
+				tfm,
+				ENCRYPT,
+				mb_cbc_cipher[i].enc.vecs,
+				mb_cbc_cipher[i].enc.count
+				);
+		if (err)
+			goto out;
+
+		err = test_mb_skcipher(
+				tfm,
+				DECRYPT,
+				mb_cbc_cipher[i].dec.vecs,
+				mb_cbc_cipher[i].dec.count
+				);
+		if (err)
+			goto out;
+
+	}
+out:
+	for (i = 0; i < MAX_REQ; i++)
+		crypto_free_skcipher(tfm[i]);
+
+	return err;
+}
+#endif /* CONFIG_CRYPTO_AES_CBC_MB */
+
 static int test_skcipher(struct crypto_skcipher *tfm, int enc,
 			 const struct cipher_testvec *template,
 			 unsigned int tcount)
@@ -1723,6 +2424,12 @@ static int alg_test_skcipher(const struct alg_test_desc *desc,
 	struct crypto_skcipher *tfm;
 	int err = 0;
 
+	#ifdef CONFIG_CRYPTO_AES_CBC_MB
+	/* invoke the comprehensive cbc multibuffer tests */
+	if (desc->alg && (strcmp(desc->alg, "cbc(aes)") == 0))
+		return alg_test_mb_skcipher(desc, driver, type, mask);
+	#endif
+
 	tfm = crypto_alloc_skcipher(driver, type, mask);
 	if (IS_ERR(tfm)) {
 		printk(KERN_ERR "alg: skcipher: Failed to load transform for "
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index 85679b3..d775415 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -34414,19 +34414,59 @@ struct comp_testvec {
 };
 
 #ifdef CONFIG_CRYPTO_AES_CBC_MB
-static struct cipher_testvec aes_cbc_enc_tv_template_rnddata_klen16[] = {
+static struct cipher_testvec aes_cbc_enc_tv_template_rnddata_klenmix[] = {
 {
-	.key =
-"\xd7\x0c\x4c\x6d\x11\x02\xb0\x31\x63\x9b\x82\x76\x9e\x03\x26\xdf",
-	.klen = 16,
-	.iv =
-"\xc1\x62\x66\x62\xb8\x65\x28\xfa\x5f\x36\xd3\x09\xb1\x2c\xa1\xa3",
-	.input =
-"\x4f\x6c\x63\xa5\xd0\x19\x08\x4e\xd4\x58\x33\xf6\x2b\xeb\x26\xb9",
-	.ilen = 16,
-	.result =
-"\xa0\x35\xb0\x33\xc0\x2e\xe5\xbb\xbc\xe6\x01\x9e\xf4\x67\x11\x14",
-	.rlen = 16,
+		.key	= "\x60\x3d\xeb\x10\x15\xca\x71\xbe"
+			  "\x2b\x73\xae\xf0\x85\x7d\x77\x81"
+			  "\x1f\x35\x2c\x07\x3b\x61\x08\xd7"
+			  "\x2d\x98\x10\xa3\x09\x14\xdf\xf4",
+		.klen	= 32,
+		.iv	= "\x00\x01\x02\x03\x04\x05\x06\x07"
+			  "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
+		.input	= "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96"
+			  "\xe9\x3d\x7e\x11\x73\x93\x17\x2a"
+			  "\xae\x2d\x8a\x57\x1e\x03\xac\x9c"
+			  "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51"
+			  "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11"
+			  "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef"
+			  "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17"
+			  "\xad\x2b\x41\x7b\xe6\x6c\x37\x10",
+		.ilen	= 64,
+		.result	= "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba"
+			  "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6"
+			  "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d"
+			  "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d"
+			  "\x39\xf2\x33\x69\xa9\xd9\xba\xcf"
+			  "\xa5\x30\xe2\x63\x04\x23\x14\x61"
+			  "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc"
+			  "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b",
+		.rlen	= 64,
+},
+{ /* From NIST SP800-38A */
+		.key	= "\x8e\x73\xb0\xf7\xda\x0e\x64\x52"
+			  "\xc8\x10\xf3\x2b\x80\x90\x79\xe5"
+			  "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b",
+		.klen	= 24,
+		.iv	= "\x00\x01\x02\x03\x04\x05\x06\x07"
+			  "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
+		.input	= "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96"
+			  "\xe9\x3d\x7e\x11\x73\x93\x17\x2a"
+			  "\xae\x2d\x8a\x57\x1e\x03\xac\x9c"
+			  "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51"
+			  "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11"
+			  "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef"
+			  "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17"
+			  "\xad\x2b\x41\x7b\xe6\x6c\x37\x10",
+		.ilen	= 64,
+		.result	= "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d"
+			  "\x71\x78\x18\x3a\x9f\xa0\x71\xe8"
+			  "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4"
+			  "\xe5\xe7\x38\x76\x3f\x69\x14\x5a"
+			  "\x57\x1b\x24\x20\x12\xfb\x7a\xe0"
+			  "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0"
+			  "\x08\xb0\xe2\x79\x88\x59\x88\x81"
+			  "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd",
+		.rlen	= 64,
 },
 {
 	.key =
-- 
1.9.1

^ permalink raw reply related	[flat|nested] 15+ messages in thread

* Re: [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
@ 2017-04-21  0:19   ` kbuild test robot
  2017-04-21  3:02   ` kbuild test robot
  2017-04-24  9:00   ` Herbert Xu
  2 siblings, 0 replies; 15+ messages in thread
From: kbuild test robot @ 2017-04-21  0:19 UTC (permalink / raw)
  To: Megha Dey
  Cc: kbuild-all, herbert, tim.c.chen, davem, linux-crypto,
	linux-kernel, megha.dey, Megha Dey

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

Hi Megha,

[auto build test ERROR on next-20170420]
[also build test ERROR on v4.11-rc7]
[cannot apply to crypto/master sparc-next/master v4.9-rc8 v4.9-rc7 v4.9-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Megha-Dey/crypto-AES-CBC-multibuffer-implementation/20170421-064210
config: x86_64-rhel (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

>> ERROR: "mcryptd_alloc_skcipher" [crypto/crypto_simd.ko] undefined!
>> ERROR: "mcryptd_free_skcipher" [crypto/crypto_simd.ko] undefined!

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 39144 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt
  2017-04-20 20:50 ` [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt Megha Dey
@ 2017-04-21  0:54   ` kbuild test robot
  2017-04-21  1:56   ` kbuild test robot
  1 sibling, 0 replies; 15+ messages in thread
From: kbuild test robot @ 2017-04-21  0:54 UTC (permalink / raw)
  To: Megha Dey
  Cc: kbuild-all, herbert, tim.c.chen, davem, linux-crypto,
	linux-kernel, megha.dey, Megha Dey

[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]

Hi Megha,

[auto build test WARNING on next-20170420]
[also build test WARNING on v4.11-rc7]
[cannot apply to crypto/master sparc-next/master v4.9-rc8 v4.9-rc7 v4.9-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Megha-Dey/crypto-AES-CBC-multibuffer-implementation/20170421-064210
config: m68k-sun3_defconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 4.9.0
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=m68k 

All warnings (new ones prefixed by >>):

   crypto/tcrypt.c: In function 'test_mb_acipher_cycles':
>> crypto/tcrypt.c:1082:3: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'long unsigned int' [-Wformat=]
      pr_cont("1 operation in %llu cycles (%d bytes)\n",
      ^

vim +1082 crypto/tcrypt.c

  1066		 * Initiate a maximum of MB_WIDTH operations per loop
  1067		 * Measure performance over MB_WIDTH iterations
  1068		 * Let do_multi_acipher_op count the cycles
  1069		 */
  1070		for (i = 0; i < ITR; i++) {
  1071			mb_start = get_cycles();
  1072			ret = do_multi_acipher_op(req, enc);
  1073	
  1074			mb_end = get_cycles();
  1075			cycles += mb_end - mb_start;
  1076			if (ret)
  1077				goto out;
  1078		}
  1079	
  1080	out:
  1081		if (ret == 0)
> 1082			pr_cont("1 operation in %llu cycles (%d bytes)\n",
  1083				(cycles + 4) / (ITR*MB_WIDTH), blen);
  1084	
  1085		return ret;
  1086	}
  1087	
  1088	static void test_mb_acipher_speed(const char *algo, int enc, unsigned int secs,
  1089					struct cipher_speed_template *template,
  1090					unsigned int tcount, u8 *keysize)

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 11963 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt
  2017-04-20 20:50 ` [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt Megha Dey
  2017-04-21  0:54   ` kbuild test robot
@ 2017-04-21  1:56   ` kbuild test robot
  1 sibling, 0 replies; 15+ messages in thread
From: kbuild test robot @ 2017-04-21  1:56 UTC (permalink / raw)
  To: Megha Dey
  Cc: kbuild-all, herbert, tim.c.chen, davem, linux-crypto,
	linux-kernel, megha.dey, Megha Dey

[-- Attachment #1: Type: text/plain, Size: 4127 bytes --]

Hi Megha,

[auto build test WARNING on next-20170420]
[also build test WARNING on v4.11-rc7]
[cannot apply to crypto/master sparc-next/master v4.9-rc8 v4.9-rc7 v4.9-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Megha-Dey/crypto-AES-CBC-multibuffer-implementation/20170421-064210
config: parisc-c3000_defconfig (attached as .config)
compiler: hppa-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=parisc 

All warnings (new ones prefixed by >>):

   In file included from include/linux/printk.h:6:0,
                    from include/linux/kernel.h:13,
                    from include/linux/crypto.h:21,
                    from include/crypto/aead.h:16,
                    from crypto/tcrypt.c:27:
   crypto/tcrypt.c: In function 'test_mb_acipher_cycles':
>> include/linux/kern_levels.h:4:18: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'long unsigned int' [-Wformat=]
    #define KERN_SOH "\001"  /* ASCII Start Of Header */
                     ^
   include/linux/kern_levels.h:23:19: note: in expansion of macro 'KERN_SOH'
    #define KERN_CONT KERN_SOH "c"
                      ^~~~~~~~
>> include/linux/printk.h:315:9: note: in expansion of macro 'KERN_CONT'
     printk(KERN_CONT fmt, ##__VA_ARGS__)
            ^~~~~~~~~
>> crypto/tcrypt.c:1082:3: note: in expansion of macro 'pr_cont'
      pr_cont("1 operation in %llu cycles (%d bytes)\n",
      ^~~~~~~

vim +/KERN_CONT +315 include/linux/printk.h

a0cba217 Linus Torvalds 2016-08-09  299  	printk(KERN_CRIT pr_fmt(fmt), ##__VA_ARGS__)
a0cba217 Linus Torvalds 2016-08-09  300  #define pr_err(fmt, ...) \
a0cba217 Linus Torvalds 2016-08-09  301  	printk(KERN_ERR pr_fmt(fmt), ##__VA_ARGS__)
a0cba217 Linus Torvalds 2016-08-09  302  #define pr_warning(fmt, ...) \
a0cba217 Linus Torvalds 2016-08-09  303  	printk(KERN_WARNING pr_fmt(fmt), ##__VA_ARGS__)
a0cba217 Linus Torvalds 2016-08-09  304  #define pr_warn pr_warning
a0cba217 Linus Torvalds 2016-08-09  305  #define pr_notice(fmt, ...) \
a0cba217 Linus Torvalds 2016-08-09  306  	printk(KERN_NOTICE pr_fmt(fmt), ##__VA_ARGS__)
a0cba217 Linus Torvalds 2016-08-09  307  #define pr_info(fmt, ...) \
a0cba217 Linus Torvalds 2016-08-09  308  	printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
7b1460ec Steven Rostedt 2015-04-15  309  /*
7b1460ec Steven Rostedt 2015-04-15  310   * Like KERN_CONT, pr_cont() should only be used when continuing
7b1460ec Steven Rostedt 2015-04-15  311   * a line with no newline ('\n') enclosed. Otherwise it defaults
7b1460ec Steven Rostedt 2015-04-15  312   * back to KERN_DEFAULT.
7b1460ec Steven Rostedt 2015-04-15  313   */
968ab183 Linus Torvalds 2010-11-15  314  #define pr_cont(fmt, ...) \
968ab183 Linus Torvalds 2010-11-15 @315  	printk(KERN_CONT fmt, ##__VA_ARGS__)
968ab183 Linus Torvalds 2010-11-15  316  
968ab183 Linus Torvalds 2010-11-15  317  /* pr_devel() should produce zero code unless DEBUG is defined */
968ab183 Linus Torvalds 2010-11-15  318  #ifdef DEBUG
968ab183 Linus Torvalds 2010-11-15  319  #define pr_devel(fmt, ...) \
968ab183 Linus Torvalds 2010-11-15  320  	printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
968ab183 Linus Torvalds 2010-11-15  321  #else
968ab183 Linus Torvalds 2010-11-15  322  #define pr_devel(fmt, ...) \
5264f2f7 Joe Perches    2011-01-12  323  	no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)

:::::: The code at line 315 was first introduced by commit
:::::: 968ab1838a5d48f02f5b471aa1d0e59e2cc2ccbc include/linux/kernel.h: Move logging bits to include/linux/printk.h

:::::: TO: Linus Torvalds <torvalds@linux-foundation.org>
:::::: CC: Linus Torvalds <torvalds@linux-foundation.org>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 14663 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
  2017-04-21  0:19   ` kbuild test robot
@ 2017-04-21  3:02   ` kbuild test robot
  2017-04-24  9:00   ` Herbert Xu
  2 siblings, 0 replies; 15+ messages in thread
From: kbuild test robot @ 2017-04-21  3:02 UTC (permalink / raw)
  To: Megha Dey
  Cc: kbuild-all, herbert, tim.c.chen, davem, linux-crypto,
	linux-kernel, megha.dey, Megha Dey

[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]

Hi Megha,

[auto build test ERROR on next-20170420]
[also build test ERROR on v4.11-rc7]
[cannot apply to crypto/master sparc-next/master v4.9-rc8 v4.9-rc7 v4.9-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Megha-Dey/crypto-AES-CBC-multibuffer-implementation/20170421-064210
config: arm64-defconfig (attached as .config)
compiler: aarch64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=arm64 

All errors (new ones prefixed by >>):

   crypto/built-in.o: In function `simd_skcipher_exit_mb':
>> crypto/simd.c:267: undefined reference to `mcryptd_free_skcipher'
   crypto/simd.c:267:(.text+0x12624): relocation truncated to fit: R_AARCH64_CALL26 against undefined symbol `mcryptd_free_skcipher'
   crypto/built-in.o: In function `simd_skcipher_init_mb':
>> crypto/simd.c:282: undefined reference to `mcryptd_alloc_skcipher'
   crypto/simd.c:282:(.text+0x12650): relocation truncated to fit: R_AARCH64_CALL26 against undefined symbol `mcryptd_alloc_skcipher'

vim +267 crypto/simd.c

   261	}
   262	
   263	static void simd_skcipher_exit_mb(struct crypto_skcipher *tfm)
   264	{
   265		struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
   266	
 > 267		mcryptd_free_skcipher(ctx->mcryptd_tfm);
   268	}
   269	
   270	static int simd_skcipher_init_mb(struct crypto_skcipher *tfm)
   271	{
   272		struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
   273		struct mcryptd_skcipher *mcryptd_tfm;
   274		struct simd_skcipher_alg *salg;
   275		struct skcipher_alg *alg;
   276		unsigned reqsize;
   277		struct mcryptd_skcipher_ctx *mctx;
   278	
   279		alg = crypto_skcipher_alg(tfm);
   280		salg = container_of(alg, struct simd_skcipher_alg, alg);
   281	
 > 282		mcryptd_tfm = mcryptd_alloc_skcipher(salg->ialg_name,
   283						   CRYPTO_ALG_INTERNAL,
   284						   CRYPTO_ALG_INTERNAL);
   285		if (IS_ERR(mcryptd_tfm))

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 35043 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
  2017-04-21  0:19   ` kbuild test robot
  2017-04-21  3:02   ` kbuild test robot
@ 2017-04-24  9:00   ` Herbert Xu
  2017-06-08 19:52     ` Megha Dey
  2 siblings, 1 reply; 15+ messages in thread
From: Herbert Xu @ 2017-04-24  9:00 UTC (permalink / raw)
  To: Megha Dey; +Cc: tim.c.chen, davem, linux-crypto, linux-kernel, megha.dey

On Thu, Apr 20, 2017 at 01:50:34PM -0700, Megha Dey wrote:
>
> +static int simd_skcipher_decrypt_mb(struct skcipher_request *req)
> +{
> +	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> +	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
> +	struct skcipher_request *subreq;
> +	struct crypto_skcipher *child;
> +
> +	subreq = skcipher_request_ctx(req);
> +	*subreq = *req;
> +
> +	child = &ctx->mcryptd_tfm->base;
> +
> +	skcipher_request_set_tfm(subreq, child);
> +
> +	return crypto_skcipher_decrypt(subreq);
> +}

Why did you add the code here in simd.c? This doesn't seem to have
anything to do with kernel SIMD instructions.

>From your later patch I see that what you want is simply a wrapper
around a synchronous internal algorithm.  That is indeed similar
in purpose to simd.c, but I still find it weird to be adding this
code here.

My suggestion is to instead move this code to mcryptd.c.  It's
a bit fiddly because mcryptd already exists as a template.  But
you should still be able to create a seaprate mcryptd interface
for skcipher in the way that simd does it.  In time we can convert
mcryptd hash to this model too.

Also you adopted the simd compat naming scheme.  Please don't do
that as you're creating something brand new so there is no need
to keep the existing compat (i.e., __XXX) naming scheme.  In the
new naming scheme, the internal algorithm should just carry the
normal alg name (e.g., ecb(aes)) and driver name, while the mcryptd
wrapped version will have the same algorithm name but carry a
prefix on the driver name (which is simd- for simd.c, but you
should probably used mcryptd-).

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-04-24  9:00   ` Herbert Xu
@ 2017-06-08 19:52     ` Megha Dey
  2017-06-23  8:32       ` Herbert Xu
  0 siblings, 1 reply; 15+ messages in thread
From: Megha Dey @ 2017-06-08 19:52 UTC (permalink / raw)
  To: Herbert Xu; +Cc: tim.c.chen, davem, linux-crypto, linux-kernel

On Mon, 2017-04-24 at 17:00 +0800, Herbert Xu wrote:
> On Thu, Apr 20, 2017 at 01:50:34PM -0700, Megha Dey wrote:
> >
> > +static int simd_skcipher_decrypt_mb(struct skcipher_request *req)
> > +{
> > +	struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> > +	struct simd_skcipher_ctx_mb *ctx = crypto_skcipher_ctx(tfm);
> > +	struct skcipher_request *subreq;
> > +	struct crypto_skcipher *child;
> > +
> > +	subreq = skcipher_request_ctx(req);
> > +	*subreq = *req;
> > +
> > +	child = &ctx->mcryptd_tfm->base;
> > +
> > +	skcipher_request_set_tfm(subreq, child);
> > +
> > +	return crypto_skcipher_decrypt(subreq);
> > +}
> 
> Why did you add the code here in simd.c? This doesn't seem to have
> anything to do with kernel SIMD instructions.
> 
> From your later patch I see that what you want is simply a wrapper
> around a synchronous internal algorithm.  That is indeed similar
> in purpose to simd.c, but I still find it weird to be adding this
> code here.
> 
> My suggestion is to instead move this code to mcryptd.c.  It's
> a bit fiddly because mcryptd already exists as a template.  But
> you should still be able to create a seaprate mcryptd interface
> for skcipher in the way that simd does it.  In time we can convert
> mcryptd hash to this model too.
> 
> Also you adopted the simd compat naming scheme.  Please don't do
> that as you're creating something brand new so there is no need
> to keep the existing compat (i.e., __XXX) naming scheme.  In the
> new naming scheme, the internal algorithm should just carry the
> normal alg name (e.g., ecb(aes)) and driver name, while the mcryptd
> wrapped version will have the same algorithm name but carry a
> prefix on the driver name (which is simd- for simd.c, but you
> should probably used mcryptd-).
> 
> Cheers,

I will move this code to the mcryptd.c.

About the naming scheme, could you give me an example where the internal
and external algorithm have the same name? I tried searching but did not
find any.

When the outer and inner algorithm have the same name, I see a crash
when testing using tcrypt. This is because the wrong algortihm (with a
higher priority) is being picked up in __crypto_alg_lookup.  

Inner alg:
Currently:
alg name:__cbc(aes), driver name:__cbc-aes-aesni-mb

expected:
alg name:cbc(aes), driver name: cbc-aes-aesni-mb

Outer alg:
Currently:
alg name:cbc(aes), driver name:cbc-aes-aesni-mb

expected:
alg name:cbc(aes), driver name:mcryptd-cbc-aes-aesni-mb

Thanks,
Megha

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support
  2017-06-08 19:52     ` Megha Dey
@ 2017-06-23  8:32       ` Herbert Xu
  0 siblings, 0 replies; 15+ messages in thread
From: Herbert Xu @ 2017-06-23  8:32 UTC (permalink / raw)
  To: Megha Dey; +Cc: tim.c.chen, davem, linux-crypto, linux-kernel

On Thu, Jun 08, 2017 at 12:52:54PM -0700, Megha Dey wrote:
>
> I will move this code to the mcryptd.c.
> 
> About the naming scheme, could you give me an example where the internal
> and external algorithm have the same name? I tried searching but did not
> find any.
> 
> When the outer and inner algorithm have the same name, I see a crash
> when testing using tcrypt. This is because the wrong algortihm (with a
> higher priority) is being picked up in __crypto_alg_lookup.  
> 
> Inner alg:
> Currently:
> alg name:__cbc(aes), driver name:__cbc-aes-aesni-mb
> 
> expected:
> alg name:cbc(aes), driver name: cbc-aes-aesni-mb
> 
> Outer alg:
> Currently:
> alg name:cbc(aes), driver name:cbc-aes-aesni-mb
> 
> expected:
> alg name:cbc(aes), driver name:mcryptd-cbc-aes-aesni-mb

This all looks right.  So I'm not sure why you're getting the crash.
We're relying on the INTERNAL flag to ensure the internal algorithm
is not picked up except when we strictly ask for it.

In fact I see something fishy in your testmgr code (the last patch
in the series I think).  It's setting the INTERNAL bit when
allocating tfms, that does not look right.

The only one that should be setting this is mcryptd.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2017-06-23  8:32 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-20 20:50 [Patch V5 0/7] crypto: AES CBC multibuffer implementation Megha Dey
2017-04-20 20:50 ` [Patch V5 1/7] crypto: Multi-buffer encryption infrastructure support Megha Dey
2017-04-21  0:19   ` kbuild test robot
2017-04-21  3:02   ` kbuild test robot
2017-04-24  9:00   ` Herbert Xu
2017-06-08 19:52     ` Megha Dey
2017-06-23  8:32       ` Herbert Xu
2017-04-20 20:50 ` [Patch V5 2/7] crypto: AES CBC multi-buffer data structures Megha Dey
2017-04-20 20:50 ` [Patch V5 3/7] crypto: AES CBC multi-buffer scheduler Megha Dey
2017-04-20 20:50 ` [Patch V5 4/7] crypto: AES CBC by8 encryption Megha Dey
2017-04-20 20:50 ` [Patch V5 5/7] crypto: AES CBC multi-buffer glue code Megha Dey
2017-04-20 20:50 ` [Patch V5 6/7] crypto: AES vectors for AES CBC multibuffer testing Megha Dey
2017-04-20 20:50 ` [Patch V5 7/7] crypto: AES CBC multi-buffer tcrypt Megha Dey
2017-04-21  0:54   ` kbuild test robot
2017-04-21  1:56   ` kbuild test robot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).