All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 0/8] crypto: aes - retire table based generic AES
@ 2017-07-18 12:06 Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 1/8] drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies Ard Biesheuvel
                   ` (8 more replies)
  0 siblings, 9 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

The generic AES driver uses 16 lookup tables of 1 KB each, and has
encryption and decryption routines that are fully unrolled. Given how
the dependencies between this code and other drivers are declared in
Kconfig files, this code is always pulled into the core kernel, even
if it is usually superseded at runtime by accelerated drivers that
exist for many architectures.

This leaves us with 25 KB of dead code in the kernel, which is negligible
in typical environments, but which is actually a big deal for the IoT
domain, where every kilobyte counts.

Also, the scalar, table based AES routines that exist for ARM, arm64, i586
and x86_64 share the lookup tables with AES generic, and may be invoked
occasionally when the time-invariant AES-NI or other special instruction
drivers are called in interrupt context, at which time the SIMD register
file cannot be used. Pulling 16 KB of code and 9 KB of instructions into
the L1s (and evicting what was already there) when a softirq happens to
be handled in the context of an interrupt taken from kernel mode (which
means no SIMD on x86) is also something that we may like to avoid, by
falling back to a much smaller and moderately less performant driver.
(Note that arm64 will be updated shortly to supply fallbacks for all
SIMD based AES implementations, which will be based on the core routines)

For the reasons above, this series refactors the way the various AES
implementations are wired up, to allow the generic version in
crypto/aes_generic.c to be omitted from the build entirely.

Patch #1 removes some bogus 'select CRYPTO_AES' statement.

Patch #2 factors out aes-generic's lookup tables, which are shared with
arch-specific implementations in arch/x86, arch/arm and arch/arm64.

Patch #3 replaces the table based aes-generic.o with a new aes.o based on
the fixed time cipher, and uses it to fulfil dependencies on CRYPTO_AES.

Patch #4 switches the fallback in the AES-NI code to the new, generic encrypt
and decrypt routines so it no longer depends on the x86 scalar code or
[transitively] on AES-generic.

Patch #5 tweaks the ARM table based code to only use 2 KB + 256 bytes worth
of lookup tables instead of 4 KB.

Patch #6 does the same for arm64

Patch #7 removes the local copy of the AES sboxes from the arm64 NEON driver,
and switches to the ones exposed by the new AES core module instead.

Patch #8 updates the Kconfig help text to be more descriptive of what they
actually control, rather than duplicating AES's wikipedia entry a number of
times.

v4: - remove aes-generic altogether instead of allow a preference to be set
    - factor out shared lookup tables (#2)
    - reduce dependency of ARM's table based code on shared lookup tables
      (#5, #6)

v3: - fix big-endian issue in refactored fixed-time AES driver
    - improve Kconfig help texts
    - add patch #4

v2: - repurpose CRYPTO_AES and avoid HAVE_AES/NEED_AES Kconfig symbols
    - don't factor out tables from AES generic to be reused by per arch drivers,
      since the space saving is moderate (the generic code only), and the
      drivers weren't made to be small anyway

Ard Biesheuvel (8):
  drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies
  crypto - aes: use dedicated lookup tables for table based asm routines
  crypto: aes - retire table based generic AES in favor of fixed time
    driver
  crypto: x86/aes-ni - switch to generic fallback
  crypto: arm/aes - avoid expanded lookup tables in the final round
  crypto: arm64/aes - avoid expanded lookup tables in the final round
  crypto: arm64/aes-neon - reuse Sboxes from AES core module
  crypto: aes - add meaningful help text to the various AES drivers

 arch/arm/crypto/Kconfig             |   16 +-
 arch/arm/crypto/aes-cipher-core.S   |   54 +-
 arch/arm64/crypto/Kconfig           |   30 +-
 arch/arm64/crypto/aes-cipher-core.S |  159 ++-
 arch/arm64/crypto/aes-neon.S        |   74 +-
 arch/x86/crypto/aes-i586-asm_32.S   |   13 +-
 arch/x86/crypto/aes-x86_64-asm_64.S |   12 +-
 arch/x86/crypto/aesni-intel_glue.c  |    4 +-
 crypto/Kconfig                      |  138 +-
 crypto/Makefile                     |    3 +-
 crypto/{aes_ti.c => aes.c}          |  169 ++-
 crypto/aes_generic.c                | 1478 --------------------
 drivers/crypto/Kconfig              |    5 -
 drivers/crypto/chelsio/chcr_algo.c  |    4 +-
 include/crypto/aes-tables.S         | 1104 +++++++++++++++
 include/crypto/aes.h                |   11 +-
 16 files changed, 1464 insertions(+), 1810 deletions(-)
 rename crypto/{aes_ti.c => aes.c} (76%)
 delete mode 100644 crypto/aes_generic.c
 create mode 100644 include/crypto/aes-tables.S

-- 
2.9.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v4 1/8] drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 2/8] crypto - aes: use dedicated lookup tables for table based asm routines Ard Biesheuvel
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

In preparation of fine tuning the dependency relations between the
accelerated AES drivers and the core support code, let's remove the
dependency declarations that are false. None of these modules have
link time dependencies on the generic AES code, nor do they declare
any AES algos with CRYPTO_ALG_NEED_FALLBACK, so they can function
perfectly fine without crypto/aes_generic.o loaded.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 drivers/crypto/Kconfig | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 5b5393f1b87a..46a48ea99fb9 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -432,7 +432,6 @@ config CRYPTO_DEV_S5P
 	tristate "Support for Samsung S5PV210/Exynos crypto accelerator"
 	depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
 	depends on HAS_IOMEM && HAS_DMA
-	select CRYPTO_AES
 	select CRYPTO_BLKCIPHER
 	help
 	  This option allows you to have support for S5P crypto acceleration.
@@ -486,7 +485,6 @@ config CRYPTO_DEV_ATMEL_AES
 	tristate "Support for Atmel AES hw accelerator"
 	depends on HAS_DMA
 	depends on ARCH_AT91 || COMPILE_TEST
-	select CRYPTO_AES
 	select CRYPTO_AEAD
 	select CRYPTO_BLKCIPHER
 	help
@@ -618,7 +616,6 @@ config CRYPTO_DEV_SUN4I_SS
 	depends on ARCH_SUNXI && !64BIT
 	select CRYPTO_MD5
 	select CRYPTO_SHA1
-	select CRYPTO_AES
 	select CRYPTO_DES
 	select CRYPTO_BLKCIPHER
 	help
@@ -641,7 +638,6 @@ config CRYPTO_DEV_SUN4I_SS_PRNG
 config CRYPTO_DEV_ROCKCHIP
 	tristate "Rockchip's Cryptographic Engine driver"
 	depends on OF && ARCH_ROCKCHIP
-	select CRYPTO_AES
 	select CRYPTO_DES
 	select CRYPTO_MD5
 	select CRYPTO_SHA1
@@ -657,7 +653,6 @@ config CRYPTO_DEV_MEDIATEK
 	tristate "MediaTek's EIP97 Cryptographic Engine driver"
 	depends on HAS_DMA
 	depends on (ARM && ARCH_MEDIATEK) || COMPILE_TEST
-	select CRYPTO_AES
 	select CRYPTO_AEAD
 	select CRYPTO_BLKCIPHER
 	select CRYPTO_CTR
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 2/8] crypto - aes: use dedicated lookup tables for table based asm routines
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 1/8] drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 3/8] crypto: aes - retire table based generic AES in favor of fixed time driver Ard Biesheuvel
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

Instead of linking against the table based AES generic C code to reuse
the lookup tables, add an assembler file that defines a couple of macros
that instantiate the tables in-place. This allows us to replace AES in
a subsequent patch.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm/crypto/aes-cipher-core.S   |    7 +
 arch/arm64/crypto/aes-cipher-core.S |    8 +-
 arch/x86/crypto/aes-i586-asm_32.S   |   13 +-
 arch/x86/crypto/aes-x86_64-asm_64.S |   12 +-
 include/crypto/aes-tables.S         | 1104 ++++++++++++++++++++
 include/crypto/aes.h                |    5 -
 6 files changed, 1132 insertions(+), 17 deletions(-)

diff --git a/arch/arm/crypto/aes-cipher-core.S b/arch/arm/crypto/aes-cipher-core.S
index c817a86c4ca8..a727692cd9c1 100644
--- a/arch/arm/crypto/aes-cipher-core.S
+++ b/arch/arm/crypto/aes-cipher-core.S
@@ -9,6 +9,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <crypto/aes-tables.S>
 #include <linux/linkage.h>
 
 	.text
@@ -170,6 +171,12 @@
 	.ltorg
 	.endm
 
+	.align			6
+	aes_table_reduced	crypto_ft_tab
+	aes_table_reduced	crypto_fl_tab
+	aes_table_reduced	crypto_it_tab
+	aes_table_reduced	crypto_il_tab
+
 ENTRY(__aes_arm_encrypt)
 	do_crypt	fround, crypto_ft_tab, crypto_fl_tab
 ENDPROC(__aes_arm_encrypt)
diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S
index f2f9cc519309..bbe5dd96135c 100644
--- a/arch/arm64/crypto/aes-cipher-core.S
+++ b/arch/arm64/crypto/aes-cipher-core.S
@@ -8,6 +8,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <crypto/aes-tables.S>
 #include <linux/linkage.h>
 #include <asm/assembler.h>
 
@@ -99,7 +100,12 @@ CPU_BE(	rev		w8, w8		)
 	ret
 	.endm
 
-	.align		5
+	.align			7
+	aes_table_reduced	crypto_ft_tab
+	aes_table_reduced	crypto_fl_tab
+	aes_table_reduced	crypto_it_tab
+	aes_table_reduced	crypto_il_tab
+
 ENTRY(__aes_arm64_encrypt)
 	do_crypt	fround, crypto_ft_tab, crypto_fl_tab
 ENDPROC(__aes_arm64_encrypt)
diff --git a/arch/x86/crypto/aes-i586-asm_32.S b/arch/x86/crypto/aes-i586-asm_32.S
index 2849dbc59e11..d68c57ca2ace 100644
--- a/arch/x86/crypto/aes-i586-asm_32.S
+++ b/arch/x86/crypto/aes-i586-asm_32.S
@@ -38,6 +38,13 @@
 
 #include <linux/linkage.h>
 #include <asm/asm-offsets.h>
+#include <crypto/aes-tables.S>
+
+.align 4
+aes_table_prerotated crypto_ft_tab
+aes_table_prerotated crypto_fl_tab
+aes_table_prerotated crypto_it_tab
+aes_table_prerotated crypto_il_tab
 
 #define tlen 1024   // length of each of 4 'xor' arrays (256 32-bit words)
 
@@ -220,9 +227,6 @@
 // AES (Rijndael) Encryption Subroutine
 /* void aes_enc_blk(struct crypto_aes_ctx *ctx, u8 *out_blk, const u8 *in_blk) */
 
-.extern  crypto_ft_tab
-.extern  crypto_fl_tab
-
 ENTRY(aes_enc_blk)
 	push    %ebp
 	mov     ctx(%esp),%ebp
@@ -292,9 +296,6 @@ ENDPROC(aes_enc_blk)
 // AES (Rijndael) Decryption Subroutine
 /* void aes_dec_blk(struct crypto_aes_ctx *ctx, u8 *out_blk, const u8 *in_blk) */
 
-.extern  crypto_it_tab
-.extern  crypto_il_tab
-
 ENTRY(aes_dec_blk)
 	push    %ebp
 	mov     ctx(%esp),%ebp
diff --git a/arch/x86/crypto/aes-x86_64-asm_64.S b/arch/x86/crypto/aes-x86_64-asm_64.S
index 8739cf7795de..7b5a9ef3e51d 100644
--- a/arch/x86/crypto/aes-x86_64-asm_64.S
+++ b/arch/x86/crypto/aes-x86_64-asm_64.S
@@ -8,15 +8,17 @@
  * including this sentence is retained in full.
  */
 
-.extern crypto_ft_tab
-.extern crypto_it_tab
-.extern crypto_fl_tab
-.extern crypto_il_tab
-
 .text
 
 #include <linux/linkage.h>
 #include <asm/asm-offsets.h>
+#include <crypto/aes-tables.S>
+
+.align 4
+aes_table_prerotated crypto_ft_tab
+aes_table_prerotated crypto_fl_tab
+aes_table_prerotated crypto_it_tab
+aes_table_prerotated crypto_il_tab
 
 #define R1	%rax
 #define R1E	%eax
diff --git a/include/crypto/aes-tables.S b/include/crypto/aes-tables.S
new file mode 100644
index 000000000000..9625c38a76fb
--- /dev/null
+++ b/include/crypto/aes-tables.S
@@ -0,0 +1,1104 @@
+/*
+ * ---------------------------------------------------------------------------
+ * Copyright (c) 2002, Dr Brian Gladman <brg@gladman.me.uk>, Worcester, UK.
+ * All rights reserved.
+ *
+ * LICENSE TERMS
+ *
+ * The free distribution and use of this software in both source and binary
+ * form is allowed (with or without changes) provided that:
+ *
+ *   1. distributions of this source code include the above copyright
+ *      notice, this list of conditions and the following disclaimer;
+ *
+ *   2. distributions in binary form include the above copyright
+ *      notice, this list of conditions and the following disclaimer
+ *      in the documentation and/or other associated materials;
+ *
+ *   3. the copyright holder's name is not used to endorse products
+ *      built using this software without specific written permission.
+ *
+ * ALTERNATIVELY, provided that this notice is retained in full, this product
+ * may be distributed under the terms of the GNU General Public License (GPL),
+ * in which case the provisions of the GPL apply INSTEAD OF those given above.
+ *
+ * DISCLAIMER
+ *
+ * This software is provided 'as is' with no explicit or implied warranties
+ * in respect of its properties, including, but not limited to, correctness
+ * and/or fitness for purpose.
+ * ---------------------------------------------------------------------------
+ */
+
+#include <linux/linkage.h>
+
+	.macro		aes_table_reduced, name
+	__aes_tab	\name, 0
+	.endm
+
+	.macro		aes_table_prerotated, name
+	__aes_tab	\name, 1
+	.endm
+
+	.macro		__aes_tab, name, full
+\name\():
+	__\name		\full
+	.size		\name, . - \name
+	.endm
+
+	.macro		__crypto_ft_tab, full=1
+	.long		0xa56363c6, 0x847c7cf8, 0x997777ee, 0x8d7b7bf6
+	.long		0x0df2f2ff, 0xbd6b6bd6, 0xb16f6fde, 0x54c5c591
+	.long		0x50303060, 0x03010102, 0xa96767ce, 0x7d2b2b56
+	.long		0x19fefee7, 0x62d7d7b5, 0xe6abab4d, 0x9a7676ec
+	.long		0x45caca8f, 0x9d82821f, 0x40c9c989, 0x877d7dfa
+	.long		0x15fafaef, 0xeb5959b2, 0xc947478e, 0x0bf0f0fb
+	.long		0xecadad41, 0x67d4d4b3, 0xfda2a25f, 0xeaafaf45
+	.long		0xbf9c9c23, 0xf7a4a453, 0x967272e4, 0x5bc0c09b
+	.long		0xc2b7b775, 0x1cfdfde1, 0xae93933d, 0x6a26264c
+	.long		0x5a36366c, 0x413f3f7e, 0x02f7f7f5, 0x4fcccc83
+	.long		0x5c343468, 0xf4a5a551, 0x34e5e5d1, 0x08f1f1f9
+	.long		0x937171e2, 0x73d8d8ab, 0x53313162, 0x3f15152a
+	.long		0x0c040408, 0x52c7c795, 0x65232346, 0x5ec3c39d
+	.long		0x28181830, 0xa1969637, 0x0f05050a, 0xb59a9a2f
+	.long		0x0907070e, 0x36121224, 0x9b80801b, 0x3de2e2df
+	.long		0x26ebebcd, 0x6927274e, 0xcdb2b27f, 0x9f7575ea
+	.long		0x1b090912, 0x9e83831d, 0x742c2c58, 0x2e1a1a34
+	.long		0x2d1b1b36, 0xb26e6edc, 0xee5a5ab4, 0xfba0a05b
+	.long		0xf65252a4, 0x4d3b3b76, 0x61d6d6b7, 0xceb3b37d
+	.long		0x7b292952, 0x3ee3e3dd, 0x712f2f5e, 0x97848413
+	.long		0xf55353a6, 0x68d1d1b9, 0x00000000, 0x2cededc1
+	.long		0x60202040, 0x1ffcfce3, 0xc8b1b179, 0xed5b5bb6
+	.long		0xbe6a6ad4, 0x46cbcb8d, 0xd9bebe67, 0x4b393972
+	.long		0xde4a4a94, 0xd44c4c98, 0xe85858b0, 0x4acfcf85
+	.long		0x6bd0d0bb, 0x2aefefc5, 0xe5aaaa4f, 0x16fbfbed
+	.long		0xc5434386, 0xd74d4d9a, 0x55333366, 0x94858511
+	.long		0xcf45458a, 0x10f9f9e9, 0x06020204, 0x817f7ffe
+	.long		0xf05050a0, 0x443c3c78, 0xba9f9f25, 0xe3a8a84b
+	.long		0xf35151a2, 0xfea3a35d, 0xc0404080, 0x8a8f8f05
+	.long		0xad92923f, 0xbc9d9d21, 0x48383870, 0x04f5f5f1
+	.long		0xdfbcbc63, 0xc1b6b677, 0x75dadaaf, 0x63212142
+	.long		0x30101020, 0x1affffe5, 0x0ef3f3fd, 0x6dd2d2bf
+	.long		0x4ccdcd81, 0x140c0c18, 0x35131326, 0x2fececc3
+	.long		0xe15f5fbe, 0xa2979735, 0xcc444488, 0x3917172e
+	.long		0x57c4c493, 0xf2a7a755, 0x827e7efc, 0x473d3d7a
+	.long		0xac6464c8, 0xe75d5dba, 0x2b191932, 0x957373e6
+	.long		0xa06060c0, 0x98818119, 0xd14f4f9e, 0x7fdcdca3
+	.long		0x66222244, 0x7e2a2a54, 0xab90903b, 0x8388880b
+	.long		0xca46468c, 0x29eeeec7, 0xd3b8b86b, 0x3c141428
+	.long		0x79dedea7, 0xe25e5ebc, 0x1d0b0b16, 0x76dbdbad
+	.long		0x3be0e0db, 0x56323264, 0x4e3a3a74, 0x1e0a0a14
+	.long		0xdb494992, 0x0a06060c, 0x6c242448, 0xe45c5cb8
+	.long		0x5dc2c29f, 0x6ed3d3bd, 0xefacac43, 0xa66262c4
+	.long		0xa8919139, 0xa4959531, 0x37e4e4d3, 0x8b7979f2
+	.long		0x32e7e7d5, 0x43c8c88b, 0x5937376e, 0xb76d6dda
+	.long		0x8c8d8d01, 0x64d5d5b1, 0xd24e4e9c, 0xe0a9a949
+	.long		0xb46c6cd8, 0xfa5656ac, 0x07f4f4f3, 0x25eaeacf
+	.long		0xaf6565ca, 0x8e7a7af4, 0xe9aeae47, 0x18080810
+	.long		0xd5baba6f, 0x887878f0, 0x6f25254a, 0x722e2e5c
+	.long		0x241c1c38, 0xf1a6a657, 0xc7b4b473, 0x51c6c697
+	.long		0x23e8e8cb, 0x7cdddda1, 0x9c7474e8, 0x211f1f3e
+	.long		0xdd4b4b96, 0xdcbdbd61, 0x868b8b0d, 0x858a8a0f
+	.long		0x907070e0, 0x423e3e7c, 0xc4b5b571, 0xaa6666cc
+	.long		0xd8484890, 0x05030306, 0x01f6f6f7, 0x120e0e1c
+	.long		0xa36161c2, 0x5f35356a, 0xf95757ae, 0xd0b9b969
+	.long		0x91868617, 0x58c1c199, 0x271d1d3a, 0xb99e9e27
+	.long		0x38e1e1d9, 0x13f8f8eb, 0xb398982b, 0x33111122
+	.long		0xbb6969d2, 0x70d9d9a9, 0x898e8e07, 0xa7949433
+	.long		0xb69b9b2d, 0x221e1e3c, 0x92878715, 0x20e9e9c9
+	.long		0x49cece87, 0xff5555aa, 0x78282850, 0x7adfdfa5
+	.long		0x8f8c8c03, 0xf8a1a159, 0x80898909, 0x170d0d1a
+	.long		0xdabfbf65, 0x31e6e6d7, 0xc6424284, 0xb86868d0
+	.long		0xc3414182, 0xb0999929, 0x772d2d5a, 0x110f0f1e
+	.long		0xcbb0b07b, 0xfc5454a8, 0xd6bbbb6d, 0x3a16162c
+
+	.if \full == 1
+	.long		0x6363c6a5, 0x7c7cf884, 0x7777ee99, 0x7b7bf68d
+	.long		0xf2f2ff0d, 0x6b6bd6bd, 0x6f6fdeb1, 0xc5c59154
+	.long		0x30306050, 0x01010203, 0x6767cea9, 0x2b2b567d
+	.long		0xfefee719, 0xd7d7b562, 0xabab4de6, 0x7676ec9a
+	.long		0xcaca8f45, 0x82821f9d, 0xc9c98940, 0x7d7dfa87
+	.long		0xfafaef15, 0x5959b2eb, 0x47478ec9, 0xf0f0fb0b
+	.long		0xadad41ec, 0xd4d4b367, 0xa2a25ffd, 0xafaf45ea
+	.long		0x9c9c23bf, 0xa4a453f7, 0x7272e496, 0xc0c09b5b
+	.long		0xb7b775c2, 0xfdfde11c, 0x93933dae, 0x26264c6a
+	.long		0x36366c5a, 0x3f3f7e41, 0xf7f7f502, 0xcccc834f
+	.long		0x3434685c, 0xa5a551f4, 0xe5e5d134, 0xf1f1f908
+	.long		0x7171e293, 0xd8d8ab73, 0x31316253, 0x15152a3f
+	.long		0x0404080c, 0xc7c79552, 0x23234665, 0xc3c39d5e
+	.long		0x18183028, 0x969637a1, 0x05050a0f, 0x9a9a2fb5
+	.long		0x07070e09, 0x12122436, 0x80801b9b, 0xe2e2df3d
+	.long		0xebebcd26, 0x27274e69, 0xb2b27fcd, 0x7575ea9f
+	.long		0x0909121b, 0x83831d9e, 0x2c2c5874, 0x1a1a342e
+	.long		0x1b1b362d, 0x6e6edcb2, 0x5a5ab4ee, 0xa0a05bfb
+	.long		0x5252a4f6, 0x3b3b764d, 0xd6d6b761, 0xb3b37dce
+	.long		0x2929527b, 0xe3e3dd3e, 0x2f2f5e71, 0x84841397
+	.long		0x5353a6f5, 0xd1d1b968, 0x00000000, 0xededc12c
+	.long		0x20204060, 0xfcfce31f, 0xb1b179c8, 0x5b5bb6ed
+	.long		0x6a6ad4be, 0xcbcb8d46, 0xbebe67d9, 0x3939724b
+	.long		0x4a4a94de, 0x4c4c98d4, 0x5858b0e8, 0xcfcf854a
+	.long		0xd0d0bb6b, 0xefefc52a, 0xaaaa4fe5, 0xfbfbed16
+	.long		0x434386c5, 0x4d4d9ad7, 0x33336655, 0x85851194
+	.long		0x45458acf, 0xf9f9e910, 0x02020406, 0x7f7ffe81
+	.long		0x5050a0f0, 0x3c3c7844, 0x9f9f25ba, 0xa8a84be3
+	.long		0x5151a2f3, 0xa3a35dfe, 0x404080c0, 0x8f8f058a
+	.long		0x92923fad, 0x9d9d21bc, 0x38387048, 0xf5f5f104
+	.long		0xbcbc63df, 0xb6b677c1, 0xdadaaf75, 0x21214263
+	.long		0x10102030, 0xffffe51a, 0xf3f3fd0e, 0xd2d2bf6d
+	.long		0xcdcd814c, 0x0c0c1814, 0x13132635, 0xececc32f
+	.long		0x5f5fbee1, 0x979735a2, 0x444488cc, 0x17172e39
+	.long		0xc4c49357, 0xa7a755f2, 0x7e7efc82, 0x3d3d7a47
+	.long		0x6464c8ac, 0x5d5dbae7, 0x1919322b, 0x7373e695
+	.long		0x6060c0a0, 0x81811998, 0x4f4f9ed1, 0xdcdca37f
+	.long		0x22224466, 0x2a2a547e, 0x90903bab, 0x88880b83
+	.long		0x46468cca, 0xeeeec729, 0xb8b86bd3, 0x1414283c
+	.long		0xdedea779, 0x5e5ebce2, 0x0b0b161d, 0xdbdbad76
+	.long		0xe0e0db3b, 0x32326456, 0x3a3a744e, 0x0a0a141e
+	.long		0x494992db, 0x06060c0a, 0x2424486c, 0x5c5cb8e4
+	.long		0xc2c29f5d, 0xd3d3bd6e, 0xacac43ef, 0x6262c4a6
+	.long		0x919139a8, 0x959531a4, 0xe4e4d337, 0x7979f28b
+	.long		0xe7e7d532, 0xc8c88b43, 0x37376e59, 0x6d6ddab7
+	.long		0x8d8d018c, 0xd5d5b164, 0x4e4e9cd2, 0xa9a949e0
+	.long		0x6c6cd8b4, 0x5656acfa, 0xf4f4f307, 0xeaeacf25
+	.long		0x6565caaf, 0x7a7af48e, 0xaeae47e9, 0x08081018
+	.long		0xbaba6fd5, 0x7878f088, 0x25254a6f, 0x2e2e5c72
+	.long		0x1c1c3824, 0xa6a657f1, 0xb4b473c7, 0xc6c69751
+	.long		0xe8e8cb23, 0xdddda17c, 0x7474e89c, 0x1f1f3e21
+	.long		0x4b4b96dd, 0xbdbd61dc, 0x8b8b0d86, 0x8a8a0f85
+	.long		0x7070e090, 0x3e3e7c42, 0xb5b571c4, 0x6666ccaa
+	.long		0x484890d8, 0x03030605, 0xf6f6f701, 0x0e0e1c12
+	.long		0x6161c2a3, 0x35356a5f, 0x5757aef9, 0xb9b969d0
+	.long		0x86861791, 0xc1c19958, 0x1d1d3a27, 0x9e9e27b9
+	.long		0xe1e1d938, 0xf8f8eb13, 0x98982bb3, 0x11112233
+	.long		0x6969d2bb, 0xd9d9a970, 0x8e8e0789, 0x949433a7
+	.long		0x9b9b2db6, 0x1e1e3c22, 0x87871592, 0xe9e9c920
+	.long		0xcece8749, 0x5555aaff, 0x28285078, 0xdfdfa57a
+	.long		0x8c8c038f, 0xa1a159f8, 0x89890980, 0x0d0d1a17
+	.long		0xbfbf65da, 0xe6e6d731, 0x424284c6, 0x6868d0b8
+	.long		0x414182c3, 0x999929b0, 0x2d2d5a77, 0x0f0f1e11
+	.long		0xb0b07bcb, 0x5454a8fc, 0xbbbb6dd6, 0x16162c3a
+
+	.long		0x63c6a563, 0x7cf8847c, 0x77ee9977, 0x7bf68d7b
+	.long		0xf2ff0df2, 0x6bd6bd6b, 0x6fdeb16f, 0xc59154c5
+	.long		0x30605030, 0x01020301, 0x67cea967, 0x2b567d2b
+	.long		0xfee719fe, 0xd7b562d7, 0xab4de6ab, 0x76ec9a76
+	.long		0xca8f45ca, 0x821f9d82, 0xc98940c9, 0x7dfa877d
+	.long		0xfaef15fa, 0x59b2eb59, 0x478ec947, 0xf0fb0bf0
+	.long		0xad41ecad, 0xd4b367d4, 0xa25ffda2, 0xaf45eaaf
+	.long		0x9c23bf9c, 0xa453f7a4, 0x72e49672, 0xc09b5bc0
+	.long		0xb775c2b7, 0xfde11cfd, 0x933dae93, 0x264c6a26
+	.long		0x366c5a36, 0x3f7e413f, 0xf7f502f7, 0xcc834fcc
+	.long		0x34685c34, 0xa551f4a5, 0xe5d134e5, 0xf1f908f1
+	.long		0x71e29371, 0xd8ab73d8, 0x31625331, 0x152a3f15
+	.long		0x04080c04, 0xc79552c7, 0x23466523, 0xc39d5ec3
+	.long		0x18302818, 0x9637a196, 0x050a0f05, 0x9a2fb59a
+	.long		0x070e0907, 0x12243612, 0x801b9b80, 0xe2df3de2
+	.long		0xebcd26eb, 0x274e6927, 0xb27fcdb2, 0x75ea9f75
+	.long		0x09121b09, 0x831d9e83, 0x2c58742c, 0x1a342e1a
+	.long		0x1b362d1b, 0x6edcb26e, 0x5ab4ee5a, 0xa05bfba0
+	.long		0x52a4f652, 0x3b764d3b, 0xd6b761d6, 0xb37dceb3
+	.long		0x29527b29, 0xe3dd3ee3, 0x2f5e712f, 0x84139784
+	.long		0x53a6f553, 0xd1b968d1, 0x00000000, 0xedc12ced
+	.long		0x20406020, 0xfce31ffc, 0xb179c8b1, 0x5bb6ed5b
+	.long		0x6ad4be6a, 0xcb8d46cb, 0xbe67d9be, 0x39724b39
+	.long		0x4a94de4a, 0x4c98d44c, 0x58b0e858, 0xcf854acf
+	.long		0xd0bb6bd0, 0xefc52aef, 0xaa4fe5aa, 0xfbed16fb
+	.long		0x4386c543, 0x4d9ad74d, 0x33665533, 0x85119485
+	.long		0x458acf45, 0xf9e910f9, 0x02040602, 0x7ffe817f
+	.long		0x50a0f050, 0x3c78443c, 0x9f25ba9f, 0xa84be3a8
+	.long		0x51a2f351, 0xa35dfea3, 0x4080c040, 0x8f058a8f
+	.long		0x923fad92, 0x9d21bc9d, 0x38704838, 0xf5f104f5
+	.long		0xbc63dfbc, 0xb677c1b6, 0xdaaf75da, 0x21426321
+	.long		0x10203010, 0xffe51aff, 0xf3fd0ef3, 0xd2bf6dd2
+	.long		0xcd814ccd, 0x0c18140c, 0x13263513, 0xecc32fec
+	.long		0x5fbee15f, 0x9735a297, 0x4488cc44, 0x172e3917
+	.long		0xc49357c4, 0xa755f2a7, 0x7efc827e, 0x3d7a473d
+	.long		0x64c8ac64, 0x5dbae75d, 0x19322b19, 0x73e69573
+	.long		0x60c0a060, 0x81199881, 0x4f9ed14f, 0xdca37fdc
+	.long		0x22446622, 0x2a547e2a, 0x903bab90, 0x880b8388
+	.long		0x468cca46, 0xeec729ee, 0xb86bd3b8, 0x14283c14
+	.long		0xdea779de, 0x5ebce25e, 0x0b161d0b, 0xdbad76db
+	.long		0xe0db3be0, 0x32645632, 0x3a744e3a, 0x0a141e0a
+	.long		0x4992db49, 0x060c0a06, 0x24486c24, 0x5cb8e45c
+	.long		0xc29f5dc2, 0xd3bd6ed3, 0xac43efac, 0x62c4a662
+	.long		0x9139a891, 0x9531a495, 0xe4d337e4, 0x79f28b79
+	.long		0xe7d532e7, 0xc88b43c8, 0x376e5937, 0x6ddab76d
+	.long		0x8d018c8d, 0xd5b164d5, 0x4e9cd24e, 0xa949e0a9
+	.long		0x6cd8b46c, 0x56acfa56, 0xf4f307f4, 0xeacf25ea
+	.long		0x65caaf65, 0x7af48e7a, 0xae47e9ae, 0x08101808
+	.long		0xba6fd5ba, 0x78f08878, 0x254a6f25, 0x2e5c722e
+	.long		0x1c38241c, 0xa657f1a6, 0xb473c7b4, 0xc69751c6
+	.long		0xe8cb23e8, 0xdda17cdd, 0x74e89c74, 0x1f3e211f
+	.long		0x4b96dd4b, 0xbd61dcbd, 0x8b0d868b, 0x8a0f858a
+	.long		0x70e09070, 0x3e7c423e, 0xb571c4b5, 0x66ccaa66
+	.long		0x4890d848, 0x03060503, 0xf6f701f6, 0x0e1c120e
+	.long		0x61c2a361, 0x356a5f35, 0x57aef957, 0xb969d0b9
+	.long		0x86179186, 0xc19958c1, 0x1d3a271d, 0x9e27b99e
+	.long		0xe1d938e1, 0xf8eb13f8, 0x982bb398, 0x11223311
+	.long		0x69d2bb69, 0xd9a970d9, 0x8e07898e, 0x9433a794
+	.long		0x9b2db69b, 0x1e3c221e, 0x87159287, 0xe9c920e9
+	.long		0xce8749ce, 0x55aaff55, 0x28507828, 0xdfa57adf
+	.long		0x8c038f8c, 0xa159f8a1, 0x89098089, 0x0d1a170d
+	.long		0xbf65dabf, 0xe6d731e6, 0x4284c642, 0x68d0b868
+	.long		0x4182c341, 0x9929b099, 0x2d5a772d, 0x0f1e110f
+	.long		0xb07bcbb0, 0x54a8fc54, 0xbb6dd6bb, 0x162c3a16
+
+	.long		0xc6a56363, 0xf8847c7c, 0xee997777, 0xf68d7b7b
+	.long		0xff0df2f2, 0xd6bd6b6b, 0xdeb16f6f, 0x9154c5c5
+	.long		0x60503030, 0x02030101, 0xcea96767, 0x567d2b2b
+	.long		0xe719fefe, 0xb562d7d7, 0x4de6abab, 0xec9a7676
+	.long		0x8f45caca, 0x1f9d8282, 0x8940c9c9, 0xfa877d7d
+	.long		0xef15fafa, 0xb2eb5959, 0x8ec94747, 0xfb0bf0f0
+	.long		0x41ecadad, 0xb367d4d4, 0x5ffda2a2, 0x45eaafaf
+	.long		0x23bf9c9c, 0x53f7a4a4, 0xe4967272, 0x9b5bc0c0
+	.long		0x75c2b7b7, 0xe11cfdfd, 0x3dae9393, 0x4c6a2626
+	.long		0x6c5a3636, 0x7e413f3f, 0xf502f7f7, 0x834fcccc
+	.long		0x685c3434, 0x51f4a5a5, 0xd134e5e5, 0xf908f1f1
+	.long		0xe2937171, 0xab73d8d8, 0x62533131, 0x2a3f1515
+	.long		0x080c0404, 0x9552c7c7, 0x46652323, 0x9d5ec3c3
+	.long		0x30281818, 0x37a19696, 0x0a0f0505, 0x2fb59a9a
+	.long		0x0e090707, 0x24361212, 0x1b9b8080, 0xdf3de2e2
+	.long		0xcd26ebeb, 0x4e692727, 0x7fcdb2b2, 0xea9f7575
+	.long		0x121b0909, 0x1d9e8383, 0x58742c2c, 0x342e1a1a
+	.long		0x362d1b1b, 0xdcb26e6e, 0xb4ee5a5a, 0x5bfba0a0
+	.long		0xa4f65252, 0x764d3b3b, 0xb761d6d6, 0x7dceb3b3
+	.long		0x527b2929, 0xdd3ee3e3, 0x5e712f2f, 0x13978484
+	.long		0xa6f55353, 0xb968d1d1, 0x00000000, 0xc12ceded
+	.long		0x40602020, 0xe31ffcfc, 0x79c8b1b1, 0xb6ed5b5b
+	.long		0xd4be6a6a, 0x8d46cbcb, 0x67d9bebe, 0x724b3939
+	.long		0x94de4a4a, 0x98d44c4c, 0xb0e85858, 0x854acfcf
+	.long		0xbb6bd0d0, 0xc52aefef, 0x4fe5aaaa, 0xed16fbfb
+	.long		0x86c54343, 0x9ad74d4d, 0x66553333, 0x11948585
+	.long		0x8acf4545, 0xe910f9f9, 0x04060202, 0xfe817f7f
+	.long		0xa0f05050, 0x78443c3c, 0x25ba9f9f, 0x4be3a8a8
+	.long		0xa2f35151, 0x5dfea3a3, 0x80c04040, 0x058a8f8f
+	.long		0x3fad9292, 0x21bc9d9d, 0x70483838, 0xf104f5f5
+	.long		0x63dfbcbc, 0x77c1b6b6, 0xaf75dada, 0x42632121
+	.long		0x20301010, 0xe51affff, 0xfd0ef3f3, 0xbf6dd2d2
+	.long		0x814ccdcd, 0x18140c0c, 0x26351313, 0xc32fecec
+	.long		0xbee15f5f, 0x35a29797, 0x88cc4444, 0x2e391717
+	.long		0x9357c4c4, 0x55f2a7a7, 0xfc827e7e, 0x7a473d3d
+	.long		0xc8ac6464, 0xbae75d5d, 0x322b1919, 0xe6957373
+	.long		0xc0a06060, 0x19988181, 0x9ed14f4f, 0xa37fdcdc
+	.long		0x44662222, 0x547e2a2a, 0x3bab9090, 0x0b838888
+	.long		0x8cca4646, 0xc729eeee, 0x6bd3b8b8, 0x283c1414
+	.long		0xa779dede, 0xbce25e5e, 0x161d0b0b, 0xad76dbdb
+	.long		0xdb3be0e0, 0x64563232, 0x744e3a3a, 0x141e0a0a
+	.long		0x92db4949, 0x0c0a0606, 0x486c2424, 0xb8e45c5c
+	.long		0x9f5dc2c2, 0xbd6ed3d3, 0x43efacac, 0xc4a66262
+	.long		0x39a89191, 0x31a49595, 0xd337e4e4, 0xf28b7979
+	.long		0xd532e7e7, 0x8b43c8c8, 0x6e593737, 0xdab76d6d
+	.long		0x018c8d8d, 0xb164d5d5, 0x9cd24e4e, 0x49e0a9a9
+	.long		0xd8b46c6c, 0xacfa5656, 0xf307f4f4, 0xcf25eaea
+	.long		0xcaaf6565, 0xf48e7a7a, 0x47e9aeae, 0x10180808
+	.long		0x6fd5baba, 0xf0887878, 0x4a6f2525, 0x5c722e2e
+	.long		0x38241c1c, 0x57f1a6a6, 0x73c7b4b4, 0x9751c6c6
+	.long		0xcb23e8e8, 0xa17cdddd, 0xe89c7474, 0x3e211f1f
+	.long		0x96dd4b4b, 0x61dcbdbd, 0x0d868b8b, 0x0f858a8a
+	.long		0xe0907070, 0x7c423e3e, 0x71c4b5b5, 0xccaa6666
+	.long		0x90d84848, 0x06050303, 0xf701f6f6, 0x1c120e0e
+	.long		0xc2a36161, 0x6a5f3535, 0xaef95757, 0x69d0b9b9
+	.long		0x17918686, 0x9958c1c1, 0x3a271d1d, 0x27b99e9e
+	.long		0xd938e1e1, 0xeb13f8f8, 0x2bb39898, 0x22331111
+	.long		0xd2bb6969, 0xa970d9d9, 0x07898e8e, 0x33a79494
+	.long		0x2db69b9b, 0x3c221e1e, 0x15928787, 0xc920e9e9
+	.long		0x8749cece, 0xaaff5555, 0x50782828, 0xa57adfdf
+	.long		0x038f8c8c, 0x59f8a1a1, 0x09808989, 0x1a170d0d
+	.long		0x65dabfbf, 0xd731e6e6, 0x84c64242, 0xd0b86868
+	.long		0x82c34141, 0x29b09999, 0x5a772d2d, 0x1e110f0f
+	.long		0x7bcbb0b0, 0xa8fc5454, 0x6dd6bbbb, 0x2c3a1616
+	.endif
+	.endm
+
+	.macro		__crypto_fl_tab, full=1
+	.long		0x00000063, 0x0000007c, 0x00000077, 0x0000007b
+	.long		0x000000f2, 0x0000006b, 0x0000006f, 0x000000c5
+	.long		0x00000030, 0x00000001, 0x00000067, 0x0000002b
+	.long		0x000000fe, 0x000000d7, 0x000000ab, 0x00000076
+	.long		0x000000ca, 0x00000082, 0x000000c9, 0x0000007d
+	.long		0x000000fa, 0x00000059, 0x00000047, 0x000000f0
+	.long		0x000000ad, 0x000000d4, 0x000000a2, 0x000000af
+	.long		0x0000009c, 0x000000a4, 0x00000072, 0x000000c0
+	.long		0x000000b7, 0x000000fd, 0x00000093, 0x00000026
+	.long		0x00000036, 0x0000003f, 0x000000f7, 0x000000cc
+	.long		0x00000034, 0x000000a5, 0x000000e5, 0x000000f1
+	.long		0x00000071, 0x000000d8, 0x00000031, 0x00000015
+	.long		0x00000004, 0x000000c7, 0x00000023, 0x000000c3
+	.long		0x00000018, 0x00000096, 0x00000005, 0x0000009a
+	.long		0x00000007, 0x00000012, 0x00000080, 0x000000e2
+	.long		0x000000eb, 0x00000027, 0x000000b2, 0x00000075
+	.long		0x00000009, 0x00000083, 0x0000002c, 0x0000001a
+	.long		0x0000001b, 0x0000006e, 0x0000005a, 0x000000a0
+	.long		0x00000052, 0x0000003b, 0x000000d6, 0x000000b3
+	.long		0x00000029, 0x000000e3, 0x0000002f, 0x00000084
+	.long		0x00000053, 0x000000d1, 0x00000000, 0x000000ed
+	.long		0x00000020, 0x000000fc, 0x000000b1, 0x0000005b
+	.long		0x0000006a, 0x000000cb, 0x000000be, 0x00000039
+	.long		0x0000004a, 0x0000004c, 0x00000058, 0x000000cf
+	.long		0x000000d0, 0x000000ef, 0x000000aa, 0x000000fb
+	.long		0x00000043, 0x0000004d, 0x00000033, 0x00000085
+	.long		0x00000045, 0x000000f9, 0x00000002, 0x0000007f
+	.long		0x00000050, 0x0000003c, 0x0000009f, 0x000000a8
+	.long		0x00000051, 0x000000a3, 0x00000040, 0x0000008f
+	.long		0x00000092, 0x0000009d, 0x00000038, 0x000000f5
+	.long		0x000000bc, 0x000000b6, 0x000000da, 0x00000021
+	.long		0x00000010, 0x000000ff, 0x000000f3, 0x000000d2
+	.long		0x000000cd, 0x0000000c, 0x00000013, 0x000000ec
+	.long		0x0000005f, 0x00000097, 0x00000044, 0x00000017
+	.long		0x000000c4, 0x000000a7, 0x0000007e, 0x0000003d
+	.long		0x00000064, 0x0000005d, 0x00000019, 0x00000073
+	.long		0x00000060, 0x00000081, 0x0000004f, 0x000000dc
+	.long		0x00000022, 0x0000002a, 0x00000090, 0x00000088
+	.long		0x00000046, 0x000000ee, 0x000000b8, 0x00000014
+	.long		0x000000de, 0x0000005e, 0x0000000b, 0x000000db
+	.long		0x000000e0, 0x00000032, 0x0000003a, 0x0000000a
+	.long		0x00000049, 0x00000006, 0x00000024, 0x0000005c
+	.long		0x000000c2, 0x000000d3, 0x000000ac, 0x00000062
+	.long		0x00000091, 0x00000095, 0x000000e4, 0x00000079
+	.long		0x000000e7, 0x000000c8, 0x00000037, 0x0000006d
+	.long		0x0000008d, 0x000000d5, 0x0000004e, 0x000000a9
+	.long		0x0000006c, 0x00000056, 0x000000f4, 0x000000ea
+	.long		0x00000065, 0x0000007a, 0x000000ae, 0x00000008
+	.long		0x000000ba, 0x00000078, 0x00000025, 0x0000002e
+	.long		0x0000001c, 0x000000a6, 0x000000b4, 0x000000c6
+	.long		0x000000e8, 0x000000dd, 0x00000074, 0x0000001f
+	.long		0x0000004b, 0x000000bd, 0x0000008b, 0x0000008a
+	.long		0x00000070, 0x0000003e, 0x000000b5, 0x00000066
+	.long		0x00000048, 0x00000003, 0x000000f6, 0x0000000e
+	.long		0x00000061, 0x00000035, 0x00000057, 0x000000b9
+	.long		0x00000086, 0x000000c1, 0x0000001d, 0x0000009e
+	.long		0x000000e1, 0x000000f8, 0x00000098, 0x00000011
+	.long		0x00000069, 0x000000d9, 0x0000008e, 0x00000094
+	.long		0x0000009b, 0x0000001e, 0x00000087, 0x000000e9
+	.long		0x000000ce, 0x00000055, 0x00000028, 0x000000df
+	.long		0x0000008c, 0x000000a1, 0x00000089, 0x0000000d
+	.long		0x000000bf, 0x000000e6, 0x00000042, 0x00000068
+	.long		0x00000041, 0x00000099, 0x0000002d, 0x0000000f
+	.long		0x000000b0, 0x00000054, 0x000000bb, 0x00000016
+
+	.if		\full == 1
+	.long		0x00006300, 0x00007c00, 0x00007700, 0x00007b00
+	.long		0x0000f200, 0x00006b00, 0x00006f00, 0x0000c500
+	.long		0x00003000, 0x00000100, 0x00006700, 0x00002b00
+	.long		0x0000fe00, 0x0000d700, 0x0000ab00, 0x00007600
+	.long		0x0000ca00, 0x00008200, 0x0000c900, 0x00007d00
+	.long		0x0000fa00, 0x00005900, 0x00004700, 0x0000f000
+	.long		0x0000ad00, 0x0000d400, 0x0000a200, 0x0000af00
+	.long		0x00009c00, 0x0000a400, 0x00007200, 0x0000c000
+	.long		0x0000b700, 0x0000fd00, 0x00009300, 0x00002600
+	.long		0x00003600, 0x00003f00, 0x0000f700, 0x0000cc00
+	.long		0x00003400, 0x0000a500, 0x0000e500, 0x0000f100
+	.long		0x00007100, 0x0000d800, 0x00003100, 0x00001500
+	.long		0x00000400, 0x0000c700, 0x00002300, 0x0000c300
+	.long		0x00001800, 0x00009600, 0x00000500, 0x00009a00
+	.long		0x00000700, 0x00001200, 0x00008000, 0x0000e200
+	.long		0x0000eb00, 0x00002700, 0x0000b200, 0x00007500
+	.long		0x00000900, 0x00008300, 0x00002c00, 0x00001a00
+	.long		0x00001b00, 0x00006e00, 0x00005a00, 0x0000a000
+	.long		0x00005200, 0x00003b00, 0x0000d600, 0x0000b300
+	.long		0x00002900, 0x0000e300, 0x00002f00, 0x00008400
+	.long		0x00005300, 0x0000d100, 0x00000000, 0x0000ed00
+	.long		0x00002000, 0x0000fc00, 0x0000b100, 0x00005b00
+	.long		0x00006a00, 0x0000cb00, 0x0000be00, 0x00003900
+	.long		0x00004a00, 0x00004c00, 0x00005800, 0x0000cf00
+	.long		0x0000d000, 0x0000ef00, 0x0000aa00, 0x0000fb00
+	.long		0x00004300, 0x00004d00, 0x00003300, 0x00008500
+	.long		0x00004500, 0x0000f900, 0x00000200, 0x00007f00
+	.long		0x00005000, 0x00003c00, 0x00009f00, 0x0000a800
+	.long		0x00005100, 0x0000a300, 0x00004000, 0x00008f00
+	.long		0x00009200, 0x00009d00, 0x00003800, 0x0000f500
+	.long		0x0000bc00, 0x0000b600, 0x0000da00, 0x00002100
+	.long		0x00001000, 0x0000ff00, 0x0000f300, 0x0000d200
+	.long		0x0000cd00, 0x00000c00, 0x00001300, 0x0000ec00
+	.long		0x00005f00, 0x00009700, 0x00004400, 0x00001700
+	.long		0x0000c400, 0x0000a700, 0x00007e00, 0x00003d00
+	.long		0x00006400, 0x00005d00, 0x00001900, 0x00007300
+	.long		0x00006000, 0x00008100, 0x00004f00, 0x0000dc00
+	.long		0x00002200, 0x00002a00, 0x00009000, 0x00008800
+	.long		0x00004600, 0x0000ee00, 0x0000b800, 0x00001400
+	.long		0x0000de00, 0x00005e00, 0x00000b00, 0x0000db00
+	.long		0x0000e000, 0x00003200, 0x00003a00, 0x00000a00
+	.long		0x00004900, 0x00000600, 0x00002400, 0x00005c00
+	.long		0x0000c200, 0x0000d300, 0x0000ac00, 0x00006200
+	.long		0x00009100, 0x00009500, 0x0000e400, 0x00007900
+	.long		0x0000e700, 0x0000c800, 0x00003700, 0x00006d00
+	.long		0x00008d00, 0x0000d500, 0x00004e00, 0x0000a900
+	.long		0x00006c00, 0x00005600, 0x0000f400, 0x0000ea00
+	.long		0x00006500, 0x00007a00, 0x0000ae00, 0x00000800
+	.long		0x0000ba00, 0x00007800, 0x00002500, 0x00002e00
+	.long		0x00001c00, 0x0000a600, 0x0000b400, 0x0000c600
+	.long		0x0000e800, 0x0000dd00, 0x00007400, 0x00001f00
+	.long		0x00004b00, 0x0000bd00, 0x00008b00, 0x00008a00
+	.long		0x00007000, 0x00003e00, 0x0000b500, 0x00006600
+	.long		0x00004800, 0x00000300, 0x0000f600, 0x00000e00
+	.long		0x00006100, 0x00003500, 0x00005700, 0x0000b900
+	.long		0x00008600, 0x0000c100, 0x00001d00, 0x00009e00
+	.long		0x0000e100, 0x0000f800, 0x00009800, 0x00001100
+	.long		0x00006900, 0x0000d900, 0x00008e00, 0x00009400
+	.long		0x00009b00, 0x00001e00, 0x00008700, 0x0000e900
+	.long		0x0000ce00, 0x00005500, 0x00002800, 0x0000df00
+	.long		0x00008c00, 0x0000a100, 0x00008900, 0x00000d00
+	.long		0x0000bf00, 0x0000e600, 0x00004200, 0x00006800
+	.long		0x00004100, 0x00009900, 0x00002d00, 0x00000f00
+	.long		0x0000b000, 0x00005400, 0x0000bb00, 0x00001600
+
+	.long		0x00630000, 0x007c0000, 0x00770000, 0x007b0000
+	.long		0x00f20000, 0x006b0000, 0x006f0000, 0x00c50000
+	.long		0x00300000, 0x00010000, 0x00670000, 0x002b0000
+	.long		0x00fe0000, 0x00d70000, 0x00ab0000, 0x00760000
+	.long		0x00ca0000, 0x00820000, 0x00c90000, 0x007d0000
+	.long		0x00fa0000, 0x00590000, 0x00470000, 0x00f00000
+	.long		0x00ad0000, 0x00d40000, 0x00a20000, 0x00af0000
+	.long		0x009c0000, 0x00a40000, 0x00720000, 0x00c00000
+	.long		0x00b70000, 0x00fd0000, 0x00930000, 0x00260000
+	.long		0x00360000, 0x003f0000, 0x00f70000, 0x00cc0000
+	.long		0x00340000, 0x00a50000, 0x00e50000, 0x00f10000
+	.long		0x00710000, 0x00d80000, 0x00310000, 0x00150000
+	.long		0x00040000, 0x00c70000, 0x00230000, 0x00c30000
+	.long		0x00180000, 0x00960000, 0x00050000, 0x009a0000
+	.long		0x00070000, 0x00120000, 0x00800000, 0x00e20000
+	.long		0x00eb0000, 0x00270000, 0x00b20000, 0x00750000
+	.long		0x00090000, 0x00830000, 0x002c0000, 0x001a0000
+	.long		0x001b0000, 0x006e0000, 0x005a0000, 0x00a00000
+	.long		0x00520000, 0x003b0000, 0x00d60000, 0x00b30000
+	.long		0x00290000, 0x00e30000, 0x002f0000, 0x00840000
+	.long		0x00530000, 0x00d10000, 0x00000000, 0x00ed0000
+	.long		0x00200000, 0x00fc0000, 0x00b10000, 0x005b0000
+	.long		0x006a0000, 0x00cb0000, 0x00be0000, 0x00390000
+	.long		0x004a0000, 0x004c0000, 0x00580000, 0x00cf0000
+	.long		0x00d00000, 0x00ef0000, 0x00aa0000, 0x00fb0000
+	.long		0x00430000, 0x004d0000, 0x00330000, 0x00850000
+	.long		0x00450000, 0x00f90000, 0x00020000, 0x007f0000
+	.long		0x00500000, 0x003c0000, 0x009f0000, 0x00a80000
+	.long		0x00510000, 0x00a30000, 0x00400000, 0x008f0000
+	.long		0x00920000, 0x009d0000, 0x00380000, 0x00f50000
+	.long		0x00bc0000, 0x00b60000, 0x00da0000, 0x00210000
+	.long		0x00100000, 0x00ff0000, 0x00f30000, 0x00d20000
+	.long		0x00cd0000, 0x000c0000, 0x00130000, 0x00ec0000
+	.long		0x005f0000, 0x00970000, 0x00440000, 0x00170000
+	.long		0x00c40000, 0x00a70000, 0x007e0000, 0x003d0000
+	.long		0x00640000, 0x005d0000, 0x00190000, 0x00730000
+	.long		0x00600000, 0x00810000, 0x004f0000, 0x00dc0000
+	.long		0x00220000, 0x002a0000, 0x00900000, 0x00880000
+	.long		0x00460000, 0x00ee0000, 0x00b80000, 0x00140000
+	.long		0x00de0000, 0x005e0000, 0x000b0000, 0x00db0000
+	.long		0x00e00000, 0x00320000, 0x003a0000, 0x000a0000
+	.long		0x00490000, 0x00060000, 0x00240000, 0x005c0000
+	.long		0x00c20000, 0x00d30000, 0x00ac0000, 0x00620000
+	.long		0x00910000, 0x00950000, 0x00e40000, 0x00790000
+	.long		0x00e70000, 0x00c80000, 0x00370000, 0x006d0000
+	.long		0x008d0000, 0x00d50000, 0x004e0000, 0x00a90000
+	.long		0x006c0000, 0x00560000, 0x00f40000, 0x00ea0000
+	.long		0x00650000, 0x007a0000, 0x00ae0000, 0x00080000
+	.long		0x00ba0000, 0x00780000, 0x00250000, 0x002e0000
+	.long		0x001c0000, 0x00a60000, 0x00b40000, 0x00c60000
+	.long		0x00e80000, 0x00dd0000, 0x00740000, 0x001f0000
+	.long		0x004b0000, 0x00bd0000, 0x008b0000, 0x008a0000
+	.long		0x00700000, 0x003e0000, 0x00b50000, 0x00660000
+	.long		0x00480000, 0x00030000, 0x00f60000, 0x000e0000
+	.long		0x00610000, 0x00350000, 0x00570000, 0x00b90000
+	.long		0x00860000, 0x00c10000, 0x001d0000, 0x009e0000
+	.long		0x00e10000, 0x00f80000, 0x00980000, 0x00110000
+	.long		0x00690000, 0x00d90000, 0x008e0000, 0x00940000
+	.long		0x009b0000, 0x001e0000, 0x00870000, 0x00e90000
+	.long		0x00ce0000, 0x00550000, 0x00280000, 0x00df0000
+	.long		0x008c0000, 0x00a10000, 0x00890000, 0x000d0000
+	.long		0x00bf0000, 0x00e60000, 0x00420000, 0x00680000
+	.long		0x00410000, 0x00990000, 0x002d0000, 0x000f0000
+	.long		0x00b00000, 0x00540000, 0x00bb0000, 0x00160000
+
+	.long		0x63000000, 0x7c000000, 0x77000000, 0x7b000000
+	.long		0xf2000000, 0x6b000000, 0x6f000000, 0xc5000000
+	.long		0x30000000, 0x01000000, 0x67000000, 0x2b000000
+	.long		0xfe000000, 0xd7000000, 0xab000000, 0x76000000
+	.long		0xca000000, 0x82000000, 0xc9000000, 0x7d000000
+	.long		0xfa000000, 0x59000000, 0x47000000, 0xf0000000
+	.long		0xad000000, 0xd4000000, 0xa2000000, 0xaf000000
+	.long		0x9c000000, 0xa4000000, 0x72000000, 0xc0000000
+	.long		0xb7000000, 0xfd000000, 0x93000000, 0x26000000
+	.long		0x36000000, 0x3f000000, 0xf7000000, 0xcc000000
+	.long		0x34000000, 0xa5000000, 0xe5000000, 0xf1000000
+	.long		0x71000000, 0xd8000000, 0x31000000, 0x15000000
+	.long		0x04000000, 0xc7000000, 0x23000000, 0xc3000000
+	.long		0x18000000, 0x96000000, 0x05000000, 0x9a000000
+	.long		0x07000000, 0x12000000, 0x80000000, 0xe2000000
+	.long		0xeb000000, 0x27000000, 0xb2000000, 0x75000000
+	.long		0x09000000, 0x83000000, 0x2c000000, 0x1a000000
+	.long		0x1b000000, 0x6e000000, 0x5a000000, 0xa0000000
+	.long		0x52000000, 0x3b000000, 0xd6000000, 0xb3000000
+	.long		0x29000000, 0xe3000000, 0x2f000000, 0x84000000
+	.long		0x53000000, 0xd1000000, 0x00000000, 0xed000000
+	.long		0x20000000, 0xfc000000, 0xb1000000, 0x5b000000
+	.long		0x6a000000, 0xcb000000, 0xbe000000, 0x39000000
+	.long		0x4a000000, 0x4c000000, 0x58000000, 0xcf000000
+	.long		0xd0000000, 0xef000000, 0xaa000000, 0xfb000000
+	.long		0x43000000, 0x4d000000, 0x33000000, 0x85000000
+	.long		0x45000000, 0xf9000000, 0x02000000, 0x7f000000
+	.long		0x50000000, 0x3c000000, 0x9f000000, 0xa8000000
+	.long		0x51000000, 0xa3000000, 0x40000000, 0x8f000000
+	.long		0x92000000, 0x9d000000, 0x38000000, 0xf5000000
+	.long		0xbc000000, 0xb6000000, 0xda000000, 0x21000000
+	.long		0x10000000, 0xff000000, 0xf3000000, 0xd2000000
+	.long		0xcd000000, 0x0c000000, 0x13000000, 0xec000000
+	.long		0x5f000000, 0x97000000, 0x44000000, 0x17000000
+	.long		0xc4000000, 0xa7000000, 0x7e000000, 0x3d000000
+	.long		0x64000000, 0x5d000000, 0x19000000, 0x73000000
+	.long		0x60000000, 0x81000000, 0x4f000000, 0xdc000000
+	.long		0x22000000, 0x2a000000, 0x90000000, 0x88000000
+	.long		0x46000000, 0xee000000, 0xb8000000, 0x14000000
+	.long		0xde000000, 0x5e000000, 0x0b000000, 0xdb000000
+	.long		0xe0000000, 0x32000000, 0x3a000000, 0x0a000000
+	.long		0x49000000, 0x06000000, 0x24000000, 0x5c000000
+	.long		0xc2000000, 0xd3000000, 0xac000000, 0x62000000
+	.long		0x91000000, 0x95000000, 0xe4000000, 0x79000000
+	.long		0xe7000000, 0xc8000000, 0x37000000, 0x6d000000
+	.long		0x8d000000, 0xd5000000, 0x4e000000, 0xa9000000
+	.long		0x6c000000, 0x56000000, 0xf4000000, 0xea000000
+	.long		0x65000000, 0x7a000000, 0xae000000, 0x08000000
+	.long		0xba000000, 0x78000000, 0x25000000, 0x2e000000
+	.long		0x1c000000, 0xa6000000, 0xb4000000, 0xc6000000
+	.long		0xe8000000, 0xdd000000, 0x74000000, 0x1f000000
+	.long		0x4b000000, 0xbd000000, 0x8b000000, 0x8a000000
+	.long		0x70000000, 0x3e000000, 0xb5000000, 0x66000000
+	.long		0x48000000, 0x03000000, 0xf6000000, 0x0e000000
+	.long		0x61000000, 0x35000000, 0x57000000, 0xb9000000
+	.long		0x86000000, 0xc1000000, 0x1d000000, 0x9e000000
+	.long		0xe1000000, 0xf8000000, 0x98000000, 0x11000000
+	.long		0x69000000, 0xd9000000, 0x8e000000, 0x94000000
+	.long		0x9b000000, 0x1e000000, 0x87000000, 0xe9000000
+	.long		0xce000000, 0x55000000, 0x28000000, 0xdf000000
+	.long		0x8c000000, 0xa1000000, 0x89000000, 0x0d000000
+	.long		0xbf000000, 0xe6000000, 0x42000000, 0x68000000
+	.long		0x41000000, 0x99000000, 0x2d000000, 0x0f000000
+	.long		0xb0000000, 0x54000000, 0xbb000000, 0x16000000
+	.endif
+	.endm
+
+	.macro		__crypto_it_tab, full=1
+	.long		0x50a7f451, 0x5365417e, 0xc3a4171a, 0x965e273a
+	.long		0xcb6bab3b, 0xf1459d1f, 0xab58faac, 0x9303e34b
+	.long		0x55fa3020, 0xf66d76ad, 0x9176cc88, 0x254c02f5
+	.long		0xfcd7e54f, 0xd7cb2ac5, 0x80443526, 0x8fa362b5
+	.long		0x495ab1de, 0x671bba25, 0x980eea45, 0xe1c0fe5d
+	.long		0x02752fc3, 0x12f04c81, 0xa397468d, 0xc6f9d36b
+	.long		0xe75f8f03, 0x959c9215, 0xeb7a6dbf, 0xda595295
+	.long		0x2d83bed4, 0xd3217458, 0x2969e049, 0x44c8c98e
+	.long		0x6a89c275, 0x78798ef4, 0x6b3e5899, 0xdd71b927
+	.long		0xb64fe1be, 0x17ad88f0, 0x66ac20c9, 0xb43ace7d
+	.long		0x184adf63, 0x82311ae5, 0x60335197, 0x457f5362
+	.long		0xe07764b1, 0x84ae6bbb, 0x1ca081fe, 0x942b08f9
+	.long		0x58684870, 0x19fd458f, 0x876cde94, 0xb7f87b52
+	.long		0x23d373ab, 0xe2024b72, 0x578f1fe3, 0x2aab5566
+	.long		0x0728ebb2, 0x03c2b52f, 0x9a7bc586, 0xa50837d3
+	.long		0xf2872830, 0xb2a5bf23, 0xba6a0302, 0x5c8216ed
+	.long		0x2b1ccf8a, 0x92b479a7, 0xf0f207f3, 0xa1e2694e
+	.long		0xcdf4da65, 0xd5be0506, 0x1f6234d1, 0x8afea6c4
+	.long		0x9d532e34, 0xa055f3a2, 0x32e18a05, 0x75ebf6a4
+	.long		0x39ec830b, 0xaaef6040, 0x069f715e, 0x51106ebd
+	.long		0xf98a213e, 0x3d06dd96, 0xae053edd, 0x46bde64d
+	.long		0xb58d5491, 0x055dc471, 0x6fd40604, 0xff155060
+	.long		0x24fb9819, 0x97e9bdd6, 0xcc434089, 0x779ed967
+	.long		0xbd42e8b0, 0x888b8907, 0x385b19e7, 0xdbeec879
+	.long		0x470a7ca1, 0xe90f427c, 0xc91e84f8, 0x00000000
+	.long		0x83868009, 0x48ed2b32, 0xac70111e, 0x4e725a6c
+	.long		0xfbff0efd, 0x5638850f, 0x1ed5ae3d, 0x27392d36
+	.long		0x64d90f0a, 0x21a65c68, 0xd1545b9b, 0x3a2e3624
+	.long		0xb1670a0c, 0x0fe75793, 0xd296eeb4, 0x9e919b1b
+	.long		0x4fc5c080, 0xa220dc61, 0x694b775a, 0x161a121c
+	.long		0x0aba93e2, 0xe52aa0c0, 0x43e0223c, 0x1d171b12
+	.long		0x0b0d090e, 0xadc78bf2, 0xb9a8b62d, 0xc8a91e14
+	.long		0x8519f157, 0x4c0775af, 0xbbdd99ee, 0xfd607fa3
+	.long		0x9f2601f7, 0xbcf5725c, 0xc53b6644, 0x347efb5b
+	.long		0x7629438b, 0xdcc623cb, 0x68fcedb6, 0x63f1e4b8
+	.long		0xcadc31d7, 0x10856342, 0x40229713, 0x2011c684
+	.long		0x7d244a85, 0xf83dbbd2, 0x1132f9ae, 0x6da129c7
+	.long		0x4b2f9e1d, 0xf330b2dc, 0xec52860d, 0xd0e3c177
+	.long		0x6c16b32b, 0x99b970a9, 0xfa489411, 0x2264e947
+	.long		0xc48cfca8, 0x1a3ff0a0, 0xd82c7d56, 0xef903322
+	.long		0xc74e4987, 0xc1d138d9, 0xfea2ca8c, 0x360bd498
+	.long		0xcf81f5a6, 0x28de7aa5, 0x268eb7da, 0xa4bfad3f
+	.long		0xe49d3a2c, 0x0d927850, 0x9bcc5f6a, 0x62467e54
+	.long		0xc2138df6, 0xe8b8d890, 0x5ef7392e, 0xf5afc382
+	.long		0xbe805d9f, 0x7c93d069, 0xa92dd56f, 0xb31225cf
+	.long		0x3b99acc8, 0xa77d1810, 0x6e639ce8, 0x7bbb3bdb
+	.long		0x097826cd, 0xf418596e, 0x01b79aec, 0xa89a4f83
+	.long		0x656e95e6, 0x7ee6ffaa, 0x08cfbc21, 0xe6e815ef
+	.long		0xd99be7ba, 0xce366f4a, 0xd4099fea, 0xd67cb029
+	.long		0xafb2a431, 0x31233f2a, 0x3094a5c6, 0xc066a235
+	.long		0x37bc4e74, 0xa6ca82fc, 0xb0d090e0, 0x15d8a733
+	.long		0x4a9804f1, 0xf7daec41, 0x0e50cd7f, 0x2ff69117
+	.long		0x8dd64d76, 0x4db0ef43, 0x544daacc, 0xdf0496e4
+	.long		0xe3b5d19e, 0x1b886a4c, 0xb81f2cc1, 0x7f516546
+	.long		0x04ea5e9d, 0x5d358c01, 0x737487fa, 0x2e410bfb
+	.long		0x5a1d67b3, 0x52d2db92, 0x335610e9, 0x1347d66d
+	.long		0x8c61d79a, 0x7a0ca137, 0x8e14f859, 0x893c13eb
+	.long		0xee27a9ce, 0x35c961b7, 0xede51ce1, 0x3cb1477a
+	.long		0x59dfd29c, 0x3f73f255, 0x79ce1418, 0xbf37c773
+	.long		0xeacdf753, 0x5baafd5f, 0x146f3ddf, 0x86db4478
+	.long		0x81f3afca, 0x3ec468b9, 0x2c342438, 0x5f40a3c2
+	.long		0x72c31d16, 0x0c25e2bc, 0x8b493c28, 0x41950dff
+	.long		0x7101a839, 0xdeb30c08, 0x9ce4b4d8, 0x90c15664
+	.long		0x6184cb7b, 0x70b632d5, 0x745c6c48, 0x4257b8d0
+
+	.if		\full == 1
+	.long		0xa7f45150, 0x65417e53, 0xa4171ac3, 0x5e273a96
+	.long		0x6bab3bcb, 0x459d1ff1, 0x58faacab, 0x03e34b93
+	.long		0xfa302055, 0x6d76adf6, 0x76cc8891, 0x4c02f525
+	.long		0xd7e54ffc, 0xcb2ac5d7, 0x44352680, 0xa362b58f
+	.long		0x5ab1de49, 0x1bba2567, 0x0eea4598, 0xc0fe5de1
+	.long		0x752fc302, 0xf04c8112, 0x97468da3, 0xf9d36bc6
+	.long		0x5f8f03e7, 0x9c921595, 0x7a6dbfeb, 0x595295da
+	.long		0x83bed42d, 0x217458d3, 0x69e04929, 0xc8c98e44
+	.long		0x89c2756a, 0x798ef478, 0x3e58996b, 0x71b927dd
+	.long		0x4fe1beb6, 0xad88f017, 0xac20c966, 0x3ace7db4
+	.long		0x4adf6318, 0x311ae582, 0x33519760, 0x7f536245
+	.long		0x7764b1e0, 0xae6bbb84, 0xa081fe1c, 0x2b08f994
+	.long		0x68487058, 0xfd458f19, 0x6cde9487, 0xf87b52b7
+	.long		0xd373ab23, 0x024b72e2, 0x8f1fe357, 0xab55662a
+	.long		0x28ebb207, 0xc2b52f03, 0x7bc5869a, 0x0837d3a5
+	.long		0x872830f2, 0xa5bf23b2, 0x6a0302ba, 0x8216ed5c
+	.long		0x1ccf8a2b, 0xb479a792, 0xf207f3f0, 0xe2694ea1
+	.long		0xf4da65cd, 0xbe0506d5, 0x6234d11f, 0xfea6c48a
+	.long		0x532e349d, 0x55f3a2a0, 0xe18a0532, 0xebf6a475
+	.long		0xec830b39, 0xef6040aa, 0x9f715e06, 0x106ebd51
+	.long		0x8a213ef9, 0x06dd963d, 0x053eddae, 0xbde64d46
+	.long		0x8d5491b5, 0x5dc47105, 0xd406046f, 0x155060ff
+	.long		0xfb981924, 0xe9bdd697, 0x434089cc, 0x9ed96777
+	.long		0x42e8b0bd, 0x8b890788, 0x5b19e738, 0xeec879db
+	.long		0x0a7ca147, 0x0f427ce9, 0x1e84f8c9, 0x00000000
+	.long		0x86800983, 0xed2b3248, 0x70111eac, 0x725a6c4e
+	.long		0xff0efdfb, 0x38850f56, 0xd5ae3d1e, 0x392d3627
+	.long		0xd90f0a64, 0xa65c6821, 0x545b9bd1, 0x2e36243a
+	.long		0x670a0cb1, 0xe757930f, 0x96eeb4d2, 0x919b1b9e
+	.long		0xc5c0804f, 0x20dc61a2, 0x4b775a69, 0x1a121c16
+	.long		0xba93e20a, 0x2aa0c0e5, 0xe0223c43, 0x171b121d
+	.long		0x0d090e0b, 0xc78bf2ad, 0xa8b62db9, 0xa91e14c8
+	.long		0x19f15785, 0x0775af4c, 0xdd99eebb, 0x607fa3fd
+	.long		0x2601f79f, 0xf5725cbc, 0x3b6644c5, 0x7efb5b34
+	.long		0x29438b76, 0xc623cbdc, 0xfcedb668, 0xf1e4b863
+	.long		0xdc31d7ca, 0x85634210, 0x22971340, 0x11c68420
+	.long		0x244a857d, 0x3dbbd2f8, 0x32f9ae11, 0xa129c76d
+	.long		0x2f9e1d4b, 0x30b2dcf3, 0x52860dec, 0xe3c177d0
+	.long		0x16b32b6c, 0xb970a999, 0x489411fa, 0x64e94722
+	.long		0x8cfca8c4, 0x3ff0a01a, 0x2c7d56d8, 0x903322ef
+	.long		0x4e4987c7, 0xd138d9c1, 0xa2ca8cfe, 0x0bd49836
+	.long		0x81f5a6cf, 0xde7aa528, 0x8eb7da26, 0xbfad3fa4
+	.long		0x9d3a2ce4, 0x9278500d, 0xcc5f6a9b, 0x467e5462
+	.long		0x138df6c2, 0xb8d890e8, 0xf7392e5e, 0xafc382f5
+	.long		0x805d9fbe, 0x93d0697c, 0x2dd56fa9, 0x1225cfb3
+	.long		0x99acc83b, 0x7d1810a7, 0x639ce86e, 0xbb3bdb7b
+	.long		0x7826cd09, 0x18596ef4, 0xb79aec01, 0x9a4f83a8
+	.long		0x6e95e665, 0xe6ffaa7e, 0xcfbc2108, 0xe815efe6
+	.long		0x9be7bad9, 0x366f4ace, 0x099fead4, 0x7cb029d6
+	.long		0xb2a431af, 0x233f2a31, 0x94a5c630, 0x66a235c0
+	.long		0xbc4e7437, 0xca82fca6, 0xd090e0b0, 0xd8a73315
+	.long		0x9804f14a, 0xdaec41f7, 0x50cd7f0e, 0xf691172f
+	.long		0xd64d768d, 0xb0ef434d, 0x4daacc54, 0x0496e4df
+	.long		0xb5d19ee3, 0x886a4c1b, 0x1f2cc1b8, 0x5165467f
+	.long		0xea5e9d04, 0x358c015d, 0x7487fa73, 0x410bfb2e
+	.long		0x1d67b35a, 0xd2db9252, 0x5610e933, 0x47d66d13
+	.long		0x61d79a8c, 0x0ca1377a, 0x14f8598e, 0x3c13eb89
+	.long		0x27a9ceee, 0xc961b735, 0xe51ce1ed, 0xb1477a3c
+	.long		0xdfd29c59, 0x73f2553f, 0xce141879, 0x37c773bf
+	.long		0xcdf753ea, 0xaafd5f5b, 0x6f3ddf14, 0xdb447886
+	.long		0xf3afca81, 0xc468b93e, 0x3424382c, 0x40a3c25f
+	.long		0xc31d1672, 0x25e2bc0c, 0x493c288b, 0x950dff41
+	.long		0x01a83971, 0xb30c08de, 0xe4b4d89c, 0xc1566490
+	.long		0x84cb7b61, 0xb632d570, 0x5c6c4874, 0x57b8d042
+
+	.long		0xf45150a7, 0x417e5365, 0x171ac3a4, 0x273a965e
+	.long		0xab3bcb6b, 0x9d1ff145, 0xfaacab58, 0xe34b9303
+	.long		0x302055fa, 0x76adf66d, 0xcc889176, 0x02f5254c
+	.long		0xe54ffcd7, 0x2ac5d7cb, 0x35268044, 0x62b58fa3
+	.long		0xb1de495a, 0xba25671b, 0xea45980e, 0xfe5de1c0
+	.long		0x2fc30275, 0x4c8112f0, 0x468da397, 0xd36bc6f9
+	.long		0x8f03e75f, 0x9215959c, 0x6dbfeb7a, 0x5295da59
+	.long		0xbed42d83, 0x7458d321, 0xe0492969, 0xc98e44c8
+	.long		0xc2756a89, 0x8ef47879, 0x58996b3e, 0xb927dd71
+	.long		0xe1beb64f, 0x88f017ad, 0x20c966ac, 0xce7db43a
+	.long		0xdf63184a, 0x1ae58231, 0x51976033, 0x5362457f
+	.long		0x64b1e077, 0x6bbb84ae, 0x81fe1ca0, 0x08f9942b
+	.long		0x48705868, 0x458f19fd, 0xde94876c, 0x7b52b7f8
+	.long		0x73ab23d3, 0x4b72e202, 0x1fe3578f, 0x55662aab
+	.long		0xebb20728, 0xb52f03c2, 0xc5869a7b, 0x37d3a508
+	.long		0x2830f287, 0xbf23b2a5, 0x0302ba6a, 0x16ed5c82
+	.long		0xcf8a2b1c, 0x79a792b4, 0x07f3f0f2, 0x694ea1e2
+	.long		0xda65cdf4, 0x0506d5be, 0x34d11f62, 0xa6c48afe
+	.long		0x2e349d53, 0xf3a2a055, 0x8a0532e1, 0xf6a475eb
+	.long		0x830b39ec, 0x6040aaef, 0x715e069f, 0x6ebd5110
+	.long		0x213ef98a, 0xdd963d06, 0x3eddae05, 0xe64d46bd
+	.long		0x5491b58d, 0xc471055d, 0x06046fd4, 0x5060ff15
+	.long		0x981924fb, 0xbdd697e9, 0x4089cc43, 0xd967779e
+	.long		0xe8b0bd42, 0x8907888b, 0x19e7385b, 0xc879dbee
+	.long		0x7ca1470a, 0x427ce90f, 0x84f8c91e, 0x00000000
+	.long		0x80098386, 0x2b3248ed, 0x111eac70, 0x5a6c4e72
+	.long		0x0efdfbff, 0x850f5638, 0xae3d1ed5, 0x2d362739
+	.long		0x0f0a64d9, 0x5c6821a6, 0x5b9bd154, 0x36243a2e
+	.long		0x0a0cb167, 0x57930fe7, 0xeeb4d296, 0x9b1b9e91
+	.long		0xc0804fc5, 0xdc61a220, 0x775a694b, 0x121c161a
+	.long		0x93e20aba, 0xa0c0e52a, 0x223c43e0, 0x1b121d17
+	.long		0x090e0b0d, 0x8bf2adc7, 0xb62db9a8, 0x1e14c8a9
+	.long		0xf1578519, 0x75af4c07, 0x99eebbdd, 0x7fa3fd60
+	.long		0x01f79f26, 0x725cbcf5, 0x6644c53b, 0xfb5b347e
+	.long		0x438b7629, 0x23cbdcc6, 0xedb668fc, 0xe4b863f1
+	.long		0x31d7cadc, 0x63421085, 0x97134022, 0xc6842011
+	.long		0x4a857d24, 0xbbd2f83d, 0xf9ae1132, 0x29c76da1
+	.long		0x9e1d4b2f, 0xb2dcf330, 0x860dec52, 0xc177d0e3
+	.long		0xb32b6c16, 0x70a999b9, 0x9411fa48, 0xe9472264
+	.long		0xfca8c48c, 0xf0a01a3f, 0x7d56d82c, 0x3322ef90
+	.long		0x4987c74e, 0x38d9c1d1, 0xca8cfea2, 0xd498360b
+	.long		0xf5a6cf81, 0x7aa528de, 0xb7da268e, 0xad3fa4bf
+	.long		0x3a2ce49d, 0x78500d92, 0x5f6a9bcc, 0x7e546246
+	.long		0x8df6c213, 0xd890e8b8, 0x392e5ef7, 0xc382f5af
+	.long		0x5d9fbe80, 0xd0697c93, 0xd56fa92d, 0x25cfb312
+	.long		0xacc83b99, 0x1810a77d, 0x9ce86e63, 0x3bdb7bbb
+	.long		0x26cd0978, 0x596ef418, 0x9aec01b7, 0x4f83a89a
+	.long		0x95e6656e, 0xffaa7ee6, 0xbc2108cf, 0x15efe6e8
+	.long		0xe7bad99b, 0x6f4ace36, 0x9fead409, 0xb029d67c
+	.long		0xa431afb2, 0x3f2a3123, 0xa5c63094, 0xa235c066
+	.long		0x4e7437bc, 0x82fca6ca, 0x90e0b0d0, 0xa73315d8
+	.long		0x04f14a98, 0xec41f7da, 0xcd7f0e50, 0x91172ff6
+	.long		0x4d768dd6, 0xef434db0, 0xaacc544d, 0x96e4df04
+	.long		0xd19ee3b5, 0x6a4c1b88, 0x2cc1b81f, 0x65467f51
+	.long		0x5e9d04ea, 0x8c015d35, 0x87fa7374, 0x0bfb2e41
+	.long		0x67b35a1d, 0xdb9252d2, 0x10e93356, 0xd66d1347
+	.long		0xd79a8c61, 0xa1377a0c, 0xf8598e14, 0x13eb893c
+	.long		0xa9ceee27, 0x61b735c9, 0x1ce1ede5, 0x477a3cb1
+	.long		0xd29c59df, 0xf2553f73, 0x141879ce, 0xc773bf37
+	.long		0xf753eacd, 0xfd5f5baa, 0x3ddf146f, 0x447886db
+	.long		0xafca81f3, 0x68b93ec4, 0x24382c34, 0xa3c25f40
+	.long		0x1d1672c3, 0xe2bc0c25, 0x3c288b49, 0x0dff4195
+	.long		0xa8397101, 0x0c08deb3, 0xb4d89ce4, 0x566490c1
+	.long		0xcb7b6184, 0x32d570b6, 0x6c48745c, 0xb8d04257
+
+	.long		0x5150a7f4, 0x7e536541, 0x1ac3a417, 0x3a965e27
+	.long		0x3bcb6bab, 0x1ff1459d, 0xacab58fa, 0x4b9303e3
+	.long		0x2055fa30, 0xadf66d76, 0x889176cc, 0xf5254c02
+	.long		0x4ffcd7e5, 0xc5d7cb2a, 0x26804435, 0xb58fa362
+	.long		0xde495ab1, 0x25671bba, 0x45980eea, 0x5de1c0fe
+	.long		0xc302752f, 0x8112f04c, 0x8da39746, 0x6bc6f9d3
+	.long		0x03e75f8f, 0x15959c92, 0xbfeb7a6d, 0x95da5952
+	.long		0xd42d83be, 0x58d32174, 0x492969e0, 0x8e44c8c9
+	.long		0x756a89c2, 0xf478798e, 0x996b3e58, 0x27dd71b9
+	.long		0xbeb64fe1, 0xf017ad88, 0xc966ac20, 0x7db43ace
+	.long		0x63184adf, 0xe582311a, 0x97603351, 0x62457f53
+	.long		0xb1e07764, 0xbb84ae6b, 0xfe1ca081, 0xf9942b08
+	.long		0x70586848, 0x8f19fd45, 0x94876cde, 0x52b7f87b
+	.long		0xab23d373, 0x72e2024b, 0xe3578f1f, 0x662aab55
+	.long		0xb20728eb, 0x2f03c2b5, 0x869a7bc5, 0xd3a50837
+	.long		0x30f28728, 0x23b2a5bf, 0x02ba6a03, 0xed5c8216
+	.long		0x8a2b1ccf, 0xa792b479, 0xf3f0f207, 0x4ea1e269
+	.long		0x65cdf4da, 0x06d5be05, 0xd11f6234, 0xc48afea6
+	.long		0x349d532e, 0xa2a055f3, 0x0532e18a, 0xa475ebf6
+	.long		0x0b39ec83, 0x40aaef60, 0x5e069f71, 0xbd51106e
+	.long		0x3ef98a21, 0x963d06dd, 0xddae053e, 0x4d46bde6
+	.long		0x91b58d54, 0x71055dc4, 0x046fd406, 0x60ff1550
+	.long		0x1924fb98, 0xd697e9bd, 0x89cc4340, 0x67779ed9
+	.long		0xb0bd42e8, 0x07888b89, 0xe7385b19, 0x79dbeec8
+	.long		0xa1470a7c, 0x7ce90f42, 0xf8c91e84, 0x00000000
+	.long		0x09838680, 0x3248ed2b, 0x1eac7011, 0x6c4e725a
+	.long		0xfdfbff0e, 0x0f563885, 0x3d1ed5ae, 0x3627392d
+	.long		0x0a64d90f, 0x6821a65c, 0x9bd1545b, 0x243a2e36
+	.long		0x0cb1670a, 0x930fe757, 0xb4d296ee, 0x1b9e919b
+	.long		0x804fc5c0, 0x61a220dc, 0x5a694b77, 0x1c161a12
+	.long		0xe20aba93, 0xc0e52aa0, 0x3c43e022, 0x121d171b
+	.long		0x0e0b0d09, 0xf2adc78b, 0x2db9a8b6, 0x14c8a91e
+	.long		0x578519f1, 0xaf4c0775, 0xeebbdd99, 0xa3fd607f
+	.long		0xf79f2601, 0x5cbcf572, 0x44c53b66, 0x5b347efb
+	.long		0x8b762943, 0xcbdcc623, 0xb668fced, 0xb863f1e4
+	.long		0xd7cadc31, 0x42108563, 0x13402297, 0x842011c6
+	.long		0x857d244a, 0xd2f83dbb, 0xae1132f9, 0xc76da129
+	.long		0x1d4b2f9e, 0xdcf330b2, 0x0dec5286, 0x77d0e3c1
+	.long		0x2b6c16b3, 0xa999b970, 0x11fa4894, 0x472264e9
+	.long		0xa8c48cfc, 0xa01a3ff0, 0x56d82c7d, 0x22ef9033
+	.long		0x87c74e49, 0xd9c1d138, 0x8cfea2ca, 0x98360bd4
+	.long		0xa6cf81f5, 0xa528de7a, 0xda268eb7, 0x3fa4bfad
+	.long		0x2ce49d3a, 0x500d9278, 0x6a9bcc5f, 0x5462467e
+	.long		0xf6c2138d, 0x90e8b8d8, 0x2e5ef739, 0x82f5afc3
+	.long		0x9fbe805d, 0x697c93d0, 0x6fa92dd5, 0xcfb31225
+	.long		0xc83b99ac, 0x10a77d18, 0xe86e639c, 0xdb7bbb3b
+	.long		0xcd097826, 0x6ef41859, 0xec01b79a, 0x83a89a4f
+	.long		0xe6656e95, 0xaa7ee6ff, 0x2108cfbc, 0xefe6e815
+	.long		0xbad99be7, 0x4ace366f, 0xead4099f, 0x29d67cb0
+	.long		0x31afb2a4, 0x2a31233f, 0xc63094a5, 0x35c066a2
+	.long		0x7437bc4e, 0xfca6ca82, 0xe0b0d090, 0x3315d8a7
+	.long		0xf14a9804, 0x41f7daec, 0x7f0e50cd, 0x172ff691
+	.long		0x768dd64d, 0x434db0ef, 0xcc544daa, 0xe4df0496
+	.long		0x9ee3b5d1, 0x4c1b886a, 0xc1b81f2c, 0x467f5165
+	.long		0x9d04ea5e, 0x015d358c, 0xfa737487, 0xfb2e410b
+	.long		0xb35a1d67, 0x9252d2db, 0xe9335610, 0x6d1347d6
+	.long		0x9a8c61d7, 0x377a0ca1, 0x598e14f8, 0xeb893c13
+	.long		0xceee27a9, 0xb735c961, 0xe1ede51c, 0x7a3cb147
+	.long		0x9c59dfd2, 0x553f73f2, 0x1879ce14, 0x73bf37c7
+	.long		0x53eacdf7, 0x5f5baafd, 0xdf146f3d, 0x7886db44
+	.long		0xca81f3af, 0xb93ec468, 0x382c3424, 0xc25f40a3
+	.long		0x1672c31d, 0xbc0c25e2, 0x288b493c, 0xff41950d
+	.long		0x397101a8, 0x08deb30c, 0xd89ce4b4, 0x6490c156
+	.long		0x7b6184cb, 0xd570b632, 0x48745c6c, 0xd04257b8
+	.endif
+	.endm
+
+	.macro		__crypto_il_tab, full=1
+	.long		0x00000052, 0x00000009, 0x0000006a, 0x000000d5
+	.long		0x00000030, 0x00000036, 0x000000a5, 0x00000038
+	.long		0x000000bf, 0x00000040, 0x000000a3, 0x0000009e
+	.long		0x00000081, 0x000000f3, 0x000000d7, 0x000000fb
+	.long		0x0000007c, 0x000000e3, 0x00000039, 0x00000082
+	.long		0x0000009b, 0x0000002f, 0x000000ff, 0x00000087
+	.long		0x00000034, 0x0000008e, 0x00000043, 0x00000044
+	.long		0x000000c4, 0x000000de, 0x000000e9, 0x000000cb
+	.long		0x00000054, 0x0000007b, 0x00000094, 0x00000032
+	.long		0x000000a6, 0x000000c2, 0x00000023, 0x0000003d
+	.long		0x000000ee, 0x0000004c, 0x00000095, 0x0000000b
+	.long		0x00000042, 0x000000fa, 0x000000c3, 0x0000004e
+	.long		0x00000008, 0x0000002e, 0x000000a1, 0x00000066
+	.long		0x00000028, 0x000000d9, 0x00000024, 0x000000b2
+	.long		0x00000076, 0x0000005b, 0x000000a2, 0x00000049
+	.long		0x0000006d, 0x0000008b, 0x000000d1, 0x00000025
+	.long		0x00000072, 0x000000f8, 0x000000f6, 0x00000064
+	.long		0x00000086, 0x00000068, 0x00000098, 0x00000016
+	.long		0x000000d4, 0x000000a4, 0x0000005c, 0x000000cc
+	.long		0x0000005d, 0x00000065, 0x000000b6, 0x00000092
+	.long		0x0000006c, 0x00000070, 0x00000048, 0x00000050
+	.long		0x000000fd, 0x000000ed, 0x000000b9, 0x000000da
+	.long		0x0000005e, 0x00000015, 0x00000046, 0x00000057
+	.long		0x000000a7, 0x0000008d, 0x0000009d, 0x00000084
+	.long		0x00000090, 0x000000d8, 0x000000ab, 0x00000000
+	.long		0x0000008c, 0x000000bc, 0x000000d3, 0x0000000a
+	.long		0x000000f7, 0x000000e4, 0x00000058, 0x00000005
+	.long		0x000000b8, 0x000000b3, 0x00000045, 0x00000006
+	.long		0x000000d0, 0x0000002c, 0x0000001e, 0x0000008f
+	.long		0x000000ca, 0x0000003f, 0x0000000f, 0x00000002
+	.long		0x000000c1, 0x000000af, 0x000000bd, 0x00000003
+	.long		0x00000001, 0x00000013, 0x0000008a, 0x0000006b
+	.long		0x0000003a, 0x00000091, 0x00000011, 0x00000041
+	.long		0x0000004f, 0x00000067, 0x000000dc, 0x000000ea
+	.long		0x00000097, 0x000000f2, 0x000000cf, 0x000000ce
+	.long		0x000000f0, 0x000000b4, 0x000000e6, 0x00000073
+	.long		0x00000096, 0x000000ac, 0x00000074, 0x00000022
+	.long		0x000000e7, 0x000000ad, 0x00000035, 0x00000085
+	.long		0x000000e2, 0x000000f9, 0x00000037, 0x000000e8
+	.long		0x0000001c, 0x00000075, 0x000000df, 0x0000006e
+	.long		0x00000047, 0x000000f1, 0x0000001a, 0x00000071
+	.long		0x0000001d, 0x00000029, 0x000000c5, 0x00000089
+	.long		0x0000006f, 0x000000b7, 0x00000062, 0x0000000e
+	.long		0x000000aa, 0x00000018, 0x000000be, 0x0000001b
+	.long		0x000000fc, 0x00000056, 0x0000003e, 0x0000004b
+	.long		0x000000c6, 0x000000d2, 0x00000079, 0x00000020
+	.long		0x0000009a, 0x000000db, 0x000000c0, 0x000000fe
+	.long		0x00000078, 0x000000cd, 0x0000005a, 0x000000f4
+	.long		0x0000001f, 0x000000dd, 0x000000a8, 0x00000033
+	.long		0x00000088, 0x00000007, 0x000000c7, 0x00000031
+	.long		0x000000b1, 0x00000012, 0x00000010, 0x00000059
+	.long		0x00000027, 0x00000080, 0x000000ec, 0x0000005f
+	.long		0x00000060, 0x00000051, 0x0000007f, 0x000000a9
+	.long		0x00000019, 0x000000b5, 0x0000004a, 0x0000000d
+	.long		0x0000002d, 0x000000e5, 0x0000007a, 0x0000009f
+	.long		0x00000093, 0x000000c9, 0x0000009c, 0x000000ef
+	.long		0x000000a0, 0x000000e0, 0x0000003b, 0x0000004d
+	.long		0x000000ae, 0x0000002a, 0x000000f5, 0x000000b0
+	.long		0x000000c8, 0x000000eb, 0x000000bb, 0x0000003c
+	.long		0x00000083, 0x00000053, 0x00000099, 0x00000061
+	.long		0x00000017, 0x0000002b, 0x00000004, 0x0000007e
+	.long		0x000000ba, 0x00000077, 0x000000d6, 0x00000026
+	.long		0x000000e1, 0x00000069, 0x00000014, 0x00000063
+	.long		0x00000055, 0x00000021, 0x0000000c, 0x0000007d
+
+	.if		\full == 1
+	.long		0x00005200, 0x00000900, 0x00006a00, 0x0000d500
+	.long		0x00003000, 0x00003600, 0x0000a500, 0x00003800
+	.long		0x0000bf00, 0x00004000, 0x0000a300, 0x00009e00
+	.long		0x00008100, 0x0000f300, 0x0000d700, 0x0000fb00
+	.long		0x00007c00, 0x0000e300, 0x00003900, 0x00008200
+	.long		0x00009b00, 0x00002f00, 0x0000ff00, 0x00008700
+	.long		0x00003400, 0x00008e00, 0x00004300, 0x00004400
+	.long		0x0000c400, 0x0000de00, 0x0000e900, 0x0000cb00
+	.long		0x00005400, 0x00007b00, 0x00009400, 0x00003200
+	.long		0x0000a600, 0x0000c200, 0x00002300, 0x00003d00
+	.long		0x0000ee00, 0x00004c00, 0x00009500, 0x00000b00
+	.long		0x00004200, 0x0000fa00, 0x0000c300, 0x00004e00
+	.long		0x00000800, 0x00002e00, 0x0000a100, 0x00006600
+	.long		0x00002800, 0x0000d900, 0x00002400, 0x0000b200
+	.long		0x00007600, 0x00005b00, 0x0000a200, 0x00004900
+	.long		0x00006d00, 0x00008b00, 0x0000d100, 0x00002500
+	.long		0x00007200, 0x0000f800, 0x0000f600, 0x00006400
+	.long		0x00008600, 0x00006800, 0x00009800, 0x00001600
+	.long		0x0000d400, 0x0000a400, 0x00005c00, 0x0000cc00
+	.long		0x00005d00, 0x00006500, 0x0000b600, 0x00009200
+	.long		0x00006c00, 0x00007000, 0x00004800, 0x00005000
+	.long		0x0000fd00, 0x0000ed00, 0x0000b900, 0x0000da00
+	.long		0x00005e00, 0x00001500, 0x00004600, 0x00005700
+	.long		0x0000a700, 0x00008d00, 0x00009d00, 0x00008400
+	.long		0x00009000, 0x0000d800, 0x0000ab00, 0x00000000
+	.long		0x00008c00, 0x0000bc00, 0x0000d300, 0x00000a00
+	.long		0x0000f700, 0x0000e400, 0x00005800, 0x00000500
+	.long		0x0000b800, 0x0000b300, 0x00004500, 0x00000600
+	.long		0x0000d000, 0x00002c00, 0x00001e00, 0x00008f00
+	.long		0x0000ca00, 0x00003f00, 0x00000f00, 0x00000200
+	.long		0x0000c100, 0x0000af00, 0x0000bd00, 0x00000300
+	.long		0x00000100, 0x00001300, 0x00008a00, 0x00006b00
+	.long		0x00003a00, 0x00009100, 0x00001100, 0x00004100
+	.long		0x00004f00, 0x00006700, 0x0000dc00, 0x0000ea00
+	.long		0x00009700, 0x0000f200, 0x0000cf00, 0x0000ce00
+	.long		0x0000f000, 0x0000b400, 0x0000e600, 0x00007300
+	.long		0x00009600, 0x0000ac00, 0x00007400, 0x00002200
+	.long		0x0000e700, 0x0000ad00, 0x00003500, 0x00008500
+	.long		0x0000e200, 0x0000f900, 0x00003700, 0x0000e800
+	.long		0x00001c00, 0x00007500, 0x0000df00, 0x00006e00
+	.long		0x00004700, 0x0000f100, 0x00001a00, 0x00007100
+	.long		0x00001d00, 0x00002900, 0x0000c500, 0x00008900
+	.long		0x00006f00, 0x0000b700, 0x00006200, 0x00000e00
+	.long		0x0000aa00, 0x00001800, 0x0000be00, 0x00001b00
+	.long		0x0000fc00, 0x00005600, 0x00003e00, 0x00004b00
+	.long		0x0000c600, 0x0000d200, 0x00007900, 0x00002000
+	.long		0x00009a00, 0x0000db00, 0x0000c000, 0x0000fe00
+	.long		0x00007800, 0x0000cd00, 0x00005a00, 0x0000f400
+	.long		0x00001f00, 0x0000dd00, 0x0000a800, 0x00003300
+	.long		0x00008800, 0x00000700, 0x0000c700, 0x00003100
+	.long		0x0000b100, 0x00001200, 0x00001000, 0x00005900
+	.long		0x00002700, 0x00008000, 0x0000ec00, 0x00005f00
+	.long		0x00006000, 0x00005100, 0x00007f00, 0x0000a900
+	.long		0x00001900, 0x0000b500, 0x00004a00, 0x00000d00
+	.long		0x00002d00, 0x0000e500, 0x00007a00, 0x00009f00
+	.long		0x00009300, 0x0000c900, 0x00009c00, 0x0000ef00
+	.long		0x0000a000, 0x0000e000, 0x00003b00, 0x00004d00
+	.long		0x0000ae00, 0x00002a00, 0x0000f500, 0x0000b000
+	.long		0x0000c800, 0x0000eb00, 0x0000bb00, 0x00003c00
+	.long		0x00008300, 0x00005300, 0x00009900, 0x00006100
+	.long		0x00001700, 0x00002b00, 0x00000400, 0x00007e00
+	.long		0x0000ba00, 0x00007700, 0x0000d600, 0x00002600
+	.long		0x0000e100, 0x00006900, 0x00001400, 0x00006300
+	.long		0x00005500, 0x00002100, 0x00000c00, 0x00007d00
+
+	.long		0x00520000, 0x00090000, 0x006a0000, 0x00d50000
+	.long		0x00300000, 0x00360000, 0x00a50000, 0x00380000
+	.long		0x00bf0000, 0x00400000, 0x00a30000, 0x009e0000
+	.long		0x00810000, 0x00f30000, 0x00d70000, 0x00fb0000
+	.long		0x007c0000, 0x00e30000, 0x00390000, 0x00820000
+	.long		0x009b0000, 0x002f0000, 0x00ff0000, 0x00870000
+	.long		0x00340000, 0x008e0000, 0x00430000, 0x00440000
+	.long		0x00c40000, 0x00de0000, 0x00e90000, 0x00cb0000
+	.long		0x00540000, 0x007b0000, 0x00940000, 0x00320000
+	.long		0x00a60000, 0x00c20000, 0x00230000, 0x003d0000
+	.long		0x00ee0000, 0x004c0000, 0x00950000, 0x000b0000
+	.long		0x00420000, 0x00fa0000, 0x00c30000, 0x004e0000
+	.long		0x00080000, 0x002e0000, 0x00a10000, 0x00660000
+	.long		0x00280000, 0x00d90000, 0x00240000, 0x00b20000
+	.long		0x00760000, 0x005b0000, 0x00a20000, 0x00490000
+	.long		0x006d0000, 0x008b0000, 0x00d10000, 0x00250000
+	.long		0x00720000, 0x00f80000, 0x00f60000, 0x00640000
+	.long		0x00860000, 0x00680000, 0x00980000, 0x00160000
+	.long		0x00d40000, 0x00a40000, 0x005c0000, 0x00cc0000
+	.long		0x005d0000, 0x00650000, 0x00b60000, 0x00920000
+	.long		0x006c0000, 0x00700000, 0x00480000, 0x00500000
+	.long		0x00fd0000, 0x00ed0000, 0x00b90000, 0x00da0000
+	.long		0x005e0000, 0x00150000, 0x00460000, 0x00570000
+	.long		0x00a70000, 0x008d0000, 0x009d0000, 0x00840000
+	.long		0x00900000, 0x00d80000, 0x00ab0000, 0x00000000
+	.long		0x008c0000, 0x00bc0000, 0x00d30000, 0x000a0000
+	.long		0x00f70000, 0x00e40000, 0x00580000, 0x00050000
+	.long		0x00b80000, 0x00b30000, 0x00450000, 0x00060000
+	.long		0x00d00000, 0x002c0000, 0x001e0000, 0x008f0000
+	.long		0x00ca0000, 0x003f0000, 0x000f0000, 0x00020000
+	.long		0x00c10000, 0x00af0000, 0x00bd0000, 0x00030000
+	.long		0x00010000, 0x00130000, 0x008a0000, 0x006b0000
+	.long		0x003a0000, 0x00910000, 0x00110000, 0x00410000
+	.long		0x004f0000, 0x00670000, 0x00dc0000, 0x00ea0000
+	.long		0x00970000, 0x00f20000, 0x00cf0000, 0x00ce0000
+	.long		0x00f00000, 0x00b40000, 0x00e60000, 0x00730000
+	.long		0x00960000, 0x00ac0000, 0x00740000, 0x00220000
+	.long		0x00e70000, 0x00ad0000, 0x00350000, 0x00850000
+	.long		0x00e20000, 0x00f90000, 0x00370000, 0x00e80000
+	.long		0x001c0000, 0x00750000, 0x00df0000, 0x006e0000
+	.long		0x00470000, 0x00f10000, 0x001a0000, 0x00710000
+	.long		0x001d0000, 0x00290000, 0x00c50000, 0x00890000
+	.long		0x006f0000, 0x00b70000, 0x00620000, 0x000e0000
+	.long		0x00aa0000, 0x00180000, 0x00be0000, 0x001b0000
+	.long		0x00fc0000, 0x00560000, 0x003e0000, 0x004b0000
+	.long		0x00c60000, 0x00d20000, 0x00790000, 0x00200000
+	.long		0x009a0000, 0x00db0000, 0x00c00000, 0x00fe0000
+	.long		0x00780000, 0x00cd0000, 0x005a0000, 0x00f40000
+	.long		0x001f0000, 0x00dd0000, 0x00a80000, 0x00330000
+	.long		0x00880000, 0x00070000, 0x00c70000, 0x00310000
+	.long		0x00b10000, 0x00120000, 0x00100000, 0x00590000
+	.long		0x00270000, 0x00800000, 0x00ec0000, 0x005f0000
+	.long		0x00600000, 0x00510000, 0x007f0000, 0x00a90000
+	.long		0x00190000, 0x00b50000, 0x004a0000, 0x000d0000
+	.long		0x002d0000, 0x00e50000, 0x007a0000, 0x009f0000
+	.long		0x00930000, 0x00c90000, 0x009c0000, 0x00ef0000
+	.long		0x00a00000, 0x00e00000, 0x003b0000, 0x004d0000
+	.long		0x00ae0000, 0x002a0000, 0x00f50000, 0x00b00000
+	.long		0x00c80000, 0x00eb0000, 0x00bb0000, 0x003c0000
+	.long		0x00830000, 0x00530000, 0x00990000, 0x00610000
+	.long		0x00170000, 0x002b0000, 0x00040000, 0x007e0000
+	.long		0x00ba0000, 0x00770000, 0x00d60000, 0x00260000
+	.long		0x00e10000, 0x00690000, 0x00140000, 0x00630000
+	.long		0x00550000, 0x00210000, 0x000c0000, 0x007d0000
+
+	.long		0x52000000, 0x09000000, 0x6a000000, 0xd5000000
+	.long		0x30000000, 0x36000000, 0xa5000000, 0x38000000
+	.long		0xbf000000, 0x40000000, 0xa3000000, 0x9e000000
+	.long		0x81000000, 0xf3000000, 0xd7000000, 0xfb000000
+	.long		0x7c000000, 0xe3000000, 0x39000000, 0x82000000
+	.long		0x9b000000, 0x2f000000, 0xff000000, 0x87000000
+	.long		0x34000000, 0x8e000000, 0x43000000, 0x44000000
+	.long		0xc4000000, 0xde000000, 0xe9000000, 0xcb000000
+	.long		0x54000000, 0x7b000000, 0x94000000, 0x32000000
+	.long		0xa6000000, 0xc2000000, 0x23000000, 0x3d000000
+	.long		0xee000000, 0x4c000000, 0x95000000, 0x0b000000
+	.long		0x42000000, 0xfa000000, 0xc3000000, 0x4e000000
+	.long		0x08000000, 0x2e000000, 0xa1000000, 0x66000000
+	.long		0x28000000, 0xd9000000, 0x24000000, 0xb2000000
+	.long		0x76000000, 0x5b000000, 0xa2000000, 0x49000000
+	.long		0x6d000000, 0x8b000000, 0xd1000000, 0x25000000
+	.long		0x72000000, 0xf8000000, 0xf6000000, 0x64000000
+	.long		0x86000000, 0x68000000, 0x98000000, 0x16000000
+	.long		0xd4000000, 0xa4000000, 0x5c000000, 0xcc000000
+	.long		0x5d000000, 0x65000000, 0xb6000000, 0x92000000
+	.long		0x6c000000, 0x70000000, 0x48000000, 0x50000000
+	.long		0xfd000000, 0xed000000, 0xb9000000, 0xda000000
+	.long		0x5e000000, 0x15000000, 0x46000000, 0x57000000
+	.long		0xa7000000, 0x8d000000, 0x9d000000, 0x84000000
+	.long		0x90000000, 0xd8000000, 0xab000000, 0x00000000
+	.long		0x8c000000, 0xbc000000, 0xd3000000, 0x0a000000
+	.long		0xf7000000, 0xe4000000, 0x58000000, 0x05000000
+	.long		0xb8000000, 0xb3000000, 0x45000000, 0x06000000
+	.long		0xd0000000, 0x2c000000, 0x1e000000, 0x8f000000
+	.long		0xca000000, 0x3f000000, 0x0f000000, 0x02000000
+	.long		0xc1000000, 0xaf000000, 0xbd000000, 0x03000000
+	.long		0x01000000, 0x13000000, 0x8a000000, 0x6b000000
+	.long		0x3a000000, 0x91000000, 0x11000000, 0x41000000
+	.long		0x4f000000, 0x67000000, 0xdc000000, 0xea000000
+	.long		0x97000000, 0xf2000000, 0xcf000000, 0xce000000
+	.long		0xf0000000, 0xb4000000, 0xe6000000, 0x73000000
+	.long		0x96000000, 0xac000000, 0x74000000, 0x22000000
+	.long		0xe7000000, 0xad000000, 0x35000000, 0x85000000
+	.long		0xe2000000, 0xf9000000, 0x37000000, 0xe8000000
+	.long		0x1c000000, 0x75000000, 0xdf000000, 0x6e000000
+	.long		0x47000000, 0xf1000000, 0x1a000000, 0x71000000
+	.long		0x1d000000, 0x29000000, 0xc5000000, 0x89000000
+	.long		0x6f000000, 0xb7000000, 0x62000000, 0x0e000000
+	.long		0xaa000000, 0x18000000, 0xbe000000, 0x1b000000
+	.long		0xfc000000, 0x56000000, 0x3e000000, 0x4b000000
+	.long		0xc6000000, 0xd2000000, 0x79000000, 0x20000000
+	.long		0x9a000000, 0xdb000000, 0xc0000000, 0xfe000000
+	.long		0x78000000, 0xcd000000, 0x5a000000, 0xf4000000
+	.long		0x1f000000, 0xdd000000, 0xa8000000, 0x33000000
+	.long		0x88000000, 0x07000000, 0xc7000000, 0x31000000
+	.long		0xb1000000, 0x12000000, 0x10000000, 0x59000000
+	.long		0x27000000, 0x80000000, 0xec000000, 0x5f000000
+	.long		0x60000000, 0x51000000, 0x7f000000, 0xa9000000
+	.long		0x19000000, 0xb5000000, 0x4a000000, 0x0d000000
+	.long		0x2d000000, 0xe5000000, 0x7a000000, 0x9f000000
+	.long		0x93000000, 0xc9000000, 0x9c000000, 0xef000000
+	.long		0xa0000000, 0xe0000000, 0x3b000000, 0x4d000000
+	.long		0xae000000, 0x2a000000, 0xf5000000, 0xb0000000
+	.long		0xc8000000, 0xeb000000, 0xbb000000, 0x3c000000
+	.long		0x83000000, 0x53000000, 0x99000000, 0x61000000
+	.long		0x17000000, 0x2b000000, 0x04000000, 0x7e000000
+	.long		0xba000000, 0x77000000, 0xd6000000, 0x26000000
+	.long		0xe1000000, 0x69000000, 0x14000000, 0x63000000
+	.long		0x55000000, 0x21000000, 0x0c000000, 0x7d000000
+	.endif
+	.endm
+
diff --git a/include/crypto/aes.h b/include/crypto/aes.h
index 7524ba3b6f3c..8a4afdca611e 100644
--- a/include/crypto/aes.h
+++ b/include/crypto/aes.h
@@ -27,11 +27,6 @@ struct crypto_aes_ctx {
 	u32 key_length;
 };
 
-extern const u32 crypto_ft_tab[4][256];
-extern const u32 crypto_fl_tab[4][256];
-extern const u32 crypto_it_tab[4][256];
-extern const u32 crypto_il_tab[4][256];
-
 int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key,
 		unsigned int key_len);
 int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 3/8] crypto: aes - retire table based generic AES in favor of fixed time driver
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 1/8] drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 2/8] crypto - aes: use dedicated lookup tables for table based asm routines Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 4/8] crypto: x86/aes-ni - switch to generic fallback Ard Biesheuvel
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

Rework the fixed time AES code so that it can fulfil dependencies of other
drivers on the shared AES key expansion routines. This way, we can remove
the table based generic AES code altogether, and use the much smaller and
time invariant fixed time driver as the global default for systems that
don't have an architecture specific accelerated implementation of the
cipher.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 crypto/Kconfig                     |   31 +-
 crypto/Makefile                    |    3 +-
 crypto/{aes_ti.c => aes.c}         |  169 ++-
 crypto/aes_generic.c               | 1478 --------------------
 drivers/crypto/chelsio/chcr_algo.c |    4 +-
 include/crypto/aes.h               |    6 +
 6 files changed, 121 insertions(+), 1570 deletions(-)

diff --git a/crypto/Kconfig b/crypto/Kconfig
index caa770e535a2..7766fea9c18e 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -895,35 +895,12 @@ config CRYPTO_GHASH_CLMUL_NI_INTEL
 comment "Ciphers"
 
 config CRYPTO_AES
-	tristate "AES cipher algorithms"
+	tristate "Generic AES cipher (fixed time)"
 	select CRYPTO_ALGAPI
 	help
-	  AES cipher algorithms (FIPS-197). AES uses the Rijndael
-	  algorithm.
-
-	  Rijndael appears to be consistently a very good performer in
-	  both hardware and software across a wide range of computing
-	  environments regardless of its use in feedback or non-feedback
-	  modes. Its key setup time is excellent, and its key agility is
-	  good. Rijndael's very low memory requirements make it very well
-	  suited for restricted-space environments, in which it also
-	  demonstrates excellent performance. Rijndael's operations are
-	  among the easiest to defend against power and timing attacks.
-
-	  The AES specifies three key sizes: 128, 192 and 256 bits
-
-	  See <http://csrc.nist.gov/CryptoToolkit/aes/> for more information.
-
-config CRYPTO_AES_TI
-	tristate "Fixed time AES cipher"
-	select CRYPTO_ALGAPI
-	help
-	  This is a generic implementation of AES that attempts to eliminate
-	  data dependent latencies as much as possible without affecting
-	  performance too much. It is intended for use by the generic CCM
-	  and GCM drivers, and other CTR or CMAC/XCBC based modes that rely
-	  solely on encryption (although decryption is supported as well, but
-	  with a more dramatic performance hit)
+	  This is a generic implementation of AES that was designed to be
+	  small (in terms of code size and D-cache footprint) and time
+	  invariant, with reasonable performance.
 
 	  Instead of using 16 lookup tables of 1 KB each, (8 for encryption and
 	  8 for decryption), this implementation only uses just two S-boxes of
diff --git a/crypto/Makefile b/crypto/Makefile
index d41f0331b085..6163d47b3e12 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -96,8 +96,7 @@ obj-$(CONFIG_CRYPTO_TWOFISH) += twofish_generic.o
 obj-$(CONFIG_CRYPTO_TWOFISH_COMMON) += twofish_common.o
 obj-$(CONFIG_CRYPTO_SERPENT) += serpent_generic.o
 CFLAGS_serpent_generic.o := $(call cc-option,-fsched-pressure)  # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149
-obj-$(CONFIG_CRYPTO_AES) += aes_generic.o
-obj-$(CONFIG_CRYPTO_AES_TI) += aes_ti.o
+obj-$(CONFIG_CRYPTO_AES) += aes.o
 obj-$(CONFIG_CRYPTO_CAMELLIA) += camellia_generic.o
 obj-$(CONFIG_CRYPTO_CAST_COMMON) += cast_common.o
 obj-$(CONFIG_CRYPTO_CAST5) += cast5_generic.o
diff --git a/crypto/aes_ti.c b/crypto/aes.c
similarity index 76%
rename from crypto/aes_ti.c
rename to crypto/aes.c
index 03023b2290e8..1c246274bfa3 100644
--- a/crypto/aes_ti.c
+++ b/crypto/aes.c
@@ -13,11 +13,7 @@
 #include <linux/module.h>
 #include <asm/unaligned.h>
 
-/*
- * Emit the sbox as volatile const to prevent the compiler from doing
- * constant folding on sbox references involving fixed indexes.
- */
-static volatile const u8 __cacheline_aligned __aesti_sbox[] = {
+static volatile const u8 __cacheline_aligned aes_sbox[] = {
 	0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5,
 	0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76,
 	0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0,
@@ -52,7 +48,7 @@ static volatile const u8 __cacheline_aligned __aesti_sbox[] = {
 	0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16,
 };
 
-static volatile const u8 __cacheline_aligned __aesti_inv_sbox[] = {
+static volatile const u8 __cacheline_aligned aes_inv_sbox[] = {
 	0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38,
 	0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb,
 	0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87,
@@ -145,30 +141,30 @@ static u32 inv_mix_columns(u32 x)
 
 static __always_inline u32 subshift(u32 in[], int pos)
 {
-	return (__aesti_sbox[in[pos] & 0xff]) ^
-	       (__aesti_sbox[(in[(pos + 1) % 4] >>  8) & 0xff] <<  8) ^
-	       (__aesti_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^
-	       (__aesti_sbox[(in[(pos + 3) % 4] >> 24) & 0xff] << 24);
+	return (aes_sbox[in[pos] & 0xff]) ^
+	       (aes_sbox[(in[(pos + 1) % 4] >>  8) & 0xff] <<  8) ^
+	       (aes_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^
+	       (aes_sbox[(in[(pos + 3) % 4] >> 24) & 0xff] << 24);
 }
 
 static __always_inline u32 inv_subshift(u32 in[], int pos)
 {
-	return (__aesti_inv_sbox[in[pos] & 0xff]) ^
-	       (__aesti_inv_sbox[(in[(pos + 3) % 4] >>  8) & 0xff] <<  8) ^
-	       (__aesti_inv_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^
-	       (__aesti_inv_sbox[(in[(pos + 1) % 4] >> 24) & 0xff] << 24);
+	return (aes_inv_sbox[in[pos] & 0xff]) ^
+	       (aes_inv_sbox[(in[(pos + 3) % 4] >>  8) & 0xff] <<  8) ^
+	       (aes_inv_sbox[(in[(pos + 2) % 4] >> 16) & 0xff] << 16) ^
+	       (aes_inv_sbox[(in[(pos + 1) % 4] >> 24) & 0xff] << 24);
 }
 
 static u32 subw(u32 in)
 {
-	return (__aesti_sbox[in & 0xff]) ^
-	       (__aesti_sbox[(in >>  8) & 0xff] <<  8) ^
-	       (__aesti_sbox[(in >> 16) & 0xff] << 16) ^
-	       (__aesti_sbox[(in >> 24) & 0xff] << 24);
+	return (aes_sbox[in & 0xff]) ^
+	       (aes_sbox[(in >>  8) & 0xff] <<  8) ^
+	       (aes_sbox[(in >> 16) & 0xff] << 16) ^
+	       (aes_sbox[(in >> 24) & 0xff] << 24);
 }
 
-static int aesti_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
-			    unsigned int key_len)
+int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+			  unsigned int key_len)
 {
 	u32 kwords = key_len / sizeof(u32);
 	u32 rc, i, j;
@@ -192,12 +188,12 @@ static int aesti_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
 		rko[2] = rko[1] ^ rki[2];
 		rko[3] = rko[2] ^ rki[3];
 
-		if (key_len == 24) {
+		if (key_len == AES_KEYSIZE_192) {
 			if (i >= 7)
 				break;
 			rko[4] = rko[3] ^ rki[4];
 			rko[5] = rko[4] ^ rki[5];
-		} else if (key_len == 32) {
+		} else if (key_len == AES_KEYSIZE_256) {
 			if (i >= 6)
 				break;
 			rko[4] = subw(rko[3]) ^ rki[4];
@@ -232,40 +228,34 @@ static int aesti_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(crypto_aes_expand_key);
 
-static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key,
-			 unsigned int key_len)
+/**
+ * crypto_aes_set_key - Set the AES key.
+ * @tfm:	The %crypto_tfm that is used in the context.
+ * @in_key:	The input key.
+ * @key_len:	The size of the key.
+ *
+ * Returns 0 on success, on failure the %CRYPTO_TFM_RES_BAD_KEY_LEN flag in tfm
+ * is set. The function uses crypto_aes_expand_key() to expand the key.
+ * &crypto_aes_ctx _must_ be the private data embedded in @tfm which is
+ * retrieved with crypto_tfm_ctx().
+ */
+int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key,
+		unsigned int key_len)
 {
 	struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-	int err;
-
-	err = aesti_expand_key(ctx, in_key, key_len);
-	if (err)
-		return err;
-
-	/*
-	 * In order to force the compiler to emit data independent Sbox lookups
-	 * at the start of each block, xor the first round key with values at
-	 * fixed indexes in the Sbox. This will need to be repeated each time
-	 * the key is used, which will pull the entire Sbox into the D-cache
-	 * before any data dependent Sbox lookups are performed.
-	 */
-	ctx->key_enc[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[128];
-	ctx->key_enc[1] ^= __aesti_sbox[32] ^ __aesti_sbox[160];
-	ctx->key_enc[2] ^= __aesti_sbox[64] ^ __aesti_sbox[192];
-	ctx->key_enc[3] ^= __aesti_sbox[96] ^ __aesti_sbox[224];
-
-	ctx->key_dec[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[128];
-	ctx->key_dec[1] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[160];
-	ctx->key_dec[2] ^= __aesti_inv_sbox[64] ^ __aesti_inv_sbox[192];
-	ctx->key_dec[3] ^= __aesti_inv_sbox[96] ^ __aesti_inv_sbox[224];
 
+	if (crypto_aes_expand_key(ctx, in_key, key_len)) {
+		tfm->crt_flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
+		return -EINVAL;
+	}
 	return 0;
 }
+EXPORT_SYMBOL_GPL(crypto_aes_set_key);
 
-static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+void crypto_aes_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in)
 {
-	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
 	const u32 *rkp = ctx->key_enc + 4;
 	int rounds = 6 + ctx->key_length / 4;
 	u32 st0[4], st1[4];
@@ -276,11 +266,6 @@ static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
 	st0[2] = ctx->key_enc[2] ^ get_unaligned_le32(in + 8);
 	st0[3] = ctx->key_enc[3] ^ get_unaligned_le32(in + 12);
 
-	st0[0] ^= __aesti_sbox[ 0] ^ __aesti_sbox[128];
-	st0[1] ^= __aesti_sbox[32] ^ __aesti_sbox[160];
-	st0[2] ^= __aesti_sbox[64] ^ __aesti_sbox[192];
-	st0[3] ^= __aesti_sbox[96] ^ __aesti_sbox[224];
-
 	for (round = 0;; round += 2, rkp += 8) {
 		st1[0] = mix_columns(subshift(st0, 0)) ^ rkp[0];
 		st1[1] = mix_columns(subshift(st0, 1)) ^ rkp[1];
@@ -301,10 +286,10 @@ static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
 	put_unaligned_le32(subshift(st1, 2) ^ rkp[6], out + 8);
 	put_unaligned_le32(subshift(st1, 3) ^ rkp[7], out + 12);
 }
+EXPORT_SYMBOL_GPL(crypto_aes_encrypt);
 
-static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+void crypto_aes_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in)
 {
-	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
 	const u32 *rkp = ctx->key_dec + 4;
 	int rounds = 6 + ctx->key_length / 4;
 	u32 st0[4], st1[4];
@@ -315,11 +300,6 @@ static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
 	st0[2] = ctx->key_dec[2] ^ get_unaligned_le32(in + 8);
 	st0[3] = ctx->key_dec[3] ^ get_unaligned_le32(in + 12);
 
-	st0[0] ^= __aesti_inv_sbox[ 0] ^ __aesti_inv_sbox[128];
-	st0[1] ^= __aesti_inv_sbox[32] ^ __aesti_inv_sbox[160];
-	st0[2] ^= __aesti_inv_sbox[64] ^ __aesti_inv_sbox[192];
-	st0[3] ^= __aesti_inv_sbox[96] ^ __aesti_inv_sbox[224];
-
 	for (round = 0;; round += 2, rkp += 8) {
 		st1[0] = inv_mix_columns(inv_subshift(st0, 0)) ^ rkp[0];
 		st1[1] = inv_mix_columns(inv_subshift(st0, 1)) ^ rkp[1];
@@ -340,11 +320,72 @@ static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
 	put_unaligned_le32(inv_subshift(st1, 2) ^ rkp[6], out + 8);
 	put_unaligned_le32(inv_subshift(st1, 3) ^ rkp[7], out + 12);
 }
+EXPORT_SYMBOL_GPL(crypto_aes_decrypt);
+
+static int aesti_set_key(struct crypto_tfm *tfm, const u8 *in_key,
+			 unsigned int key_len)
+{
+	struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	int err;
+
+	err = crypto_aes_expand_key(ctx, in_key, key_len);
+	if (err)
+		return err;
+
+	/*
+	 * In order to force the compiler to emit data independent Sbox lookups
+	 * at the start of each block, xor the first round key with values at
+	 * fixed indexes in the Sbox. This will need to be repeated each time
+	 * the key is used, which will pull the entire Sbox into the D-cache
+	 * before any data dependent Sbox lookups are performed.
+	 */
+	ctx->key_enc[0] ^= aes_sbox[ 0] ^ aes_sbox[128];
+	ctx->key_enc[1] ^= aes_sbox[32] ^ aes_sbox[160];
+	ctx->key_enc[2] ^= aes_sbox[64] ^ aes_sbox[192];
+	ctx->key_enc[3] ^= aes_sbox[96] ^ aes_sbox[224];
+
+	ctx->key_dec[0] ^= aes_inv_sbox[ 0] ^ aes_inv_sbox[128];
+	ctx->key_dec[1] ^= aes_inv_sbox[32] ^ aes_inv_sbox[160];
+	ctx->key_dec[2] ^= aes_inv_sbox[64] ^ aes_inv_sbox[192];
+	ctx->key_dec[3] ^= aes_inv_sbox[96] ^ aes_inv_sbox[224];
+
+	return 0;
+}
+
+static void aesti_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+{
+	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	u8 src[AES_BLOCK_SIZE];
+
+	memcpy(src, in, AES_BLOCK_SIZE);
+
+	src[ 0] ^= aes_sbox[ 0] ^ aes_sbox[128];
+	src[ 4] ^= aes_sbox[32] ^ aes_sbox[160];
+	src[ 8] ^= aes_sbox[64] ^ aes_sbox[192];
+	src[12] ^= aes_sbox[96] ^ aes_sbox[224];
+
+	crypto_aes_encrypt(ctx, out, src);
+}
+
+static void aesti_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+{
+	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	u8 src[AES_BLOCK_SIZE];
+
+	memcpy(src, in, AES_BLOCK_SIZE);
+
+	src[ 0] ^= aes_inv_sbox[ 0] ^ aes_inv_sbox[128];
+	src[ 4] ^= aes_inv_sbox[32] ^ aes_inv_sbox[160];
+	src[ 8] ^= aes_inv_sbox[64] ^ aes_inv_sbox[192];
+	src[12] ^= aes_inv_sbox[96] ^ aes_inv_sbox[224];
+
+	crypto_aes_decrypt(ctx, out, src);
+}
 
 static struct crypto_alg aes_alg = {
 	.cra_name			= "aes",
 	.cra_driver_name		= "aes-fixed-time",
-	.cra_priority			= 100 + 1,
+	.cra_priority			= 100,
 	.cra_flags			= CRYPTO_ALG_TYPE_CIPHER,
 	.cra_blocksize			= AES_BLOCK_SIZE,
 	.cra_ctxsize			= sizeof(struct crypto_aes_ctx),
@@ -367,6 +408,12 @@ static void __exit aes_fini(void)
 	crypto_unregister_alg(&aes_alg);
 }
 
+extern const u8 crypto_aes_sbox[256] __alias(aes_sbox);
+EXPORT_SYMBOL_GPL(crypto_aes_sbox);
+
+extern const u8 crypto_aes_inv_sbox[256] __alias(aes_inv_sbox);
+EXPORT_SYMBOL_GPL(crypto_aes_inv_sbox);
+
 module_init(aes_init);
 module_exit(aes_fini);
 
diff --git a/crypto/aes_generic.c b/crypto/aes_generic.c
deleted file mode 100644
index ca554d57d01e..000000000000
--- a/crypto/aes_generic.c
+++ /dev/null
@@ -1,1478 +0,0 @@
-/*
- * Cryptographic API.
- *
- * AES Cipher Algorithm.
- *
- * Based on Brian Gladman's code.
- *
- * Linux developers:
- *  Alexander Kjeldaas <astor@fast.no>
- *  Herbert Valerio Riedel <hvr@hvrlab.org>
- *  Kyle McMartin <kyle@debian.org>
- *  Adam J. Richter <adam@yggdrasil.com> (conversion to 2.5 API).
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * ---------------------------------------------------------------------------
- * Copyright (c) 2002, Dr Brian Gladman <brg@gladman.me.uk>, Worcester, UK.
- * All rights reserved.
- *
- * LICENSE TERMS
- *
- * The free distribution and use of this software in both source and binary
- * form is allowed (with or without changes) provided that:
- *
- *   1. distributions of this source code include the above copyright
- *      notice, this list of conditions and the following disclaimer;
- *
- *   2. distributions in binary form include the above copyright
- *      notice, this list of conditions and the following disclaimer
- *      in the documentation and/or other associated materials;
- *
- *   3. the copyright holder's name is not used to endorse products
- *      built using this software without specific written permission.
- *
- * ALTERNATIVELY, provided that this notice is retained in full, this product
- * may be distributed under the terms of the GNU General Public License (GPL),
- * in which case the provisions of the GPL apply INSTEAD OF those given above.
- *
- * DISCLAIMER
- *
- * This software is provided 'as is' with no explicit or implied warranties
- * in respect of its properties, including, but not limited to, correctness
- * and/or fitness for purpose.
- * ---------------------------------------------------------------------------
- */
-
-#include <crypto/aes.h>
-#include <linux/module.h>
-#include <linux/init.h>
-#include <linux/types.h>
-#include <linux/errno.h>
-#include <linux/crypto.h>
-#include <asm/byteorder.h>
-#include <asm/unaligned.h>
-
-static inline u8 byte(const u32 x, const unsigned n)
-{
-	return x >> (n << 3);
-}
-
-static const u32 rco_tab[10] = { 1, 2, 4, 8, 16, 32, 64, 128, 27, 54 };
-
-__visible const u32 crypto_ft_tab[4][256] = {
-	{
-		0xa56363c6, 0x847c7cf8, 0x997777ee, 0x8d7b7bf6,
-		0x0df2f2ff, 0xbd6b6bd6, 0xb16f6fde, 0x54c5c591,
-		0x50303060, 0x03010102, 0xa96767ce, 0x7d2b2b56,
-		0x19fefee7, 0x62d7d7b5, 0xe6abab4d, 0x9a7676ec,
-		0x45caca8f, 0x9d82821f, 0x40c9c989, 0x877d7dfa,
-		0x15fafaef, 0xeb5959b2, 0xc947478e, 0x0bf0f0fb,
-		0xecadad41, 0x67d4d4b3, 0xfda2a25f, 0xeaafaf45,
-		0xbf9c9c23, 0xf7a4a453, 0x967272e4, 0x5bc0c09b,
-		0xc2b7b775, 0x1cfdfde1, 0xae93933d, 0x6a26264c,
-		0x5a36366c, 0x413f3f7e, 0x02f7f7f5, 0x4fcccc83,
-		0x5c343468, 0xf4a5a551, 0x34e5e5d1, 0x08f1f1f9,
-		0x937171e2, 0x73d8d8ab, 0x53313162, 0x3f15152a,
-		0x0c040408, 0x52c7c795, 0x65232346, 0x5ec3c39d,
-		0x28181830, 0xa1969637, 0x0f05050a, 0xb59a9a2f,
-		0x0907070e, 0x36121224, 0x9b80801b, 0x3de2e2df,
-		0x26ebebcd, 0x6927274e, 0xcdb2b27f, 0x9f7575ea,
-		0x1b090912, 0x9e83831d, 0x742c2c58, 0x2e1a1a34,
-		0x2d1b1b36, 0xb26e6edc, 0xee5a5ab4, 0xfba0a05b,
-		0xf65252a4, 0x4d3b3b76, 0x61d6d6b7, 0xceb3b37d,
-		0x7b292952, 0x3ee3e3dd, 0x712f2f5e, 0x97848413,
-		0xf55353a6, 0x68d1d1b9, 0x00000000, 0x2cededc1,
-		0x60202040, 0x1ffcfce3, 0xc8b1b179, 0xed5b5bb6,
-		0xbe6a6ad4, 0x46cbcb8d, 0xd9bebe67, 0x4b393972,
-		0xde4a4a94, 0xd44c4c98, 0xe85858b0, 0x4acfcf85,
-		0x6bd0d0bb, 0x2aefefc5, 0xe5aaaa4f, 0x16fbfbed,
-		0xc5434386, 0xd74d4d9a, 0x55333366, 0x94858511,
-		0xcf45458a, 0x10f9f9e9, 0x06020204, 0x817f7ffe,
-		0xf05050a0, 0x443c3c78, 0xba9f9f25, 0xe3a8a84b,
-		0xf35151a2, 0xfea3a35d, 0xc0404080, 0x8a8f8f05,
-		0xad92923f, 0xbc9d9d21, 0x48383870, 0x04f5f5f1,
-		0xdfbcbc63, 0xc1b6b677, 0x75dadaaf, 0x63212142,
-		0x30101020, 0x1affffe5, 0x0ef3f3fd, 0x6dd2d2bf,
-		0x4ccdcd81, 0x140c0c18, 0x35131326, 0x2fececc3,
-		0xe15f5fbe, 0xa2979735, 0xcc444488, 0x3917172e,
-		0x57c4c493, 0xf2a7a755, 0x827e7efc, 0x473d3d7a,
-		0xac6464c8, 0xe75d5dba, 0x2b191932, 0x957373e6,
-		0xa06060c0, 0x98818119, 0xd14f4f9e, 0x7fdcdca3,
-		0x66222244, 0x7e2a2a54, 0xab90903b, 0x8388880b,
-		0xca46468c, 0x29eeeec7, 0xd3b8b86b, 0x3c141428,
-		0x79dedea7, 0xe25e5ebc, 0x1d0b0b16, 0x76dbdbad,
-		0x3be0e0db, 0x56323264, 0x4e3a3a74, 0x1e0a0a14,
-		0xdb494992, 0x0a06060c, 0x6c242448, 0xe45c5cb8,
-		0x5dc2c29f, 0x6ed3d3bd, 0xefacac43, 0xa66262c4,
-		0xa8919139, 0xa4959531, 0x37e4e4d3, 0x8b7979f2,
-		0x32e7e7d5, 0x43c8c88b, 0x5937376e, 0xb76d6dda,
-		0x8c8d8d01, 0x64d5d5b1, 0xd24e4e9c, 0xe0a9a949,
-		0xb46c6cd8, 0xfa5656ac, 0x07f4f4f3, 0x25eaeacf,
-		0xaf6565ca, 0x8e7a7af4, 0xe9aeae47, 0x18080810,
-		0xd5baba6f, 0x887878f0, 0x6f25254a, 0x722e2e5c,
-		0x241c1c38, 0xf1a6a657, 0xc7b4b473, 0x51c6c697,
-		0x23e8e8cb, 0x7cdddda1, 0x9c7474e8, 0x211f1f3e,
-		0xdd4b4b96, 0xdcbdbd61, 0x868b8b0d, 0x858a8a0f,
-		0x907070e0, 0x423e3e7c, 0xc4b5b571, 0xaa6666cc,
-		0xd8484890, 0x05030306, 0x01f6f6f7, 0x120e0e1c,
-		0xa36161c2, 0x5f35356a, 0xf95757ae, 0xd0b9b969,
-		0x91868617, 0x58c1c199, 0x271d1d3a, 0xb99e9e27,
-		0x38e1e1d9, 0x13f8f8eb, 0xb398982b, 0x33111122,
-		0xbb6969d2, 0x70d9d9a9, 0x898e8e07, 0xa7949433,
-		0xb69b9b2d, 0x221e1e3c, 0x92878715, 0x20e9e9c9,
-		0x49cece87, 0xff5555aa, 0x78282850, 0x7adfdfa5,
-		0x8f8c8c03, 0xf8a1a159, 0x80898909, 0x170d0d1a,
-		0xdabfbf65, 0x31e6e6d7, 0xc6424284, 0xb86868d0,
-		0xc3414182, 0xb0999929, 0x772d2d5a, 0x110f0f1e,
-		0xcbb0b07b, 0xfc5454a8, 0xd6bbbb6d, 0x3a16162c,
-	}, {
-		0x6363c6a5, 0x7c7cf884, 0x7777ee99, 0x7b7bf68d,
-		0xf2f2ff0d, 0x6b6bd6bd, 0x6f6fdeb1, 0xc5c59154,
-		0x30306050, 0x01010203, 0x6767cea9, 0x2b2b567d,
-		0xfefee719, 0xd7d7b562, 0xabab4de6, 0x7676ec9a,
-		0xcaca8f45, 0x82821f9d, 0xc9c98940, 0x7d7dfa87,
-		0xfafaef15, 0x5959b2eb, 0x47478ec9, 0xf0f0fb0b,
-		0xadad41ec, 0xd4d4b367, 0xa2a25ffd, 0xafaf45ea,
-		0x9c9c23bf, 0xa4a453f7, 0x7272e496, 0xc0c09b5b,
-		0xb7b775c2, 0xfdfde11c, 0x93933dae, 0x26264c6a,
-		0x36366c5a, 0x3f3f7e41, 0xf7f7f502, 0xcccc834f,
-		0x3434685c, 0xa5a551f4, 0xe5e5d134, 0xf1f1f908,
-		0x7171e293, 0xd8d8ab73, 0x31316253, 0x15152a3f,
-		0x0404080c, 0xc7c79552, 0x23234665, 0xc3c39d5e,
-		0x18183028, 0x969637a1, 0x05050a0f, 0x9a9a2fb5,
-		0x07070e09, 0x12122436, 0x80801b9b, 0xe2e2df3d,
-		0xebebcd26, 0x27274e69, 0xb2b27fcd, 0x7575ea9f,
-		0x0909121b, 0x83831d9e, 0x2c2c5874, 0x1a1a342e,
-		0x1b1b362d, 0x6e6edcb2, 0x5a5ab4ee, 0xa0a05bfb,
-		0x5252a4f6, 0x3b3b764d, 0xd6d6b761, 0xb3b37dce,
-		0x2929527b, 0xe3e3dd3e, 0x2f2f5e71, 0x84841397,
-		0x5353a6f5, 0xd1d1b968, 0x00000000, 0xededc12c,
-		0x20204060, 0xfcfce31f, 0xb1b179c8, 0x5b5bb6ed,
-		0x6a6ad4be, 0xcbcb8d46, 0xbebe67d9, 0x3939724b,
-		0x4a4a94de, 0x4c4c98d4, 0x5858b0e8, 0xcfcf854a,
-		0xd0d0bb6b, 0xefefc52a, 0xaaaa4fe5, 0xfbfbed16,
-		0x434386c5, 0x4d4d9ad7, 0x33336655, 0x85851194,
-		0x45458acf, 0xf9f9e910, 0x02020406, 0x7f7ffe81,
-		0x5050a0f0, 0x3c3c7844, 0x9f9f25ba, 0xa8a84be3,
-		0x5151a2f3, 0xa3a35dfe, 0x404080c0, 0x8f8f058a,
-		0x92923fad, 0x9d9d21bc, 0x38387048, 0xf5f5f104,
-		0xbcbc63df, 0xb6b677c1, 0xdadaaf75, 0x21214263,
-		0x10102030, 0xffffe51a, 0xf3f3fd0e, 0xd2d2bf6d,
-		0xcdcd814c, 0x0c0c1814, 0x13132635, 0xececc32f,
-		0x5f5fbee1, 0x979735a2, 0x444488cc, 0x17172e39,
-		0xc4c49357, 0xa7a755f2, 0x7e7efc82, 0x3d3d7a47,
-		0x6464c8ac, 0x5d5dbae7, 0x1919322b, 0x7373e695,
-		0x6060c0a0, 0x81811998, 0x4f4f9ed1, 0xdcdca37f,
-		0x22224466, 0x2a2a547e, 0x90903bab, 0x88880b83,
-		0x46468cca, 0xeeeec729, 0xb8b86bd3, 0x1414283c,
-		0xdedea779, 0x5e5ebce2, 0x0b0b161d, 0xdbdbad76,
-		0xe0e0db3b, 0x32326456, 0x3a3a744e, 0x0a0a141e,
-		0x494992db, 0x06060c0a, 0x2424486c, 0x5c5cb8e4,
-		0xc2c29f5d, 0xd3d3bd6e, 0xacac43ef, 0x6262c4a6,
-		0x919139a8, 0x959531a4, 0xe4e4d337, 0x7979f28b,
-		0xe7e7d532, 0xc8c88b43, 0x37376e59, 0x6d6ddab7,
-		0x8d8d018c, 0xd5d5b164, 0x4e4e9cd2, 0xa9a949e0,
-		0x6c6cd8b4, 0x5656acfa, 0xf4f4f307, 0xeaeacf25,
-		0x6565caaf, 0x7a7af48e, 0xaeae47e9, 0x08081018,
-		0xbaba6fd5, 0x7878f088, 0x25254a6f, 0x2e2e5c72,
-		0x1c1c3824, 0xa6a657f1, 0xb4b473c7, 0xc6c69751,
-		0xe8e8cb23, 0xdddda17c, 0x7474e89c, 0x1f1f3e21,
-		0x4b4b96dd, 0xbdbd61dc, 0x8b8b0d86, 0x8a8a0f85,
-		0x7070e090, 0x3e3e7c42, 0xb5b571c4, 0x6666ccaa,
-		0x484890d8, 0x03030605, 0xf6f6f701, 0x0e0e1c12,
-		0x6161c2a3, 0x35356a5f, 0x5757aef9, 0xb9b969d0,
-		0x86861791, 0xc1c19958, 0x1d1d3a27, 0x9e9e27b9,
-		0xe1e1d938, 0xf8f8eb13, 0x98982bb3, 0x11112233,
-		0x6969d2bb, 0xd9d9a970, 0x8e8e0789, 0x949433a7,
-		0x9b9b2db6, 0x1e1e3c22, 0x87871592, 0xe9e9c920,
-		0xcece8749, 0x5555aaff, 0x28285078, 0xdfdfa57a,
-		0x8c8c038f, 0xa1a159f8, 0x89890980, 0x0d0d1a17,
-		0xbfbf65da, 0xe6e6d731, 0x424284c6, 0x6868d0b8,
-		0x414182c3, 0x999929b0, 0x2d2d5a77, 0x0f0f1e11,
-		0xb0b07bcb, 0x5454a8fc, 0xbbbb6dd6, 0x16162c3a,
-	}, {
-		0x63c6a563, 0x7cf8847c, 0x77ee9977, 0x7bf68d7b,
-		0xf2ff0df2, 0x6bd6bd6b, 0x6fdeb16f, 0xc59154c5,
-		0x30605030, 0x01020301, 0x67cea967, 0x2b567d2b,
-		0xfee719fe, 0xd7b562d7, 0xab4de6ab, 0x76ec9a76,
-		0xca8f45ca, 0x821f9d82, 0xc98940c9, 0x7dfa877d,
-		0xfaef15fa, 0x59b2eb59, 0x478ec947, 0xf0fb0bf0,
-		0xad41ecad, 0xd4b367d4, 0xa25ffda2, 0xaf45eaaf,
-		0x9c23bf9c, 0xa453f7a4, 0x72e49672, 0xc09b5bc0,
-		0xb775c2b7, 0xfde11cfd, 0x933dae93, 0x264c6a26,
-		0x366c5a36, 0x3f7e413f, 0xf7f502f7, 0xcc834fcc,
-		0x34685c34, 0xa551f4a5, 0xe5d134e5, 0xf1f908f1,
-		0x71e29371, 0xd8ab73d8, 0x31625331, 0x152a3f15,
-		0x04080c04, 0xc79552c7, 0x23466523, 0xc39d5ec3,
-		0x18302818, 0x9637a196, 0x050a0f05, 0x9a2fb59a,
-		0x070e0907, 0x12243612, 0x801b9b80, 0xe2df3de2,
-		0xebcd26eb, 0x274e6927, 0xb27fcdb2, 0x75ea9f75,
-		0x09121b09, 0x831d9e83, 0x2c58742c, 0x1a342e1a,
-		0x1b362d1b, 0x6edcb26e, 0x5ab4ee5a, 0xa05bfba0,
-		0x52a4f652, 0x3b764d3b, 0xd6b761d6, 0xb37dceb3,
-		0x29527b29, 0xe3dd3ee3, 0x2f5e712f, 0x84139784,
-		0x53a6f553, 0xd1b968d1, 0x00000000, 0xedc12ced,
-		0x20406020, 0xfce31ffc, 0xb179c8b1, 0x5bb6ed5b,
-		0x6ad4be6a, 0xcb8d46cb, 0xbe67d9be, 0x39724b39,
-		0x4a94de4a, 0x4c98d44c, 0x58b0e858, 0xcf854acf,
-		0xd0bb6bd0, 0xefc52aef, 0xaa4fe5aa, 0xfbed16fb,
-		0x4386c543, 0x4d9ad74d, 0x33665533, 0x85119485,
-		0x458acf45, 0xf9e910f9, 0x02040602, 0x7ffe817f,
-		0x50a0f050, 0x3c78443c, 0x9f25ba9f, 0xa84be3a8,
-		0x51a2f351, 0xa35dfea3, 0x4080c040, 0x8f058a8f,
-		0x923fad92, 0x9d21bc9d, 0x38704838, 0xf5f104f5,
-		0xbc63dfbc, 0xb677c1b6, 0xdaaf75da, 0x21426321,
-		0x10203010, 0xffe51aff, 0xf3fd0ef3, 0xd2bf6dd2,
-		0xcd814ccd, 0x0c18140c, 0x13263513, 0xecc32fec,
-		0x5fbee15f, 0x9735a297, 0x4488cc44, 0x172e3917,
-		0xc49357c4, 0xa755f2a7, 0x7efc827e, 0x3d7a473d,
-		0x64c8ac64, 0x5dbae75d, 0x19322b19, 0x73e69573,
-		0x60c0a060, 0x81199881, 0x4f9ed14f, 0xdca37fdc,
-		0x22446622, 0x2a547e2a, 0x903bab90, 0x880b8388,
-		0x468cca46, 0xeec729ee, 0xb86bd3b8, 0x14283c14,
-		0xdea779de, 0x5ebce25e, 0x0b161d0b, 0xdbad76db,
-		0xe0db3be0, 0x32645632, 0x3a744e3a, 0x0a141e0a,
-		0x4992db49, 0x060c0a06, 0x24486c24, 0x5cb8e45c,
-		0xc29f5dc2, 0xd3bd6ed3, 0xac43efac, 0x62c4a662,
-		0x9139a891, 0x9531a495, 0xe4d337e4, 0x79f28b79,
-		0xe7d532e7, 0xc88b43c8, 0x376e5937, 0x6ddab76d,
-		0x8d018c8d, 0xd5b164d5, 0x4e9cd24e, 0xa949e0a9,
-		0x6cd8b46c, 0x56acfa56, 0xf4f307f4, 0xeacf25ea,
-		0x65caaf65, 0x7af48e7a, 0xae47e9ae, 0x08101808,
-		0xba6fd5ba, 0x78f08878, 0x254a6f25, 0x2e5c722e,
-		0x1c38241c, 0xa657f1a6, 0xb473c7b4, 0xc69751c6,
-		0xe8cb23e8, 0xdda17cdd, 0x74e89c74, 0x1f3e211f,
-		0x4b96dd4b, 0xbd61dcbd, 0x8b0d868b, 0x8a0f858a,
-		0x70e09070, 0x3e7c423e, 0xb571c4b5, 0x66ccaa66,
-		0x4890d848, 0x03060503, 0xf6f701f6, 0x0e1c120e,
-		0x61c2a361, 0x356a5f35, 0x57aef957, 0xb969d0b9,
-		0x86179186, 0xc19958c1, 0x1d3a271d, 0x9e27b99e,
-		0xe1d938e1, 0xf8eb13f8, 0x982bb398, 0x11223311,
-		0x69d2bb69, 0xd9a970d9, 0x8e07898e, 0x9433a794,
-		0x9b2db69b, 0x1e3c221e, 0x87159287, 0xe9c920e9,
-		0xce8749ce, 0x55aaff55, 0x28507828, 0xdfa57adf,
-		0x8c038f8c, 0xa159f8a1, 0x89098089, 0x0d1a170d,
-		0xbf65dabf, 0xe6d731e6, 0x4284c642, 0x68d0b868,
-		0x4182c341, 0x9929b099, 0x2d5a772d, 0x0f1e110f,
-		0xb07bcbb0, 0x54a8fc54, 0xbb6dd6bb, 0x162c3a16,
-	}, {
-		0xc6a56363, 0xf8847c7c, 0xee997777, 0xf68d7b7b,
-		0xff0df2f2, 0xd6bd6b6b, 0xdeb16f6f, 0x9154c5c5,
-		0x60503030, 0x02030101, 0xcea96767, 0x567d2b2b,
-		0xe719fefe, 0xb562d7d7, 0x4de6abab, 0xec9a7676,
-		0x8f45caca, 0x1f9d8282, 0x8940c9c9, 0xfa877d7d,
-		0xef15fafa, 0xb2eb5959, 0x8ec94747, 0xfb0bf0f0,
-		0x41ecadad, 0xb367d4d4, 0x5ffda2a2, 0x45eaafaf,
-		0x23bf9c9c, 0x53f7a4a4, 0xe4967272, 0x9b5bc0c0,
-		0x75c2b7b7, 0xe11cfdfd, 0x3dae9393, 0x4c6a2626,
-		0x6c5a3636, 0x7e413f3f, 0xf502f7f7, 0x834fcccc,
-		0x685c3434, 0x51f4a5a5, 0xd134e5e5, 0xf908f1f1,
-		0xe2937171, 0xab73d8d8, 0x62533131, 0x2a3f1515,
-		0x080c0404, 0x9552c7c7, 0x46652323, 0x9d5ec3c3,
-		0x30281818, 0x37a19696, 0x0a0f0505, 0x2fb59a9a,
-		0x0e090707, 0x24361212, 0x1b9b8080, 0xdf3de2e2,
-		0xcd26ebeb, 0x4e692727, 0x7fcdb2b2, 0xea9f7575,
-		0x121b0909, 0x1d9e8383, 0x58742c2c, 0x342e1a1a,
-		0x362d1b1b, 0xdcb26e6e, 0xb4ee5a5a, 0x5bfba0a0,
-		0xa4f65252, 0x764d3b3b, 0xb761d6d6, 0x7dceb3b3,
-		0x527b2929, 0xdd3ee3e3, 0x5e712f2f, 0x13978484,
-		0xa6f55353, 0xb968d1d1, 0x00000000, 0xc12ceded,
-		0x40602020, 0xe31ffcfc, 0x79c8b1b1, 0xb6ed5b5b,
-		0xd4be6a6a, 0x8d46cbcb, 0x67d9bebe, 0x724b3939,
-		0x94de4a4a, 0x98d44c4c, 0xb0e85858, 0x854acfcf,
-		0xbb6bd0d0, 0xc52aefef, 0x4fe5aaaa, 0xed16fbfb,
-		0x86c54343, 0x9ad74d4d, 0x66553333, 0x11948585,
-		0x8acf4545, 0xe910f9f9, 0x04060202, 0xfe817f7f,
-		0xa0f05050, 0x78443c3c, 0x25ba9f9f, 0x4be3a8a8,
-		0xa2f35151, 0x5dfea3a3, 0x80c04040, 0x058a8f8f,
-		0x3fad9292, 0x21bc9d9d, 0x70483838, 0xf104f5f5,
-		0x63dfbcbc, 0x77c1b6b6, 0xaf75dada, 0x42632121,
-		0x20301010, 0xe51affff, 0xfd0ef3f3, 0xbf6dd2d2,
-		0x814ccdcd, 0x18140c0c, 0x26351313, 0xc32fecec,
-		0xbee15f5f, 0x35a29797, 0x88cc4444, 0x2e391717,
-		0x9357c4c4, 0x55f2a7a7, 0xfc827e7e, 0x7a473d3d,
-		0xc8ac6464, 0xbae75d5d, 0x322b1919, 0xe6957373,
-		0xc0a06060, 0x19988181, 0x9ed14f4f, 0xa37fdcdc,
-		0x44662222, 0x547e2a2a, 0x3bab9090, 0x0b838888,
-		0x8cca4646, 0xc729eeee, 0x6bd3b8b8, 0x283c1414,
-		0xa779dede, 0xbce25e5e, 0x161d0b0b, 0xad76dbdb,
-		0xdb3be0e0, 0x64563232, 0x744e3a3a, 0x141e0a0a,
-		0x92db4949, 0x0c0a0606, 0x486c2424, 0xb8e45c5c,
-		0x9f5dc2c2, 0xbd6ed3d3, 0x43efacac, 0xc4a66262,
-		0x39a89191, 0x31a49595, 0xd337e4e4, 0xf28b7979,
-		0xd532e7e7, 0x8b43c8c8, 0x6e593737, 0xdab76d6d,
-		0x018c8d8d, 0xb164d5d5, 0x9cd24e4e, 0x49e0a9a9,
-		0xd8b46c6c, 0xacfa5656, 0xf307f4f4, 0xcf25eaea,
-		0xcaaf6565, 0xf48e7a7a, 0x47e9aeae, 0x10180808,
-		0x6fd5baba, 0xf0887878, 0x4a6f2525, 0x5c722e2e,
-		0x38241c1c, 0x57f1a6a6, 0x73c7b4b4, 0x9751c6c6,
-		0xcb23e8e8, 0xa17cdddd, 0xe89c7474, 0x3e211f1f,
-		0x96dd4b4b, 0x61dcbdbd, 0x0d868b8b, 0x0f858a8a,
-		0xe0907070, 0x7c423e3e, 0x71c4b5b5, 0xccaa6666,
-		0x90d84848, 0x06050303, 0xf701f6f6, 0x1c120e0e,
-		0xc2a36161, 0x6a5f3535, 0xaef95757, 0x69d0b9b9,
-		0x17918686, 0x9958c1c1, 0x3a271d1d, 0x27b99e9e,
-		0xd938e1e1, 0xeb13f8f8, 0x2bb39898, 0x22331111,
-		0xd2bb6969, 0xa970d9d9, 0x07898e8e, 0x33a79494,
-		0x2db69b9b, 0x3c221e1e, 0x15928787, 0xc920e9e9,
-		0x8749cece, 0xaaff5555, 0x50782828, 0xa57adfdf,
-		0x038f8c8c, 0x59f8a1a1, 0x09808989, 0x1a170d0d,
-		0x65dabfbf, 0xd731e6e6, 0x84c64242, 0xd0b86868,
-		0x82c34141, 0x29b09999, 0x5a772d2d, 0x1e110f0f,
-		0x7bcbb0b0, 0xa8fc5454, 0x6dd6bbbb, 0x2c3a1616,
-	}
-};
-
-__visible const u32 crypto_fl_tab[4][256] = {
-	{
-		0x00000063, 0x0000007c, 0x00000077, 0x0000007b,
-		0x000000f2, 0x0000006b, 0x0000006f, 0x000000c5,
-		0x00000030, 0x00000001, 0x00000067, 0x0000002b,
-		0x000000fe, 0x000000d7, 0x000000ab, 0x00000076,
-		0x000000ca, 0x00000082, 0x000000c9, 0x0000007d,
-		0x000000fa, 0x00000059, 0x00000047, 0x000000f0,
-		0x000000ad, 0x000000d4, 0x000000a2, 0x000000af,
-		0x0000009c, 0x000000a4, 0x00000072, 0x000000c0,
-		0x000000b7, 0x000000fd, 0x00000093, 0x00000026,
-		0x00000036, 0x0000003f, 0x000000f7, 0x000000cc,
-		0x00000034, 0x000000a5, 0x000000e5, 0x000000f1,
-		0x00000071, 0x000000d8, 0x00000031, 0x00000015,
-		0x00000004, 0x000000c7, 0x00000023, 0x000000c3,
-		0x00000018, 0x00000096, 0x00000005, 0x0000009a,
-		0x00000007, 0x00000012, 0x00000080, 0x000000e2,
-		0x000000eb, 0x00000027, 0x000000b2, 0x00000075,
-		0x00000009, 0x00000083, 0x0000002c, 0x0000001a,
-		0x0000001b, 0x0000006e, 0x0000005a, 0x000000a0,
-		0x00000052, 0x0000003b, 0x000000d6, 0x000000b3,
-		0x00000029, 0x000000e3, 0x0000002f, 0x00000084,
-		0x00000053, 0x000000d1, 0x00000000, 0x000000ed,
-		0x00000020, 0x000000fc, 0x000000b1, 0x0000005b,
-		0x0000006a, 0x000000cb, 0x000000be, 0x00000039,
-		0x0000004a, 0x0000004c, 0x00000058, 0x000000cf,
-		0x000000d0, 0x000000ef, 0x000000aa, 0x000000fb,
-		0x00000043, 0x0000004d, 0x00000033, 0x00000085,
-		0x00000045, 0x000000f9, 0x00000002, 0x0000007f,
-		0x00000050, 0x0000003c, 0x0000009f, 0x000000a8,
-		0x00000051, 0x000000a3, 0x00000040, 0x0000008f,
-		0x00000092, 0x0000009d, 0x00000038, 0x000000f5,
-		0x000000bc, 0x000000b6, 0x000000da, 0x00000021,
-		0x00000010, 0x000000ff, 0x000000f3, 0x000000d2,
-		0x000000cd, 0x0000000c, 0x00000013, 0x000000ec,
-		0x0000005f, 0x00000097, 0x00000044, 0x00000017,
-		0x000000c4, 0x000000a7, 0x0000007e, 0x0000003d,
-		0x00000064, 0x0000005d, 0x00000019, 0x00000073,
-		0x00000060, 0x00000081, 0x0000004f, 0x000000dc,
-		0x00000022, 0x0000002a, 0x00000090, 0x00000088,
-		0x00000046, 0x000000ee, 0x000000b8, 0x00000014,
-		0x000000de, 0x0000005e, 0x0000000b, 0x000000db,
-		0x000000e0, 0x00000032, 0x0000003a, 0x0000000a,
-		0x00000049, 0x00000006, 0x00000024, 0x0000005c,
-		0x000000c2, 0x000000d3, 0x000000ac, 0x00000062,
-		0x00000091, 0x00000095, 0x000000e4, 0x00000079,
-		0x000000e7, 0x000000c8, 0x00000037, 0x0000006d,
-		0x0000008d, 0x000000d5, 0x0000004e, 0x000000a9,
-		0x0000006c, 0x00000056, 0x000000f4, 0x000000ea,
-		0x00000065, 0x0000007a, 0x000000ae, 0x00000008,
-		0x000000ba, 0x00000078, 0x00000025, 0x0000002e,
-		0x0000001c, 0x000000a6, 0x000000b4, 0x000000c6,
-		0x000000e8, 0x000000dd, 0x00000074, 0x0000001f,
-		0x0000004b, 0x000000bd, 0x0000008b, 0x0000008a,
-		0x00000070, 0x0000003e, 0x000000b5, 0x00000066,
-		0x00000048, 0x00000003, 0x000000f6, 0x0000000e,
-		0x00000061, 0x00000035, 0x00000057, 0x000000b9,
-		0x00000086, 0x000000c1, 0x0000001d, 0x0000009e,
-		0x000000e1, 0x000000f8, 0x00000098, 0x00000011,
-		0x00000069, 0x000000d9, 0x0000008e, 0x00000094,
-		0x0000009b, 0x0000001e, 0x00000087, 0x000000e9,
-		0x000000ce, 0x00000055, 0x00000028, 0x000000df,
-		0x0000008c, 0x000000a1, 0x00000089, 0x0000000d,
-		0x000000bf, 0x000000e6, 0x00000042, 0x00000068,
-		0x00000041, 0x00000099, 0x0000002d, 0x0000000f,
-		0x000000b0, 0x00000054, 0x000000bb, 0x00000016,
-	}, {
-		0x00006300, 0x00007c00, 0x00007700, 0x00007b00,
-		0x0000f200, 0x00006b00, 0x00006f00, 0x0000c500,
-		0x00003000, 0x00000100, 0x00006700, 0x00002b00,
-		0x0000fe00, 0x0000d700, 0x0000ab00, 0x00007600,
-		0x0000ca00, 0x00008200, 0x0000c900, 0x00007d00,
-		0x0000fa00, 0x00005900, 0x00004700, 0x0000f000,
-		0x0000ad00, 0x0000d400, 0x0000a200, 0x0000af00,
-		0x00009c00, 0x0000a400, 0x00007200, 0x0000c000,
-		0x0000b700, 0x0000fd00, 0x00009300, 0x00002600,
-		0x00003600, 0x00003f00, 0x0000f700, 0x0000cc00,
-		0x00003400, 0x0000a500, 0x0000e500, 0x0000f100,
-		0x00007100, 0x0000d800, 0x00003100, 0x00001500,
-		0x00000400, 0x0000c700, 0x00002300, 0x0000c300,
-		0x00001800, 0x00009600, 0x00000500, 0x00009a00,
-		0x00000700, 0x00001200, 0x00008000, 0x0000e200,
-		0x0000eb00, 0x00002700, 0x0000b200, 0x00007500,
-		0x00000900, 0x00008300, 0x00002c00, 0x00001a00,
-		0x00001b00, 0x00006e00, 0x00005a00, 0x0000a000,
-		0x00005200, 0x00003b00, 0x0000d600, 0x0000b300,
-		0x00002900, 0x0000e300, 0x00002f00, 0x00008400,
-		0x00005300, 0x0000d100, 0x00000000, 0x0000ed00,
-		0x00002000, 0x0000fc00, 0x0000b100, 0x00005b00,
-		0x00006a00, 0x0000cb00, 0x0000be00, 0x00003900,
-		0x00004a00, 0x00004c00, 0x00005800, 0x0000cf00,
-		0x0000d000, 0x0000ef00, 0x0000aa00, 0x0000fb00,
-		0x00004300, 0x00004d00, 0x00003300, 0x00008500,
-		0x00004500, 0x0000f900, 0x00000200, 0x00007f00,
-		0x00005000, 0x00003c00, 0x00009f00, 0x0000a800,
-		0x00005100, 0x0000a300, 0x00004000, 0x00008f00,
-		0x00009200, 0x00009d00, 0x00003800, 0x0000f500,
-		0x0000bc00, 0x0000b600, 0x0000da00, 0x00002100,
-		0x00001000, 0x0000ff00, 0x0000f300, 0x0000d200,
-		0x0000cd00, 0x00000c00, 0x00001300, 0x0000ec00,
-		0x00005f00, 0x00009700, 0x00004400, 0x00001700,
-		0x0000c400, 0x0000a700, 0x00007e00, 0x00003d00,
-		0x00006400, 0x00005d00, 0x00001900, 0x00007300,
-		0x00006000, 0x00008100, 0x00004f00, 0x0000dc00,
-		0x00002200, 0x00002a00, 0x00009000, 0x00008800,
-		0x00004600, 0x0000ee00, 0x0000b800, 0x00001400,
-		0x0000de00, 0x00005e00, 0x00000b00, 0x0000db00,
-		0x0000e000, 0x00003200, 0x00003a00, 0x00000a00,
-		0x00004900, 0x00000600, 0x00002400, 0x00005c00,
-		0x0000c200, 0x0000d300, 0x0000ac00, 0x00006200,
-		0x00009100, 0x00009500, 0x0000e400, 0x00007900,
-		0x0000e700, 0x0000c800, 0x00003700, 0x00006d00,
-		0x00008d00, 0x0000d500, 0x00004e00, 0x0000a900,
-		0x00006c00, 0x00005600, 0x0000f400, 0x0000ea00,
-		0x00006500, 0x00007a00, 0x0000ae00, 0x00000800,
-		0x0000ba00, 0x00007800, 0x00002500, 0x00002e00,
-		0x00001c00, 0x0000a600, 0x0000b400, 0x0000c600,
-		0x0000e800, 0x0000dd00, 0x00007400, 0x00001f00,
-		0x00004b00, 0x0000bd00, 0x00008b00, 0x00008a00,
-		0x00007000, 0x00003e00, 0x0000b500, 0x00006600,
-		0x00004800, 0x00000300, 0x0000f600, 0x00000e00,
-		0x00006100, 0x00003500, 0x00005700, 0x0000b900,
-		0x00008600, 0x0000c100, 0x00001d00, 0x00009e00,
-		0x0000e100, 0x0000f800, 0x00009800, 0x00001100,
-		0x00006900, 0x0000d900, 0x00008e00, 0x00009400,
-		0x00009b00, 0x00001e00, 0x00008700, 0x0000e900,
-		0x0000ce00, 0x00005500, 0x00002800, 0x0000df00,
-		0x00008c00, 0x0000a100, 0x00008900, 0x00000d00,
-		0x0000bf00, 0x0000e600, 0x00004200, 0x00006800,
-		0x00004100, 0x00009900, 0x00002d00, 0x00000f00,
-		0x0000b000, 0x00005400, 0x0000bb00, 0x00001600,
-	}, {
-		0x00630000, 0x007c0000, 0x00770000, 0x007b0000,
-		0x00f20000, 0x006b0000, 0x006f0000, 0x00c50000,
-		0x00300000, 0x00010000, 0x00670000, 0x002b0000,
-		0x00fe0000, 0x00d70000, 0x00ab0000, 0x00760000,
-		0x00ca0000, 0x00820000, 0x00c90000, 0x007d0000,
-		0x00fa0000, 0x00590000, 0x00470000, 0x00f00000,
-		0x00ad0000, 0x00d40000, 0x00a20000, 0x00af0000,
-		0x009c0000, 0x00a40000, 0x00720000, 0x00c00000,
-		0x00b70000, 0x00fd0000, 0x00930000, 0x00260000,
-		0x00360000, 0x003f0000, 0x00f70000, 0x00cc0000,
-		0x00340000, 0x00a50000, 0x00e50000, 0x00f10000,
-		0x00710000, 0x00d80000, 0x00310000, 0x00150000,
-		0x00040000, 0x00c70000, 0x00230000, 0x00c30000,
-		0x00180000, 0x00960000, 0x00050000, 0x009a0000,
-		0x00070000, 0x00120000, 0x00800000, 0x00e20000,
-		0x00eb0000, 0x00270000, 0x00b20000, 0x00750000,
-		0x00090000, 0x00830000, 0x002c0000, 0x001a0000,
-		0x001b0000, 0x006e0000, 0x005a0000, 0x00a00000,
-		0x00520000, 0x003b0000, 0x00d60000, 0x00b30000,
-		0x00290000, 0x00e30000, 0x002f0000, 0x00840000,
-		0x00530000, 0x00d10000, 0x00000000, 0x00ed0000,
-		0x00200000, 0x00fc0000, 0x00b10000, 0x005b0000,
-		0x006a0000, 0x00cb0000, 0x00be0000, 0x00390000,
-		0x004a0000, 0x004c0000, 0x00580000, 0x00cf0000,
-		0x00d00000, 0x00ef0000, 0x00aa0000, 0x00fb0000,
-		0x00430000, 0x004d0000, 0x00330000, 0x00850000,
-		0x00450000, 0x00f90000, 0x00020000, 0x007f0000,
-		0x00500000, 0x003c0000, 0x009f0000, 0x00a80000,
-		0x00510000, 0x00a30000, 0x00400000, 0x008f0000,
-		0x00920000, 0x009d0000, 0x00380000, 0x00f50000,
-		0x00bc0000, 0x00b60000, 0x00da0000, 0x00210000,
-		0x00100000, 0x00ff0000, 0x00f30000, 0x00d20000,
-		0x00cd0000, 0x000c0000, 0x00130000, 0x00ec0000,
-		0x005f0000, 0x00970000, 0x00440000, 0x00170000,
-		0x00c40000, 0x00a70000, 0x007e0000, 0x003d0000,
-		0x00640000, 0x005d0000, 0x00190000, 0x00730000,
-		0x00600000, 0x00810000, 0x004f0000, 0x00dc0000,
-		0x00220000, 0x002a0000, 0x00900000, 0x00880000,
-		0x00460000, 0x00ee0000, 0x00b80000, 0x00140000,
-		0x00de0000, 0x005e0000, 0x000b0000, 0x00db0000,
-		0x00e00000, 0x00320000, 0x003a0000, 0x000a0000,
-		0x00490000, 0x00060000, 0x00240000, 0x005c0000,
-		0x00c20000, 0x00d30000, 0x00ac0000, 0x00620000,
-		0x00910000, 0x00950000, 0x00e40000, 0x00790000,
-		0x00e70000, 0x00c80000, 0x00370000, 0x006d0000,
-		0x008d0000, 0x00d50000, 0x004e0000, 0x00a90000,
-		0x006c0000, 0x00560000, 0x00f40000, 0x00ea0000,
-		0x00650000, 0x007a0000, 0x00ae0000, 0x00080000,
-		0x00ba0000, 0x00780000, 0x00250000, 0x002e0000,
-		0x001c0000, 0x00a60000, 0x00b40000, 0x00c60000,
-		0x00e80000, 0x00dd0000, 0x00740000, 0x001f0000,
-		0x004b0000, 0x00bd0000, 0x008b0000, 0x008a0000,
-		0x00700000, 0x003e0000, 0x00b50000, 0x00660000,
-		0x00480000, 0x00030000, 0x00f60000, 0x000e0000,
-		0x00610000, 0x00350000, 0x00570000, 0x00b90000,
-		0x00860000, 0x00c10000, 0x001d0000, 0x009e0000,
-		0x00e10000, 0x00f80000, 0x00980000, 0x00110000,
-		0x00690000, 0x00d90000, 0x008e0000, 0x00940000,
-		0x009b0000, 0x001e0000, 0x00870000, 0x00e90000,
-		0x00ce0000, 0x00550000, 0x00280000, 0x00df0000,
-		0x008c0000, 0x00a10000, 0x00890000, 0x000d0000,
-		0x00bf0000, 0x00e60000, 0x00420000, 0x00680000,
-		0x00410000, 0x00990000, 0x002d0000, 0x000f0000,
-		0x00b00000, 0x00540000, 0x00bb0000, 0x00160000,
-	}, {
-		0x63000000, 0x7c000000, 0x77000000, 0x7b000000,
-		0xf2000000, 0x6b000000, 0x6f000000, 0xc5000000,
-		0x30000000, 0x01000000, 0x67000000, 0x2b000000,
-		0xfe000000, 0xd7000000, 0xab000000, 0x76000000,
-		0xca000000, 0x82000000, 0xc9000000, 0x7d000000,
-		0xfa000000, 0x59000000, 0x47000000, 0xf0000000,
-		0xad000000, 0xd4000000, 0xa2000000, 0xaf000000,
-		0x9c000000, 0xa4000000, 0x72000000, 0xc0000000,
-		0xb7000000, 0xfd000000, 0x93000000, 0x26000000,
-		0x36000000, 0x3f000000, 0xf7000000, 0xcc000000,
-		0x34000000, 0xa5000000, 0xe5000000, 0xf1000000,
-		0x71000000, 0xd8000000, 0x31000000, 0x15000000,
-		0x04000000, 0xc7000000, 0x23000000, 0xc3000000,
-		0x18000000, 0x96000000, 0x05000000, 0x9a000000,
-		0x07000000, 0x12000000, 0x80000000, 0xe2000000,
-		0xeb000000, 0x27000000, 0xb2000000, 0x75000000,
-		0x09000000, 0x83000000, 0x2c000000, 0x1a000000,
-		0x1b000000, 0x6e000000, 0x5a000000, 0xa0000000,
-		0x52000000, 0x3b000000, 0xd6000000, 0xb3000000,
-		0x29000000, 0xe3000000, 0x2f000000, 0x84000000,
-		0x53000000, 0xd1000000, 0x00000000, 0xed000000,
-		0x20000000, 0xfc000000, 0xb1000000, 0x5b000000,
-		0x6a000000, 0xcb000000, 0xbe000000, 0x39000000,
-		0x4a000000, 0x4c000000, 0x58000000, 0xcf000000,
-		0xd0000000, 0xef000000, 0xaa000000, 0xfb000000,
-		0x43000000, 0x4d000000, 0x33000000, 0x85000000,
-		0x45000000, 0xf9000000, 0x02000000, 0x7f000000,
-		0x50000000, 0x3c000000, 0x9f000000, 0xa8000000,
-		0x51000000, 0xa3000000, 0x40000000, 0x8f000000,
-		0x92000000, 0x9d000000, 0x38000000, 0xf5000000,
-		0xbc000000, 0xb6000000, 0xda000000, 0x21000000,
-		0x10000000, 0xff000000, 0xf3000000, 0xd2000000,
-		0xcd000000, 0x0c000000, 0x13000000, 0xec000000,
-		0x5f000000, 0x97000000, 0x44000000, 0x17000000,
-		0xc4000000, 0xa7000000, 0x7e000000, 0x3d000000,
-		0x64000000, 0x5d000000, 0x19000000, 0x73000000,
-		0x60000000, 0x81000000, 0x4f000000, 0xdc000000,
-		0x22000000, 0x2a000000, 0x90000000, 0x88000000,
-		0x46000000, 0xee000000, 0xb8000000, 0x14000000,
-		0xde000000, 0x5e000000, 0x0b000000, 0xdb000000,
-		0xe0000000, 0x32000000, 0x3a000000, 0x0a000000,
-		0x49000000, 0x06000000, 0x24000000, 0x5c000000,
-		0xc2000000, 0xd3000000, 0xac000000, 0x62000000,
-		0x91000000, 0x95000000, 0xe4000000, 0x79000000,
-		0xe7000000, 0xc8000000, 0x37000000, 0x6d000000,
-		0x8d000000, 0xd5000000, 0x4e000000, 0xa9000000,
-		0x6c000000, 0x56000000, 0xf4000000, 0xea000000,
-		0x65000000, 0x7a000000, 0xae000000, 0x08000000,
-		0xba000000, 0x78000000, 0x25000000, 0x2e000000,
-		0x1c000000, 0xa6000000, 0xb4000000, 0xc6000000,
-		0xe8000000, 0xdd000000, 0x74000000, 0x1f000000,
-		0x4b000000, 0xbd000000, 0x8b000000, 0x8a000000,
-		0x70000000, 0x3e000000, 0xb5000000, 0x66000000,
-		0x48000000, 0x03000000, 0xf6000000, 0x0e000000,
-		0x61000000, 0x35000000, 0x57000000, 0xb9000000,
-		0x86000000, 0xc1000000, 0x1d000000, 0x9e000000,
-		0xe1000000, 0xf8000000, 0x98000000, 0x11000000,
-		0x69000000, 0xd9000000, 0x8e000000, 0x94000000,
-		0x9b000000, 0x1e000000, 0x87000000, 0xe9000000,
-		0xce000000, 0x55000000, 0x28000000, 0xdf000000,
-		0x8c000000, 0xa1000000, 0x89000000, 0x0d000000,
-		0xbf000000, 0xe6000000, 0x42000000, 0x68000000,
-		0x41000000, 0x99000000, 0x2d000000, 0x0f000000,
-		0xb0000000, 0x54000000, 0xbb000000, 0x16000000,
-	}
-};
-
-__visible const u32 crypto_it_tab[4][256] = {
-	{
-		0x50a7f451, 0x5365417e, 0xc3a4171a, 0x965e273a,
-		0xcb6bab3b, 0xf1459d1f, 0xab58faac, 0x9303e34b,
-		0x55fa3020, 0xf66d76ad, 0x9176cc88, 0x254c02f5,
-		0xfcd7e54f, 0xd7cb2ac5, 0x80443526, 0x8fa362b5,
-		0x495ab1de, 0x671bba25, 0x980eea45, 0xe1c0fe5d,
-		0x02752fc3, 0x12f04c81, 0xa397468d, 0xc6f9d36b,
-		0xe75f8f03, 0x959c9215, 0xeb7a6dbf, 0xda595295,
-		0x2d83bed4, 0xd3217458, 0x2969e049, 0x44c8c98e,
-		0x6a89c275, 0x78798ef4, 0x6b3e5899, 0xdd71b927,
-		0xb64fe1be, 0x17ad88f0, 0x66ac20c9, 0xb43ace7d,
-		0x184adf63, 0x82311ae5, 0x60335197, 0x457f5362,
-		0xe07764b1, 0x84ae6bbb, 0x1ca081fe, 0x942b08f9,
-		0x58684870, 0x19fd458f, 0x876cde94, 0xb7f87b52,
-		0x23d373ab, 0xe2024b72, 0x578f1fe3, 0x2aab5566,
-		0x0728ebb2, 0x03c2b52f, 0x9a7bc586, 0xa50837d3,
-		0xf2872830, 0xb2a5bf23, 0xba6a0302, 0x5c8216ed,
-		0x2b1ccf8a, 0x92b479a7, 0xf0f207f3, 0xa1e2694e,
-		0xcdf4da65, 0xd5be0506, 0x1f6234d1, 0x8afea6c4,
-		0x9d532e34, 0xa055f3a2, 0x32e18a05, 0x75ebf6a4,
-		0x39ec830b, 0xaaef6040, 0x069f715e, 0x51106ebd,
-		0xf98a213e, 0x3d06dd96, 0xae053edd, 0x46bde64d,
-		0xb58d5491, 0x055dc471, 0x6fd40604, 0xff155060,
-		0x24fb9819, 0x97e9bdd6, 0xcc434089, 0x779ed967,
-		0xbd42e8b0, 0x888b8907, 0x385b19e7, 0xdbeec879,
-		0x470a7ca1, 0xe90f427c, 0xc91e84f8, 0x00000000,
-		0x83868009, 0x48ed2b32, 0xac70111e, 0x4e725a6c,
-		0xfbff0efd, 0x5638850f, 0x1ed5ae3d, 0x27392d36,
-		0x64d90f0a, 0x21a65c68, 0xd1545b9b, 0x3a2e3624,
-		0xb1670a0c, 0x0fe75793, 0xd296eeb4, 0x9e919b1b,
-		0x4fc5c080, 0xa220dc61, 0x694b775a, 0x161a121c,
-		0x0aba93e2, 0xe52aa0c0, 0x43e0223c, 0x1d171b12,
-		0x0b0d090e, 0xadc78bf2, 0xb9a8b62d, 0xc8a91e14,
-		0x8519f157, 0x4c0775af, 0xbbdd99ee, 0xfd607fa3,
-		0x9f2601f7, 0xbcf5725c, 0xc53b6644, 0x347efb5b,
-		0x7629438b, 0xdcc623cb, 0x68fcedb6, 0x63f1e4b8,
-		0xcadc31d7, 0x10856342, 0x40229713, 0x2011c684,
-		0x7d244a85, 0xf83dbbd2, 0x1132f9ae, 0x6da129c7,
-		0x4b2f9e1d, 0xf330b2dc, 0xec52860d, 0xd0e3c177,
-		0x6c16b32b, 0x99b970a9, 0xfa489411, 0x2264e947,
-		0xc48cfca8, 0x1a3ff0a0, 0xd82c7d56, 0xef903322,
-		0xc74e4987, 0xc1d138d9, 0xfea2ca8c, 0x360bd498,
-		0xcf81f5a6, 0x28de7aa5, 0x268eb7da, 0xa4bfad3f,
-		0xe49d3a2c, 0x0d927850, 0x9bcc5f6a, 0x62467e54,
-		0xc2138df6, 0xe8b8d890, 0x5ef7392e, 0xf5afc382,
-		0xbe805d9f, 0x7c93d069, 0xa92dd56f, 0xb31225cf,
-		0x3b99acc8, 0xa77d1810, 0x6e639ce8, 0x7bbb3bdb,
-		0x097826cd, 0xf418596e, 0x01b79aec, 0xa89a4f83,
-		0x656e95e6, 0x7ee6ffaa, 0x08cfbc21, 0xe6e815ef,
-		0xd99be7ba, 0xce366f4a, 0xd4099fea, 0xd67cb029,
-		0xafb2a431, 0x31233f2a, 0x3094a5c6, 0xc066a235,
-		0x37bc4e74, 0xa6ca82fc, 0xb0d090e0, 0x15d8a733,
-		0x4a9804f1, 0xf7daec41, 0x0e50cd7f, 0x2ff69117,
-		0x8dd64d76, 0x4db0ef43, 0x544daacc, 0xdf0496e4,
-		0xe3b5d19e, 0x1b886a4c, 0xb81f2cc1, 0x7f516546,
-		0x04ea5e9d, 0x5d358c01, 0x737487fa, 0x2e410bfb,
-		0x5a1d67b3, 0x52d2db92, 0x335610e9, 0x1347d66d,
-		0x8c61d79a, 0x7a0ca137, 0x8e14f859, 0x893c13eb,
-		0xee27a9ce, 0x35c961b7, 0xede51ce1, 0x3cb1477a,
-		0x59dfd29c, 0x3f73f255, 0x79ce1418, 0xbf37c773,
-		0xeacdf753, 0x5baafd5f, 0x146f3ddf, 0x86db4478,
-		0x81f3afca, 0x3ec468b9, 0x2c342438, 0x5f40a3c2,
-		0x72c31d16, 0x0c25e2bc, 0x8b493c28, 0x41950dff,
-		0x7101a839, 0xdeb30c08, 0x9ce4b4d8, 0x90c15664,
-		0x6184cb7b, 0x70b632d5, 0x745c6c48, 0x4257b8d0,
-	}, {
-		0xa7f45150, 0x65417e53, 0xa4171ac3, 0x5e273a96,
-		0x6bab3bcb, 0x459d1ff1, 0x58faacab, 0x03e34b93,
-		0xfa302055, 0x6d76adf6, 0x76cc8891, 0x4c02f525,
-		0xd7e54ffc, 0xcb2ac5d7, 0x44352680, 0xa362b58f,
-		0x5ab1de49, 0x1bba2567, 0x0eea4598, 0xc0fe5de1,
-		0x752fc302, 0xf04c8112, 0x97468da3, 0xf9d36bc6,
-		0x5f8f03e7, 0x9c921595, 0x7a6dbfeb, 0x595295da,
-		0x83bed42d, 0x217458d3, 0x69e04929, 0xc8c98e44,
-		0x89c2756a, 0x798ef478, 0x3e58996b, 0x71b927dd,
-		0x4fe1beb6, 0xad88f017, 0xac20c966, 0x3ace7db4,
-		0x4adf6318, 0x311ae582, 0x33519760, 0x7f536245,
-		0x7764b1e0, 0xae6bbb84, 0xa081fe1c, 0x2b08f994,
-		0x68487058, 0xfd458f19, 0x6cde9487, 0xf87b52b7,
-		0xd373ab23, 0x024b72e2, 0x8f1fe357, 0xab55662a,
-		0x28ebb207, 0xc2b52f03, 0x7bc5869a, 0x0837d3a5,
-		0x872830f2, 0xa5bf23b2, 0x6a0302ba, 0x8216ed5c,
-		0x1ccf8a2b, 0xb479a792, 0xf207f3f0, 0xe2694ea1,
-		0xf4da65cd, 0xbe0506d5, 0x6234d11f, 0xfea6c48a,
-		0x532e349d, 0x55f3a2a0, 0xe18a0532, 0xebf6a475,
-		0xec830b39, 0xef6040aa, 0x9f715e06, 0x106ebd51,
-		0x8a213ef9, 0x06dd963d, 0x053eddae, 0xbde64d46,
-		0x8d5491b5, 0x5dc47105, 0xd406046f, 0x155060ff,
-		0xfb981924, 0xe9bdd697, 0x434089cc, 0x9ed96777,
-		0x42e8b0bd, 0x8b890788, 0x5b19e738, 0xeec879db,
-		0x0a7ca147, 0x0f427ce9, 0x1e84f8c9, 0x00000000,
-		0x86800983, 0xed2b3248, 0x70111eac, 0x725a6c4e,
-		0xff0efdfb, 0x38850f56, 0xd5ae3d1e, 0x392d3627,
-		0xd90f0a64, 0xa65c6821, 0x545b9bd1, 0x2e36243a,
-		0x670a0cb1, 0xe757930f, 0x96eeb4d2, 0x919b1b9e,
-		0xc5c0804f, 0x20dc61a2, 0x4b775a69, 0x1a121c16,
-		0xba93e20a, 0x2aa0c0e5, 0xe0223c43, 0x171b121d,
-		0x0d090e0b, 0xc78bf2ad, 0xa8b62db9, 0xa91e14c8,
-		0x19f15785, 0x0775af4c, 0xdd99eebb, 0x607fa3fd,
-		0x2601f79f, 0xf5725cbc, 0x3b6644c5, 0x7efb5b34,
-		0x29438b76, 0xc623cbdc, 0xfcedb668, 0xf1e4b863,
-		0xdc31d7ca, 0x85634210, 0x22971340, 0x11c68420,
-		0x244a857d, 0x3dbbd2f8, 0x32f9ae11, 0xa129c76d,
-		0x2f9e1d4b, 0x30b2dcf3, 0x52860dec, 0xe3c177d0,
-		0x16b32b6c, 0xb970a999, 0x489411fa, 0x64e94722,
-		0x8cfca8c4, 0x3ff0a01a, 0x2c7d56d8, 0x903322ef,
-		0x4e4987c7, 0xd138d9c1, 0xa2ca8cfe, 0x0bd49836,
-		0x81f5a6cf, 0xde7aa528, 0x8eb7da26, 0xbfad3fa4,
-		0x9d3a2ce4, 0x9278500d, 0xcc5f6a9b, 0x467e5462,
-		0x138df6c2, 0xb8d890e8, 0xf7392e5e, 0xafc382f5,
-		0x805d9fbe, 0x93d0697c, 0x2dd56fa9, 0x1225cfb3,
-		0x99acc83b, 0x7d1810a7, 0x639ce86e, 0xbb3bdb7b,
-		0x7826cd09, 0x18596ef4, 0xb79aec01, 0x9a4f83a8,
-		0x6e95e665, 0xe6ffaa7e, 0xcfbc2108, 0xe815efe6,
-		0x9be7bad9, 0x366f4ace, 0x099fead4, 0x7cb029d6,
-		0xb2a431af, 0x233f2a31, 0x94a5c630, 0x66a235c0,
-		0xbc4e7437, 0xca82fca6, 0xd090e0b0, 0xd8a73315,
-		0x9804f14a, 0xdaec41f7, 0x50cd7f0e, 0xf691172f,
-		0xd64d768d, 0xb0ef434d, 0x4daacc54, 0x0496e4df,
-		0xb5d19ee3, 0x886a4c1b, 0x1f2cc1b8, 0x5165467f,
-		0xea5e9d04, 0x358c015d, 0x7487fa73, 0x410bfb2e,
-		0x1d67b35a, 0xd2db9252, 0x5610e933, 0x47d66d13,
-		0x61d79a8c, 0x0ca1377a, 0x14f8598e, 0x3c13eb89,
-		0x27a9ceee, 0xc961b735, 0xe51ce1ed, 0xb1477a3c,
-		0xdfd29c59, 0x73f2553f, 0xce141879, 0x37c773bf,
-		0xcdf753ea, 0xaafd5f5b, 0x6f3ddf14, 0xdb447886,
-		0xf3afca81, 0xc468b93e, 0x3424382c, 0x40a3c25f,
-		0xc31d1672, 0x25e2bc0c, 0x493c288b, 0x950dff41,
-		0x01a83971, 0xb30c08de, 0xe4b4d89c, 0xc1566490,
-		0x84cb7b61, 0xb632d570, 0x5c6c4874, 0x57b8d042,
-	}, {
-		0xf45150a7, 0x417e5365, 0x171ac3a4, 0x273a965e,
-		0xab3bcb6b, 0x9d1ff145, 0xfaacab58, 0xe34b9303,
-		0x302055fa, 0x76adf66d, 0xcc889176, 0x02f5254c,
-		0xe54ffcd7, 0x2ac5d7cb, 0x35268044, 0x62b58fa3,
-		0xb1de495a, 0xba25671b, 0xea45980e, 0xfe5de1c0,
-		0x2fc30275, 0x4c8112f0, 0x468da397, 0xd36bc6f9,
-		0x8f03e75f, 0x9215959c, 0x6dbfeb7a, 0x5295da59,
-		0xbed42d83, 0x7458d321, 0xe0492969, 0xc98e44c8,
-		0xc2756a89, 0x8ef47879, 0x58996b3e, 0xb927dd71,
-		0xe1beb64f, 0x88f017ad, 0x20c966ac, 0xce7db43a,
-		0xdf63184a, 0x1ae58231, 0x51976033, 0x5362457f,
-		0x64b1e077, 0x6bbb84ae, 0x81fe1ca0, 0x08f9942b,
-		0x48705868, 0x458f19fd, 0xde94876c, 0x7b52b7f8,
-		0x73ab23d3, 0x4b72e202, 0x1fe3578f, 0x55662aab,
-		0xebb20728, 0xb52f03c2, 0xc5869a7b, 0x37d3a508,
-		0x2830f287, 0xbf23b2a5, 0x0302ba6a, 0x16ed5c82,
-		0xcf8a2b1c, 0x79a792b4, 0x07f3f0f2, 0x694ea1e2,
-		0xda65cdf4, 0x0506d5be, 0x34d11f62, 0xa6c48afe,
-		0x2e349d53, 0xf3a2a055, 0x8a0532e1, 0xf6a475eb,
-		0x830b39ec, 0x6040aaef, 0x715e069f, 0x6ebd5110,
-		0x213ef98a, 0xdd963d06, 0x3eddae05, 0xe64d46bd,
-		0x5491b58d, 0xc471055d, 0x06046fd4, 0x5060ff15,
-		0x981924fb, 0xbdd697e9, 0x4089cc43, 0xd967779e,
-		0xe8b0bd42, 0x8907888b, 0x19e7385b, 0xc879dbee,
-		0x7ca1470a, 0x427ce90f, 0x84f8c91e, 0x00000000,
-		0x80098386, 0x2b3248ed, 0x111eac70, 0x5a6c4e72,
-		0x0efdfbff, 0x850f5638, 0xae3d1ed5, 0x2d362739,
-		0x0f0a64d9, 0x5c6821a6, 0x5b9bd154, 0x36243a2e,
-		0x0a0cb167, 0x57930fe7, 0xeeb4d296, 0x9b1b9e91,
-		0xc0804fc5, 0xdc61a220, 0x775a694b, 0x121c161a,
-		0x93e20aba, 0xa0c0e52a, 0x223c43e0, 0x1b121d17,
-		0x090e0b0d, 0x8bf2adc7, 0xb62db9a8, 0x1e14c8a9,
-		0xf1578519, 0x75af4c07, 0x99eebbdd, 0x7fa3fd60,
-		0x01f79f26, 0x725cbcf5, 0x6644c53b, 0xfb5b347e,
-		0x438b7629, 0x23cbdcc6, 0xedb668fc, 0xe4b863f1,
-		0x31d7cadc, 0x63421085, 0x97134022, 0xc6842011,
-		0x4a857d24, 0xbbd2f83d, 0xf9ae1132, 0x29c76da1,
-		0x9e1d4b2f, 0xb2dcf330, 0x860dec52, 0xc177d0e3,
-		0xb32b6c16, 0x70a999b9, 0x9411fa48, 0xe9472264,
-		0xfca8c48c, 0xf0a01a3f, 0x7d56d82c, 0x3322ef90,
-		0x4987c74e, 0x38d9c1d1, 0xca8cfea2, 0xd498360b,
-		0xf5a6cf81, 0x7aa528de, 0xb7da268e, 0xad3fa4bf,
-		0x3a2ce49d, 0x78500d92, 0x5f6a9bcc, 0x7e546246,
-		0x8df6c213, 0xd890e8b8, 0x392e5ef7, 0xc382f5af,
-		0x5d9fbe80, 0xd0697c93, 0xd56fa92d, 0x25cfb312,
-		0xacc83b99, 0x1810a77d, 0x9ce86e63, 0x3bdb7bbb,
-		0x26cd0978, 0x596ef418, 0x9aec01b7, 0x4f83a89a,
-		0x95e6656e, 0xffaa7ee6, 0xbc2108cf, 0x15efe6e8,
-		0xe7bad99b, 0x6f4ace36, 0x9fead409, 0xb029d67c,
-		0xa431afb2, 0x3f2a3123, 0xa5c63094, 0xa235c066,
-		0x4e7437bc, 0x82fca6ca, 0x90e0b0d0, 0xa73315d8,
-		0x04f14a98, 0xec41f7da, 0xcd7f0e50, 0x91172ff6,
-		0x4d768dd6, 0xef434db0, 0xaacc544d, 0x96e4df04,
-		0xd19ee3b5, 0x6a4c1b88, 0x2cc1b81f, 0x65467f51,
-		0x5e9d04ea, 0x8c015d35, 0x87fa7374, 0x0bfb2e41,
-		0x67b35a1d, 0xdb9252d2, 0x10e93356, 0xd66d1347,
-		0xd79a8c61, 0xa1377a0c, 0xf8598e14, 0x13eb893c,
-		0xa9ceee27, 0x61b735c9, 0x1ce1ede5, 0x477a3cb1,
-		0xd29c59df, 0xf2553f73, 0x141879ce, 0xc773bf37,
-		0xf753eacd, 0xfd5f5baa, 0x3ddf146f, 0x447886db,
-		0xafca81f3, 0x68b93ec4, 0x24382c34, 0xa3c25f40,
-		0x1d1672c3, 0xe2bc0c25, 0x3c288b49, 0x0dff4195,
-		0xa8397101, 0x0c08deb3, 0xb4d89ce4, 0x566490c1,
-		0xcb7b6184, 0x32d570b6, 0x6c48745c, 0xb8d04257,
-	}, {
-		0x5150a7f4, 0x7e536541, 0x1ac3a417, 0x3a965e27,
-		0x3bcb6bab, 0x1ff1459d, 0xacab58fa, 0x4b9303e3,
-		0x2055fa30, 0xadf66d76, 0x889176cc, 0xf5254c02,
-		0x4ffcd7e5, 0xc5d7cb2a, 0x26804435, 0xb58fa362,
-		0xde495ab1, 0x25671bba, 0x45980eea, 0x5de1c0fe,
-		0xc302752f, 0x8112f04c, 0x8da39746, 0x6bc6f9d3,
-		0x03e75f8f, 0x15959c92, 0xbfeb7a6d, 0x95da5952,
-		0xd42d83be, 0x58d32174, 0x492969e0, 0x8e44c8c9,
-		0x756a89c2, 0xf478798e, 0x996b3e58, 0x27dd71b9,
-		0xbeb64fe1, 0xf017ad88, 0xc966ac20, 0x7db43ace,
-		0x63184adf, 0xe582311a, 0x97603351, 0x62457f53,
-		0xb1e07764, 0xbb84ae6b, 0xfe1ca081, 0xf9942b08,
-		0x70586848, 0x8f19fd45, 0x94876cde, 0x52b7f87b,
-		0xab23d373, 0x72e2024b, 0xe3578f1f, 0x662aab55,
-		0xb20728eb, 0x2f03c2b5, 0x869a7bc5, 0xd3a50837,
-		0x30f28728, 0x23b2a5bf, 0x02ba6a03, 0xed5c8216,
-		0x8a2b1ccf, 0xa792b479, 0xf3f0f207, 0x4ea1e269,
-		0x65cdf4da, 0x06d5be05, 0xd11f6234, 0xc48afea6,
-		0x349d532e, 0xa2a055f3, 0x0532e18a, 0xa475ebf6,
-		0x0b39ec83, 0x40aaef60, 0x5e069f71, 0xbd51106e,
-		0x3ef98a21, 0x963d06dd, 0xddae053e, 0x4d46bde6,
-		0x91b58d54, 0x71055dc4, 0x046fd406, 0x60ff1550,
-		0x1924fb98, 0xd697e9bd, 0x89cc4340, 0x67779ed9,
-		0xb0bd42e8, 0x07888b89, 0xe7385b19, 0x79dbeec8,
-		0xa1470a7c, 0x7ce90f42, 0xf8c91e84, 0x00000000,
-		0x09838680, 0x3248ed2b, 0x1eac7011, 0x6c4e725a,
-		0xfdfbff0e, 0x0f563885, 0x3d1ed5ae, 0x3627392d,
-		0x0a64d90f, 0x6821a65c, 0x9bd1545b, 0x243a2e36,
-		0x0cb1670a, 0x930fe757, 0xb4d296ee, 0x1b9e919b,
-		0x804fc5c0, 0x61a220dc, 0x5a694b77, 0x1c161a12,
-		0xe20aba93, 0xc0e52aa0, 0x3c43e022, 0x121d171b,
-		0x0e0b0d09, 0xf2adc78b, 0x2db9a8b6, 0x14c8a91e,
-		0x578519f1, 0xaf4c0775, 0xeebbdd99, 0xa3fd607f,
-		0xf79f2601, 0x5cbcf572, 0x44c53b66, 0x5b347efb,
-		0x8b762943, 0xcbdcc623, 0xb668fced, 0xb863f1e4,
-		0xd7cadc31, 0x42108563, 0x13402297, 0x842011c6,
-		0x857d244a, 0xd2f83dbb, 0xae1132f9, 0xc76da129,
-		0x1d4b2f9e, 0xdcf330b2, 0x0dec5286, 0x77d0e3c1,
-		0x2b6c16b3, 0xa999b970, 0x11fa4894, 0x472264e9,
-		0xa8c48cfc, 0xa01a3ff0, 0x56d82c7d, 0x22ef9033,
-		0x87c74e49, 0xd9c1d138, 0x8cfea2ca, 0x98360bd4,
-		0xa6cf81f5, 0xa528de7a, 0xda268eb7, 0x3fa4bfad,
-		0x2ce49d3a, 0x500d9278, 0x6a9bcc5f, 0x5462467e,
-		0xf6c2138d, 0x90e8b8d8, 0x2e5ef739, 0x82f5afc3,
-		0x9fbe805d, 0x697c93d0, 0x6fa92dd5, 0xcfb31225,
-		0xc83b99ac, 0x10a77d18, 0xe86e639c, 0xdb7bbb3b,
-		0xcd097826, 0x6ef41859, 0xec01b79a, 0x83a89a4f,
-		0xe6656e95, 0xaa7ee6ff, 0x2108cfbc, 0xefe6e815,
-		0xbad99be7, 0x4ace366f, 0xead4099f, 0x29d67cb0,
-		0x31afb2a4, 0x2a31233f, 0xc63094a5, 0x35c066a2,
-		0x7437bc4e, 0xfca6ca82, 0xe0b0d090, 0x3315d8a7,
-		0xf14a9804, 0x41f7daec, 0x7f0e50cd, 0x172ff691,
-		0x768dd64d, 0x434db0ef, 0xcc544daa, 0xe4df0496,
-		0x9ee3b5d1, 0x4c1b886a, 0xc1b81f2c, 0x467f5165,
-		0x9d04ea5e, 0x015d358c, 0xfa737487, 0xfb2e410b,
-		0xb35a1d67, 0x9252d2db, 0xe9335610, 0x6d1347d6,
-		0x9a8c61d7, 0x377a0ca1, 0x598e14f8, 0xeb893c13,
-		0xceee27a9, 0xb735c961, 0xe1ede51c, 0x7a3cb147,
-		0x9c59dfd2, 0x553f73f2, 0x1879ce14, 0x73bf37c7,
-		0x53eacdf7, 0x5f5baafd, 0xdf146f3d, 0x7886db44,
-		0xca81f3af, 0xb93ec468, 0x382c3424, 0xc25f40a3,
-		0x1672c31d, 0xbc0c25e2, 0x288b493c, 0xff41950d,
-		0x397101a8, 0x08deb30c, 0xd89ce4b4, 0x6490c156,
-		0x7b6184cb, 0xd570b632, 0x48745c6c, 0xd04257b8,
-	}
-};
-
-__visible const u32 crypto_il_tab[4][256] = {
-	{
-		0x00000052, 0x00000009, 0x0000006a, 0x000000d5,
-		0x00000030, 0x00000036, 0x000000a5, 0x00000038,
-		0x000000bf, 0x00000040, 0x000000a3, 0x0000009e,
-		0x00000081, 0x000000f3, 0x000000d7, 0x000000fb,
-		0x0000007c, 0x000000e3, 0x00000039, 0x00000082,
-		0x0000009b, 0x0000002f, 0x000000ff, 0x00000087,
-		0x00000034, 0x0000008e, 0x00000043, 0x00000044,
-		0x000000c4, 0x000000de, 0x000000e9, 0x000000cb,
-		0x00000054, 0x0000007b, 0x00000094, 0x00000032,
-		0x000000a6, 0x000000c2, 0x00000023, 0x0000003d,
-		0x000000ee, 0x0000004c, 0x00000095, 0x0000000b,
-		0x00000042, 0x000000fa, 0x000000c3, 0x0000004e,
-		0x00000008, 0x0000002e, 0x000000a1, 0x00000066,
-		0x00000028, 0x000000d9, 0x00000024, 0x000000b2,
-		0x00000076, 0x0000005b, 0x000000a2, 0x00000049,
-		0x0000006d, 0x0000008b, 0x000000d1, 0x00000025,
-		0x00000072, 0x000000f8, 0x000000f6, 0x00000064,
-		0x00000086, 0x00000068, 0x00000098, 0x00000016,
-		0x000000d4, 0x000000a4, 0x0000005c, 0x000000cc,
-		0x0000005d, 0x00000065, 0x000000b6, 0x00000092,
-		0x0000006c, 0x00000070, 0x00000048, 0x00000050,
-		0x000000fd, 0x000000ed, 0x000000b9, 0x000000da,
-		0x0000005e, 0x00000015, 0x00000046, 0x00000057,
-		0x000000a7, 0x0000008d, 0x0000009d, 0x00000084,
-		0x00000090, 0x000000d8, 0x000000ab, 0x00000000,
-		0x0000008c, 0x000000bc, 0x000000d3, 0x0000000a,
-		0x000000f7, 0x000000e4, 0x00000058, 0x00000005,
-		0x000000b8, 0x000000b3, 0x00000045, 0x00000006,
-		0x000000d0, 0x0000002c, 0x0000001e, 0x0000008f,
-		0x000000ca, 0x0000003f, 0x0000000f, 0x00000002,
-		0x000000c1, 0x000000af, 0x000000bd, 0x00000003,
-		0x00000001, 0x00000013, 0x0000008a, 0x0000006b,
-		0x0000003a, 0x00000091, 0x00000011, 0x00000041,
-		0x0000004f, 0x00000067, 0x000000dc, 0x000000ea,
-		0x00000097, 0x000000f2, 0x000000cf, 0x000000ce,
-		0x000000f0, 0x000000b4, 0x000000e6, 0x00000073,
-		0x00000096, 0x000000ac, 0x00000074, 0x00000022,
-		0x000000e7, 0x000000ad, 0x00000035, 0x00000085,
-		0x000000e2, 0x000000f9, 0x00000037, 0x000000e8,
-		0x0000001c, 0x00000075, 0x000000df, 0x0000006e,
-		0x00000047, 0x000000f1, 0x0000001a, 0x00000071,
-		0x0000001d, 0x00000029, 0x000000c5, 0x00000089,
-		0x0000006f, 0x000000b7, 0x00000062, 0x0000000e,
-		0x000000aa, 0x00000018, 0x000000be, 0x0000001b,
-		0x000000fc, 0x00000056, 0x0000003e, 0x0000004b,
-		0x000000c6, 0x000000d2, 0x00000079, 0x00000020,
-		0x0000009a, 0x000000db, 0x000000c0, 0x000000fe,
-		0x00000078, 0x000000cd, 0x0000005a, 0x000000f4,
-		0x0000001f, 0x000000dd, 0x000000a8, 0x00000033,
-		0x00000088, 0x00000007, 0x000000c7, 0x00000031,
-		0x000000b1, 0x00000012, 0x00000010, 0x00000059,
-		0x00000027, 0x00000080, 0x000000ec, 0x0000005f,
-		0x00000060, 0x00000051, 0x0000007f, 0x000000a9,
-		0x00000019, 0x000000b5, 0x0000004a, 0x0000000d,
-		0x0000002d, 0x000000e5, 0x0000007a, 0x0000009f,
-		0x00000093, 0x000000c9, 0x0000009c, 0x000000ef,
-		0x000000a0, 0x000000e0, 0x0000003b, 0x0000004d,
-		0x000000ae, 0x0000002a, 0x000000f5, 0x000000b0,
-		0x000000c8, 0x000000eb, 0x000000bb, 0x0000003c,
-		0x00000083, 0x00000053, 0x00000099, 0x00000061,
-		0x00000017, 0x0000002b, 0x00000004, 0x0000007e,
-		0x000000ba, 0x00000077, 0x000000d6, 0x00000026,
-		0x000000e1, 0x00000069, 0x00000014, 0x00000063,
-		0x00000055, 0x00000021, 0x0000000c, 0x0000007d,
-	}, {
-		0x00005200, 0x00000900, 0x00006a00, 0x0000d500,
-		0x00003000, 0x00003600, 0x0000a500, 0x00003800,
-		0x0000bf00, 0x00004000, 0x0000a300, 0x00009e00,
-		0x00008100, 0x0000f300, 0x0000d700, 0x0000fb00,
-		0x00007c00, 0x0000e300, 0x00003900, 0x00008200,
-		0x00009b00, 0x00002f00, 0x0000ff00, 0x00008700,
-		0x00003400, 0x00008e00, 0x00004300, 0x00004400,
-		0x0000c400, 0x0000de00, 0x0000e900, 0x0000cb00,
-		0x00005400, 0x00007b00, 0x00009400, 0x00003200,
-		0x0000a600, 0x0000c200, 0x00002300, 0x00003d00,
-		0x0000ee00, 0x00004c00, 0x00009500, 0x00000b00,
-		0x00004200, 0x0000fa00, 0x0000c300, 0x00004e00,
-		0x00000800, 0x00002e00, 0x0000a100, 0x00006600,
-		0x00002800, 0x0000d900, 0x00002400, 0x0000b200,
-		0x00007600, 0x00005b00, 0x0000a200, 0x00004900,
-		0x00006d00, 0x00008b00, 0x0000d100, 0x00002500,
-		0x00007200, 0x0000f800, 0x0000f600, 0x00006400,
-		0x00008600, 0x00006800, 0x00009800, 0x00001600,
-		0x0000d400, 0x0000a400, 0x00005c00, 0x0000cc00,
-		0x00005d00, 0x00006500, 0x0000b600, 0x00009200,
-		0x00006c00, 0x00007000, 0x00004800, 0x00005000,
-		0x0000fd00, 0x0000ed00, 0x0000b900, 0x0000da00,
-		0x00005e00, 0x00001500, 0x00004600, 0x00005700,
-		0x0000a700, 0x00008d00, 0x00009d00, 0x00008400,
-		0x00009000, 0x0000d800, 0x0000ab00, 0x00000000,
-		0x00008c00, 0x0000bc00, 0x0000d300, 0x00000a00,
-		0x0000f700, 0x0000e400, 0x00005800, 0x00000500,
-		0x0000b800, 0x0000b300, 0x00004500, 0x00000600,
-		0x0000d000, 0x00002c00, 0x00001e00, 0x00008f00,
-		0x0000ca00, 0x00003f00, 0x00000f00, 0x00000200,
-		0x0000c100, 0x0000af00, 0x0000bd00, 0x00000300,
-		0x00000100, 0x00001300, 0x00008a00, 0x00006b00,
-		0x00003a00, 0x00009100, 0x00001100, 0x00004100,
-		0x00004f00, 0x00006700, 0x0000dc00, 0x0000ea00,
-		0x00009700, 0x0000f200, 0x0000cf00, 0x0000ce00,
-		0x0000f000, 0x0000b400, 0x0000e600, 0x00007300,
-		0x00009600, 0x0000ac00, 0x00007400, 0x00002200,
-		0x0000e700, 0x0000ad00, 0x00003500, 0x00008500,
-		0x0000e200, 0x0000f900, 0x00003700, 0x0000e800,
-		0x00001c00, 0x00007500, 0x0000df00, 0x00006e00,
-		0x00004700, 0x0000f100, 0x00001a00, 0x00007100,
-		0x00001d00, 0x00002900, 0x0000c500, 0x00008900,
-		0x00006f00, 0x0000b700, 0x00006200, 0x00000e00,
-		0x0000aa00, 0x00001800, 0x0000be00, 0x00001b00,
-		0x0000fc00, 0x00005600, 0x00003e00, 0x00004b00,
-		0x0000c600, 0x0000d200, 0x00007900, 0x00002000,
-		0x00009a00, 0x0000db00, 0x0000c000, 0x0000fe00,
-		0x00007800, 0x0000cd00, 0x00005a00, 0x0000f400,
-		0x00001f00, 0x0000dd00, 0x0000a800, 0x00003300,
-		0x00008800, 0x00000700, 0x0000c700, 0x00003100,
-		0x0000b100, 0x00001200, 0x00001000, 0x00005900,
-		0x00002700, 0x00008000, 0x0000ec00, 0x00005f00,
-		0x00006000, 0x00005100, 0x00007f00, 0x0000a900,
-		0x00001900, 0x0000b500, 0x00004a00, 0x00000d00,
-		0x00002d00, 0x0000e500, 0x00007a00, 0x00009f00,
-		0x00009300, 0x0000c900, 0x00009c00, 0x0000ef00,
-		0x0000a000, 0x0000e000, 0x00003b00, 0x00004d00,
-		0x0000ae00, 0x00002a00, 0x0000f500, 0x0000b000,
-		0x0000c800, 0x0000eb00, 0x0000bb00, 0x00003c00,
-		0x00008300, 0x00005300, 0x00009900, 0x00006100,
-		0x00001700, 0x00002b00, 0x00000400, 0x00007e00,
-		0x0000ba00, 0x00007700, 0x0000d600, 0x00002600,
-		0x0000e100, 0x00006900, 0x00001400, 0x00006300,
-		0x00005500, 0x00002100, 0x00000c00, 0x00007d00,
-	}, {
-		0x00520000, 0x00090000, 0x006a0000, 0x00d50000,
-		0x00300000, 0x00360000, 0x00a50000, 0x00380000,
-		0x00bf0000, 0x00400000, 0x00a30000, 0x009e0000,
-		0x00810000, 0x00f30000, 0x00d70000, 0x00fb0000,
-		0x007c0000, 0x00e30000, 0x00390000, 0x00820000,
-		0x009b0000, 0x002f0000, 0x00ff0000, 0x00870000,
-		0x00340000, 0x008e0000, 0x00430000, 0x00440000,
-		0x00c40000, 0x00de0000, 0x00e90000, 0x00cb0000,
-		0x00540000, 0x007b0000, 0x00940000, 0x00320000,
-		0x00a60000, 0x00c20000, 0x00230000, 0x003d0000,
-		0x00ee0000, 0x004c0000, 0x00950000, 0x000b0000,
-		0x00420000, 0x00fa0000, 0x00c30000, 0x004e0000,
-		0x00080000, 0x002e0000, 0x00a10000, 0x00660000,
-		0x00280000, 0x00d90000, 0x00240000, 0x00b20000,
-		0x00760000, 0x005b0000, 0x00a20000, 0x00490000,
-		0x006d0000, 0x008b0000, 0x00d10000, 0x00250000,
-		0x00720000, 0x00f80000, 0x00f60000, 0x00640000,
-		0x00860000, 0x00680000, 0x00980000, 0x00160000,
-		0x00d40000, 0x00a40000, 0x005c0000, 0x00cc0000,
-		0x005d0000, 0x00650000, 0x00b60000, 0x00920000,
-		0x006c0000, 0x00700000, 0x00480000, 0x00500000,
-		0x00fd0000, 0x00ed0000, 0x00b90000, 0x00da0000,
-		0x005e0000, 0x00150000, 0x00460000, 0x00570000,
-		0x00a70000, 0x008d0000, 0x009d0000, 0x00840000,
-		0x00900000, 0x00d80000, 0x00ab0000, 0x00000000,
-		0x008c0000, 0x00bc0000, 0x00d30000, 0x000a0000,
-		0x00f70000, 0x00e40000, 0x00580000, 0x00050000,
-		0x00b80000, 0x00b30000, 0x00450000, 0x00060000,
-		0x00d00000, 0x002c0000, 0x001e0000, 0x008f0000,
-		0x00ca0000, 0x003f0000, 0x000f0000, 0x00020000,
-		0x00c10000, 0x00af0000, 0x00bd0000, 0x00030000,
-		0x00010000, 0x00130000, 0x008a0000, 0x006b0000,
-		0x003a0000, 0x00910000, 0x00110000, 0x00410000,
-		0x004f0000, 0x00670000, 0x00dc0000, 0x00ea0000,
-		0x00970000, 0x00f20000, 0x00cf0000, 0x00ce0000,
-		0x00f00000, 0x00b40000, 0x00e60000, 0x00730000,
-		0x00960000, 0x00ac0000, 0x00740000, 0x00220000,
-		0x00e70000, 0x00ad0000, 0x00350000, 0x00850000,
-		0x00e20000, 0x00f90000, 0x00370000, 0x00e80000,
-		0x001c0000, 0x00750000, 0x00df0000, 0x006e0000,
-		0x00470000, 0x00f10000, 0x001a0000, 0x00710000,
-		0x001d0000, 0x00290000, 0x00c50000, 0x00890000,
-		0x006f0000, 0x00b70000, 0x00620000, 0x000e0000,
-		0x00aa0000, 0x00180000, 0x00be0000, 0x001b0000,
-		0x00fc0000, 0x00560000, 0x003e0000, 0x004b0000,
-		0x00c60000, 0x00d20000, 0x00790000, 0x00200000,
-		0x009a0000, 0x00db0000, 0x00c00000, 0x00fe0000,
-		0x00780000, 0x00cd0000, 0x005a0000, 0x00f40000,
-		0x001f0000, 0x00dd0000, 0x00a80000, 0x00330000,
-		0x00880000, 0x00070000, 0x00c70000, 0x00310000,
-		0x00b10000, 0x00120000, 0x00100000, 0x00590000,
-		0x00270000, 0x00800000, 0x00ec0000, 0x005f0000,
-		0x00600000, 0x00510000, 0x007f0000, 0x00a90000,
-		0x00190000, 0x00b50000, 0x004a0000, 0x000d0000,
-		0x002d0000, 0x00e50000, 0x007a0000, 0x009f0000,
-		0x00930000, 0x00c90000, 0x009c0000, 0x00ef0000,
-		0x00a00000, 0x00e00000, 0x003b0000, 0x004d0000,
-		0x00ae0000, 0x002a0000, 0x00f50000, 0x00b00000,
-		0x00c80000, 0x00eb0000, 0x00bb0000, 0x003c0000,
-		0x00830000, 0x00530000, 0x00990000, 0x00610000,
-		0x00170000, 0x002b0000, 0x00040000, 0x007e0000,
-		0x00ba0000, 0x00770000, 0x00d60000, 0x00260000,
-		0x00e10000, 0x00690000, 0x00140000, 0x00630000,
-		0x00550000, 0x00210000, 0x000c0000, 0x007d0000,
-	}, {
-		0x52000000, 0x09000000, 0x6a000000, 0xd5000000,
-		0x30000000, 0x36000000, 0xa5000000, 0x38000000,
-		0xbf000000, 0x40000000, 0xa3000000, 0x9e000000,
-		0x81000000, 0xf3000000, 0xd7000000, 0xfb000000,
-		0x7c000000, 0xe3000000, 0x39000000, 0x82000000,
-		0x9b000000, 0x2f000000, 0xff000000, 0x87000000,
-		0x34000000, 0x8e000000, 0x43000000, 0x44000000,
-		0xc4000000, 0xde000000, 0xe9000000, 0xcb000000,
-		0x54000000, 0x7b000000, 0x94000000, 0x32000000,
-		0xa6000000, 0xc2000000, 0x23000000, 0x3d000000,
-		0xee000000, 0x4c000000, 0x95000000, 0x0b000000,
-		0x42000000, 0xfa000000, 0xc3000000, 0x4e000000,
-		0x08000000, 0x2e000000, 0xa1000000, 0x66000000,
-		0x28000000, 0xd9000000, 0x24000000, 0xb2000000,
-		0x76000000, 0x5b000000, 0xa2000000, 0x49000000,
-		0x6d000000, 0x8b000000, 0xd1000000, 0x25000000,
-		0x72000000, 0xf8000000, 0xf6000000, 0x64000000,
-		0x86000000, 0x68000000, 0x98000000, 0x16000000,
-		0xd4000000, 0xa4000000, 0x5c000000, 0xcc000000,
-		0x5d000000, 0x65000000, 0xb6000000, 0x92000000,
-		0x6c000000, 0x70000000, 0x48000000, 0x50000000,
-		0xfd000000, 0xed000000, 0xb9000000, 0xda000000,
-		0x5e000000, 0x15000000, 0x46000000, 0x57000000,
-		0xa7000000, 0x8d000000, 0x9d000000, 0x84000000,
-		0x90000000, 0xd8000000, 0xab000000, 0x00000000,
-		0x8c000000, 0xbc000000, 0xd3000000, 0x0a000000,
-		0xf7000000, 0xe4000000, 0x58000000, 0x05000000,
-		0xb8000000, 0xb3000000, 0x45000000, 0x06000000,
-		0xd0000000, 0x2c000000, 0x1e000000, 0x8f000000,
-		0xca000000, 0x3f000000, 0x0f000000, 0x02000000,
-		0xc1000000, 0xaf000000, 0xbd000000, 0x03000000,
-		0x01000000, 0x13000000, 0x8a000000, 0x6b000000,
-		0x3a000000, 0x91000000, 0x11000000, 0x41000000,
-		0x4f000000, 0x67000000, 0xdc000000, 0xea000000,
-		0x97000000, 0xf2000000, 0xcf000000, 0xce000000,
-		0xf0000000, 0xb4000000, 0xe6000000, 0x73000000,
-		0x96000000, 0xac000000, 0x74000000, 0x22000000,
-		0xe7000000, 0xad000000, 0x35000000, 0x85000000,
-		0xe2000000, 0xf9000000, 0x37000000, 0xe8000000,
-		0x1c000000, 0x75000000, 0xdf000000, 0x6e000000,
-		0x47000000, 0xf1000000, 0x1a000000, 0x71000000,
-		0x1d000000, 0x29000000, 0xc5000000, 0x89000000,
-		0x6f000000, 0xb7000000, 0x62000000, 0x0e000000,
-		0xaa000000, 0x18000000, 0xbe000000, 0x1b000000,
-		0xfc000000, 0x56000000, 0x3e000000, 0x4b000000,
-		0xc6000000, 0xd2000000, 0x79000000, 0x20000000,
-		0x9a000000, 0xdb000000, 0xc0000000, 0xfe000000,
-		0x78000000, 0xcd000000, 0x5a000000, 0xf4000000,
-		0x1f000000, 0xdd000000, 0xa8000000, 0x33000000,
-		0x88000000, 0x07000000, 0xc7000000, 0x31000000,
-		0xb1000000, 0x12000000, 0x10000000, 0x59000000,
-		0x27000000, 0x80000000, 0xec000000, 0x5f000000,
-		0x60000000, 0x51000000, 0x7f000000, 0xa9000000,
-		0x19000000, 0xb5000000, 0x4a000000, 0x0d000000,
-		0x2d000000, 0xe5000000, 0x7a000000, 0x9f000000,
-		0x93000000, 0xc9000000, 0x9c000000, 0xef000000,
-		0xa0000000, 0xe0000000, 0x3b000000, 0x4d000000,
-		0xae000000, 0x2a000000, 0xf5000000, 0xb0000000,
-		0xc8000000, 0xeb000000, 0xbb000000, 0x3c000000,
-		0x83000000, 0x53000000, 0x99000000, 0x61000000,
-		0x17000000, 0x2b000000, 0x04000000, 0x7e000000,
-		0xba000000, 0x77000000, 0xd6000000, 0x26000000,
-		0xe1000000, 0x69000000, 0x14000000, 0x63000000,
-		0x55000000, 0x21000000, 0x0c000000, 0x7d000000,
-	}
-};
-
-EXPORT_SYMBOL_GPL(crypto_ft_tab);
-EXPORT_SYMBOL_GPL(crypto_fl_tab);
-EXPORT_SYMBOL_GPL(crypto_it_tab);
-EXPORT_SYMBOL_GPL(crypto_il_tab);
-
-/* initialise the key schedule from the user supplied key */
-
-#define star_x(x) (((x) & 0x7f7f7f7f) << 1) ^ ((((x) & 0x80808080) >> 7) * 0x1b)
-
-#define imix_col(y, x)	do {		\
-	u	= star_x(x);		\
-	v	= star_x(u);		\
-	w	= star_x(v);		\
-	t	= w ^ (x);		\
-	(y)	= u ^ v ^ w;		\
-	(y)	^= ror32(u ^ t, 8) ^	\
-		ror32(v ^ t, 16) ^	\
-		ror32(t, 24);		\
-} while (0)
-
-#define ls_box(x)		\
-	crypto_fl_tab[0][byte(x, 0)] ^	\
-	crypto_fl_tab[1][byte(x, 1)] ^	\
-	crypto_fl_tab[2][byte(x, 2)] ^	\
-	crypto_fl_tab[3][byte(x, 3)]
-
-#define loop4(i)	do {		\
-	t = ror32(t, 8);		\
-	t = ls_box(t) ^ rco_tab[i];	\
-	t ^= ctx->key_enc[4 * i];		\
-	ctx->key_enc[4 * i + 4] = t;		\
-	t ^= ctx->key_enc[4 * i + 1];		\
-	ctx->key_enc[4 * i + 5] = t;		\
-	t ^= ctx->key_enc[4 * i + 2];		\
-	ctx->key_enc[4 * i + 6] = t;		\
-	t ^= ctx->key_enc[4 * i + 3];		\
-	ctx->key_enc[4 * i + 7] = t;		\
-} while (0)
-
-#define loop6(i)	do {		\
-	t = ror32(t, 8);		\
-	t = ls_box(t) ^ rco_tab[i];	\
-	t ^= ctx->key_enc[6 * i];		\
-	ctx->key_enc[6 * i + 6] = t;		\
-	t ^= ctx->key_enc[6 * i + 1];		\
-	ctx->key_enc[6 * i + 7] = t;		\
-	t ^= ctx->key_enc[6 * i + 2];		\
-	ctx->key_enc[6 * i + 8] = t;		\
-	t ^= ctx->key_enc[6 * i + 3];		\
-	ctx->key_enc[6 * i + 9] = t;		\
-	t ^= ctx->key_enc[6 * i + 4];		\
-	ctx->key_enc[6 * i + 10] = t;		\
-	t ^= ctx->key_enc[6 * i + 5];		\
-	ctx->key_enc[6 * i + 11] = t;		\
-} while (0)
-
-#define loop8tophalf(i)	do {			\
-	t = ror32(t, 8);			\
-	t = ls_box(t) ^ rco_tab[i];		\
-	t ^= ctx->key_enc[8 * i];			\
-	ctx->key_enc[8 * i + 8] = t;			\
-	t ^= ctx->key_enc[8 * i + 1];			\
-	ctx->key_enc[8 * i + 9] = t;			\
-	t ^= ctx->key_enc[8 * i + 2];			\
-	ctx->key_enc[8 * i + 10] = t;			\
-	t ^= ctx->key_enc[8 * i + 3];			\
-	ctx->key_enc[8 * i + 11] = t;			\
-} while (0)
-
-#define loop8(i)	do {				\
-	loop8tophalf(i);				\
-	t  = ctx->key_enc[8 * i + 4] ^ ls_box(t);	\
-	ctx->key_enc[8 * i + 12] = t;			\
-	t ^= ctx->key_enc[8 * i + 5];			\
-	ctx->key_enc[8 * i + 13] = t;			\
-	t ^= ctx->key_enc[8 * i + 6];			\
-	ctx->key_enc[8 * i + 14] = t;			\
-	t ^= ctx->key_enc[8 * i + 7];			\
-	ctx->key_enc[8 * i + 15] = t;			\
-} while (0)
-
-/**
- * crypto_aes_expand_key - Expands the AES key as described in FIPS-197
- * @ctx:	The location where the computed key will be stored.
- * @in_key:	The supplied key.
- * @key_len:	The length of the supplied key.
- *
- * Returns 0 on success. The function fails only if an invalid key size (or
- * pointer) is supplied.
- * The expanded key size is 240 bytes (max of 14 rounds with a unique 16 bytes
- * key schedule plus a 16 bytes key which is used before the first round).
- * The decryption key is prepared for the "Equivalent Inverse Cipher" as
- * described in FIPS-197. The first slot (16 bytes) of each key (enc or dec) is
- * for the initial combination, the second slot for the first round and so on.
- */
-int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
-		unsigned int key_len)
-{
-	u32 i, t, u, v, w, j;
-
-	if (key_len != AES_KEYSIZE_128 && key_len != AES_KEYSIZE_192 &&
-			key_len != AES_KEYSIZE_256)
-		return -EINVAL;
-
-	ctx->key_length = key_len;
-
-	ctx->key_enc[0] = get_unaligned_le32(in_key);
-	ctx->key_enc[1] = get_unaligned_le32(in_key + 4);
-	ctx->key_enc[2] = get_unaligned_le32(in_key + 8);
-	ctx->key_enc[3] = get_unaligned_le32(in_key + 12);
-
-	ctx->key_dec[key_len + 24] = ctx->key_enc[0];
-	ctx->key_dec[key_len + 25] = ctx->key_enc[1];
-	ctx->key_dec[key_len + 26] = ctx->key_enc[2];
-	ctx->key_dec[key_len + 27] = ctx->key_enc[3];
-
-	switch (key_len) {
-	case AES_KEYSIZE_128:
-		t = ctx->key_enc[3];
-		for (i = 0; i < 10; ++i)
-			loop4(i);
-		break;
-
-	case AES_KEYSIZE_192:
-		ctx->key_enc[4] = get_unaligned_le32(in_key + 16);
-		t = ctx->key_enc[5] = get_unaligned_le32(in_key + 20);
-		for (i = 0; i < 8; ++i)
-			loop6(i);
-		break;
-
-	case AES_KEYSIZE_256:
-		ctx->key_enc[4] = get_unaligned_le32(in_key + 16);
-		ctx->key_enc[5] = get_unaligned_le32(in_key + 20);
-		ctx->key_enc[6] = get_unaligned_le32(in_key + 24);
-		t = ctx->key_enc[7] = get_unaligned_le32(in_key + 28);
-		for (i = 0; i < 6; ++i)
-			loop8(i);
-		loop8tophalf(i);
-		break;
-	}
-
-	ctx->key_dec[0] = ctx->key_enc[key_len + 24];
-	ctx->key_dec[1] = ctx->key_enc[key_len + 25];
-	ctx->key_dec[2] = ctx->key_enc[key_len + 26];
-	ctx->key_dec[3] = ctx->key_enc[key_len + 27];
-
-	for (i = 4; i < key_len + 24; ++i) {
-		j = key_len + 24 - (i & ~3) + (i & 3);
-		imix_col(ctx->key_dec[j], ctx->key_enc[i]);
-	}
-	return 0;
-}
-EXPORT_SYMBOL_GPL(crypto_aes_expand_key);
-
-/**
- * crypto_aes_set_key - Set the AES key.
- * @tfm:	The %crypto_tfm that is used in the context.
- * @in_key:	The input key.
- * @key_len:	The size of the key.
- *
- * Returns 0 on success, on failure the %CRYPTO_TFM_RES_BAD_KEY_LEN flag in tfm
- * is set. The function uses crypto_aes_expand_key() to expand the key.
- * &crypto_aes_ctx _must_ be the private data embedded in @tfm which is
- * retrieved with crypto_tfm_ctx().
- */
-int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key,
-		unsigned int key_len)
-{
-	struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-	u32 *flags = &tfm->crt_flags;
-	int ret;
-
-	ret = crypto_aes_expand_key(ctx, in_key, key_len);
-	if (!ret)
-		return 0;
-
-	*flags |= CRYPTO_TFM_RES_BAD_KEY_LEN;
-	return -EINVAL;
-}
-EXPORT_SYMBOL_GPL(crypto_aes_set_key);
-
-/* encrypt a block of text */
-
-#define f_rn(bo, bi, n, k)	do {				\
-	bo[n] = crypto_ft_tab[0][byte(bi[n], 0)] ^			\
-		crypto_ft_tab[1][byte(bi[(n + 1) & 3], 1)] ^		\
-		crypto_ft_tab[2][byte(bi[(n + 2) & 3], 2)] ^		\
-		crypto_ft_tab[3][byte(bi[(n + 3) & 3], 3)] ^ *(k + n);	\
-} while (0)
-
-#define f_nround(bo, bi, k)	do {\
-	f_rn(bo, bi, 0, k);	\
-	f_rn(bo, bi, 1, k);	\
-	f_rn(bo, bi, 2, k);	\
-	f_rn(bo, bi, 3, k);	\
-	k += 4;			\
-} while (0)
-
-#define f_rl(bo, bi, n, k)	do {				\
-	bo[n] = crypto_fl_tab[0][byte(bi[n], 0)] ^			\
-		crypto_fl_tab[1][byte(bi[(n + 1) & 3], 1)] ^		\
-		crypto_fl_tab[2][byte(bi[(n + 2) & 3], 2)] ^		\
-		crypto_fl_tab[3][byte(bi[(n + 3) & 3], 3)] ^ *(k + n);	\
-} while (0)
-
-#define f_lround(bo, bi, k)	do {\
-	f_rl(bo, bi, 0, k);	\
-	f_rl(bo, bi, 1, k);	\
-	f_rl(bo, bi, 2, k);	\
-	f_rl(bo, bi, 3, k);	\
-} while (0)
-
-static void aes_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
-{
-	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-	u32 b0[4], b1[4];
-	const u32 *kp = ctx->key_enc + 4;
-	const int key_len = ctx->key_length;
-
-	b0[0] = ctx->key_enc[0] ^ get_unaligned_le32(in);
-	b0[1] = ctx->key_enc[1] ^ get_unaligned_le32(in + 4);
-	b0[2] = ctx->key_enc[2] ^ get_unaligned_le32(in + 8);
-	b0[3] = ctx->key_enc[3] ^ get_unaligned_le32(in + 12);
-
-	if (key_len > 24) {
-		f_nround(b1, b0, kp);
-		f_nround(b0, b1, kp);
-	}
-
-	if (key_len > 16) {
-		f_nround(b1, b0, kp);
-		f_nround(b0, b1, kp);
-	}
-
-	f_nround(b1, b0, kp);
-	f_nround(b0, b1, kp);
-	f_nround(b1, b0, kp);
-	f_nround(b0, b1, kp);
-	f_nround(b1, b0, kp);
-	f_nround(b0, b1, kp);
-	f_nround(b1, b0, kp);
-	f_nround(b0, b1, kp);
-	f_nround(b1, b0, kp);
-	f_lround(b0, b1, kp);
-
-	put_unaligned_le32(b0[0], out);
-	put_unaligned_le32(b0[1], out + 4);
-	put_unaligned_le32(b0[2], out + 8);
-	put_unaligned_le32(b0[3], out + 12);
-}
-
-/* decrypt a block of text */
-
-#define i_rn(bo, bi, n, k)	do {				\
-	bo[n] = crypto_it_tab[0][byte(bi[n], 0)] ^			\
-		crypto_it_tab[1][byte(bi[(n + 3) & 3], 1)] ^		\
-		crypto_it_tab[2][byte(bi[(n + 2) & 3], 2)] ^		\
-		crypto_it_tab[3][byte(bi[(n + 1) & 3], 3)] ^ *(k + n);	\
-} while (0)
-
-#define i_nround(bo, bi, k)	do {\
-	i_rn(bo, bi, 0, k);	\
-	i_rn(bo, bi, 1, k);	\
-	i_rn(bo, bi, 2, k);	\
-	i_rn(bo, bi, 3, k);	\
-	k += 4;			\
-} while (0)
-
-#define i_rl(bo, bi, n, k)	do {			\
-	bo[n] = crypto_il_tab[0][byte(bi[n], 0)] ^		\
-	crypto_il_tab[1][byte(bi[(n + 3) & 3], 1)] ^		\
-	crypto_il_tab[2][byte(bi[(n + 2) & 3], 2)] ^		\
-	crypto_il_tab[3][byte(bi[(n + 1) & 3], 3)] ^ *(k + n);	\
-} while (0)
-
-#define i_lround(bo, bi, k)	do {\
-	i_rl(bo, bi, 0, k);	\
-	i_rl(bo, bi, 1, k);	\
-	i_rl(bo, bi, 2, k);	\
-	i_rl(bo, bi, 3, k);	\
-} while (0)
-
-static void aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
-{
-	const struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
-	u32 b0[4], b1[4];
-	const int key_len = ctx->key_length;
-	const u32 *kp = ctx->key_dec + 4;
-
-	b0[0] = ctx->key_dec[0] ^ get_unaligned_le32(in);
-	b0[1] = ctx->key_dec[1] ^ get_unaligned_le32(in + 4);
-	b0[2] = ctx->key_dec[2] ^ get_unaligned_le32(in + 8);
-	b0[3] = ctx->key_dec[3] ^ get_unaligned_le32(in + 12);
-
-	if (key_len > 24) {
-		i_nround(b1, b0, kp);
-		i_nround(b0, b1, kp);
-	}
-
-	if (key_len > 16) {
-		i_nround(b1, b0, kp);
-		i_nround(b0, b1, kp);
-	}
-
-	i_nround(b1, b0, kp);
-	i_nround(b0, b1, kp);
-	i_nround(b1, b0, kp);
-	i_nround(b0, b1, kp);
-	i_nround(b1, b0, kp);
-	i_nround(b0, b1, kp);
-	i_nround(b1, b0, kp);
-	i_nround(b0, b1, kp);
-	i_nround(b1, b0, kp);
-	i_lround(b0, b1, kp);
-
-	put_unaligned_le32(b0[0], out);
-	put_unaligned_le32(b0[1], out + 4);
-	put_unaligned_le32(b0[2], out + 8);
-	put_unaligned_le32(b0[3], out + 12);
-}
-
-static struct crypto_alg aes_alg = {
-	.cra_name		=	"aes",
-	.cra_driver_name	=	"aes-generic",
-	.cra_priority		=	100,
-	.cra_flags		=	CRYPTO_ALG_TYPE_CIPHER,
-	.cra_blocksize		=	AES_BLOCK_SIZE,
-	.cra_ctxsize		=	sizeof(struct crypto_aes_ctx),
-	.cra_module		=	THIS_MODULE,
-	.cra_u			=	{
-		.cipher = {
-			.cia_min_keysize	=	AES_MIN_KEY_SIZE,
-			.cia_max_keysize	=	AES_MAX_KEY_SIZE,
-			.cia_setkey		=	crypto_aes_set_key,
-			.cia_encrypt		=	aes_encrypt,
-			.cia_decrypt		=	aes_decrypt
-		}
-	}
-};
-
-static int __init aes_init(void)
-{
-	return crypto_register_alg(&aes_alg);
-}
-
-static void __exit aes_fini(void)
-{
-	crypto_unregister_alg(&aes_alg);
-}
-
-module_init(aes_init);
-module_exit(aes_fini);
-
-MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm");
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_ALIAS_CRYPTO("aes");
-MODULE_ALIAS_CRYPTO("aes-generic");
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 0e8160701833..11f4f1d232ff 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -1258,7 +1258,7 @@ static int chcr_cra_init(struct crypto_tfm *tfm)
 
 	if (get_cryptoalg_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_XTS) {
 		/* To update tweak*/
-		ablkctx->aes_generic = crypto_alloc_cipher("aes-generic", 0, 0);
+		ablkctx->aes_generic = crypto_alloc_cipher("aes", 0, 0);
 		if (IS_ERR(ablkctx->aes_generic)) {
 			pr_err("failed to allocate aes cipher for tweak\n");
 			return PTR_ERR(ablkctx->aes_generic);
@@ -2889,7 +2889,7 @@ static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key,
 	/* Calculate the H = CIPH(K, 0 repeated 16 times).
 	 * It will go in key context
 	 */
-	cipher = crypto_alloc_cipher("aes-generic", 0, 0);
+	cipher = crypto_alloc_cipher("aes", 0, 0);
 	if (IS_ERR(cipher)) {
 		aeadctx->enckey_len = 0;
 		ret = -ENOMEM;
diff --git a/include/crypto/aes.h b/include/crypto/aes.h
index 8a4afdca611e..f0cbe4aacba2 100644
--- a/include/crypto/aes.h
+++ b/include/crypto/aes.h
@@ -31,4 +31,10 @@ int crypto_aes_set_key(struct crypto_tfm *tfm, const u8 *in_key,
 		unsigned int key_len);
 int crypto_aes_expand_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
 		unsigned int key_len);
+
+void crypto_aes_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+			const u8 *in);
+void crypto_aes_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+			const u8 *in);
+
 #endif
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 4/8] crypto: x86/aes-ni - switch to generic fallback
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (2 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 3/8] crypto: aes - retire table based generic AES in favor of fixed time driver Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 5/8] crypto: arm/aes - avoid expanded lookup tables in the final round Ard Biesheuvel
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

The time invariant AES-NI implementation is SIMD based, and so it needs
a fallback in case the code is called from a context where SIMD is not
allowed. On x86, this is really only when executing in the context of an
interrupt taken while in kernel mode, since SIMD is allowed in all other
cases.

There is very little code in the kernel that actually performs AES in
interrupt context, and the code that does (mac80211) only does so when
running on 802.11 devices that have no support for AES in hardware, and
those are rare these days.

So switch to the new AES core code as a fallback. It is much smaller, as
well as more resistant to cache timing attacks, and removing the
dependency allows us to disable the time variant drivers altogether if
desired.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/x86/crypto/aesni-intel_glue.c | 4 ++--
 crypto/Kconfig                     | 3 +--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 4a55cdcdc008..1734e6185800 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -334,7 +334,7 @@ static void aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
 	struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
 
 	if (!irq_fpu_usable())
-		crypto_aes_encrypt_x86(ctx, dst, src);
+		crypto_aes_encrypt(ctx, dst, src);
 	else {
 		kernel_fpu_begin();
 		aesni_enc(ctx, dst, src);
@@ -347,7 +347,7 @@ static void aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
 	struct crypto_aes_ctx *ctx = aes_ctx(crypto_tfm_ctx(tfm));
 
 	if (!irq_fpu_usable())
-		crypto_aes_decrypt_x86(ctx, dst, src);
+		crypto_aes_decrypt(ctx, dst, src);
 	else {
 		kernel_fpu_begin();
 		aesni_dec(ctx, dst, src);
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 7766fea9c18e..8f4b9f3381e2 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -956,8 +956,7 @@ config CRYPTO_AES_NI_INTEL
 	tristate "AES cipher algorithms (AES-NI)"
 	depends on X86
 	select CRYPTO_AEAD
-	select CRYPTO_AES_X86_64 if 64BIT
-	select CRYPTO_AES_586 if !64BIT
+	select CRYPTO_AES
 	select CRYPTO_ALGAPI
 	select CRYPTO_BLKCIPHER
 	select CRYPTO_GLUE_HELPER_X86 if 64BIT
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 5/8] crypto: arm/aes - avoid expanded lookup tables in the final round
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (3 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 4/8] crypto: x86/aes-ni - switch to generic fallback Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 6/8] crypto: arm64/aes " Ard Biesheuvel
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

For the final round, avoid the expanded and padded lookup tables
exported by the generic AES driver. Instead, for encryption, we can
perform byte loads from the same table we used for the inner rounds,
which will still be hot in the caches. For decryption, use the inverse
AES Sbox exported by the generic AES driver, which is 4x smaller than
the inverse table exported by the generic driver.

This significantly reduces the Dcache footprint of our code, and does
not introduce any additional module dependencies, given that we already
rely on the core AES module for the shared key expansion routines.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm/crypto/aes-cipher-core.S | 51 ++++++++++----------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/arch/arm/crypto/aes-cipher-core.S b/arch/arm/crypto/aes-cipher-core.S
index a727692cd9c1..5e9ddc576ec1 100644
--- a/arch/arm/crypto/aes-cipher-core.S
+++ b/arch/arm/crypto/aes-cipher-core.S
@@ -33,19 +33,19 @@
 	.endif
 	.endm
 
-	.macro		__load, out, in, idx
+	.macro		__load, out, in, idx, sz, op
 	.if		__LINUX_ARM_ARCH__ < 7 && \idx > 0
-	ldr		\out, [ttab, \in, lsr #(8 * \idx) - 2]
+	ldr\op		\out, [ttab, \in, lsr #(8 * \idx) - \sz]
 	.else
-	ldr		\out, [ttab, \in, lsl #2]
+	ldr\op		\out, [ttab, \in, lsl #\sz]
 	.endif
 	.endm
 
-	.macro		__hround, out0, out1, in0, in1, in2, in3, t3, t4, enc
+	.macro		__hround, out0, out1, in0, in1, in2, in3, t3, t4, enc, sz, op
 	__select	\out0, \in0, 0
 	__select	t0, \in1, 1
-	__load		\out0, \out0, 0
-	__load		t0, t0, 1
+	__load		\out0, \out0, 0, \sz, \op
+	__load		t0, t0, 1, \sz, \op
 
 	.if		\enc
 	__select	\out1, \in1, 0
@@ -54,10 +54,10 @@
 	__select	\out1, \in3, 0
 	__select	t1, \in0, 1
 	.endif
-	__load		\out1, \out1, 0
+	__load		\out1, \out1, 0, \sz, \op
 	__select	t2, \in2, 2
-	__load		t1, t1, 1
-	__load		t2, t2, 2
+	__load		t1, t1, 1, \sz, \op
+	__load		t2, t2, 2, \sz, \op
 
 	eor		\out0, \out0, t0, ror #24
 
@@ -69,9 +69,9 @@
 	__select	\t3, \in1, 2
 	__select	\t4, \in2, 3
 	.endif
-	__load		\t3, \t3, 2
-	__load		t0, t0, 3
-	__load		\t4, \t4, 3
+	__load		\t3, \t3, 2, \sz, \op
+	__load		t0, t0, 3, \sz, \op
+	__load		\t4, \t4, 3, \sz, \op
 
 	eor		\out1, \out1, t1, ror #24
 	eor		\out0, \out0, t2, ror #16
@@ -83,14 +83,14 @@
 	eor		\out1, \out1, t2
 	.endm
 
-	.macro		fround, out0, out1, out2, out3, in0, in1, in2, in3
-	__hround	\out0, \out1, \in0, \in1, \in2, \in3, \out2, \out3, 1
-	__hround	\out2, \out3, \in2, \in3, \in0, \in1, \in1, \in2, 1
+	.macro		fround, out0, out1, out2, out3, in0, in1, in2, in3, sz=2, op
+	__hround	\out0, \out1, \in0, \in1, \in2, \in3, \out2, \out3, 1, \sz, \op
+	__hround	\out2, \out3, \in2, \in3, \in0, \in1, \in1, \in2, 1, \sz, \op
 	.endm
 
-	.macro		iround, out0, out1, out2, out3, in0, in1, in2, in3
-	__hround	\out0, \out1, \in0, \in3, \in2, \in1, \out2, \out3, 0
-	__hround	\out2, \out3, \in2, \in1, \in0, \in3, \in1, \in0, 0
+	.macro		iround, out0, out1, out2, out3, in0, in1, in2, in3, sz=2, op
+	__hround	\out0, \out1, \in0, \in3, \in2, \in1, \out2, \out3, 0, \sz, \op
+	__hround	\out2, \out3, \in2, \in1, \in0, \in3, \in1, \in0, 0, \sz, \op
 	.endm
 
 	.macro		__rev, out, in
@@ -115,7 +115,7 @@
 	.endif
 	.endm
 
-	.macro		do_crypt, round, ttab, ltab
+	.macro		do_crypt, round, ttab, ltab, bsz
 	push		{r3-r11, lr}
 
 	ldr		r4, [in]
@@ -147,9 +147,12 @@
 
 1:	subs		rounds, rounds, #4
 	\round		r8, r9, r10, r11, r4, r5, r6, r7
-	__adrl		ttab, \ltab, ls
+	bls		2f
 	\round		r4, r5, r6, r7, r8, r9, r10, r11
-	bhi		0b
+	b		0b
+
+2:	__adrl		ttab, \ltab
+	\round		r4, r5, r6, r7, r8, r9, r10, r11, \bsz, b
 
 #ifdef CONFIG_CPU_BIG_ENDIAN
 	__rev		r4, r4
@@ -173,14 +176,12 @@
 
 	.align			6
 	aes_table_reduced	crypto_ft_tab
-	aes_table_reduced	crypto_fl_tab
 	aes_table_reduced	crypto_it_tab
-	aes_table_reduced	crypto_il_tab
 
 ENTRY(__aes_arm_encrypt)
-	do_crypt	fround, crypto_ft_tab, crypto_fl_tab
+	do_crypt	fround, crypto_ft_tab, crypto_ft_tab + 1, 2
 ENDPROC(__aes_arm_encrypt)
 
 ENTRY(__aes_arm_decrypt)
-	do_crypt	iround, crypto_it_tab, crypto_il_tab
+	do_crypt	iround, crypto_it_tab, crypto_aes_inv_sbox, 0
 ENDPROC(__aes_arm_decrypt)
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 6/8] crypto: arm64/aes - avoid expanded lookup tables in the final round
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (4 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 5/8] crypto: arm/aes - avoid expanded lookup tables in the final round Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 7/8] crypto: arm64/aes-neon - reuse Sboxes from AES core module Ard Biesheuvel
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

For the final round, avoid the expanded and padded lookup tables
exported by the generic AES driver. Instead, for encryption, we can
perform byte loads from the same table we used for the inner rounds,
which will still be hot in the caches. For decryption, use the inverse
AES Sbox exported by the generic AES driver, which is 4x smaller than
the inverse table exported by the generic driver.

This significantly reduces the Dcache footprint of our code, and does
not introduce any additional module dependencies, given that we already
rely on the core AES module for the shared key expansion routines. It
also frees up register x18, which is not available as a scratch register
on all platforms, which and so avoiding it improves shareability of this
code.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/crypto/aes-cipher-core.S | 155 ++++++++++++++------
 1 file changed, 108 insertions(+), 47 deletions(-)

diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S
index bbe5dd96135c..fe807f164d83 100644
--- a/arch/arm64/crypto/aes-cipher-core.S
+++ b/arch/arm64/crypto/aes-cipher-core.S
@@ -18,99 +18,160 @@
 	out		.req	x1
 	in		.req	x2
 	rounds		.req	x3
-	tt		.req	x4
-	lt		.req	x2
+	tt		.req	x2
 
-	.macro		__pair, enc, reg0, reg1, in0, in1e, in1d, shift
+	.macro		__ubf1, reg0, reg1, in0, in1e, in1d, sz, shift
 	ubfx		\reg0, \in0, #\shift, #8
-	.if		\enc
 	ubfx		\reg1, \in1e, #\shift, #8
-	.else
+	.endm
+
+	.macro		__ubf0, reg0, reg1, in0, in1e, in1d, sz, shift
+	ubfx		\reg0, \in0, #\shift, #8
 	ubfx		\reg1, \in1d, #\shift, #8
+	.endm
+
+	.macro		__ubf1b, reg0, reg1, in0, in1e, in1d, sz, shift
+	.if		\shift == 0 && \sz > 0
+	ubfiz		\reg0, \in0, #\sz, #8
+	ubfiz		\reg1, \in1e, #\sz, #8
+	.else
+	__ubf1		\reg0, \reg1, \in0, \in1e, \in1d, \sz, \shift
+	.endif
+	.endm
+
+	.macro		__ubf0b, reg0, reg1, in0, in1e, in1d, sz, shift
+	.if		\shift == 0 && \sz > 0
+	ubfiz		\reg0, \in0, #\sz, #8
+	ubfiz		\reg1, \in1d, #\sz, #8
+	.else
+	__ubf0		\reg0, \reg1, \in0, \in1e, \in1d, \sz, \shift
 	.endif
+	.endm
+
+	/*
+	 * AArch64 cannot do byte size indexed loads from a table containing
+	 * 32-bit quantities, i.e., 'ldrb w12, [tt, w12, uxtw #2]' is not a
+	 * valid instruction.
+	 *
+	 * For shift == 0, we can simply fold the size shift of the index
+	 * into the ubfx instruction, by switcing to ubfiz and using \sz as
+	 * the destination offset.
+	 * For shift > 0, we perform a 32-byte wide load instead, which does
+	 * allow an index shift of 2, and discard the high bytes later using
+	 * uxtb or lsl #24.
+	 */
+	.macro		__pair, enc, sz, op, reg0, reg1, in0, in1e, in1d, shift
+	__ubf\enc\op	\reg0, \reg1, \in0, \in1e, \in1d, \sz, \shift
+	.ifnc		\op\sz, b2
+	ldr\op		\reg0, [tt, \reg0, uxtw #\sz]
+	ldr\op		\reg1, [tt, \reg1, uxtw #\sz]
+	.elseif		\shift == 0
+	ldrb		\reg0, [tt, \reg0, uxtw]
+	ldrb		\reg1, [tt, \reg1, uxtw]
+	.else
 	ldr		\reg0, [tt, \reg0, uxtw #2]
 	ldr		\reg1, [tt, \reg1, uxtw #2]
+	.endif
 	.endm
 
-	.macro		__hround, out0, out1, in0, in1, in2, in3, t0, t1, enc
+	.macro		__hround, out0, out1, in0, in1, in2, in3, t0, t1, enc, sz, op
 	ldp		\out0, \out1, [rk], #8
 
-	__pair		\enc, w13, w14, \in0, \in1, \in3, 0
-	__pair		\enc, w15, w16, \in1, \in2, \in0, 8
-	__pair		\enc, w17, w18, \in2, \in3, \in1, 16
-	__pair		\enc, \t0, \t1, \in3, \in0, \in2, 24
-
-	eor		\out0, \out0, w13
-	eor		\out1, \out1, w14
-	eor		\out0, \out0, w15, ror #24
-	eor		\out1, \out1, w16, ror #24
-	eor		\out0, \out0, w17, ror #16
-	eor		\out1, \out1, w18, ror #16
-	eor		\out0, \out0, \t0, ror #8
-	eor		\out1, \out1, \t1, ror #8
+	__pair		\enc, \sz, \op, w12, w13, \in0, \in1, \in3, 0
+	__pair		\enc, \sz, \op, w14, w15, \in3, \in0, \in2, 24
+	__pair		\enc, \sz, \op, w16, w17, \in2, \in3, \in1, 16
+	__pair		\enc, \sz, \op, \t0, \t1, \in1, \in2, \in0, 8
+
+	eor		\out0, \out0, w12
+	eor		\out1, \out1, w13
+
+	.ifnc		\op\sz, b2
+	eor		\out0, \out0, w14, ror #8
+	eor		\out1, \out1, w15, ror #8
+	.else
+CPU_BE(	lsr		w14, w14, #24		)
+CPU_BE(	lsr		w15, w15, #24		)
+
+	eor		\out0, \out0, w14, lsl #24
+	eor		\out1, \out1, w15, lsl #24
+
+CPU_LE(	uxtb		w16, w16		)
+CPU_LE(	uxtb		w17, w17		)
+CPU_LE(	uxtb		\t0, \t0		)
+CPU_LE(	uxtb		\t1, \t1		)
+
+CPU_BE(	lsr		w16, w16, #24		)
+CPU_BE(	lsr		w17, w17, #24		)
+CPU_BE(	lsr		\t0, \t0, #24		)
+CPU_BE(	lsr		\t1, \t1, #24		)
+	.endif
+
+	eor		\out0, \out0, w16, ror #16
+	eor		\out1, \out1, w17, ror #16
+	eor		\out0, \out0, \t0, ror #24
+	eor		\out1, \out1, \t1, ror #24
 	.endm
 
-	.macro		fround, out0, out1, out2, out3, in0, in1, in2, in3
-	__hround	\out0, \out1, \in0, \in1, \in2, \in3, \out2, \out3, 1
-	__hround	\out2, \out3, \in2, \in3, \in0, \in1, \in1, \in2, 1
+	.macro		fround, out0, out1, out2, out3, in0, in1, in2, in3, sz=2, op
+	__hround	\out0, \out1, \in0, \in1, \in2, \in3, \out2, \out3, 1, \sz, \op
+	__hround	\out2, \out3, \in2, \in3, \in0, \in1, \in1, \in2, 1, \sz, \op
 	.endm
 
-	.macro		iround, out0, out1, out2, out3, in0, in1, in2, in3
-	__hround	\out0, \out1, \in0, \in3, \in2, \in1, \out2, \out3, 0
-	__hround	\out2, \out3, \in2, \in1, \in0, \in3, \in1, \in0, 0
+	.macro		iround, out0, out1, out2, out3, in0, in1, in2, in3, sz=2, op
+	__hround	\out0, \out1, \in0, \in3, \in2, \in1, \out2, \out3, 0, \sz, \op
+	__hround	\out2, \out3, \in2, \in1, \in0, \in3, \in1, \in0, 0, \sz, \op
 	.endm
 
-	.macro		do_crypt, round, ttab, ltab
-	ldp		w5, w6, [in]
-	ldp		w7, w8, [in, #8]
-	ldp		w9, w10, [rk], #16
-	ldp		w11, w12, [rk, #-8]
+	.macro		do_crypt, round, ttab, ltab, bsz
+	ldp		w4, w5, [in]
+	ldp		w6, w7, [in, #8]
+	ldp		w8, w9, [rk], #16
+	ldp		w10, w11, [rk, #-8]
 
+CPU_BE(	rev		w4, w4		)
 CPU_BE(	rev		w5, w5		)
 CPU_BE(	rev		w6, w6		)
 CPU_BE(	rev		w7, w7		)
-CPU_BE(	rev		w8, w8		)
 
+	eor		w4, w4, w8
 	eor		w5, w5, w9
 	eor		w6, w6, w10
 	eor		w7, w7, w11
-	eor		w8, w8, w12
 
 	adr_l		tt, \ttab
-	adr_l		lt, \ltab
 
 	tbnz		rounds, #1, 1f
 
-0:	\round		w9, w10, w11, w12, w5, w6, w7, w8
-	\round		w5, w6, w7, w8, w9, w10, w11, w12
+0:	\round		w8, w9, w10, w11, w4, w5, w6, w7
+	\round		w4, w5, w6, w7, w8, w9, w10, w11
 
 1:	subs		rounds, rounds, #4
-	\round		w9, w10, w11, w12, w5, w6, w7, w8
-	csel		tt, tt, lt, hi
-	\round		w5, w6, w7, w8, w9, w10, w11, w12
-	b.hi		0b
-
+	\round		w8, w9, w10, w11, w4, w5, w6, w7
+	b.ls		3f
+2:	\round		w4, w5, w6, w7, w8, w9, w10, w11
+	b		0b
+3:	adr_l		tt, \ltab
+	\round		w4, w5, w6, w7, w8, w9, w10, w11, \bsz, b
+
+CPU_BE(	rev		w4, w4		)
 CPU_BE(	rev		w5, w5		)
 CPU_BE(	rev		w6, w6		)
 CPU_BE(	rev		w7, w7		)
-CPU_BE(	rev		w8, w8		)
 
-	stp		w5, w6, [out]
-	stp		w7, w8, [out, #8]
+	stp		w4, w5, [out]
+	stp		w6, w7, [out, #8]
 	ret
 	.endm
 
 	.align			7
 	aes_table_reduced	crypto_ft_tab
-	aes_table_reduced	crypto_fl_tab
 	aes_table_reduced	crypto_it_tab
-	aes_table_reduced	crypto_il_tab
 
 ENTRY(__aes_arm64_encrypt)
-	do_crypt	fround, crypto_ft_tab, crypto_fl_tab
+	do_crypt	fround, crypto_ft_tab, crypto_ft_tab + 1, 2
 ENDPROC(__aes_arm64_encrypt)
 
 	.align		5
 ENTRY(__aes_arm64_decrypt)
-	do_crypt	iround, crypto_it_tab, crypto_il_tab
+	do_crypt	iround, crypto_it_tab, crypto_aes_inv_sbox, 0
 ENDPROC(__aes_arm64_decrypt)
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 7/8] crypto: arm64/aes-neon - reuse Sboxes from AES core module
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (5 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 6/8] crypto: arm64/aes " Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-18 12:06 ` [PATCH v4 8/8] crypto: aes - add meaningful help text to the various AES drivers Ard Biesheuvel
  2017-07-24  6:59 ` [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

The newly introduced AES core module exposes its Sboxes for the benefit
of the fixed time AES driver. Since the arm64 NEON based implementation
already depends on the same core module for its key expansion routines,
let's use its Sboxes as well, and remove the local copy.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/crypto/aes-neon.S | 74 +-------------------
 1 file changed, 3 insertions(+), 71 deletions(-)

diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S
index f1e3aa2732f9..2acb5f81dcdb 100644
--- a/arch/arm64/crypto/aes-neon.S
+++ b/arch/arm64/crypto/aes-neon.S
@@ -32,7 +32,7 @@
 
 	/* preload the entire Sbox */
 	.macro		prepare, sbox, shiftrows, temp
-	adr		\temp, \sbox
+	adr_l		\temp, \sbox
 	movi		v12.16b, #0x1b
 	ldr		q13, \shiftrows
 	ldr		q14, .Lror32by8
@@ -44,7 +44,7 @@
 
 	/* do preload for encryption */
 	.macro		enc_prepare, ignore0, ignore1, temp
-	prepare		.LForward_Sbox, .LForward_ShiftRows, \temp
+	prepare		crypto_aes_sbox, .LForward_ShiftRows, \temp
 	.endm
 
 	.macro		enc_switch_key, ignore0, ignore1, temp
@@ -53,7 +53,7 @@
 
 	/* do preload for decryption */
 	.macro		dec_prepare, ignore0, ignore1, temp
-	prepare		.LReverse_Sbox, .LReverse_ShiftRows, \temp
+	prepare		crypto_aes_inv_sbox, .LReverse_ShiftRows, \temp
 	.endm
 
 	/* apply SubBytes transformation using the the preloaded Sbox */
@@ -274,74 +274,6 @@
 
 	.text
 	.align		6
-.LForward_Sbox:
-	.byte		0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5
-	.byte		0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76
-	.byte		0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0
-	.byte		0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0
-	.byte		0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc
-	.byte		0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15
-	.byte		0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a
-	.byte		0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75
-	.byte		0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0
-	.byte		0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84
-	.byte		0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b
-	.byte		0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf
-	.byte		0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85
-	.byte		0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8
-	.byte		0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5
-	.byte		0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2
-	.byte		0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17
-	.byte		0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73
-	.byte		0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88
-	.byte		0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb
-	.byte		0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c
-	.byte		0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79
-	.byte		0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9
-	.byte		0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08
-	.byte		0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6
-	.byte		0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a
-	.byte		0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e
-	.byte		0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e
-	.byte		0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94
-	.byte		0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf
-	.byte		0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68
-	.byte		0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16
-
-.LReverse_Sbox:
-	.byte		0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38
-	.byte		0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb
-	.byte		0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87
-	.byte		0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb
-	.byte		0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d
-	.byte		0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e
-	.byte		0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2
-	.byte		0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25
-	.byte		0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16
-	.byte		0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92
-	.byte		0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda
-	.byte		0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84
-	.byte		0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a
-	.byte		0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06
-	.byte		0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02
-	.byte		0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b
-	.byte		0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea
-	.byte		0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73
-	.byte		0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85
-	.byte		0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e
-	.byte		0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89
-	.byte		0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b
-	.byte		0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20
-	.byte		0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4
-	.byte		0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31
-	.byte		0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f
-	.byte		0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d
-	.byte		0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef
-	.byte		0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0
-	.byte		0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61
-	.byte		0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26
-	.byte		0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d
-
 .LForward_ShiftRows:
 	.octa		0x0b06010c07020d08030e09040f0a0500
 
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH v4 8/8] crypto: aes - add meaningful help text to the various AES drivers
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (6 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 7/8] crypto: arm64/aes-neon - reuse Sboxes from AES core module Ard Biesheuvel
@ 2017-07-18 12:06 ` Ard Biesheuvel
  2017-07-24  6:59 ` [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
  8 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-18 12:06 UTC (permalink / raw)
  To: linux-crypto, herbert, nico, ebiggers; +Cc: Ard Biesheuvel

Remove the duplicated boilerplate help text and add a bit of
explanation about the nature of the various AES implementations that
exist for various architectures. In particular, highlight the time
variant nature of some implementations, and the fact that they can be
omitted if required.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm/crypto/Kconfig   |  16 ++-
 arch/arm64/crypto/Kconfig |  30 +++++-
 crypto/Kconfig            | 104 +++++++-------------
 3 files changed, 75 insertions(+), 75 deletions(-)

diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig
index b9adedcc5b2e..f611127c5ef9 100644
--- a/arch/arm/crypto/Kconfig
+++ b/arch/arm/crypto/Kconfig
@@ -62,11 +62,23 @@ config CRYPTO_SHA512_ARM
 	  using optimized ARM assembler and NEON, when available.
 
 config CRYPTO_AES_ARM
-	tristate "Scalar AES cipher for ARM"
+	tristate "Table based AES cipher for 32-bit ARM"
 	select CRYPTO_ALGAPI
 	select CRYPTO_AES
 	help
-	  Use optimized AES assembler routines for ARM platforms.
+	  Table based implementation in 32-bit ARM assembler of the FIPS-197
+	  Advanced Encryption Standard (AES) symmetric cipher algorithm. This
+	  driver reuses the tables exposed by the generic AES driver.
+
+	  For CPUs that lack the special ARMv8-CE instructions, this is the
+	  fastest implementation available of the core cipher, but it may be
+	  susceptible to known-plaintext attacks on the key due to the
+	  correlation between the processing time and the input of the first
+	  round. Therefore, it is recommended to also enable the time invariant
+	  NEON based driver below (CRYPTO_AES_ARM_BS), which will supersede
+	  this driver on NEON capable CPUs when using AES in CBC, CTR and XTS
+	  modes. If time invariance is a requirement, this driver should not
+	  be enabled.
 
 config CRYPTO_AES_ARM_BS
 	tristate "Bit sliced AES using NEON instructions"
diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig
index d92293747d63..bf38680a2dbb 100644
--- a/arch/arm64/crypto/Kconfig
+++ b/arch/arm64/crypto/Kconfig
@@ -42,13 +42,37 @@ config CRYPTO_CRC32_ARM64_CE
 	select CRYPTO_HASH
 
 config CRYPTO_AES_ARM64
-	tristate "AES core cipher using scalar instructions"
+	tristate "Table based AES cipher for 64-bit ARM"
 	select CRYPTO_AES
+	help
+	  Table based implementation in 64-bit ARM assembler of the FIPS-197
+	  Advanced Encryption Standard (AES) symmetric cipher algorithm. This
+	  driver reuses the tables exposed by the generic AES driver.
+
+	  For CPUs that lack the special ARMv8-CE instructions, this is the
+	  fastest implementation available of the core cipher, but it may be
+	  susceptible to known-plaintext attacks on the key due to the
+	  correlation between the processing time and the input of the first
+	  round. Therefore, it is recommended to also enable the time invariant
+	  drivers below (CRYPTO_AES_ARM64_NEON_BLK and CRYPTO_AES_ARM64_BS),
+	  which will supersede this driver when using AES in the specific modes
+	  that they implement. If time invariance is a requirement, this driver
+	  should not be enabled.
 
 config CRYPTO_AES_ARM64_CE
-	tristate "AES core cipher using ARMv8 Crypto Extensions"
-	depends on ARM64 && KERNEL_MODE_NEON
+	tristate "AES cipher using ARMv8 Crypto Extensions"
+	depends on KERNEL_MODE_NEON
 	select CRYPTO_ALGAPI
+	help
+	  Implementation in assembler of the FIPS-197 Advanced Encryption
+	  Standard (AES) symmetric cipher algorithm, using instructions from
+	  ARM's optional ARMv8 Crypto Extensions. This implementation is time
+	  invariant, and is by far the preferred option for CPUs that support
+	  this extension.
+
+	  If in doubt, enable as a module: it will be loaded automatically on
+	  CPUs that support it, and supersede other implementations of the AES
+	  cipher.
 
 config CRYPTO_AES_ARM64_CE_CCM
 	tristate "AES in CCM mode using ARMv8 Crypto Extensions"
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 8f4b9f3381e2..9bec9f7a81d9 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -909,51 +909,37 @@ config CRYPTO_AES
 	  block.
 
 config CRYPTO_AES_586
-	tristate "AES cipher algorithms (i586)"
+	tristate "Table based AES cipher for 32-bit x86"
 	depends on (X86 || UML_X86) && !64BIT
 	select CRYPTO_ALGAPI
 	select CRYPTO_AES
 	help
-	  AES cipher algorithms (FIPS-197). AES uses the Rijndael
-	  algorithm.
-
-	  Rijndael appears to be consistently a very good performer in
-	  both hardware and software across a wide range of computing
-	  environments regardless of its use in feedback or non-feedback
-	  modes. Its key setup time is excellent, and its key agility is
-	  good. Rijndael's very low memory requirements make it very well
-	  suited for restricted-space environments, in which it also
-	  demonstrates excellent performance. Rijndael's operations are
-	  among the easiest to defend against power and timing attacks.
-
-	  The AES specifies three key sizes: 128, 192 and 256 bits
-
-	  See <http://csrc.nist.gov/encryption/aes/> for more information.
+	  Table based implementation in 32-bit x86 assembler of the FIPS-197
+	  Advanced Encryption Standard (AES) symmetric cipher algorithm. For
+	  older 32-bit x86 CPUs that lack the special AES-NI instructions, it
+	  is the fastest implementation available, but it may be susceptible to
+	  known-plaintext attacks on the key due to the correlation between the
+	  processing time and the input of the first round. It reuses the
+	  tables exposed by the generic AES driver. If time invariance is a
+	  requirement, this driver should not be enabled.
 
 config CRYPTO_AES_X86_64
-	tristate "AES cipher algorithms (x86_64)"
+	tristate "Table based AES cipher for 64-bit x86"
 	depends on (X86 || UML_X86) && 64BIT
 	select CRYPTO_ALGAPI
 	select CRYPTO_AES
 	help
-	  AES cipher algorithms (FIPS-197). AES uses the Rijndael
-	  algorithm.
-
-	  Rijndael appears to be consistently a very good performer in
-	  both hardware and software across a wide range of computing
-	  environments regardless of its use in feedback or non-feedback
-	  modes. Its key setup time is excellent, and its key agility is
-	  good. Rijndael's very low memory requirements make it very well
-	  suited for restricted-space environments, in which it also
-	  demonstrates excellent performance. Rijndael's operations are
-	  among the easiest to defend against power and timing attacks.
-
-	  The AES specifies three key sizes: 128, 192 and 256 bits
-
-	  See <http://csrc.nist.gov/encryption/aes/> for more information.
+	  Table based implementation in 64-bit x86 assembler of the FIPS-197
+	  Advanced Encryption Standard (AES) symmetric cipher algorithm. For
+	  older 64-bit x86 CPUs that lack the special AES-NI instructions, it
+	  is the fastest implementation available, but it may be susceptible to
+	  known-plaintext attacks on the key due to the correlation between the
+	  processing time and the input of the first round. It reuses the
+	  tables exposed by the generic AES driver. If time invariance is a
+	  requirement, this driver should not be enabled.
 
 config CRYPTO_AES_NI_INTEL
-	tristate "AES cipher algorithms (AES-NI)"
+	tristate "AES cipher for x86 using AES-NI instructions"
 	depends on X86
 	select CRYPTO_AEAD
 	select CRYPTO_AES_CORE
@@ -962,52 +948,29 @@ config CRYPTO_AES_NI_INTEL
 	select CRYPTO_GLUE_HELPER_X86 if 64BIT
 	select CRYPTO_SIMD
 	help
-	  Use Intel AES-NI instructions for AES algorithm.
-
-	  AES cipher algorithms (FIPS-197). AES uses the Rijndael
-	  algorithm.
-
-	  Rijndael appears to be consistently a very good performer in
-	  both hardware and software across a wide range of computing
-	  environments regardless of its use in feedback or non-feedback
-	  modes. Its key setup time is excellent, and its key agility is
-	  good. Rijndael's very low memory requirements make it very well
-	  suited for restricted-space environments, in which it also
-	  demonstrates excellent performance. Rijndael's operations are
-	  among the easiest to defend against power and timing attacks.
-
-	  The AES specifies three key sizes: 128, 192 and 256 bits
-
-	  See <http://csrc.nist.gov/encryption/aes/> for more information.
+	  Implementation in x86 assembler of the FIPS-197 Advanced Encryption
+	  Standard (AES) symmetric cipher algorithm, using instructions from
+	  Intel's optional AES-NI ISA extension. This implementation is time
+	  invariant, and is by far the preferred option for CPUs that support
+	  this extension.
 
 	  In addition to AES cipher algorithm support, the acceleration
 	  for some popular block cipher mode is supported too, including
 	  ECB, CBC, LRW, PCBC, XTS. The 64 bit version has additional
 	  acceleration for CTR.
 
+	  If in doubt, enable as a module: it will be loaded automatically on
+	  CPUs that support it, and supersede other implementations of the AES
+	  cipher.
+
 config CRYPTO_AES_SPARC64
-	tristate "AES cipher algorithms (SPARC64)"
+	tristate "AES cipher for SPARC64 using crypto opcodes"
 	depends on SPARC64
 	select CRYPTO_CRYPTD
 	select CRYPTO_ALGAPI
 	help
-	  Use SPARC64 crypto opcodes for AES algorithm.
-
-	  AES cipher algorithms (FIPS-197). AES uses the Rijndael
-	  algorithm.
-
-	  Rijndael appears to be consistently a very good performer in
-	  both hardware and software across a wide range of computing
-	  environments regardless of its use in feedback or non-feedback
-	  modes. Its key setup time is excellent, and its key agility is
-	  good. Rijndael's very low memory requirements make it very well
-	  suited for restricted-space environments, in which it also
-	  demonstrates excellent performance. Rijndael's operations are
-	  among the easiest to defend against power and timing attacks.
-
-	  The AES specifies three key sizes: 128, 192 and 256 bits
-
-	  See <http://csrc.nist.gov/encryption/aes/> for more information.
+	  Implementation of the FIPS-197 Advanced Encryption Standard (AES)
+	  symmetric cipher algorithm, using SPARC64 crypto opcodes.
 
 	  In addition to AES cipher algorithm support, the acceleration
 	  for some popular block cipher mode is supported too, including
@@ -1017,8 +980,9 @@ config CRYPTO_AES_PPC_SPE
 	tristate "AES cipher algorithms (PPC SPE)"
 	depends on PPC && SPE
 	help
-	  AES cipher algorithms (FIPS-197). Additionally the acceleration
-	  for popular block cipher modes ECB, CBC, CTR and XTS is supported.
+	  Implementation of the FIPS-197 Advanced Encryption Standard (AES)
+	  symmetric cipher algorithm. Additionally, the acceleration for
+	  popular block cipher modes ECB, CBC, CTR and XTS is supported.
 	  This module should only be used for low power (router) devices
 	  without hardware AES acceleration (e.g. caam crypto). It reduces the
 	  size of the AES tables from 16KB to 8KB + 256 bytes and mitigates
-- 
2.9.3

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH v4 0/8] crypto: aes - retire table based generic AES
  2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
                   ` (7 preceding siblings ...)
  2017-07-18 12:06 ` [PATCH v4 8/8] crypto: aes - add meaningful help text to the various AES drivers Ard Biesheuvel
@ 2017-07-24  6:59 ` Ard Biesheuvel
  2017-07-24 16:57   ` Eric Biggers
  8 siblings, 1 reply; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-24  6:59 UTC (permalink / raw)
  To: linux-crypto, Herbert Xu, nico, Eric Biggers; +Cc: Ard Biesheuvel

On 18 July 2017 at 13:06, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> The generic AES driver uses 16 lookup tables of 1 KB each, and has
> encryption and decryption routines that are fully unrolled. Given how
> the dependencies between this code and other drivers are declared in
> Kconfig files, this code is always pulled into the core kernel, even
> if it is usually superseded at runtime by accelerated drivers that
> exist for many architectures.
>
> This leaves us with 25 KB of dead code in the kernel, which is negligible
> in typical environments, but which is actually a big deal for the IoT
> domain, where every kilobyte counts.
>
> Also, the scalar, table based AES routines that exist for ARM, arm64, i586
> and x86_64 share the lookup tables with AES generic, and may be invoked
> occasionally when the time-invariant AES-NI or other special instruction
> drivers are called in interrupt context, at which time the SIMD register
> file cannot be used. Pulling 16 KB of code and 9 KB of instructions into
> the L1s (and evicting what was already there) when a softirq happens to
> be handled in the context of an interrupt taken from kernel mode (which
> means no SIMD on x86) is also something that we may like to avoid, by
> falling back to a much smaller and moderately less performant driver.
> (Note that arm64 will be updated shortly to supply fallbacks for all
> SIMD based AES implementations, which will be based on the core routines)
>
> For the reasons above, this series refactors the way the various AES
> implementations are wired up, to allow the generic version in
> crypto/aes_generic.c to be omitted from the build entirely.
>
> Patch #1 removes some bogus 'select CRYPTO_AES' statement.
>
> Patch #2 factors out aes-generic's lookup tables, which are shared with
> arch-specific implementations in arch/x86, arch/arm and arch/arm64.
>
> Patch #3 replaces the table based aes-generic.o with a new aes.o based on
> the fixed time cipher, and uses it to fulfil dependencies on CRYPTO_AES.
>
> Patch #4 switches the fallback in the AES-NI code to the new, generic encrypt
> and decrypt routines so it no longer depends on the x86 scalar code or
> [transitively] on AES-generic.
>
> Patch #5 tweaks the ARM table based code to only use 2 KB + 256 bytes worth
> of lookup tables instead of 4 KB.
>
> Patch #6 does the same for arm64
>
> Patch #7 removes the local copy of the AES sboxes from the arm64 NEON driver,
> and switches to the ones exposed by the new AES core module instead.
>
> Patch #8 updates the Kconfig help text to be more descriptive of what they
> actually control, rather than duplicating AES's wikipedia entry a number of
> times.
>
> v4: - remove aes-generic altogether instead of allow a preference to be set

Actually, after benchmarking the x86_64 asm AES code, I am not so sure
we should remove AES_GENERIC at all, since it turns out to be faster.
Interestingly, I found a remark by Eric in the git log stating the
same, so if we want to cut down on AES variants, we should probably
start by deleting the x86 code instead.

So please disregard this for now: I will rework the other stuff I have
so it no longer depends on this, and repost, because it is much more
important for me that that makes it into v4.14. This can wait for
v4.15, as far as I am concerned, and I will benchmark a bit more to
see if we can get rid of the i586 code as well.

-- 
Ard.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH v4 0/8] crypto: aes - retire table based generic AES
  2017-07-24  6:59 ` [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
@ 2017-07-24 16:57   ` Eric Biggers
  2017-07-24 17:03     ` Ard Biesheuvel
  0 siblings, 1 reply; 12+ messages in thread
From: Eric Biggers @ 2017-07-24 16:57 UTC (permalink / raw)
  To: Ard Biesheuvel; +Cc: linux-crypto, Herbert Xu, nico, Eric Biggers

On Mon, Jul 24, 2017 at 07:59:43AM +0100, Ard Biesheuvel wrote:
> On 18 July 2017 at 13:06, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > The generic AES driver uses 16 lookup tables of 1 KB each, and has
> > encryption and decryption routines that are fully unrolled. Given how
> > the dependencies between this code and other drivers are declared in
> > Kconfig files, this code is always pulled into the core kernel, even
> > if it is usually superseded at runtime by accelerated drivers that
> > exist for many architectures.
> >
> > This leaves us with 25 KB of dead code in the kernel, which is negligible
> > in typical environments, but which is actually a big deal for the IoT
> > domain, where every kilobyte counts.
> >
> > Also, the scalar, table based AES routines that exist for ARM, arm64, i586
> > and x86_64 share the lookup tables with AES generic, and may be invoked
> > occasionally when the time-invariant AES-NI or other special instruction
> > drivers are called in interrupt context, at which time the SIMD register
> > file cannot be used. Pulling 16 KB of code and 9 KB of instructions into
> > the L1s (and evicting what was already there) when a softirq happens to
> > be handled in the context of an interrupt taken from kernel mode (which
> > means no SIMD on x86) is also something that we may like to avoid, by
> > falling back to a much smaller and moderately less performant driver.
> > (Note that arm64 will be updated shortly to supply fallbacks for all
> > SIMD based AES implementations, which will be based on the core routines)
> >
> > For the reasons above, this series refactors the way the various AES
> > implementations are wired up, to allow the generic version in
> > crypto/aes_generic.c to be omitted from the build entirely.
> >
> > Patch #1 removes some bogus 'select CRYPTO_AES' statement.
> >
> > Patch #2 factors out aes-generic's lookup tables, which are shared with
> > arch-specific implementations in arch/x86, arch/arm and arch/arm64.
> >
> > Patch #3 replaces the table based aes-generic.o with a new aes.o based on
> > the fixed time cipher, and uses it to fulfil dependencies on CRYPTO_AES.
> >
> > Patch #4 switches the fallback in the AES-NI code to the new, generic encrypt
> > and decrypt routines so it no longer depends on the x86 scalar code or
> > [transitively] on AES-generic.
> >
> > Patch #5 tweaks the ARM table based code to only use 2 KB + 256 bytes worth
> > of lookup tables instead of 4 KB.
> >
> > Patch #6 does the same for arm64
> >
> > Patch #7 removes the local copy of the AES sboxes from the arm64 NEON driver,
> > and switches to the ones exposed by the new AES core module instead.
> >
> > Patch #8 updates the Kconfig help text to be more descriptive of what they
> > actually control, rather than duplicating AES's wikipedia entry a number of
> > times.
> >
> > v4: - remove aes-generic altogether instead of allow a preference to be set
> 
> Actually, after benchmarking the x86_64 asm AES code, I am not so sure
> we should remove AES_GENERIC at all, since it turns out to be faster.
> Interestingly, I found a remark by Eric in the git log stating the
> same, so if we want to cut down on AES variants, we should probably
> start by deleting the x86 code instead.
> 
> So please disregard this for now: I will rework the other stuff I have
> so it no longer depends on this, and repost, because it is much more
> important for me that that makes it into v4.14. This can wait for
> v4.15, as far as I am concerned, and I will benchmark a bit more to
> see if we can get rid of the i586 code as well.
> 

Yes I did notice that aes-generic was actually faster.  Probably the x86_64-asm
implementation should be removed, but it may be worthwhile to try a few
different gcc versions to see how well they compile aes-generic.  I expect that
x86_64-asm used to be faster but gcc has gotten smarter.  Also x86_64-asm is
only really useful on older CPUs, so ideally it should be benchmarked on an
older CPU.

Eric

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH v4 0/8] crypto: aes - retire table based generic AES
  2017-07-24 16:57   ` Eric Biggers
@ 2017-07-24 17:03     ` Ard Biesheuvel
  0 siblings, 0 replies; 12+ messages in thread
From: Ard Biesheuvel @ 2017-07-24 17:03 UTC (permalink / raw)
  To: Eric Biggers; +Cc: linux-crypto, Herbert Xu, nico, Eric Biggers

On 24 July 2017 at 17:57, Eric Biggers <ebiggers3@gmail.com> wrote:
> On Mon, Jul 24, 2017 at 07:59:43AM +0100, Ard Biesheuvel wrote:
>> On 18 July 2017 at 13:06, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>> > The generic AES driver uses 16 lookup tables of 1 KB each, and has
>> > encryption and decryption routines that are fully unrolled. Given how
>> > the dependencies between this code and other drivers are declared in
>> > Kconfig files, this code is always pulled into the core kernel, even
>> > if it is usually superseded at runtime by accelerated drivers that
>> > exist for many architectures.
>> >
>> > This leaves us with 25 KB of dead code in the kernel, which is negligible
>> > in typical environments, but which is actually a big deal for the IoT
>> > domain, where every kilobyte counts.
>> >
>> > Also, the scalar, table based AES routines that exist for ARM, arm64, i586
>> > and x86_64 share the lookup tables with AES generic, and may be invoked
>> > occasionally when the time-invariant AES-NI or other special instruction
>> > drivers are called in interrupt context, at which time the SIMD register
>> > file cannot be used. Pulling 16 KB of code and 9 KB of instructions into
>> > the L1s (and evicting what was already there) when a softirq happens to
>> > be handled in the context of an interrupt taken from kernel mode (which
>> > means no SIMD on x86) is also something that we may like to avoid, by
>> > falling back to a much smaller and moderately less performant driver.
>> > (Note that arm64 will be updated shortly to supply fallbacks for all
>> > SIMD based AES implementations, which will be based on the core routines)
>> >
>> > For the reasons above, this series refactors the way the various AES
>> > implementations are wired up, to allow the generic version in
>> > crypto/aes_generic.c to be omitted from the build entirely.
>> >
>> > Patch #1 removes some bogus 'select CRYPTO_AES' statement.
>> >
>> > Patch #2 factors out aes-generic's lookup tables, which are shared with
>> > arch-specific implementations in arch/x86, arch/arm and arch/arm64.
>> >
>> > Patch #3 replaces the table based aes-generic.o with a new aes.o based on
>> > the fixed time cipher, and uses it to fulfil dependencies on CRYPTO_AES.
>> >
>> > Patch #4 switches the fallback in the AES-NI code to the new, generic encrypt
>> > and decrypt routines so it no longer depends on the x86 scalar code or
>> > [transitively] on AES-generic.
>> >
>> > Patch #5 tweaks the ARM table based code to only use 2 KB + 256 bytes worth
>> > of lookup tables instead of 4 KB.
>> >
>> > Patch #6 does the same for arm64
>> >
>> > Patch #7 removes the local copy of the AES sboxes from the arm64 NEON driver,
>> > and switches to the ones exposed by the new AES core module instead.
>> >
>> > Patch #8 updates the Kconfig help text to be more descriptive of what they
>> > actually control, rather than duplicating AES's wikipedia entry a number of
>> > times.
>> >
>> > v4: - remove aes-generic altogether instead of allow a preference to be set
>>
>> Actually, after benchmarking the x86_64 asm AES code, I am not so sure
>> we should remove AES_GENERIC at all, since it turns out to be faster.
>> Interestingly, I found a remark by Eric in the git log stating the
>> same, so if we want to cut down on AES variants, we should probably
>> start by deleting the x86 code instead.
>>
>> So please disregard this for now: I will rework the other stuff I have
>> so it no longer depends on this, and repost, because it is much more
>> important for me that that makes it into v4.14. This can wait for
>> v4.15, as far as I am concerned, and I will benchmark a bit more to
>> see if we can get rid of the i586 code as well.
>>
>
> Yes I did notice that aes-generic was actually faster.  Probably the x86_64-asm
> implementation should be removed, but it may be worthwhile to try a few
> different gcc versions to see how well they compile aes-generic.  I expect that
> x86_64-asm used to be faster but gcc has gotten smarter.  Also x86_64-asm is
> only really useful on older CPUs, so ideally it should be benchmarked on an
> older CPU.
>

I tried it on a ~10 year old E2200 chip, and aes-generic was slightly
faster, i.e., 5 - 10 % as you noticed as well.

If current GCC creates faster code, we should just remove the asm code
IMO. No point in keeping it around for the sake of people who insist
on building current linux with an outdated compiler.

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2017-07-24 17:03 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-18 12:06 [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 1/8] drivers/crypto/Kconfig: drop bogus CRYPTO_AES dependencies Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 2/8] crypto - aes: use dedicated lookup tables for table based asm routines Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 3/8] crypto: aes - retire table based generic AES in favor of fixed time driver Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 4/8] crypto: x86/aes-ni - switch to generic fallback Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 5/8] crypto: arm/aes - avoid expanded lookup tables in the final round Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 6/8] crypto: arm64/aes " Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 7/8] crypto: arm64/aes-neon - reuse Sboxes from AES core module Ard Biesheuvel
2017-07-18 12:06 ` [PATCH v4 8/8] crypto: aes - add meaningful help text to the various AES drivers Ard Biesheuvel
2017-07-24  6:59 ` [PATCH v4 0/8] crypto: aes - retire table based generic AES Ard Biesheuvel
2017-07-24 16:57   ` Eric Biggers
2017-07-24 17:03     ` Ard Biesheuvel

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.