* [RFC PATCH v2 00/11] x86: Support Intel Key Locker
@ 2021-05-14 20:14 Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature Chang S. Bae
` (11 more replies)
0 siblings, 12 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:14 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Key Locker [1][2] is a new security feature available in new Intel CPUs to
protect data encryption keys for the Advanced Encryption Standard
algorithm. The protection limits the amount of time an AES key is exposed
in memory by sealing a key and referencing it with new AES instructions.
The new AES instruction set is a successor of Intel's AES-NI (AES New
Instruction). Users may switch to the Key Locker version from crypto
libraries. This series includes a new AES implementation for the Crypto
API, which was validated through the crypto unit tests. The performance in
the test cases was measured and found comparable to the AES-NI version.
Key Locker introduces a (CPU-)internal key to encode AES keys. The kernel
needs to load it and ensure it unchanged as long as CPUs are operational.
The series has three parts:
* PATCH1-7: Implement the internal key management
* PATCH8-10: Add a new AES implementation in the Crypto library
* PATCH11: Provide the hardware randomization option
Dan has asked for this to go out as another RFC to have more conversation
before asking the maintainers to consider this implementation.
Changes from RFC v1 [3]:
* Refactored the AES-NI implementation and fix the AES-KL in the Crypto
API. (Ard Biesheuvel)
* Revised the AES implementation description. (Dave Hansen and Peter
Zijlsta).
* Noted the binutils version and made it prerequisite. (Peter Zijlstra and
Borislav Petkov)
* Reorganized helper functions. With that, simplified the feature
enablement check.
* Added to flush cache lines when removing key from memory.
* Separated the opcode map update into a new patch. Also, included AES
instructions.
* Refactored the LOADIWKEY instruction in a new helper.
* Folded the backup error warning. (Rafael Wysocki)
* Massaged changelog accordingly.
[1] Intel Architecture Instruction Set Extensions Programming Reference:
https://software.intel.com/content/dam/develop/external/us/en/documents-tps/architecture-instruction-set-extensions-programming-reference.pdf
[2] Intel Key Locker Specification:
https://software.intel.com/content/dam/develop/external/us/en/documents/343965-intel-key-locker-specification.pdf
[3] RFC v1: https://lore.kernel.org/lkml/20201216174146.10446-1-chang.seok.bae@intel.com/
Chang S. Bae (11):
x86/cpufeature: Enumerate Key Locker feature
x86/insn: Add Key Locker instructions to the opcode map
x86/cpu: Load Key Locker internal key at boot-time
x86/msr-index: Add MSRs for Key Locker internal key
x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep
states
x86/cpu: Add a config option and a chicken bit for Key Locker
selftests/x86: Test Key Locker internal key maintenance
crypto: x86/aes-ni - Improve error handling
crypto: x86/aes-ni - Refactor to prepare a new AES implementation
crypto: x86/aes-kl - Support AES algorithm using Key Locker
instructions
x86/cpu: Support the hardware randomization option for Key Locker
internal key
.../admin-guide/kernel-parameters.txt | 2 +
arch/x86/Kconfig | 14 +
arch/x86/crypto/Makefile | 5 +-
arch/x86/crypto/aes-intel_asm.S | 26 +
arch/x86/crypto/aes-intel_glue.c | 208 +++
arch/x86/crypto/aes-intel_glue.h | 61 +
arch/x86/crypto/aeskl-intel_asm.S | 1181 +++++++++++++++++
arch/x86/crypto/aeskl-intel_glue.c | 390 ++++++
arch/x86/crypto/aesni-intel_asm.S | 90 +-
arch/x86/crypto/aesni-intel_glue.c | 310 +----
arch/x86/crypto/aesni-intel_glue.h | 88 ++
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/disabled-features.h | 8 +-
arch/x86/include/asm/keylocker.h | 30 +
arch/x86/include/asm/msr-index.h | 6 +
arch/x86/include/asm/special_insns.h | 36 +
arch/x86/include/uapi/asm/processor-flags.h | 2 +
arch/x86/kernel/Makefile | 1 +
arch/x86/kernel/cpu/common.c | 21 +-
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
arch/x86/kernel/keylocker.c | 264 ++++
arch/x86/kernel/smpboot.c | 2 +
arch/x86/lib/x86-opcode-map.txt | 11 +-
arch/x86/power/cpu.c | 2 +
crypto/Kconfig | 23 +
drivers/char/random.c | 6 +
include/linux/random.h | 2 +
tools/arch/x86/lib/x86-opcode-map.txt | 11 +-
tools/testing/selftests/x86/Makefile | 2 +-
tools/testing/selftests/x86/keylocker.c | 177 +++
30 files changed, 2638 insertions(+), 343 deletions(-)
create mode 100644 arch/x86/crypto/aes-intel_asm.S
create mode 100644 arch/x86/crypto/aes-intel_glue.c
create mode 100644 arch/x86/crypto/aes-intel_glue.h
create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
create mode 100644 arch/x86/crypto/aesni-intel_glue.h
create mode 100644 arch/x86/include/asm/keylocker.h
create mode 100644 arch/x86/kernel/keylocker.c
create mode 100644 tools/testing/selftests/x86/keylocker.c
base-commit: 6efb943b8616ec53a5e444193dccf1af9ad627b5
--
2.17.1
^ permalink raw reply [flat|nested] 28+ messages in thread
* [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
@ 2021-05-14 20:14 ` Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 02/11] x86/insn: Add Key Locker instructions to the opcode map Chang S. Bae
` (10 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:14 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Intel's Key Locker is a new security feature providing a mechanism to
protect data encryption keys when processing the Advanced Encryption
Standard (AES) algorithm.
The feature accompanies new AES instructions as a successor to Intel's AES
New Instructions (AES-NI). It also follows AES-NI's other feature
dependency.
Here add it to enumerate the hardware capability, shown as 'keylocker' in
/proc/cpuinfo.
Define the feature-specific CPUID leaf and bits for the feature enablement.
Add X86_FEATURE_KEYLOCKER to the disabled features mask for the use of
compile-time configuration.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Updated the changelog.
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/disabled-features.h | 8 +++++++-
arch/x86/include/asm/keylocker.h | 18 ++++++++++++++++++
arch/x86/include/uapi/asm/processor-flags.h | 2 ++
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
5 files changed, 29 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/include/asm/keylocker.h
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index ac37830ae941..578cf3fe7182 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -359,6 +359,7 @@
#define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */
#define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */
#define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */
+#define X86_FEATURE_KEYLOCKER (16*32+23) /* Key Locker */
#define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */
#define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */
#define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */
diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h
index b7dd944dc867..96acda147a50 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -44,6 +44,12 @@
# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))
#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */
+#ifdef CONFIG_X86_KEYLOCKER
+# define DISABLE_KEYLOCKER 0
+#else
+# define DISABLE_KEYLOCKER (1<<(X86_FEATURE_KEYLOCKER & 31))
+#endif /* CONFIG_X86_KEYLOCKER */
+
#ifdef CONFIG_X86_5LEVEL
# define DISABLE_LA57 0
#else
@@ -88,7 +94,7 @@
#define DISABLED_MASK14 0
#define DISABLED_MASK15 0
#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
- DISABLE_ENQCMD)
+ DISABLE_ENQCMD|DISABLE_KEYLOCKER)
#define DISABLED_MASK17 0
#define DISABLED_MASK18 0
#define DISABLED_MASK19 0
diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
new file mode 100644
index 000000000000..0597d6b1cd05
--- /dev/null
+++ b/arch/x86/include/asm/keylocker.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef _ASM_KEYLOCKER_H
+#define _ASM_KEYLOCKER_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/bits.h>
+
+#define KEYLOCKER_CPUID 0x019
+#define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0)
+#define KEYLOCKER_CPUID_EBX_AESKLE BIT(0)
+#define KEYLOCKER_CPUID_EBX_WIDE BIT(2)
+#define KEYLOCKER_CPUID_EBX_BACKUP BIT(4)
+#define KEYLOCKER_CPUID_ECX_RAND BIT(1)
+
+#endif /*__ASSEMBLY__ */
+#endif /* _ASM_KEYLOCKER_H */
diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h
index bcba3c643e63..b958a95a0908 100644
--- a/arch/x86/include/uapi/asm/processor-flags.h
+++ b/arch/x86/include/uapi/asm/processor-flags.h
@@ -124,6 +124,8 @@
#define X86_CR4_PCIDE _BITUL(X86_CR4_PCIDE_BIT)
#define X86_CR4_OSXSAVE_BIT 18 /* enable xsave and xrestore */
#define X86_CR4_OSXSAVE _BITUL(X86_CR4_OSXSAVE_BIT)
+#define X86_CR4_KEYLOCKER_BIT 19 /* enable Key Locker */
+#define X86_CR4_KEYLOCKER _BITUL(X86_CR4_KEYLOCKER_BIT)
#define X86_CR4_SMEP_BIT 20 /* enable SMEP support */
#define X86_CR4_SMEP _BITUL(X86_CR4_SMEP_BIT)
#define X86_CR4_SMAP_BIT 21 /* enable SMAP support */
diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c
index defda61f372d..ce2020b9ff4d 100644
--- a/arch/x86/kernel/cpu/cpuid-deps.c
+++ b/arch/x86/kernel/cpu/cpuid-deps.c
@@ -75,6 +75,7 @@ static const struct cpuid_dep cpuid_deps[] = {
{ X86_FEATURE_SGX_LC, X86_FEATURE_SGX },
{ X86_FEATURE_SGX1, X86_FEATURE_SGX },
{ X86_FEATURE_SGX2, X86_FEATURE_SGX1 },
+ { X86_FEATURE_KEYLOCKER, X86_FEATURE_XMM2 },
{}
};
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 02/11] x86/insn: Add Key Locker instructions to the opcode map
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature Chang S. Bae
@ 2021-05-14 20:14 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 03/11] x86/cpu: Load Key Locker internal key at boot-time Chang S. Bae
` (9 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:14 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Add the following Key Locker instructions to the opcode map:
LOADIWKEY:
Load an CPU-internal wrapping key.
ENCODEKEY128:
Wrap a 128-bit AES key to a key handle.
ENCODEKEY256:
Wrap a 256-bit AES key to a key handle.
AESENC128KL:
Encrypt a 128-bit block of data using a 128-bit AES key indicated
by a key handle.
AESENC256KL:
Encrypt a 128-bit block of data using a 256-bit AES key indicated
by a key handle.
AESDEC128KL:
Decrypt a 128-bit block of data using a 128-bit AES key indicated
by a key handle.
AESDEC256KL:
Decrypt a 128-bit block of data using a 256-bit AES key indicated
by a key handle.
AESENCWIDE128KL:
Encrypt 8 128-bit blocks of data using a 128-bit AES key indicated
by a key handle.
AESENCWIDE256KL:
Encrypt 8 128-bit blocks of data using a 256-bit AES key indicated
by a key handle.
AESDECWIDE128KL:
Decrypt 8 128-bit blocks of data using a 128-bit AES key indicated
by a key handle.
AESDECWIDE256KL:
Decrypt 8 128-bit blocks of data using a 256-bit AES key indicated
by a key handle.
Details of these instructions can be found in Intel Software Developer's
Manual.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Separated out the LOADIWKEY addition in a new patch.
* Included AES instructions to avoid warning messages when the AES Key
Locker module is built.
---
arch/x86/lib/x86-opcode-map.txt | 11 +++++++----
tools/arch/x86/lib/x86-opcode-map.txt | 11 +++++++----
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt
index ec31f5b60323..eb702fc6572e 100644
--- a/arch/x86/lib/x86-opcode-map.txt
+++ b/arch/x86/lib/x86-opcode-map.txt
@@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
f2: ANDN Gy,By,Ey (v)
@@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
EndTable
Table: 3-byte opcode 2 (0x0f 0x3a)
diff --git a/tools/arch/x86/lib/x86-opcode-map.txt b/tools/arch/x86/lib/x86-opcode-map.txt
index ec31f5b60323..eb702fc6572e 100644
--- a/tools/arch/x86/lib/x86-opcode-map.txt
+++ b/tools/arch/x86/lib/x86-opcode-map.txt
@@ -794,11 +794,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
f2: ANDN Gy,By,Ey (v)
@@ -808,6 +809,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
EndTable
Table: 3-byte opcode 2 (0x0f 0x3a)
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 03/11] x86/cpu: Load Key Locker internal key at boot-time
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 02/11] x86/insn: Add Key Locker instructions to the opcode map Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 04/11] x86/msr-index: Add MSRs for Key Locker internal key Chang S. Bae
` (8 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Internal (Wrapping) Key is a new entity of Intel Key Locker feature. This
internal key is loaded in a software-inaccessible CPU state and used to
encode a data encryption key.
The kernel makes random data and loads it as the internal key in each CPU.
The data need to be flushed as soon as the load is done.
System firmware may disable the feature. Check the dynamic CPUID bit
(KEYLOCKER_CPUID_EBX_AESKLE) at first.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Added to flush cache line when removing an internal key value. Also,
renamed the helper.
* Reorganized the helper functions.
* Changed to reference boot CPU's feature flag consistently, and to update
it if the feature is disabled.
* Refactored LOADIWKEY instruction in a new helper.
* Added function descriptions.
---
arch/x86/include/asm/keylocker.h | 9 ++
arch/x86/include/asm/special_insns.h | 36 ++++++++
arch/x86/kernel/Makefile | 1 +
arch/x86/kernel/cpu/common.c | 5 +-
arch/x86/kernel/keylocker.c | 125 +++++++++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +
6 files changed, 177 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/kernel/keylocker.c
diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
index 0597d6b1cd05..870832f007ec 100644
--- a/arch/x86/include/asm/keylocker.h
+++ b/arch/x86/include/asm/keylocker.h
@@ -5,6 +5,7 @@
#ifndef __ASSEMBLY__
+#include <asm/processor.h>
#include <linux/bits.h>
#define KEYLOCKER_CPUID 0x019
@@ -14,5 +15,13 @@
#define KEYLOCKER_CPUID_EBX_BACKUP BIT(4)
#define KEYLOCKER_CPUID_ECX_RAND BIT(1)
+#ifdef CONFIG_X86_KEYLOCKER
+void setup_keylocker(struct cpuinfo_x86 *c);
+void flush_keylocker_data(void);
+#else
+#define setup_keylocker(c) do { } while (0)
+#define flush_keylocker_data() do { } while (0)
+#endif
+
#endif /*__ASSEMBLY__ */
#endif /* _ASM_KEYLOCKER_H */
diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index 2acd6cb62328..0511b2d72ab8 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -306,6 +306,42 @@ static inline int enqcmds(void __iomem *dst, const void *src)
return 0;
}
+/**
+ * loadiwkey() - Load CPU-internal wrapping key
+ * @keyconf: Indicate the key source and the key backup allowance.
+ *
+ * The key value is presumed in XMM0-2 registers. So make sure the states
+ * are available before calling this.
+ *
+ * If bit 0 of @keyconf is set, key backup is not permitted. If bit 1 is
+ * set, a hardware-randomized key is loaded.
+ *
+ * Returns: -EAGAIN if hardware randomization fails; otherwise, 0.
+ *
+ */
+static inline int __must_check loadiwkey(u32 keyconf)
+{
+ int zf;
+
+ /*
+ * LOADIWKEY %xmm1,%xmm2
+ *
+ * EAX and XMM0 are implicit operands. Load key value from XMM0-2
+ * to software-invisible CPU state.
+ *
+ * This instruction is supported by binutils >= 2.36.
+ */
+ asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1"
+ CC_SET(z)
+ : CC_OUT(z) (zf)
+ : "a"(keyconf));
+
+ /* Hardware randomization failure is indicated via EFLAGS.ZF=1. */
+ if (zf)
+ return -EAGAIN;
+ return 0;
+}
+
#endif /* __KERNEL__ */
#endif /* _ASM_X86_SPECIAL_INSNS_H */
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 0704c2a94272..5662876e175b 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -143,6 +143,7 @@ obj-$(CONFIG_PERF_EVENTS) += perf_regs.o
obj-$(CONFIG_TRACING) += tracepoint.o
obj-$(CONFIG_SCHED_MC_PRIO) += itmt.o
obj-$(CONFIG_X86_UMIP) += umip.o
+obj-$(CONFIG_X86_KEYLOCKER) += keylocker.o
obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o
obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index a1b756c49a93..2e64371acb81 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -57,6 +57,8 @@
#include <asm/microcode_intel.h>
#include <asm/intel-family.h>
#include <asm/cpu_device_id.h>
+#include <asm/keylocker.h>
+
#include <asm/uv/uv.h>
#include "cpu.h"
@@ -1555,10 +1557,11 @@ static void identify_cpu(struct cpuinfo_x86 *c)
/* Disable the PN if appropriate */
squash_the_stupid_serial_number(c);
- /* Set up SMEP/SMAP/UMIP */
+ /* Setup various Intel-specific CPU security features */
setup_smep(c);
setup_smap(c);
setup_umip(c);
+ setup_keylocker(c);
/* Enable FSGSBASE instructions if available. */
if (cpu_has(c, X86_FEATURE_FSGSBASE)) {
diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
new file mode 100644
index 000000000000..d590815de508
--- /dev/null
+++ b/arch/x86/kernel/keylocker.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+/*
+ * Setup Key Locker feature and support the internal key management.
+ */
+
+#include <linux/random.h>
+
+#include <asm/cacheflush.h>
+#include <asm/fpu/api.h>
+#include <asm/fpu/types.h>
+#include <asm/keylocker.h>
+#include <asm/tlbflush.h>
+
+/* Internal (Wrapping) Key size fits in three 128-bit registers. */
+#define KEYSIZE_128BIT 3
+
+static struct _keydata {
+ struct reg_128_bit value[KEYSIZE_128BIT];
+} keydata;
+
+/**
+ * make_keylocker_data() - Generate the internal key.
+ *
+ * Return: Nothing
+ */
+static void make_keylocker_data(void)
+{
+ int i;
+
+ for (i = 0; i < KEYSIZE_128BIT; i++)
+ get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit));
+}
+
+/**
+ * flush_keylocker_data() - Flush the internal key in memory.
+ *
+ * Returns: Nothing
+ */
+void flush_keylocker_data(void)
+{
+ unsigned int size = sizeof(struct reg_128_bit) * KEYSIZE_128BIT;
+ void *keyaddr = &keydata.value;
+
+ if (!boot_cpu_has(X86_FEATURE_KEYLOCKER))
+ return;
+
+ memset(keyaddr, 0, size);
+ clflush_cache_range(keyaddr, size);
+}
+
+#define KEYSRC_SWRAND 0
+
+/**
+ * load_keylocker() - Load the internal key.
+ *
+ * Returns: -EAGAIN if hardware randomization fails; otherwise, 0.
+ */
+static int load_keylocker(void)
+{
+ struct reg_128_bit zeros = { 0 };
+ u32 keysrc = KEYSRC_SWRAND;
+ int err;
+
+ kernel_fpu_begin();
+
+ asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;"
+ :: "m"(keydata.value[0]),
+ "m"(keydata.value[1]),
+ "m"(keydata.value[2]));
+
+ err = loadiwkey(keysrc);
+
+ asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;"
+ :: "m"(zeros));
+
+ kernel_fpu_end();
+
+ return err;
+}
+
+/**
+ * setup_keylocker() - Enable the feature if supported.
+ * @c: A pointer to struct cpuinfo_x86
+ *
+ * Returns: Nothing
+ */
+void setup_keylocker(struct cpuinfo_x86 *c)
+{
+ int err;
+
+ if (!boot_cpu_has(X86_FEATURE_KEYLOCKER))
+ goto out;
+
+ cr4_set_bits(X86_CR4_KEYLOCKER);
+
+ if (c == &boot_cpu_data) {
+ u32 eax, ebx, ecx, edx;
+
+ cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx);
+ /* BIOS may not enable it on some systems. */
+ if (!(ebx & KEYLOCKER_CPUID_EBX_AESKLE)) {
+ pr_debug("x86/keylocker: not fully supported.\n");
+ goto disable;
+ }
+
+ make_keylocker_data();
+ }
+
+ err = load_keylocker();
+ if (err) {
+ pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err);
+ goto disable;
+ }
+
+ pr_info_once("x86/keylocker: Enabled.\n");
+ return;
+
+disable:
+ setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER);
+ pr_info_once("x86/keylocker: Disabled.\n");
+out:
+ /* Make sure the feature disabled for kexec-reboot. */
+ cr4_clear_bits(X86_CR4_KEYLOCKER);
+}
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 0ad5214f598a..0266d70a44be 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -82,6 +82,7 @@
#include <asm/spec-ctrl.h>
#include <asm/hw_irq.h>
#include <asm/stackprotector.h>
+#include <asm/keylocker.h>
#ifdef CONFIG_ACPI_CPPC_LIB
#include <acpi/cppc_acpi.h>
@@ -1429,6 +1430,7 @@ void __init native_smp_cpus_done(unsigned int max_cpus)
nmi_selftest();
impress_friends();
mtrr_aps_init();
+ flush_keylocker_data();
}
static int __initdata setup_possible_cpus = -1;
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 04/11] x86/msr-index: Add MSRs for Key Locker internal key
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (2 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 03/11] x86/cpu: Load Key Locker internal key at boot-time Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states Chang S. Bae
` (7 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Key Locker internal key in a CPU state can be backed up in a platform
register. The backup can be also copied back to a CPU state. This mechanism
is useful to restore the key (after system sleep).
Add MSRs for the internal key backup, copy, and status check.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
---
arch/x86/include/asm/msr-index.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 742d89a00721..f8e78782c8f4 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -935,4 +935,10 @@
#define MSR_VM_IGNNE 0xc0010115
#define MSR_VM_HSAVE_PA 0xc0010117
+/* MSRs for Key Locker Internal (Wrapping) Key management */
+#define MSR_IA32_COPY_LOCAL_TO_PLATFORM 0x00000d91
+#define MSR_IA32_COPY_PLATFORM_TO_LOCAL 0x00000d92
+#define MSR_IA32_COPY_STATUS 0x00000990
+#define MSR_IA32_IWKEYBACKUP_STATUS 0x00000991
+
#endif /* _ASM_X86_MSR_INDEX_H */
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (3 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 04/11] x86/msr-index: Add MSRs for Key Locker internal key Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-24 14:21 ` Rafael J. Wysocki
2021-05-14 20:15 ` [RFC PATCH v2 06/11] x86/cpu: Add a config option and a chicken bit for Key Locker Chang S. Bae
` (6 subsequent siblings)
11 siblings, 1 reply; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae, linux-pm
When the system state switches to these sleep states, the internal key gets
reset. Since this system transition is transparent to userspace, the
internal key needs to be restored properly.
Key Locker provides a mechanism to back up the internal key in non-volatile
memory. The kernel requests a backup right after the key loaded at
boot-time and copies it later when the system wakes up.
The backup during the S5 sleep state is not trusted. It is overwritten by a
new key at the next boot.
On a system with the S3/4 states, enable the feature only when the backup
mechanism is supported.
Disable the feature when the copy fails (or the backup corrupts). The
shutdown is considered too noisy. A new key is considerable only when
threads can be synchronously suspended.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-pm@vger.kernel.org
---
Changes from RFC v1:
* Folded the warning message into the if condition check. (Rafael Wysocki)
* Rebased on the changes of the previous patches.
* Added error code for key restoration failures.
* Moved the restore helper.
* Added function descriptions.
---
arch/x86/include/asm/keylocker.h | 3 +
arch/x86/kernel/keylocker.c | 122 +++++++++++++++++++++++++++++--
arch/x86/power/cpu.c | 2 +
3 files changed, 122 insertions(+), 5 deletions(-)
diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
index 870832f007ec..74b806346bee 100644
--- a/arch/x86/include/asm/keylocker.h
+++ b/arch/x86/include/asm/keylocker.h
@@ -7,6 +7,7 @@
#include <asm/processor.h>
#include <linux/bits.h>
+#include <asm/msr.h>
#define KEYLOCKER_CPUID 0x019
#define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0)
@@ -18,9 +19,11 @@
#ifdef CONFIG_X86_KEYLOCKER
void setup_keylocker(struct cpuinfo_x86 *c);
void flush_keylocker_data(void);
+void restore_keylocker(void);
#else
#define setup_keylocker(c) do { } while (0)
#define flush_keylocker_data() do { } while (0)
+#define restore_keylocker() do { } while (0)
#endif
#endif /*__ASSEMBLY__ */
diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
index d590815de508..0f60350944fa 100644
--- a/arch/x86/kernel/keylocker.c
+++ b/arch/x86/kernel/keylocker.c
@@ -5,6 +5,8 @@
*/
#include <linux/random.h>
+#include <linux/acpi.h>
+#include <linux/delay.h>
#include <asm/cacheflush.h>
#include <asm/fpu/api.h>
@@ -12,10 +14,13 @@
#include <asm/keylocker.h>
#include <asm/tlbflush.h>
+static bool keybackup_available;
+
/* Internal (Wrapping) Key size fits in three 128-bit registers. */
#define KEYSIZE_128BIT 3
static struct _keydata {
+ bool valid;
struct reg_128_bit value[KEYSIZE_128BIT];
} keydata;
@@ -30,6 +35,8 @@ static void make_keylocker_data(void)
for (i = 0; i < KEYSIZE_128BIT; i++)
get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit));
+
+ keydata.valid = true;
}
/**
@@ -47,6 +54,8 @@ void flush_keylocker_data(void)
memset(keyaddr, 0, size);
clflush_cache_range(keyaddr, size);
+
+ keydata.valid = false;
}
#define KEYSRC_SWRAND 0
@@ -79,6 +88,40 @@ static int load_keylocker(void)
return err;
}
+#define KEYRESTORE_RETRY 1
+
+/**
+ * copy_keylocker() - Copy the internal key from the backup.
+ *
+ * Request hardware to copy the key in non-volatile memory to the CPU state. Do this
+ * again if the copy fails. The key may not be ready when the precedent backup is
+ * still in progress.
+ *
+ * Returns: -EBUSY if the copy fails, -ENODEV if no backup is available, or 0 if
+ * successful.
+ */
+static int copy_keylocker(void)
+{
+ int i;
+
+ if (!keybackup_available)
+ return -ENODEV;
+
+ wrmsrl(MSR_IA32_COPY_PLATFORM_TO_LOCAL, 1);
+
+ for (i = 0; i <= KEYRESTORE_RETRY; i++) {
+ u64 status;
+
+ if (i)
+ udelay(1);
+
+ rdmsrl(MSR_IA32_COPY_STATUS, status);
+ if (status & BIT(0))
+ return 0;
+ }
+ return -EBUSY;
+}
+
/**
* setup_keylocker() - Enable the feature if supported.
* @c: A pointer to struct cpuinfo_x86
@@ -104,13 +147,43 @@ void setup_keylocker(struct cpuinfo_x86 *c)
goto disable;
}
+ keybackup_available = (ebx & KEYLOCKER_CPUID_EBX_BACKUP);
+ /* Internal key backup is essential with S3/4 states. */
+ if (!keybackup_available &&
+ (acpi_sleep_state_supported(ACPI_STATE_S3) ||
+ acpi_sleep_state_supported(ACPI_STATE_S4))) {
+ pr_debug("x86/keylocker: no key backup support with possible S3/4.\n");
+ goto disable;
+ }
+
make_keylocker_data();
- }
- err = load_keylocker();
- if (err) {
- pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err);
- goto disable;
+ err = load_keylocker();
+ if (err) {
+ pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err);
+ goto disable;
+ }
+
+ /* Back up the internal key in non-volatile memory if supported. */
+ if (keybackup_available)
+ wrmsrl(MSR_IA32_COPY_LOCAL_TO_PLATFORM, 1);
+ } else {
+
+ /*
+ * Load the internal key directly when available in memory, which is only
+ * possible at boot-time.
+ *
+ * NB: When system wakes up, this path also recovers the internal key.
+ */
+ if (keydata.valid)
+ err = load_keylocker();
+ else
+ err = copy_keylocker();
+ if (err) {
+ pr_err_once("x86/keylocker: Fail to %s internal key (rc: %d).\n",
+ keydata.valid ? "load" : "copy", err);
+ goto disable;
+ }
}
pr_info_once("x86/keylocker: Enabled.\n");
@@ -123,3 +196,42 @@ void setup_keylocker(struct cpuinfo_x86 *c)
/* Make sure the feature disabled for kexec-reboot. */
cr4_clear_bits(X86_CR4_KEYLOCKER);
}
+
+/**
+ * restore_keylocker() - Restore the internal key.
+ *
+ * The boot CPU executes this while other CPUs restore it through the setup function.
+ *
+ * Returns: Nothing
+ */
+void restore_keylocker(void)
+{
+ u64 backup_status;
+ int err;
+
+ if (!boot_cpu_has(X86_FEATURE_KEYLOCKER))
+ return;
+
+ /*
+ * IA32_IWKEYBACKUP_STATUS MSR contains a bitmap that indicates an invalid backup if bit 0
+ * is set and a read (or write) error if bit 2 is set.
+ */
+ rdmsrl(MSR_IA32_IWKEYBACKUP_STATUS, backup_status);
+ if (WARN(!(backup_status & BIT(0)),
+ "x86/keylocker: Internal key backup access failed with %s.\n",
+ (backup_status & BIT(2)) ? "read error" : "invalid status"))
+ goto disable_out;
+
+ err = copy_keylocker();
+ if (err) {
+ pr_err("x86/keylocker: Internal key restoration failed (rc: %d).\n", err);
+ goto disable_out;
+ }
+
+ return;
+
+disable_out:
+ pr_info("x86/keylocker: Disabled with internal key restoration failure.\n");
+ setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER);
+ cr4_clear_bits(X86_CR4_KEYLOCKER);
+}
diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index 3a070e7cdb8b..ace94f07701a 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -25,6 +25,7 @@
#include <asm/cpu.h>
#include <asm/mmu_context.h>
#include <asm/cpu_device_id.h>
+#include <asm/keylocker.h>
#ifdef CONFIG_X86_32
__visible unsigned long saved_context_ebx;
@@ -261,6 +262,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt)
mtrr_bp_restore();
perf_restore_debug_store();
msr_restore_context(ctxt);
+ restore_keylocker();
c = &cpu_data(smp_processor_id());
if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL))
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 06/11] x86/cpu: Add a config option and a chicken bit for Key Locker
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (4 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 07/11] selftests/x86: Test Key Locker internal key maintenance Chang S. Bae
` (5 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae, linux-doc
Add a kernel config option to enable the feature (disabled by default) at
compile-time.
Also, add a new command-line parameter -- 'nokeylocker' to disable the
feature at boot-time.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Documentation/admin-guide/kernel-parameters.txt | 2 ++
arch/x86/Kconfig | 14 ++++++++++++++
arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++
3 files changed, 32 insertions(+)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index cb89dbdedc46..d0afe36c1802 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3284,6 +3284,8 @@
nohugevmalloc [PPC] Disable kernel huge vmalloc mappings.
+ nokeylocker [X86] Disables Key Locker hardware feature.
+
nosmt [KNL,S390] Disable symmetric multithreading (SMT).
Equivalent to smt=1.
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 0045e1b44190..de8eeb705ed8 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1885,6 +1885,20 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
If unsure, say y.
+config X86_KEYLOCKER
+ prompt "Key Locker"
+ def_bool n
+ depends on CPU_SUP_INTEL
+ help
+ Key Locker is a new security feature to protect data encryption
+ keys for the Advanced Encryption Standard (AES) algorithm.
+
+ When enabled, every CPU has a unique internal key to wrap AES
+ keys in an encoded format. The internal key is not accessible
+ to software once loaded.
+
+ If unsure, say y.
+
choice
prompt "TSX enable mode"
depends on CPU_SUP_INTEL
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 2e64371acb81..c655cce30c2b 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -353,6 +353,22 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c)
/* These bits should not change their value after CPU init is finished. */
static const unsigned long cr4_pinned_mask =
X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | X86_CR4_FSGSBASE;
+
+static __init int x86_nokeylocker_setup(char *arg)
+{
+ /* Expect an exact match without trailing characters. */
+ if (strlen(arg))
+ return 0;
+
+ if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER))
+ return 1;
+
+ setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER);
+ pr_info("x86/keylocker: Disabled by kernel command line.\n");
+ return 1;
+}
+__setup("nokeylocker", x86_nokeylocker_setup);
+
static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning);
static unsigned long cr4_pinned_bits __ro_after_init;
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 07/11] selftests/x86: Test Key Locker internal key maintenance
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (5 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 06/11] x86/cpu: Add a config option and a chicken bit for Key Locker Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 08/11] crypto: x86/aes-ni - Improve error handling Chang S. Bae
` (4 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae, linux-kselftest
The test validates the internal key to be the same in all CPUs.
It performs the validation again with the Suspend-To-RAM (ACPI S3) state.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-kselftest@vger.kernel.org
---
Changes from RFC v1:
* Commented the binutils version number for ENCODEKEY128 (Peter Zijlstra)
---
tools/testing/selftests/x86/Makefile | 2 +-
tools/testing/selftests/x86/keylocker.c | 177 ++++++++++++++++++++++++
2 files changed, 178 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/x86/keylocker.c
diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile
index 333980375bc7..09237cc84108 100644
--- a/tools/testing/selftests/x86/Makefile
+++ b/tools/testing/selftests/x86/Makefile
@@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh $(CC) trivial_program.c -no-pie)
TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \
check_initial_reg_state sigreturn iopl ioperm \
test_vsyscall mov_ss_trap \
- syscall_arg_fault fsgsbase_restore
+ syscall_arg_fault fsgsbase_restore keylocker
TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \
test_FCMOV test_FCOMI test_FISTTP \
vdso_restorer
diff --git a/tools/testing/selftests/x86/keylocker.c b/tools/testing/selftests/x86/keylocker.c
new file mode 100644
index 000000000000..78bbb7939b1a
--- /dev/null
+++ b/tools/testing/selftests/x86/keylocker.c
@@ -0,0 +1,177 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * keylocker.c, validate the internal key management.
+ */
+#undef _GNU_SOURCE
+#define _GNU_SOURCE 1
+
+#include <stdio.h>
+#include <stdbool.h>
+#include <string.h>
+#include <fcntl.h>
+#include <err.h>
+#include <sched.h>
+#include <setjmp.h>
+#include <signal.h>
+#include <unistd.h>
+
+#define HANDLE_SIZE 48
+
+static bool keylocker_disabled;
+
+/* Encode a 128-bit key to a 384-bit handle */
+static inline void __encode_key(char *handle)
+{
+ static const unsigned char aeskey[] = { 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38,
+ 0x71, 0x77, 0x74, 0x69, 0x6f, 0x6b, 0x6c, 0x78 };
+
+ asm volatile ("movdqu %0, %%xmm0" : : "m" (*aeskey) :);
+
+ /* Set no restriction to the handle */
+ asm volatile ("mov $0, %%eax" :);
+
+ /* ENCODEKEY128 %EAX (supported by binutils >= 2.36) */
+ asm volatile (".byte 0xf3, 0xf, 0x38, 0xfa, 0xc0");
+
+ asm volatile ("movdqu %%xmm0, %0; movdqu %%xmm1, %1; movdqu %%xmm2, %2;"
+ : "=m" (handle[0]), "=m" (handle[0x10]), "=m" (handle[0x20]));
+}
+
+static jmp_buf jmpbuf;
+
+static void handle_sigill(int sig, siginfo_t *si, void *ctx_void)
+{
+ keylocker_disabled = true;
+ siglongjmp(jmpbuf, 1);
+}
+
+static bool encode_key(char *handle)
+{
+ bool success = true;
+ struct sigaction sa;
+ int ret;
+
+ memset(&sa, 0, sizeof(sa));
+
+ /* Set signal handler */
+ sa.sa_flags = SA_SIGINFO;
+ sa.sa_sigaction = handle_sigill;
+ sigemptyset(&sa.sa_mask);
+ ret = sigaction(SIGILL, &sa, 0);
+ if (ret)
+ err(1, "sigaction");
+
+ if (sigsetjmp(jmpbuf, 1))
+ success = false;
+ else
+ __encode_key(handle);
+
+ /* Clear signal handler */
+ sa.sa_flags = 0;
+ sa.sa_sigaction = NULL;
+ sa.sa_handler = SIG_DFL;
+ sigemptyset(&sa.sa_mask);
+ ret = sigaction(SIGILL, &sa, 0);
+ if (ret)
+ err(1, "sigaction");
+
+ return success;
+}
+
+/*
+ * Test if the internal key is the same in all the CPUs:
+ *
+ * Since the value is not readable, compare the encoded output of a AES key
+ * between CPUs.
+ */
+
+static int nerrs;
+
+static unsigned char cpu0_handle[HANDLE_SIZE] = { 0 };
+
+static void test_internal_key(bool slept, long cpus)
+{
+ int cpu, errs;
+
+ printf("Test the internal key consistency between CPUs\n");
+
+ for (cpu = 0, errs = 0; cpu < cpus; cpu++) {
+ char handle[HANDLE_SIZE] = { 0 };
+ cpu_set_t mask;
+ bool success;
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu, &mask);
+ sched_setaffinity(0, sizeof(cpu_set_t), &mask);
+
+ success = encode_key(handle);
+ if (!success) {
+ /* The encode should success after the S3 sleep */
+ if (slept)
+ errs++;
+ printf("[%s]\tKey Locker disabled at CPU%d\n",
+ slept ? "FAIL" : "NOTE", cpu);
+ continue;
+ }
+
+ if (cpu == 0 && !slept) {
+ /* Record the first handle value as reference */
+ memcpy(cpu0_handle, handle, HANDLE_SIZE);
+ } else if (memcmp(cpu0_handle, handle, HANDLE_SIZE)) {
+ printf("[FAIL]\tMismatched internal key at CPU%d\n",
+ cpu);
+ errs++;
+ }
+ }
+
+ if (errs == 0 && !keylocker_disabled)
+ printf("[OK]\tAll the internal keys are the same\n");
+ else
+ nerrs += errs;
+}
+
+static void switch_to_sleep(bool *slept)
+{
+ ssize_t bytes;
+ int fd;
+
+ printf("Transition to Suspend-To-RAM state\n");
+
+ fd = open("/sys/power/mem_sleep", O_RDWR);
+ if (fd < 0)
+ err(1, "Open /sys/power/mem_sleep");
+
+ bytes = write(fd, "deep", strlen("deep"));
+ if (bytes != strlen("deep"))
+ err(1, "Write /sys/power/mem_sleep");
+ close(fd);
+
+ fd = open("/sys/power/state", O_RDWR);
+ if (fd < 0)
+ err(1, "Open /sys/power/state");
+
+ bytes = write(fd, "mem", strlen("mem"));
+ if (bytes != strlen("mem"))
+ err(1, "Write /sys/power/state");
+ close(fd);
+
+ printf("Wake up from Suspend-To-RAM state\n");
+ *slept = true;
+}
+
+int main(void)
+{
+ bool slept = false;
+ long cpus;
+
+ cpus = sysconf(_SC_NPROCESSORS_ONLN);
+ printf("%ld CPUs in the system\n", cpus);
+
+ test_internal_key(slept, cpus);
+ if (keylocker_disabled)
+ return nerrs ? 1 : 0;
+
+ switch_to_sleep(&slept);
+ test_internal_key(slept, cpus);
+ return nerrs ? 1 : 0;
+}
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 08/11] crypto: x86/aes-ni - Improve error handling
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (6 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 07/11] selftests/x86: Test Key Locker internal key maintenance Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 09/11] crypto: x86/aes-ni - Refactor to prepare a new AES implementation Chang S. Bae
` (3 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Some error case in the glue code is possibly ignored and thus not handled
correctly. Make sure each error code is not overwritten.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: x86@kernel.org
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Added as a new patch. This change prepares to address Ard's feedback.
---
arch/x86/crypto/aesni-intel_glue.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 2144e54a6c89..685943f0e5a3 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -291,7 +291,7 @@ static int ecb_encrypt(struct skcipher_request *req)
nbytes & AES_BLOCK_MASK);
kernel_fpu_end();
nbytes &= AES_BLOCK_SIZE - 1;
- err = skcipher_walk_done(&walk, nbytes);
+ err |= skcipher_walk_done(&walk, nbytes);
}
return err;
@@ -313,7 +313,7 @@ static int ecb_decrypt(struct skcipher_request *req)
nbytes & AES_BLOCK_MASK);
kernel_fpu_end();
nbytes &= AES_BLOCK_SIZE - 1;
- err = skcipher_walk_done(&walk, nbytes);
+ err |= skcipher_walk_done(&walk, nbytes);
}
return err;
@@ -335,7 +335,7 @@ static int cbc_encrypt(struct skcipher_request *req)
nbytes & AES_BLOCK_MASK, walk.iv);
kernel_fpu_end();
nbytes &= AES_BLOCK_SIZE - 1;
- err = skcipher_walk_done(&walk, nbytes);
+ err |= skcipher_walk_done(&walk, nbytes);
}
return err;
@@ -357,7 +357,7 @@ static int cbc_decrypt(struct skcipher_request *req)
nbytes & AES_BLOCK_MASK, walk.iv);
kernel_fpu_end();
nbytes &= AES_BLOCK_SIZE - 1;
- err = skcipher_walk_done(&walk, nbytes);
+ err |= skcipher_walk_done(&walk, nbytes);
}
return err;
@@ -522,7 +522,7 @@ static int ctr_crypt(struct skcipher_request *req)
nbytes = 0;
}
kernel_fpu_end();
- err = skcipher_walk_done(&walk, nbytes);
+ err |= skcipher_walk_done(&walk, nbytes);
}
return err;
}
@@ -691,7 +691,7 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
}
kernel_fpu_end();
- err = skcipher_walk_done(&walk, 0);
+ err |= skcipher_walk_done(&walk, 0);
}
if (err)
@@ -862,7 +862,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt)
skcipher_request_set_crypt(&subreq, req->src, req->dst,
blocks * AES_BLOCK_SIZE, req->iv);
req = &subreq;
- err = skcipher_walk_virt(&walk, req, false);
+ err |= skcipher_walk_virt(&walk, req, false);
} else {
tail = 0;
}
@@ -888,7 +888,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt)
nbytes, walk.iv);
kernel_fpu_end();
- err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+ err |= skcipher_walk_done(&walk, walk.nbytes - nbytes);
if (walk.nbytes > 0)
kernel_fpu_begin();
@@ -905,7 +905,7 @@ static int xts_crypt(struct skcipher_request *req, bool encrypt)
skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
req->iv);
- err = skcipher_walk_virt(&walk, &subreq, false);
+ err |= skcipher_walk_virt(&walk, &subreq, false);
if (err)
return err;
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 09/11] crypto: x86/aes-ni - Refactor to prepare a new AES implementation
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (7 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 08/11] crypto: x86/aes-ni - Improve error handling Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Chang S. Bae
` (2 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae, Ard Biesheuvel
Refactor some glue code and constant values of assembly code in preparation
for a new AES implementation.
Also, introduce wrappers for data transformation functions to return an
error value. New AES instructions may populate an error condition while
AES-NI does not.
No functional change.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: x86@kernel.org
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Added as a new patch. (Ard Biesheuvel)
---
arch/x86/crypto/Makefile | 2 +-
arch/x86/crypto/aes-intel_asm.S | 26 +++
arch/x86/crypto/aes-intel_glue.c | 208 +++++++++++++++++++
arch/x86/crypto/aes-intel_glue.h | 61 ++++++
arch/x86/crypto/aesni-intel_asm.S | 90 ++++-----
arch/x86/crypto/aesni-intel_glue.c | 308 ++++-------------------------
arch/x86/crypto/aesni-intel_glue.h | 88 +++++++++
7 files changed, 452 insertions(+), 331 deletions(-)
create mode 100644 arch/x86/crypto/aes-intel_asm.S
create mode 100644 arch/x86/crypto/aes-intel_glue.c
create mode 100644 arch/x86/crypto/aes-intel_glue.h
create mode 100644 arch/x86/crypto/aesni-intel_glue.h
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index d0959e7b809f..30d558df67cd 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -47,7 +47,7 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o
chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
-aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
+aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o
aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
diff --git a/arch/x86/crypto/aes-intel_asm.S b/arch/x86/crypto/aes-intel_asm.S
new file mode 100644
index 000000000000..98abf875af79
--- /dev/null
+++ b/arch/x86/crypto/aes-intel_asm.S
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+/*
+ * Constant values shared between AES implementations:
+ */
+
+.pushsection .rodata
+.align 16
+.Lcts_permute_table:
+ .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+ .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+ .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
+ .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
+ .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+ .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
+#ifdef __x86_64__
+.Lbswap_mask:
+ .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
+#endif
+.popsection
+
+.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
+.align 16
+.Lgf128mul_x_ble_mask:
+ .octa 0x00000000000000010000000000000087
+.previous
diff --git a/arch/x86/crypto/aes-intel_glue.c b/arch/x86/crypto/aes-intel_glue.c
new file mode 100644
index 000000000000..66c67f1610c9
--- /dev/null
+++ b/arch/x86/crypto/aes-intel_glue.c
@@ -0,0 +1,208 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/err.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/xts.h>
+#include <crypto/scatterwalk.h>
+#include <crypto/internal/aead.h>
+#include <crypto/internal/simd.h>
+#include "aes-intel_glue.h"
+
+int ecb_crypt_common(struct skcipher_request *req,
+ int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len))
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+ struct skcipher_walk walk;
+ unsigned int nbytes;
+ int err;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while ((nbytes = walk.nbytes)) {
+ kernel_fpu_begin();
+ err = fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK);
+ kernel_fpu_end();
+ if (err)
+ return err;
+
+ nbytes &= AES_BLOCK_SIZE - 1;
+ err |= skcipher_walk_done(&walk, nbytes);
+ }
+
+ return err;
+}
+
+int cbc_crypt_common(struct skcipher_request *req,
+ int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv))
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+ struct skcipher_walk walk;
+ unsigned int nbytes;
+ int err;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while ((nbytes = walk.nbytes)) {
+ kernel_fpu_begin();
+ err = fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK,
+ walk.iv);
+ kernel_fpu_end();
+ if (err)
+ return err;
+
+ nbytes &= AES_BLOCK_SIZE - 1;
+ err |= skcipher_walk_done(&walk, nbytes);
+ }
+
+ return err;
+}
+
+int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen,
+ int (*fn)(struct crypto_tfm *tfm, void *raw_ctx,
+ const u8 *in_key, unsigned int key_len))
+{
+ struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
+ int err;
+
+ err = xts_verify_key(tfm, key, keylen);
+ if (err)
+ return err;
+
+ keylen /= 2;
+
+ /* first half of xts-key is for crypt */
+ err = fn(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, key, keylen);
+ if (err)
+ return err;
+
+ /* second half of xts-key is for tweak */
+ return fn(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, key + keylen, keylen);
+}
+
+int xts_crypt_common(struct skcipher_request *req,
+ int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv),
+ int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in))
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
+ int tail = req->cryptlen % AES_BLOCK_SIZE;
+ struct skcipher_request subreq;
+ struct skcipher_walk walk;
+ int err;
+
+ if (req->cryptlen < AES_BLOCK_SIZE)
+ return -EINVAL;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ if (unlikely(tail > 0 && walk.nbytes < walk.total)) {
+ int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2;
+
+ skcipher_walk_abort(&walk);
+
+ skcipher_request_set_tfm(&subreq, tfm);
+ skcipher_request_set_callback(&subreq,
+ skcipher_request_flags(req),
+ NULL, NULL);
+ skcipher_request_set_crypt(&subreq, req->src, req->dst,
+ blocks * AES_BLOCK_SIZE, req->iv);
+ req = &subreq;
+ err |= skcipher_walk_virt(&walk, req, false);
+ } else {
+ tail = 0;
+ }
+
+ kernel_fpu_begin();
+
+ /* calculate first value of T */
+ err |= crypt1_fn(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv);
+
+ while (walk.nbytes > 0) {
+ int nbytes = walk.nbytes;
+
+ if (nbytes < walk.total)
+ nbytes &= ~(AES_BLOCK_SIZE - 1);
+
+ err |= crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr,
+ nbytes, walk.iv);
+ kernel_fpu_end();
+
+ err |= skcipher_walk_done(&walk, walk.nbytes - nbytes);
+
+ if (walk.nbytes > 0)
+ kernel_fpu_begin();
+ }
+
+ if (unlikely(tail > 0 && !err)) {
+ struct scatterlist sg_src[2], sg_dst[2];
+ struct scatterlist *src, *dst;
+
+ dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
+ if (req->dst != req->src)
+ dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
+
+ skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
+ req->iv);
+
+ err |= skcipher_walk_virt(&walk, &subreq, false);
+ if (err)
+ return err;
+
+ kernel_fpu_begin();
+ err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr,
+ walk.nbytes, walk.iv);
+ kernel_fpu_end();
+
+ err |= skcipher_walk_done(&walk, 0);
+ }
+ return err;
+}
+
+#ifdef CONFIG_X86_64
+
+int ctr_crypt_common(struct skcipher_request *req,
+ int (*crypt_fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv),
+ int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in))
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+ u8 keystream[AES_BLOCK_SIZE];
+ struct skcipher_walk walk;
+ unsigned int nbytes;
+ int err;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while ((nbytes = walk.nbytes) > 0) {
+ kernel_fpu_begin();
+ if (nbytes & AES_BLOCK_MASK) {
+ err = crypt_fn(ctx, walk.dst.virt.addr, walk.src.virt.addr,
+ nbytes & AES_BLOCK_MASK, walk.iv);
+ }
+ nbytes &= ~AES_BLOCK_MASK;
+
+ if (walk.nbytes == walk.total && nbytes > 0) {
+ err = crypt1_fn(ctx, keystream, walk.iv);
+ crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes,
+ walk.src.virt.addr + walk.nbytes - nbytes,
+ keystream, nbytes);
+ crypto_inc(walk.iv, AES_BLOCK_SIZE);
+ nbytes = 0;
+ }
+ kernel_fpu_end();
+ if (err)
+ return err;
+
+ err |= skcipher_walk_done(&walk, nbytes);
+ }
+ return err;
+}
+
+#endif /* CONFIG_X86_64 */
diff --git a/arch/x86/crypto/aes-intel_glue.h b/arch/x86/crypto/aes-intel_glue.h
new file mode 100644
index 000000000000..41e93c0f7932
--- /dev/null
+++ b/arch/x86/crypto/aes-intel_glue.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Shared glue code between AES implementations, refactored from the AES-NI's.
+ */
+
+#ifndef _AES_INTEL_GLUE_H
+#define _AES_INTEL_GLUE_H
+
+#include <crypto/aes.h>
+
+#define AES_ALIGN 16
+#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN)))
+#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1))
+#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
+#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AES_ALIGN_EXTRA)
+#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA)
+
+struct aes_xts_ctx {
+ u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR;
+ u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR;
+};
+
+static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx)
+{
+ unsigned long addr = (unsigned long)raw_ctx;
+ unsigned long align = AES_ALIGN;
+
+ if (align <= crypto_tfm_ctx_alignment())
+ align = 1;
+
+ return (struct crypto_aes_ctx *)ALIGN(addr, align);
+}
+
+int ecb_crypt_common(struct skcipher_request *req,
+ int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len));
+
+int cbc_crypt_common(struct skcipher_request *req,
+ int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv));
+
+int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen,
+ int (*fn)(struct crypto_tfm *tfm, void *raw_ctx,
+ const u8 *in_key, unsigned int key_len));
+
+int xts_crypt_common(struct skcipher_request *req,
+ int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv),
+ int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in));
+
+#ifdef CONFIG_X86_64
+
+int ctr_crypt_common(struct skcipher_request *req,
+ int (*crypt_fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv),
+ int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in));
+
+#endif /*CONFIG_X86_64 */
+
+#endif /* _AES_INTEL_GLUE_H */
diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S
index 4e3972570916..06dfd71ffdf4 100644
--- a/arch/x86/crypto/aesni-intel_asm.S
+++ b/arch/x86/crypto/aesni-intel_asm.S
@@ -28,6 +28,7 @@
#include <linux/linkage.h>
#include <asm/frame.h>
#include <asm/nospec-branch.h>
+#include "aes-intel_asm.S"
/*
* The following macros are used to move an (un)aligned 16 byte value to/from
@@ -1937,9 +1938,9 @@ SYM_FUNC_START(aesni_set_key)
SYM_FUNC_END(aesni_set_key)
/*
- * void aesni_enc(const void *ctx, u8 *dst, const u8 *src)
+ * void _aesni_enc(const void *ctx, u8 *dst, const u8 *src)
*/
-SYM_FUNC_START(aesni_enc)
+SYM_FUNC_START(_aesni_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
@@ -1958,7 +1959,7 @@ SYM_FUNC_START(aesni_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_enc)
+SYM_FUNC_END(_aesni_enc)
/*
* _aesni_enc1: internal ABI
@@ -2126,9 +2127,9 @@ SYM_FUNC_START_LOCAL(_aesni_enc4)
SYM_FUNC_END(_aesni_enc4)
/*
- * void aesni_dec (const void *ctx, u8 *dst, const u8 *src)
+ * void _aesni_dec (const void *ctx, u8 *dst, const u8 *src)
*/
-SYM_FUNC_START(aesni_dec)
+SYM_FUNC_START(_aesni_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
@@ -2148,7 +2149,7 @@ SYM_FUNC_START(aesni_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_dec)
+SYM_FUNC_END(_aesni_dec)
/*
* _aesni_dec1: internal ABI
@@ -2316,10 +2317,10 @@ SYM_FUNC_START_LOCAL(_aesni_dec4)
SYM_FUNC_END(_aesni_dec4)
/*
- * void aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- * size_t len)
+ * void _aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len)
*/
-SYM_FUNC_START(aesni_ecb_enc)
+SYM_FUNC_START(_aesni_ecb_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl LEN
@@ -2373,13 +2374,13 @@ SYM_FUNC_START(aesni_ecb_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_ecb_enc)
+SYM_FUNC_END(_aesni_ecb_enc)
/*
- * void aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- * size_t len);
+ * void _aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len);
*/
-SYM_FUNC_START(aesni_ecb_dec)
+SYM_FUNC_START(_aesni_ecb_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl LEN
@@ -2434,13 +2435,13 @@ SYM_FUNC_START(aesni_ecb_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_ecb_dec)
+SYM_FUNC_END(_aesni_ecb_dec)
/*
- * void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- * size_t len, u8 *iv)
+ * void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cbc_enc)
+SYM_FUNC_START(_aesni_cbc_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2478,13 +2479,13 @@ SYM_FUNC_START(aesni_cbc_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cbc_enc)
+SYM_FUNC_END(_aesni_cbc_enc)
/*
- * void aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- * size_t len, u8 *iv)
+ * void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cbc_dec)
+SYM_FUNC_START(_aesni_cbc_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2571,7 +2572,7 @@ SYM_FUNC_START(aesni_cbc_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cbc_dec)
+SYM_FUNC_END(_aesni_cbc_dec)
/*
* void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
@@ -2691,21 +2692,6 @@ SYM_FUNC_START(aesni_cts_cbc_dec)
ret
SYM_FUNC_END(aesni_cts_cbc_dec)
-.pushsection .rodata
-.align 16
-.Lcts_permute_table:
- .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
- .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
- .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
- .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f
- .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
- .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80
-#ifdef __x86_64__
-.Lbswap_mask:
- .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
-#endif
-.popsection
-
#ifdef __x86_64__
/*
* _aesni_inc_init: internal ABI
@@ -2757,10 +2743,10 @@ SYM_FUNC_START_LOCAL(_aesni_inc)
SYM_FUNC_END(_aesni_inc)
/*
- * void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
- * size_t len, u8 *iv)
+ * void _aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_ctr_enc)
+SYM_FUNC_START(_aesni_ctr_enc)
FRAME_BEGIN
cmp $16, LEN
jb .Lctr_enc_just_ret
@@ -2817,16 +2803,10 @@ SYM_FUNC_START(aesni_ctr_enc)
.Lctr_enc_just_ret:
FRAME_END
ret
-SYM_FUNC_END(aesni_ctr_enc)
+SYM_FUNC_END(_aesni_ctr_enc)
#endif
-.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
-.align 16
-.Lgf128mul_x_ble_mask:
- .octa 0x00000000000000010000000000000087
-.previous
-
/*
* _aesni_gf128mul_x_ble: internal ABI
* Multiply in GF(2^128) for XTS IVs
@@ -2846,10 +2826,10 @@ SYM_FUNC_END(aesni_ctr_enc)
pxor KEY, IV;
/*
- * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
- * const u8 *src, unsigned int len, le128 *iv)
+ * void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
+ * const u8 *src, unsigned int len, le128 *iv)
*/
-SYM_FUNC_START(aesni_xts_encrypt)
+SYM_FUNC_START(_aesni_xts_encrypt)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2998,13 +2978,13 @@ SYM_FUNC_START(aesni_xts_encrypt)
movups STATE, (OUTP)
jmp .Lxts_enc_ret
-SYM_FUNC_END(aesni_xts_encrypt)
+SYM_FUNC_END(_aesni_xts_encrypt)
/*
- * void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
- * const u8 *src, unsigned int len, le128 *iv)
+ * void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
+ * const u8 *src, unsigned int len, le128 *iv)
*/
-SYM_FUNC_START(aesni_xts_decrypt)
+SYM_FUNC_START(_aesni_xts_decrypt)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -3160,4 +3140,4 @@ SYM_FUNC_START(aesni_xts_decrypt)
movups STATE, (OUTP)
jmp .Lxts_dec_ret
-SYM_FUNC_END(aesni_xts_decrypt)
+SYM_FUNC_END(_aesni_xts_decrypt)
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 685943f0e5a3..4c6733c40ac4 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -35,34 +35,24 @@
#include <linux/workqueue.h>
#include <linux/spinlock.h>
#include <linux/static_call.h>
+#include "aes-intel_glue.h"
+#include "aesni-intel_glue.h"
-
-#define AESNI_ALIGN 16
-#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN)))
-#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1))
#define RFC4106_HASH_SUBKEY_SIZE 16
-#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
-#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_EXTRA)
-#define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA)
/* This data is stored at the end of the crypto_tfm struct.
* It's a type of per "session" data storage location.
* This needs to be 16 byte aligned.
*/
struct aesni_rfc4106_gcm_ctx {
- u8 hash_subkey[16] AESNI_ALIGN_ATTR;
- struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR;
+ u8 hash_subkey[16] AES_ALIGN_ATTR;
+ struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR;
u8 nonce[4];
};
struct generic_gcmaes_ctx {
- u8 hash_subkey[16] AESNI_ALIGN_ATTR;
- struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR;
-};
-
-struct aesni_xts_ctx {
- u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR;
- u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR;
+ u8 hash_subkey[16] AES_ALIGN_ATTR;
+ struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR;
};
#define GCM_BLOCK_LEN 16
@@ -80,18 +70,6 @@ struct gcm_context_data {
u8 hash_keys[GCM_BLOCK_LEN * 16];
};
-asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
- unsigned int key_len);
-asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in);
-asmlinkage void aesni_dec(const void *ctx, u8 *out, const u8 *in);
-asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len);
-asmlinkage void aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len);
-asmlinkage void aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
-asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out,
const u8 *in, unsigned int len, u8 *iv);
asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
@@ -100,16 +78,8 @@ asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
#define AVX_GEN2_OPTSIZE 640
#define AVX_GEN4_OPTSIZE 4096
-asmlinkage void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
-
-asmlinkage void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
-
#ifdef CONFIG_X86_64
-asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
DEFINE_STATIC_CALL(aesni_ctr_enc_tfm, aesni_ctr_enc);
/* Scatter / Gather routines, with args similar to above */
@@ -187,7 +157,7 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(gcm_use_avx2);
static inline struct
aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm)
{
- unsigned long align = AESNI_ALIGN;
+ unsigned long align = AES_ALIGN;
if (align <= crypto_tfm_ctx_alignment())
align = 1;
@@ -197,7 +167,7 @@ aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm)
static inline struct
generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm)
{
- unsigned long align = AESNI_ALIGN;
+ unsigned long align = AES_ALIGN;
if (align <= crypto_tfm_ctx_alignment())
align = 1;
@@ -205,16 +175,6 @@ generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm)
}
#endif
-static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx)
-{
- unsigned long addr = (unsigned long)raw_ctx;
- unsigned long align = AESNI_ALIGN;
-
- if (align <= crypto_tfm_ctx_alignment())
- align = 1;
- return (struct crypto_aes_ctx *)ALIGN(addr, align);
-}
-
static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx,
const u8 *in_key, unsigned int key_len)
{
@@ -250,7 +210,7 @@ static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
aes_encrypt(ctx, dst, src);
} else {
kernel_fpu_begin();
- aesni_enc(ctx, dst, src);
+ _aesni_enc(ctx, dst, src);
kernel_fpu_end();
}
}
@@ -263,7 +223,7 @@ static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
aes_decrypt(ctx, dst, src);
} else {
kernel_fpu_begin();
- aesni_dec(ctx, dst, src);
+ _aesni_dec(ctx, dst, src);
kernel_fpu_end();
}
}
@@ -277,90 +237,22 @@ static int aesni_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
static int ecb_encrypt(struct skcipher_request *req)
{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
- struct skcipher_walk walk;
- unsigned int nbytes;
- int err;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- while ((nbytes = walk.nbytes)) {
- kernel_fpu_begin();
- aesni_ecb_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr,
- nbytes & AES_BLOCK_MASK);
- kernel_fpu_end();
- nbytes &= AES_BLOCK_SIZE - 1;
- err |= skcipher_walk_done(&walk, nbytes);
- }
-
- return err;
+ return ecb_crypt_common(req, aesni_ecb_enc);
}
static int ecb_decrypt(struct skcipher_request *req)
{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
- struct skcipher_walk walk;
- unsigned int nbytes;
- int err;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- while ((nbytes = walk.nbytes)) {
- kernel_fpu_begin();
- aesni_ecb_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr,
- nbytes & AES_BLOCK_MASK);
- kernel_fpu_end();
- nbytes &= AES_BLOCK_SIZE - 1;
- err |= skcipher_walk_done(&walk, nbytes);
- }
-
- return err;
+ return ecb_crypt_common(req, aesni_ecb_dec);
}
static int cbc_encrypt(struct skcipher_request *req)
{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
- struct skcipher_walk walk;
- unsigned int nbytes;
- int err;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- while ((nbytes = walk.nbytes)) {
- kernel_fpu_begin();
- aesni_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr,
- nbytes & AES_BLOCK_MASK, walk.iv);
- kernel_fpu_end();
- nbytes &= AES_BLOCK_SIZE - 1;
- err |= skcipher_walk_done(&walk, nbytes);
- }
-
- return err;
+ return cbc_crypt_common(req, aesni_cbc_enc);
}
static int cbc_decrypt(struct skcipher_request *req)
{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
- struct skcipher_walk walk;
- unsigned int nbytes;
- int err;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- while ((nbytes = walk.nbytes)) {
- kernel_fpu_begin();
- aesni_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr,
- nbytes & AES_BLOCK_MASK, walk.iv);
- kernel_fpu_end();
- nbytes &= AES_BLOCK_SIZE - 1;
- err |= skcipher_walk_done(&walk, nbytes);
- }
-
- return err;
+ return cbc_crypt_common(req, aesni_cbc_dec);
}
static int cts_cbc_encrypt(struct skcipher_request *req)
@@ -476,8 +368,8 @@ static int cts_cbc_decrypt(struct skcipher_request *req)
}
#ifdef CONFIG_X86_64
-static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv)
+static int aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out,
+ const u8 *in, unsigned int len, u8 *iv)
{
/*
* based on key length, override with the by8 version
@@ -491,40 +383,12 @@ static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out,
aes_ctr_enc_192_avx_by8(in, iv, (void *)ctx, out, len);
else
aes_ctr_enc_256_avx_by8(in, iv, (void *)ctx, out, len);
+ return 0;
}
static int ctr_crypt(struct skcipher_request *req)
{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
- u8 keystream[AES_BLOCK_SIZE];
- struct skcipher_walk walk;
- unsigned int nbytes;
- int err;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- while ((nbytes = walk.nbytes) > 0) {
- kernel_fpu_begin();
- if (nbytes & AES_BLOCK_MASK)
- static_call(aesni_ctr_enc_tfm)(ctx, walk.dst.virt.addr,
- walk.src.virt.addr,
- nbytes & AES_BLOCK_MASK,
- walk.iv);
- nbytes &= ~AES_BLOCK_MASK;
-
- if (walk.nbytes == walk.total && nbytes > 0) {
- aesni_enc(ctx, keystream, walk.iv);
- crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes,
- walk.src.virt.addr + walk.nbytes - nbytes,
- keystream, nbytes);
- crypto_inc(walk.iv, AES_BLOCK_SIZE);
- nbytes = 0;
- }
- kernel_fpu_end();
- err |= skcipher_walk_done(&walk, nbytes);
- }
- return err;
+ return ctr_crypt_common(req, static_call(aesni_ctr_enc_tfm), aesni_enc);
}
static int
@@ -606,8 +470,8 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
u8 *iv, void *aes_ctx, u8 *auth_tag,
unsigned long auth_tag_len)
{
- u8 databuf[sizeof(struct gcm_context_data) + (AESNI_ALIGN - 8)] __aligned(8);
- struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AESNI_ALIGN);
+ u8 databuf[sizeof(struct gcm_context_data) + (AES_ALIGN - 8)] __aligned(8);
+ struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AES_ALIGN);
unsigned long left = req->cryptlen;
struct scatter_walk assoc_sg_walk;
struct skcipher_walk walk;
@@ -762,8 +626,8 @@ static int helper_rfc4106_encrypt(struct aead_request *req)
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm);
void *aes_ctx = &(ctx->aes_key_expanded);
- u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8);
- u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN);
+ u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8);
+ u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN);
unsigned int i;
__be32 counter = cpu_to_be32(1);
@@ -790,8 +654,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req)
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm);
void *aes_ctx = &(ctx->aes_key_expanded);
- u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8);
- u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN);
+ u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8);
+ u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN);
unsigned int i;
if (unlikely(req->assoclen != 16 && req->assoclen != 20))
@@ -816,123 +680,17 @@ static int helper_rfc4106_decrypt(struct aead_request *req)
static int xts_aesni_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int keylen)
{
- struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
- int err;
-
- err = xts_verify_key(tfm, key, keylen);
- if (err)
- return err;
-
- keylen /= 2;
-
- /* first half of xts-key is for crypt */
- err = aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx,
- key, keylen);
- if (err)
- return err;
-
- /* second half of xts-key is for tweak */
- return aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx,
- key + keylen, keylen);
-}
-
-static int xts_crypt(struct skcipher_request *req, bool encrypt)
-{
- struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
- int tail = req->cryptlen % AES_BLOCK_SIZE;
- struct skcipher_request subreq;
- struct skcipher_walk walk;
- int err;
-
- if (req->cryptlen < AES_BLOCK_SIZE)
- return -EINVAL;
-
- err = skcipher_walk_virt(&walk, req, false);
-
- if (unlikely(tail > 0 && walk.nbytes < walk.total)) {
- int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2;
-
- skcipher_walk_abort(&walk);
-
- skcipher_request_set_tfm(&subreq, tfm);
- skcipher_request_set_callback(&subreq,
- skcipher_request_flags(req),
- NULL, NULL);
- skcipher_request_set_crypt(&subreq, req->src, req->dst,
- blocks * AES_BLOCK_SIZE, req->iv);
- req = &subreq;
- err |= skcipher_walk_virt(&walk, req, false);
- } else {
- tail = 0;
- }
-
- kernel_fpu_begin();
-
- /* calculate first value of T */
- aesni_enc(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv);
-
- while (walk.nbytes > 0) {
- int nbytes = walk.nbytes;
-
- if (nbytes < walk.total)
- nbytes &= ~(AES_BLOCK_SIZE - 1);
-
- if (encrypt)
- aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx),
- walk.dst.virt.addr, walk.src.virt.addr,
- nbytes, walk.iv);
- else
- aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx),
- walk.dst.virt.addr, walk.src.virt.addr,
- nbytes, walk.iv);
- kernel_fpu_end();
-
- err |= skcipher_walk_done(&walk, walk.nbytes - nbytes);
-
- if (walk.nbytes > 0)
- kernel_fpu_begin();
- }
-
- if (unlikely(tail > 0 && !err)) {
- struct scatterlist sg_src[2], sg_dst[2];
- struct scatterlist *src, *dst;
-
- dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
- if (req->dst != req->src)
- dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
-
- skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
- req->iv);
-
- err |= skcipher_walk_virt(&walk, &subreq, false);
- if (err)
- return err;
-
- kernel_fpu_begin();
- if (encrypt)
- aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx),
- walk.dst.virt.addr, walk.src.virt.addr,
- walk.nbytes, walk.iv);
- else
- aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx),
- walk.dst.virt.addr, walk.src.virt.addr,
- walk.nbytes, walk.iv);
- kernel_fpu_end();
-
- err = skcipher_walk_done(&walk, 0);
- }
- return err;
+ return xts_setkey_common(tfm, key, keylen, aes_set_key_common);
}
static int xts_encrypt(struct skcipher_request *req)
{
- return xts_crypt(req, true);
+ return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc);
}
static int xts_decrypt(struct skcipher_request *req)
{
- return xts_crypt(req, false);
+ return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc);
}
static struct crypto_alg aesni_cipher_alg = {
@@ -1061,8 +819,8 @@ static int generic_gcmaes_encrypt(struct aead_request *req)
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm);
void *aes_ctx = &(ctx->aes_key_expanded);
- u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8);
- u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN);
+ u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8);
+ u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN);
__be32 counter = cpu_to_be32(1);
memcpy(iv, req->iv, 12);
@@ -1078,8 +836,8 @@ static int generic_gcmaes_decrypt(struct aead_request *req)
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm);
void *aes_ctx = &(ctx->aes_key_expanded);
- u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8);
- u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN);
+ u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8);
+ u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN);
memcpy(iv, req->iv, 12);
*((__be32 *)(iv+12)) = counter;
@@ -1102,7 +860,7 @@ static struct aead_alg aesni_aeads[] = { {
.cra_flags = CRYPTO_ALG_INTERNAL,
.cra_blocksize = 1,
.cra_ctxsize = sizeof(struct aesni_rfc4106_gcm_ctx),
- .cra_alignmask = AESNI_ALIGN - 1,
+ .cra_alignmask = AES_ALIGN - 1,
.cra_module = THIS_MODULE,
},
}, {
@@ -1119,7 +877,7 @@ static struct aead_alg aesni_aeads[] = { {
.cra_flags = CRYPTO_ALG_INTERNAL,
.cra_blocksize = 1,
.cra_ctxsize = sizeof(struct generic_gcmaes_ctx),
- .cra_alignmask = AESNI_ALIGN - 1,
+ .cra_alignmask = AES_ALIGN - 1,
.cra_module = THIS_MODULE,
},
} };
diff --git a/arch/x86/crypto/aesni-intel_glue.h b/arch/x86/crypto/aesni-intel_glue.h
new file mode 100644
index 000000000000..51049f86d78e
--- /dev/null
+++ b/arch/x86/crypto/aesni-intel_glue.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Support for Intel AES-NI instructions. This file contains function
+ * prototypes to be referenced for other AES implementations
+ */
+
+asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+ unsigned int key_len);
+asmlinkage void _aesni_enc(const void *ctx, u8 *out, const u8 *in);
+asmlinkage void _aesni_dec(const void *ctx, u8 *out, const u8 *in);
+asmlinkage void _aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len);
+asmlinkage void _aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len);
+asmlinkage void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+asmlinkage void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+asmlinkage void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+asmlinkage void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+
+static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in)
+{
+ _aesni_enc(ctx, out, in);
+ return 0;
+}
+
+static inline int aesni_dec(const void *ctx, u8 *out, const u8 *in)
+{
+ _aesni_dec(ctx, out, in);
+ return 0;
+}
+
+static inline int aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len)
+{
+ _aesni_ecb_enc(ctx, out, in, len);
+ return 0;
+}
+
+static inline int aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len)
+{
+ _aesni_ecb_dec(ctx, out, in, len);
+ return 0;
+}
+
+static inline int aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ _aesni_cbc_enc(ctx, out, in, len, iv);
+ return 0;
+}
+
+static inline int aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ _aesni_cbc_dec(ctx, out, in, len, iv);
+ return 0;
+}
+
+static inline int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ _aesni_xts_encrypt(ctx, out, in, len, iv);
+ return 0;
+}
+
+static inline int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ _aesni_xts_decrypt(ctx, out, in, len, iv);
+ return 0;
+}
+
+#ifdef CONFIG_X86_64
+
+asmlinkage void _aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+
+static inline int aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ _aesni_ctr_enc(ctx, out, in, len, iv);
+ return 0;
+}
+
+#endif /* CONFIG_X86_64 */
+
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (8 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 09/11] crypto: x86/aes-ni - Refactor to prepare a new AES implementation Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-17 21:34 ` Eric Biggers
2021-05-18 16:57 ` Andy Lutomirski
2021-05-14 20:15 ` [RFC PATCH v2 11/11] x86/cpu: Support the hardware randomization option for Key Locker internal key Chang S. Bae
2021-05-15 18:01 ` [RFC PATCH v2 00/11] x86: Support Intel Key Locker Andy Lutomirski
11 siblings, 2 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae
Key Locker (KL) is Intel's new security feature that protects the AES key
at the time of data transformation. New AES SIMD instructions -- as a
successor of Intel's AES-NI -- are provided to encode an AES key and
reference it for the AES algorithm.
New instructions support 128/256-bit keys. While it is not desirable to
receive any 192-bit key, AES-NI instructions are taken to serve this size.
New instructions are operational in both 32-/64-bit modes.
Included are methods for ECB, CBC, CTR, and XTS modes. They are not
compatible with other implementations as referencing an encrypted form
only.
XTS needs a separate 32-bit assembly code as the 64-bit version clobbers
more than eight 128-bit registers. Support 64-bit mode only.
setkey() encodes an AES key. Users may displace the AES key right after
this as encrypt()/decrypt() methods do not need the key.
Most glue code reuses the refactored functions from the AES-NI part. When
enabled, it will have a bit higher priority than the AES-NI as providing
key protection.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: x86@kernel.org
Cc: linux-crypto@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
Changes from RFC v1:
* Rebased on the refactored code. (Ard Biesheuvel)
* Dropped exporting the single block interface. (Ard Biesheuvel)
* Fixed the fallback and error handling paths. (Ard Biesheuvel)
* Revised the module description. (Dave Hansen and Peter Zijlsta)
* Made the build depend on the binutils version to support new instructions.
(Borislav Petkov and Peter Zijlstra)
* Updated the changelog accordingly.
---
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aeskl-intel_asm.S | 1181 ++++++++++++++++++++++++++++
arch/x86/crypto/aeskl-intel_glue.c | 390 +++++++++
crypto/Kconfig | 23 +
4 files changed, 1597 insertions(+)
create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 30d558df67cd..6bb4e2fb2b47 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o
aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aesni-intel_asm.o aeskl-intel_glue.o aes-intel_glue.o
+
obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S
new file mode 100644
index 000000000000..34426f11219d
--- /dev/null
+++ b/arch/x86/crypto/aeskl-intel_asm.S
@@ -0,0 +1,1181 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Implement AES algorithm using Intel AES Key Locker instructions.
+ *
+ * Most code is based from the AES-NI implementation, aesni-intel_asm.S
+ *
+ */
+
+#include <linux/linkage.h>
+#include <asm/inst.h>
+#include <asm/frame.h>
+#include "aes-intel_asm.S"
+
+.text
+
+#define STATE1 %xmm0
+#define STATE2 %xmm1
+#define STATE3 %xmm2
+#define STATE4 %xmm3
+#define STATE5 %xmm4
+#define STATE6 %xmm5
+#define STATE7 %xmm6
+#define STATE8 %xmm7
+#define STATE STATE1
+
+#define IV %xmm9
+#define KEY %xmm10
+#define BSWAP_MASK %xmm11
+#define CTR %xmm12
+#define INC %xmm13
+
+#ifdef __x86_64__
+#define IN1 %xmm8
+#define IN2 %xmm9
+#define IN3 %xmm10
+#define IN4 %xmm11
+#define IN5 %xmm12
+#define IN6 %xmm13
+#define IN7 %xmm14
+#define IN8 %xmm15
+#define IN IN1
+#define TCTR_LOW %r11
+#else
+#define IN %xmm1
+#endif
+
+#ifdef __x86_64__
+#define AREG %rax
+#define HANDLEP %rdi
+#define OUTP %rsi
+#define KLEN %r9d
+#define INP %rdx
+#define T1 %r10
+#define LEN %rcx
+#define IVP %r8
+#else
+#define AREG %eax
+#define HANDLEP %edi
+#define OUTP AREG
+#define KLEN %ebx
+#define INP %edx
+#define T1 %ecx
+#define LEN %esi
+#define IVP %ebp
+#endif
+
+#define UKEYP OUTP
+#define GF128MUL_MASK %xmm11
+
+/*
+ * int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len)
+ */
+SYM_FUNC_START(aeskl_setkey)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ push HANDLEP
+ movl (FRAME_OFFSET+8)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+12)(%esp), UKEYP # in_key
+ movl (FRAME_OFFSET+16)(%esp), %edx # key_len
+#endif
+ movl %edx, 480(HANDLEP)
+ movdqu (UKEYP), STATE1
+ mov $1, %eax
+ cmp $16, %dl
+ je .Lsetkey_128
+
+ movdqu 0x10(UKEYP), STATE2
+ encodekey256 %eax, %eax
+ movdqu STATE4, 0x30(HANDLEP)
+ jmp .Lsetkey_end
+.Lsetkey_128:
+ encodekey128 %eax, %eax
+
+.Lsetkey_end:
+ movdqu STATE1, (HANDLEP)
+ movdqu STATE2, 0x10(HANDLEP)
+ movdqu STATE3, 0x20(HANDLEP)
+
+ xor AREG, AREG
+#ifndef __x86_64__
+ popl HANDLEP
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(aeskl_setkey)
+
+/*
+ * int _aeskl_enc(const void *ctx, u8 *dst, const u8 *src)
+ */
+SYM_FUNC_START(_aeskl_enc)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+16)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+20)(%esp), INP # src
+#endif
+ movdqu (INP), STATE
+ movl 480(HANDLEP), KLEN
+
+ cmp $16, KLEN
+ je .Lenc_128
+ aesenc256kl (HANDLEP), STATE
+ jz .Lenc_err
+ jmp .Lenc_noerr
+.Lenc_128:
+ aesenc128kl (HANDLEP), STATE
+ jz .Lenc_err
+
+.Lenc_noerr:
+ xor AREG, AREG
+ jmp .Lenc_end
+.Lenc_err:
+ mov $1, AREG
+.Lenc_end:
+ movdqu STATE, (OUTP)
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_enc)
+
+/*
+ * int _aeskl_dec(const void *ctx, u8 *dst, const u8 *src)
+ */
+SYM_FUNC_START(_aeskl_dec)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+12)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+16)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+20)(%esp), INP # src
+#endif
+ movdqu (INP), STATE
+ mov 480(HANDLEP), KLEN
+
+ cmp $16, KLEN
+ je .Ldec_128
+ aesdec256kl (HANDLEP), STATE
+ jz .Ldec_err
+ jmp .Ldec_noerr
+.Ldec_128:
+ aesdec128kl (HANDLEP), STATE
+ jz .Ldec_err
+
+.Ldec_noerr:
+ xor AREG, AREG
+ jmp .Ldec_end
+.Ldec_err:
+ mov $1, AREG
+.Ldec_end:
+ movdqu STATE, (OUTP)
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_dec)
+
+/*
+ * int _aeskl_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len)
+ */
+SYM_FUNC_START(_aeskl_ecb_enc)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl LEN
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+16)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+20)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+24)(%esp), INP # src
+ movl (FRAME_OFFSET+28)(%esp), LEN # len
+#endif
+ test LEN, LEN
+ jz .Lecb_enc_noerr
+ mov 480(HANDLEP), KLEN
+ cmp $16, LEN
+ jb .Lecb_enc_noerr
+ cmp $128, LEN
+ jb .Lecb_enc1
+
+.align 4
+.Lecb_enc8:
+ movdqu (INP), STATE1
+ movdqu 0x10(INP), STATE2
+ movdqu 0x20(INP), STATE3
+ movdqu 0x30(INP), STATE4
+ movdqu 0x40(INP), STATE5
+ movdqu 0x50(INP), STATE6
+ movdqu 0x60(INP), STATE7
+ movdqu 0x70(INP), STATE8
+
+ cmp $16, KLEN
+ je .Lecb_enc8_128
+ aesencwide256kl (HANDLEP)
+ jz .Lecb_enc_err
+ jmp .Lecb_enc8_end
+.Lecb_enc8_128:
+ aesencwide128kl (HANDLEP)
+ jz .Lecb_enc_err
+
+.Lecb_enc8_end:
+ movdqu STATE1, (OUTP)
+ movdqu STATE2, 0x10(OUTP)
+ movdqu STATE3, 0x20(OUTP)
+ movdqu STATE4, 0x30(OUTP)
+ movdqu STATE5, 0x40(OUTP)
+ movdqu STATE6, 0x50(OUTP)
+ movdqu STATE7, 0x60(OUTP)
+ movdqu STATE8, 0x70(OUTP)
+
+ sub $128, LEN
+ add $128, INP
+ add $128, OUTP
+ cmp $128, LEN
+ jge .Lecb_enc8
+ cmp $16, LEN
+ jb .Lecb_enc_noerr
+
+.align 4
+.Lecb_enc1:
+ movdqu (INP), STATE1
+ cmp $16, KLEN
+ je .Lecb_enc1_128
+ aesenc256kl (HANDLEP), STATE
+ jz .Lecb_enc_err
+ jmp .Lecb_enc1_end
+.Lecb_enc1_128:
+ aesenc128kl (HANDLEP), STATE
+ jz .Lecb_enc_err
+
+.Lecb_enc1_end:
+ movdqu STATE1, (OUTP)
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lecb_enc1
+
+.Lecb_enc_noerr:
+ xor AREG, AREG
+ jmp .Lecb_enc_end
+.Lecb_enc_err:
+ mov $1, AREG
+.Lecb_enc_end:
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+ popl LEN
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_ecb_enc)
+
+/*
+ * int _aeskl_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len);
+ */
+SYM_FUNC_START(_aeskl_ecb_dec)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl LEN
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+16)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+20)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+24)(%esp), INP # src
+ movl (FRAME_OFFSET+28)(%esp), LEN # len
+#endif
+
+ test LEN, LEN
+ jz .Lecb_dec_noerr
+ mov 480(HANDLEP), KLEN
+ cmp $16, LEN
+ jb .Lecb_dec_noerr
+ cmp $128, LEN
+ jb .Lecb_dec1
+
+.align 4
+.Lecb_dec8:
+ movdqu (INP), STATE1
+ movdqu 0x10(INP), STATE2
+ movdqu 0x20(INP), STATE3
+ movdqu 0x30(INP), STATE4
+ movdqu 0x40(INP), STATE5
+ movdqu 0x50(INP), STATE6
+ movdqu 0x60(INP), STATE7
+ movdqu 0x70(INP), STATE8
+
+ cmp $16, KLEN
+ je .Lecb_dec8_128
+ aesdecwide256kl (HANDLEP)
+ jz .Lecb_dec_err
+ jmp .Lecb_dec8_end
+.Lecb_dec8_128:
+ aesdecwide128kl (HANDLEP)
+ jz .Lecb_dec_err
+
+.Lecb_dec8_end:
+ movdqu STATE1, (OUTP)
+ movdqu STATE2, 0x10(OUTP)
+ movdqu STATE3, 0x20(OUTP)
+ movdqu STATE4, 0x30(OUTP)
+ movdqu STATE5, 0x40(OUTP)
+ movdqu STATE6, 0x50(OUTP)
+ movdqu STATE7, 0x60(OUTP)
+ movdqu STATE8, 0x70(OUTP)
+
+ sub $128, LEN
+ add $128, INP
+ add $128, OUTP
+ cmp $128, LEN
+ jge .Lecb_dec8
+ cmp $16, LEN
+ jb .Lecb_dec_noerr
+
+.align 4
+.Lecb_dec1:
+ movdqu (INP), STATE1
+ cmp $16, KLEN
+ je .Lecb_dec1_128
+ aesdec256kl (HANDLEP), STATE
+ jz .Lecb_dec_err
+ jmp .Lecb_dec1_end
+.Lecb_dec1_128:
+ aesdec128kl (HANDLEP), STATE
+ jz .Lecb_dec_err
+
+.Lecb_dec1_end:
+ movdqu STATE1, (OUTP)
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lecb_dec1
+
+.Lecb_dec_noerr:
+ xor AREG, AREG
+ jmp .Lecb_dec_end
+.Lecb_dec_err:
+ mov $1, AREG
+.Lecb_dec_end:
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+ popl LEN
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_ecb_dec)
+
+/*
+ * int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_cbc_enc)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl IVP
+ pushl LEN
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+20)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+24)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+28)(%esp), INP # src
+ movl (FRAME_OFFSET+32)(%esp), LEN # len
+ movl (FRAME_OFFSET+36)(%esp), IVP # iv
+#endif
+
+ cmp $16, LEN
+ jb .Lcbc_enc_noerr
+ mov 480(HANDLEP), KLEN
+ movdqu (IVP), STATE
+
+.align 4
+.Lcbc_enc1:
+ movdqu (INP), IN
+ pxor IN, STATE
+
+ cmp $16, KLEN
+ je .Lcbc_enc1_128
+ aesenc256kl (HANDLEP), STATE
+ jz .Lcbc_enc_err
+ jmp .Lcbc_enc1_end
+.Lcbc_enc1_128:
+ aesenc128kl (HANDLEP), STATE
+ jz .Lcbc_enc_err
+
+.Lcbc_enc1_end:
+ movdqu STATE, (OUTP)
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lcbc_enc1
+ movdqu STATE, (IVP)
+
+.Lcbc_enc_noerr:
+ xor AREG, AREG
+ jmp .Lcbc_enc_end
+.Lcbc_enc_err:
+ mov $1, AREG
+.Lcbc_enc_end:
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+ popl LEN
+ popl IVP
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_cbc_enc)
+
+/*
+ * int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_cbc_dec)
+ FRAME_BEGIN
+#ifndef __x86_64__
+ pushl IVP
+ pushl LEN
+ pushl HANDLEP
+ pushl KLEN
+ movl (FRAME_OFFSET+20)(%esp), HANDLEP # ctx
+ movl (FRAME_OFFSET+24)(%esp), OUTP # dst
+ movl (FRAME_OFFSET+28)(%esp), INP # src
+ movl (FRAME_OFFSET+32)(%esp), LEN # len
+ movl (FRAME_OFFSET+36)(%esp), IVP # iv
+#endif
+
+ cmp $16, LEN
+ jb .Lcbc_dec_noerr
+ mov 480(HANDLEP), KLEN
+#ifdef __x86_64__
+ cmp $128, LEN
+ jb .Lcbc_dec1_pre
+
+.align 4
+.Lcbc_dec8:
+ movdqu 0x0(INP), STATE1
+ movdqu 0x10(INP), STATE2
+ movdqu 0x20(INP), STATE3
+ movdqu 0x30(INP), STATE4
+ movdqu 0x40(INP), STATE5
+ movdqu 0x50(INP), STATE6
+ movdqu 0x60(INP), STATE7
+ movdqu 0x70(INP), STATE8
+
+ movdqu (IVP), IN1
+ movdqa STATE1, IN2
+ movdqa STATE2, IN3
+ movdqa STATE3, IN4
+ movdqa STATE4, IN5
+ movdqa STATE5, IN6
+ movdqa STATE6, IN7
+ movdqa STATE7, IN8
+ movdqu STATE8, (IVP)
+
+ cmp $16, KLEN
+ je .Lcbc_dec8_128
+ aesdecwide256kl (HANDLEP)
+ jz .Lcbc_dec_err
+ jmp .Lcbc_dec8_end
+.Lcbc_dec8_128:
+ aesdecwide128kl (HANDLEP)
+ jz .Lcbc_dec_err
+
+.Lcbc_dec8_end:
+ pxor IN1, STATE1
+ pxor IN2, STATE2
+ pxor IN3, STATE3
+ pxor IN4, STATE4
+ pxor IN5, STATE5
+ pxor IN6, STATE6
+ pxor IN7, STATE7
+ pxor IN8, STATE8
+
+ movdqu STATE1, 0x0(OUTP)
+ movdqu STATE2, 0x10(OUTP)
+ movdqu STATE3, 0x20(OUTP)
+ movdqu STATE4, 0x30(OUTP)
+ movdqu STATE5, 0x40(OUTP)
+ movdqu STATE6, 0x50(OUTP)
+ movdqu STATE7, 0x60(OUTP)
+ movdqu STATE8, 0x70(OUTP)
+
+ sub $128, LEN
+ add $128, INP
+ add $128, OUTP
+ cmp $128, LEN
+ jge .Lcbc_dec8
+ cmp $16, LEN
+ jb .Lcbc_dec_noerr
+#endif
+
+.align 4
+.Lcbc_dec1_pre:
+ movdqu (IVP), STATE3
+.Lcbc_dec1:
+ movdqu (INP), STATE2
+ movdqa STATE2, STATE1
+
+ cmp $16, KLEN
+ je .Lcbc_dec1_128
+ aesdec256kl (HANDLEP), STATE1
+ jz .Lcbc_dec_err
+ jmp .Lcbc_dec1_end
+.Lcbc_dec1_128:
+ aesdec128kl (HANDLEP), STATE1
+ jz .Lcbc_dec_err
+
+.Lcbc_dec1_end:
+ pxor STATE3, STATE1
+ movdqu STATE1, (OUTP)
+ movdqa STATE2, STATE3
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lcbc_dec1
+ movdqu STATE3, (IVP)
+
+.Lcbc_dec_noerr:
+ xor AREG, AREG
+ jmp .Lcbc_dec_end
+.Lcbc_dec_err:
+ mov $1, AREG
+.Lcbc_dec_end:
+#ifndef __x86_64__
+ popl KLEN
+ popl HANDLEP
+ popl LEN
+ popl IVP
+#endif
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_cbc_dec)
+
+#ifdef __x86_64__
+
+/*
+ * _aeskl_ctr_inc_init: internal ABI
+ * setup registers used by _aesni_inc
+ * input:
+ * IV
+ * output:
+ * CTR: == IV, in little endian
+ * TCTR_LOW: == lower qword of CTR
+ * INC: == 1, in little endian
+ * BSWAP_MASK == endian swapping mask
+ */
+SYM_FUNC_START_LOCAL(_aeskl_ctr_inc_init)
+ movaps .Lbswap_mask, BSWAP_MASK
+ movaps IV, CTR
+ pshufb BSWAP_MASK, CTR
+ mov $1, TCTR_LOW
+ movq TCTR_LOW, INC
+ movq CTR, TCTR_LOW
+ ret
+SYM_FUNC_END(_aeskl_ctr_inc_init)
+
+/*
+ * _aeskl_ctr_inc: internal ABI
+ * Increase IV by 1, IV is in big endian
+ * input:
+ * IV
+ * CTR: == IV, in little endian
+ * TCTR_LOW: == lower qword of CTR
+ * INC: == 1, in little endian
+ * BSWAP_MASK == endian swapping mask
+ * output:
+ * IV: Increase by 1
+ * changed:
+ * CTR: == output IV, in little endian
+ * TCTR_LOW: == lower qword of CTR
+ */
+SYM_FUNC_START_LOCAL(_aeskl_ctr_inc)
+ paddq INC, CTR
+ add $1, TCTR_LOW
+ jnc .Linc_low
+ pslldq $8, INC
+ paddq INC, CTR
+ psrldq $8, INC
+.Linc_low:
+ movaps CTR, IV
+ pshufb BSWAP_MASK, IV
+ ret
+SYM_FUNC_END(_aeskl_ctr_inc)
+
+/*
+ * CTR implementations
+ */
+
+/*
+ * int _aeskl_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, size_t len, u8 *iv)
+ */
+SYM_FUNC_START(_aeskl_ctr_enc)
+ FRAME_BEGIN
+ cmp $16, LEN
+ jb .Lctr_enc_noerr
+ mov 480(HANDLEP), KLEN
+ movdqu (IVP), IV
+ call _aeskl_ctr_inc_init
+ cmp $128, LEN
+ jb .Lctr_enc1
+
+.align 4
+.Lctr_enc8:
+ movaps IV, STATE1
+ call _aeskl_ctr_inc
+ movaps IV, STATE2
+ call _aeskl_ctr_inc
+ movaps IV, STATE3
+ call _aeskl_ctr_inc
+ movaps IV, STATE4
+ call _aeskl_ctr_inc
+ movaps IV, STATE5
+ call _aeskl_ctr_inc
+ movaps IV, STATE6
+ call _aeskl_ctr_inc
+ movaps IV, STATE7
+ call _aeskl_ctr_inc
+ movaps IV, STATE8
+ call _aeskl_ctr_inc
+
+ cmp $16, KLEN
+ je .Lctr_enc8_128
+ aesencwide256kl (%rdi)
+ jz .Lctr_enc_err
+ jmp .Lctr_enc8_end
+.Lctr_enc8_128:
+ aesencwide128kl (%rdi)
+ jz .Lctr_enc_err
+.Lctr_enc8_end:
+
+ movups (INP), IN1
+ pxor IN1, STATE1
+ movups STATE1, (OUTP)
+
+ movups 0x10(INP), IN1
+ pxor IN1, STATE2
+ movups STATE2, 0x10(OUTP)
+
+ movups 0x20(INP), IN1
+ pxor IN1, STATE3
+ movups STATE3, 0x20(OUTP)
+
+ movups 0x30(INP), IN1
+ pxor IN1, STATE4
+ movups STATE4, 0x30(OUTP)
+
+ movups 0x40(INP), IN1
+ pxor IN1, STATE5
+ movups STATE5, 0x40(OUTP)
+
+ movups 0x50(INP), IN1
+ pxor IN1, STATE6
+ movups STATE6, 0x50(OUTP)
+
+ movups 0x60(INP), IN1
+ pxor IN1, STATE7
+ movups STATE7, 0x60(OUTP)
+
+ movups 0x70(INP), IN1
+ pxor IN1, STATE8
+ movups STATE8, 0x70(OUTP)
+
+ sub $128, LEN
+ add $128, INP
+ add $128, OUTP
+ cmp $128, LEN
+ jge .Lctr_enc8
+ cmp $16, LEN
+ jb .Lctr_enc_end
+
+.align 4
+.Lctr_enc1:
+ movaps IV, STATE1
+ call _aeskl_ctr_inc
+
+ cmp $16, KLEN
+ je .Lctr_enc1_128
+ aesenc256kl (HANDLEP), STATE1
+ jz .Lctr_enc_err
+ jmp .Lctr_enc1_end
+.Lctr_enc1_128:
+ aesenc128kl (HANDLEP), STATE1
+ jz .Lctr_enc_err
+
+.Lctr_enc1_end:
+ movups (INP), IN1
+ pxor IN1, STATE1
+ movups STATE1, (OUTP)
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lctr_enc1
+
+.Lctr_enc_end:
+ movdqu IV, (IVP)
+.Lctr_enc_noerr:
+ xor AREG, AREG
+ jmp .Lctr_enc_ret
+.Lctr_enc_err:
+ mov $1, AREG
+.Lctr_enc_ret:
+ FRAME_END
+ ret
+SYM_FUNC_END(_aeskl_ctr_enc)
+
+/*
+ * XTS implementation
+ */
+
+/*
+ * _aeskl_gf128mul_x_ble: internal ABI
+ * Multiply in GF(2^128) for XTS IVs
+ * input:
+ * IV: current IV
+ * GF128MUL_MASK == mask with 0x87 and 0x01
+ * output:
+ * IV: next IV
+ * changed:
+ * CTR: == temporary value
+ */
+#define _aeskl_gf128mul_x_ble() \
+ pshufd $0x13, IV, KEY; \
+ paddq IV, IV; \
+ psrad $31, KEY; \
+ pand GF128MUL_MASK, KEY; \
+ pxor KEY, IV;
+
+/*
+ * int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src,
+ * unsigned int len, le128 *iv)
+ */
+SYM_FUNC_START(_aeskl_xts_encrypt)
+ FRAME_BEGIN
+ movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
+ movups (IVP), IV
+
+ mov 480(HANDLEP), KLEN
+
+.Lxts_enc8:
+ sub $128, LEN
+ jl .Lxts_enc1_pre
+
+ movdqa IV, STATE1
+ movdqu (INP), INC
+ pxor INC, STATE1
+ movdqu IV, (OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE2
+ movdqu 0x10(INP), INC
+ pxor INC, STATE2
+ movdqu IV, 0x10(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE3
+ movdqu 0x20(INP), INC
+ pxor INC, STATE3
+ movdqu IV, 0x20(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE4
+ movdqu 0x30(INP), INC
+ pxor INC, STATE4
+ movdqu IV, 0x30(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE5
+ movdqu 0x40(INP), INC
+ pxor INC, STATE5
+ movdqu IV, 0x40(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE6
+ movdqu 0x50(INP), INC
+ pxor INC, STATE6
+ movdqu IV, 0x50(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE7
+ movdqu 0x60(INP), INC
+ pxor INC, STATE7
+ movdqu IV, 0x60(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE8
+ movdqu 0x70(INP), INC
+ pxor INC, STATE8
+ movdqu IV, 0x70(OUTP)
+
+ cmp $16, KLEN
+ je .Lxts_enc8_128
+ aesencwide256kl (%rdi)
+ jz .Lxts_enc_ret_err
+ jmp .Lxts_enc8_end
+.Lxts_enc8_128:
+ aesencwide128kl (%rdi)
+ jz .Lxts_enc_ret_err
+
+.Lxts_enc8_end:
+ movdqu 0x00(OUTP), INC
+ pxor INC, STATE1
+ movdqu STATE1, 0x00(OUTP)
+
+ movdqu 0x10(OUTP), INC
+ pxor INC, STATE2
+ movdqu STATE2, 0x10(OUTP)
+
+ movdqu 0x20(OUTP), INC
+ pxor INC, STATE3
+ movdqu STATE3, 0x20(OUTP)
+
+ movdqu 0x30(OUTP), INC
+ pxor INC, STATE4
+ movdqu STATE4, 0x30(OUTP)
+
+ movdqu 0x40(OUTP), INC
+ pxor INC, STATE5
+ movdqu STATE5, 0x40(OUTP)
+
+ movdqu 0x50(OUTP), INC
+ pxor INC, STATE6
+ movdqu STATE6, 0x50(OUTP)
+
+ movdqu 0x60(OUTP), INC
+ pxor INC, STATE7
+ movdqu STATE7, 0x60(OUTP)
+
+ movdqu 0x70(OUTP), INC
+ pxor INC, STATE8
+ movdqu STATE8, 0x70(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+
+ add $128, INP
+ add $128, OUTP
+ test LEN, LEN
+ jnz .Lxts_enc8
+
+.Lxts_enc_ret_iv:
+ movups IV, (IVP)
+.Lxts_enc_ret_noerr:
+ xor AREG, AREG
+ jmp .Lxts_enc_ret
+.Lxts_enc_ret_err:
+ mov $1, AREG
+.Lxts_enc_ret:
+ FRAME_END
+ ret
+
+.Lxts_enc1_pre:
+ add $128, LEN
+ jz .Lxts_enc_ret_iv
+ sub $16, LEN
+ jl .Lxts_enc_cts4
+
+.Lxts_enc1:
+ movdqu (INP), STATE1
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_enc1_128
+ aesenc256kl (HANDLEP), STATE1
+ jz .Lxts_enc_ret_err
+ jmp .Lxts_enc1_end
+.Lxts_enc1_128:
+ aesenc128kl (HANDLEP), STATE1
+ jz .Lxts_enc_ret_err
+
+.Lxts_enc1_end:
+ pxor IV, STATE1
+ _aeskl_gf128mul_x_ble()
+
+ test LEN, LEN
+ jz .Lxts_enc1_out
+
+ add $16, INP
+ sub $16, LEN
+ jl .Lxts_enc_cts1
+
+ movdqu STATE1, (OUTP)
+ add $16, OUTP
+ jmp .Lxts_enc1
+
+.Lxts_enc1_out:
+ movdqu STATE1, (OUTP)
+ jmp .Lxts_enc_ret_iv
+
+.Lxts_enc_cts4:
+ movdqu STATE8, STATE1
+ sub $16, OUTP
+
+.Lxts_enc_cts1:
+ lea .Lcts_permute_table(%rip), T1
+ add LEN, INP /* rewind input pointer */
+ add $16, LEN /* # bytes in final block */
+ movups (INP), IN1
+
+ mov T1, IVP
+ add $32, IVP
+ add LEN, T1
+ sub LEN, IVP
+ add OUTP, LEN
+
+ movups (T1), STATE2
+ movaps STATE1, STATE3
+ pshufb STATE2, STATE1
+ movups STATE1, (LEN)
+
+ movups (IVP), STATE1
+ pshufb STATE1, IN1
+ pblendvb STATE3, IN1
+ movaps IN1, STATE1
+
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_enc1_cts_128
+ aesenc256kl (HANDLEP), STATE1
+ jz .Lxts_enc_ret_err
+ jmp .Lxts_enc1_cts_end
+.Lxts_enc1_cts_128:
+ aesenc128kl (HANDLEP), STATE1
+ jz .Lxts_enc_ret_err
+
+.Lxts_enc1_cts_end:
+ pxor IV, STATE1
+ movups STATE1, (OUTP)
+ jmp .Lxts_enc_ret_noerr
+SYM_FUNC_END(_aeskl_xts_encrypt)
+
+/*
+ * int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, const u8 *src,
+ * unsigned int len, le128 *iv)
+ */
+SYM_FUNC_START(_aeskl_xts_decrypt)
+ FRAME_BEGIN
+ movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
+ movups (IVP), IV
+
+ mov 480(HANDLEP), KLEN
+
+ test $15, LEN
+ jz .Lxts_dec8
+ sub $16, LEN
+
+.Lxts_dec8:
+ sub $128, LEN
+ jl .Lxts_dec1_pre
+
+ movdqa IV, STATE1
+ movdqu (INP), INC
+ pxor INC, STATE1
+ movdqu IV, (OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE2
+ movdqu 0x10(INP), INC
+ pxor INC, STATE2
+ movdqu IV, 0x10(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE3
+ movdqu 0x20(INP), INC
+ pxor INC, STATE3
+ movdqu IV, 0x20(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE4
+ movdqu 0x30(INP), INC
+ pxor INC, STATE4
+ movdqu IV, 0x30(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE5
+ movdqu 0x40(INP), INC
+ pxor INC, STATE5
+ movdqu IV, 0x40(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE6
+ movdqu 0x50(INP), INC
+ pxor INC, STATE6
+ movdqu IV, 0x50(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE7
+ movdqu 0x60(INP), INC
+ pxor INC, STATE7
+ movdqu IV, 0x60(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE8
+ movdqu 0x70(INP), INC
+ pxor INC, STATE8
+ movdqu IV, 0x70(OUTP)
+
+ cmp $16, KLEN
+ je .Lxts_dec8_128
+ aesdecwide256kl (%rdi)
+ jz .Lxts_dec_ret_err
+ jmp .Lxts_dec8_end
+.Lxts_dec8_128:
+ aesdecwide128kl (%rdi)
+ jz .Lxts_dec_ret_err
+
+.Lxts_dec8_end:
+ movdqu 0x00(OUTP), INC
+ pxor INC, STATE1
+ movdqu STATE1, 0x00(OUTP)
+
+ movdqu 0x10(OUTP), INC
+ pxor INC, STATE2
+ movdqu STATE2, 0x10(OUTP)
+
+ movdqu 0x20(OUTP), INC
+ pxor INC, STATE3
+ movdqu STATE3, 0x20(OUTP)
+
+ movdqu 0x30(OUTP), INC
+ pxor INC, STATE4
+ movdqu STATE4, 0x30(OUTP)
+
+ movdqu 0x40(OUTP), INC
+ pxor INC, STATE5
+ movdqu STATE5, 0x40(OUTP)
+
+ movdqu 0x50(OUTP), INC
+ pxor INC, STATE6
+ movdqu STATE6, 0x50(OUTP)
+
+ movdqu 0x60(OUTP), INC
+ pxor INC, STATE7
+ movdqu STATE7, 0x60(OUTP)
+
+ movdqu 0x70(OUTP), INC
+ pxor INC, STATE8
+ movdqu STATE8, 0x70(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+
+ add $128, INP
+ add $128, OUTP
+ test LEN, LEN
+ jnz .Lxts_dec8
+
+.Lxts_dec_ret_iv:
+ movups IV, (IVP)
+.Lxts_dec_ret_noerr:
+ xor AREG, AREG
+ jmp .Lxts_dec_ret
+.Lxts_dec_ret_err:
+ mov $1, AREG
+.Lxts_dec_ret:
+ FRAME_END
+ ret
+
+.Lxts_dec1_pre:
+ add $128, LEN
+ jz .Lxts_dec_ret_iv
+
+.Lxts_dec1:
+ movdqu (INP), STATE1
+
+ add $16, INP
+ sub $16, LEN
+ jl .Lxts_dec_cts1
+
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_dec1_128
+ aesdec256kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+ jmp .Lxts_dec1_end
+.Lxts_dec1_128:
+ aesdec128kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+
+.Lxts_dec1_end:
+ pxor IV, STATE1
+ _aeskl_gf128mul_x_ble()
+
+ test LEN, LEN
+ jz .Lxts_dec1_out
+
+ movdqu STATE1, (OUTP)
+ add $16, OUTP
+ jmp .Lxts_dec1
+
+.Lxts_dec1_out:
+ movdqu STATE1, (OUTP)
+ jmp .Lxts_dec_ret_iv
+
+.Lxts_dec_cts1:
+ movdqa IV, STATE5
+ _aeskl_gf128mul_x_ble()
+
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_dec1_cts_pre_128
+ aesdec256kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+ jmp .Lxts_dec1_cts_pre_end
+.Lxts_dec1_cts_pre_128:
+ aesdec128kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+
+.Lxts_dec1_cts_pre_end:
+ pxor IV, STATE1
+
+ lea .Lcts_permute_table(%rip), T1
+ add LEN, INP /* rewind input pointer */
+ add $16, LEN /* # bytes in final block */
+ movups (INP), IN1
+
+ mov T1, IVP
+ add $32, IVP
+ add LEN, T1
+ sub LEN, IVP
+ add OUTP, LEN
+
+ movups (T1), STATE2
+ movaps STATE1, STATE3
+ pshufb STATE2, STATE1
+ movups STATE1, (LEN)
+
+ movups (IVP), STATE1
+ pshufb STATE1, IN1
+ pblendvb STATE3, IN1
+ movaps IN1, STATE1
+
+ pxor STATE5, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_dec1_cts_128
+ aesdec256kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+ jmp .Lxts_dec1_cts_end
+.Lxts_dec1_cts_128:
+ aesdec128kl (HANDLEP), STATE1
+ jz .Lxts_dec_ret_err
+
+.Lxts_dec1_cts_end:
+ pxor STATE5, STATE1
+
+ movups STATE1, (OUTP)
+ jmp .Lxts_dec_ret_noerr
+
+SYM_FUNC_END(_aeskl_xts_decrypt)
+
+#endif
diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c
new file mode 100644
index 000000000000..e0a052e89fa0
--- /dev/null
+++ b/arch/x86/crypto/aeskl-intel_glue.c
@@ -0,0 +1,390 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Support for AES Key Locker instructions. This file contains glue
+ * code and the real AES implementation is in aeskl-intel_asm.S.
+ *
+ * Most code is based on AES-NI glue code, aesni-intel_glue.c
+ */
+
+#include <linux/types.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/xts.h>
+#include <asm/cpu_device_id.h>
+#include <asm/fpu/api.h>
+#include <asm/simd.h>
+#include <asm/keylocker.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/internal/simd.h>
+#include "aes-intel_glue.h"
+#include "aesni-intel_glue.h"
+
+asmlinkage int aeskl_setkey(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len);
+
+asmlinkage int _aeskl_enc(const void *ctx, u8 *out, const u8 *in);
+asmlinkage int _aeskl_dec(const void *ctx, u8 *out, const u8 *in);
+
+asmlinkage int _aeskl_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len);
+asmlinkage int _aeskl_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len);
+
+asmlinkage int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+asmlinkage int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+
+#ifdef CONFIG_X86_64
+asmlinkage int _aeskl_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv);
+
+asmlinkage int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+asmlinkage int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+#endif
+
+static int aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key,
+ unsigned int key_len)
+{
+ struct crypto_aes_ctx *ctx = aes_ctx(raw_ctx);
+ int err;
+
+ if (!crypto_simd_usable())
+ return -EBUSY;
+
+ if ((key_len != AES_KEYSIZE_128) && (key_len != AES_KEYSIZE_192) &&
+ (key_len != AES_KEYSIZE_256))
+ return -EINVAL;
+
+ kernel_fpu_begin();
+ /*
+ * For 128-bit and 256-bit keys, encode to a handle that is usable only
+ * in kernel mode. Use AES-NI instructions for 192-bit keys.
+ */
+ if (likely(key_len != AES_KEYSIZE_192))
+ err = aeskl_setkey(ctx, in_key, key_len);
+ else
+ err = aesni_set_key(ctx, in_key, key_len);
+ kernel_fpu_end();
+
+ return err;
+}
+
+static inline u32 keylength(const void *raw_ctx)
+{
+ struct crypto_aes_ctx *ctx = aes_ctx((void *)raw_ctx);
+
+ return ctx->key_length;
+}
+
+static inline int aeskl_enc(const void *ctx, u8 *out, const u8 *in)
+{
+ if (unlikely(keylength(ctx) == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_enc(ctx, out, in))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static inline int aeskl_dec(const void *ctx, u8 *out, const u8 *in)
+{
+ if (unlikely(keylength(ctx) == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_dec(ctx, out, in))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static int aeskl_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_ecb_enc(ctx, out, in, len))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static int aeskl_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_ecb_dec(ctx, out, in, len))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static int aeskl_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_cbc_enc(ctx, out, in, len, iv))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static int aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_cbc_dec(ctx, out, in, len, iv))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+#ifdef CONFIG_X86_64
+
+static int aeskl_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len,
+ u8 *iv)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_ctr_enc(ctx, out, in, len, iv))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static inline int aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_xts_encrypt(ctx, out, in, len, iv))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+static inline int aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ if (unlikely(ctx->key_length == AES_KEYSIZE_192))
+ return -EINVAL;
+
+ if (_aeskl_xts_decrypt(ctx, out, in, len, iv))
+ return -EINVAL;
+ else
+ return 0;
+}
+
+#endif /* CONFIG_X86_64 */
+
+static int aeskl_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int len)
+{
+ struct crypto_tfm *crypto_tfm = crypto_skcipher_tfm(tfm);
+ void *raw_ctx = crypto_skcipher_ctx(tfm);
+
+ return aeskl_setkey_common(crypto_tfm, raw_ctx, key, len);
+}
+
+static int ecb_encrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return ecb_crypt_common(req, aeskl_ecb_enc);
+ else
+ return ecb_crypt_common(req, aesni_ecb_enc);
+}
+
+static int ecb_decrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return ecb_crypt_common(req, aeskl_ecb_dec);
+ else
+ return ecb_crypt_common(req, aesni_ecb_dec);
+}
+
+static int cbc_encrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return cbc_crypt_common(req, aeskl_cbc_enc);
+ else
+ return cbc_crypt_common(req, aesni_cbc_enc);
+}
+
+static int cbc_decrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return cbc_crypt_common(req, aeskl_cbc_dec);
+ else
+ return cbc_crypt_common(req, aesni_cbc_dec);
+}
+
+#ifdef CONFIG_X86_64
+
+static int ctr_crypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return ctr_crypt_common(req, aeskl_ctr_enc, aeskl_enc);
+ else
+ return ctr_crypt_common(req, aesni_ctr_enc, aesni_enc);
+}
+
+static int aeskl_xts_setkey(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ return xts_setkey_common(tfm, key, keylen, aeskl_setkey_common);
+}
+
+static int xts_encrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return xts_crypt_common(req, aeskl_xts_encrypt, aeskl_enc);
+ else
+ return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc);
+}
+
+static int xts_decrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192))
+ return xts_crypt_common(req, aeskl_xts_decrypt, aeskl_enc);
+ else
+ return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc);
+}
+
+#endif /* CONFIG_X86_64 */
+
+static struct skcipher_alg aeskl_skciphers[] = {
+ {
+ .base = {
+ .cra_name = "__ecb(aes)",
+ .cra_driver_name = "__ecb-aes-aeskl",
+ .cra_priority = 401,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = CRYPTO_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .setkey = aeskl_skcipher_setkey,
+ .encrypt = ecb_encrypt,
+ .decrypt = ecb_decrypt,
+ }, {
+ .base = {
+ .cra_name = "__cbc(aes)",
+ .cra_driver_name = "__cbc-aes-aeskl",
+ .cra_priority = 401,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = CRYPTO_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = aeskl_skcipher_setkey,
+ .encrypt = cbc_encrypt,
+ .decrypt = cbc_decrypt,
+#ifdef CONFIG_X86_64
+ }, {
+ .base = {
+ .cra_name = "__ctr(aes)",
+ .cra_driver_name = "__ctr-aes-aeskl",
+ .cra_priority = 401,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = 1,
+ .cra_ctxsize = CRYPTO_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ .setkey = aeskl_skcipher_setkey,
+ .encrypt = ctr_crypt,
+ .decrypt = ctr_crypt,
+ }, {
+ .base = {
+ .cra_name = "__xts(aes)",
+ .cra_driver_name = "__xts-aes-aeskl",
+ .cra_priority = 400,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = XTS_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = 2 * AES_MIN_KEY_SIZE,
+ .max_keysize = 2 * AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .walksize = 2 * AES_BLOCK_SIZE,
+ .setkey = aeskl_xts_setkey,
+ .encrypt = xts_encrypt,
+ .decrypt = xts_decrypt,
+#endif
+ }
+};
+
+static struct simd_skcipher_alg *aeskl_simd_skciphers[ARRAY_SIZE(aeskl_skciphers)];
+
+static const struct x86_cpu_id aes_keylocker_cpuid[] = {
+ X86_MATCH_FEATURE(X86_FEATURE_AES, NULL),
+ X86_MATCH_FEATURE(X86_FEATURE_KEYLOCKER, NULL),
+ {}
+};
+
+static int __init aeskl_init(void)
+{
+ u32 eax, ebx, ecx, edx;
+ int err;
+
+ if (!x86_match_cpu(aes_keylocker_cpuid))
+ return -ENODEV;
+
+ cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx);
+ if (!(ebx & KEYLOCKER_CPUID_EBX_AESKLE) ||
+ !(eax & KEYLOCKER_CPUID_EAX_SUPERVISOR) ||
+ !(ebx & KEYLOCKER_CPUID_EBX_WIDE))
+ return -ENODEV;
+
+ err = simd_register_skciphers_compat(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers),
+ aeskl_simd_skciphers);
+ if (err)
+ return err;
+
+ return 0;
+}
+
+static void __exit aeskl_exit(void)
+{
+ simd_unregister_skciphers(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers),
+ aeskl_simd_skciphers);
+}
+
+late_initcall(aeskl_init);
+module_exit(aeskl_exit);
+
+MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm, AES Key Locker implementation");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_CRYPTO("aes");
diff --git a/crypto/Kconfig b/crypto/Kconfig
index ca3b02dcbbfa..5f3d4052fa3c 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1113,6 +1113,29 @@ config CRYPTO_AES_NI_INTEL
ECB, CBC, LRW, XTS. The 64 bit version has additional
acceleration for CTR.
+config CRYPTO_AES_KL
+ tristate "AES cipher algorithms (AES-KL)"
+ depends on X86_KEYLOCKER
+ depends on LD_VERSION >= 23600
+ select CRYPTO_AES_NI_INTEL
+ help
+ Key Locker provides AES SIMD instructions (AES-KL) to protect AES
+ keys. They are analogous to AES-NI. AES-KL supports to encode AES
+ keys and use an encoded form to transform data instead of
+ referencing an AES key all the way.
+
+ The CPU-internal key is used to wrap AES keys. AES-KL relies on
+ the x86 core code that manages the internal key.
+
+ AES-KL supports 128-/256-bit keys. This module uses the AES-NI
+ methods for 192-bit keys. It includes such block cipher mode
+ support as ECB, CBC, CTR, and XTS.
+
+ binutils version 2.36 or later is needed to assemble AES-KL.
+
+ See the CRYPTO_AES_NI_INTEL description for more about the AES
+ algorithm.
+
config CRYPTO_AES_SPARC64
tristate "AES cipher algorithms (SPARC64)"
depends on SPARC64
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [RFC PATCH v2 11/11] x86/cpu: Support the hardware randomization option for Key Locker internal key
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (9 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Chang S. Bae
@ 2021-05-14 20:15 ` Chang S. Bae
2021-05-15 18:01 ` [RFC PATCH v2 00/11] x86: Support Intel Key Locker Andy Lutomirski
11 siblings, 0 replies; 28+ messages in thread
From: Chang S. Bae @ 2021-05-14 20:15 UTC (permalink / raw)
To: tglx, mingo, bp, luto, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto,
linux-kernel, chang.seok.bae, Mark Brown, linux-doc
Hardware can load the internal key with randomization. The random.trust_cpu
parameter determines the use of the CPU's random number generator. Take it
to use the CPU's internal key randomization.
The feature's backup mechanism is required to distribute an internal key.
This is the only way to copy it to other CPUs if using hardware
randomization.
This option is disabled when hardware does not support the key backup.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Cc: Mark Brown <broonie@kernel.org>
Cc: x86@kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
arch/x86/kernel/keylocker.c | 37 ++++++++++++++++++++++++++++++++-----
drivers/char/random.c | 6 ++++++
include/linux/random.h | 2 ++
3 files changed, 40 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
index 0f60350944fa..5a784492195b 100644
--- a/arch/x86/kernel/keylocker.c
+++ b/arch/x86/kernel/keylocker.c
@@ -7,6 +7,7 @@
#include <linux/random.h>
#include <linux/acpi.h>
#include <linux/delay.h>
+#include <linux/random.h>
#include <asm/cacheflush.h>
#include <asm/fpu/api.h>
@@ -15,27 +16,34 @@
#include <asm/tlbflush.h>
static bool keybackup_available;
+static bool keyhwrand_available;
/* Internal (Wrapping) Key size fits in three 128-bit registers. */
#define KEYSIZE_128BIT 3
static struct _keydata {
bool valid;
+ bool hwrand;
struct reg_128_bit value[KEYSIZE_128BIT];
} keydata;
/**
* make_keylocker_data() - Generate the internal key.
+ * @use_hwrand: True if use hardware randomization; otherwise, false.
*
* Return: Nothing
*/
-static void make_keylocker_data(void)
+static void make_keylocker_data(bool use_hwrand)
{
int i;
for (i = 0; i < KEYSIZE_128BIT; i++)
get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit));
+ keydata.hwrand = (use_hwrand && keyhwrand_available && keybackup_available);
+ if (use_hwrand && !keydata.hwrand)
+ pr_warn("x86/keylocker: hardware random key is not fully supported\n");
+
keydata.valid = true;
}
@@ -59,6 +67,8 @@ void flush_keylocker_data(void)
}
#define KEYSRC_SWRAND 0
+#define KEYSRC_HWRAND BIT(1)
+#define KEYSRC_HWRAND_RETRY 10
/**
* load_keylocker() - Load the internal key.
@@ -68,8 +78,16 @@ void flush_keylocker_data(void)
static int load_keylocker(void)
{
struct reg_128_bit zeros = { 0 };
- u32 keysrc = KEYSRC_SWRAND;
- int err;
+ int retry, err;
+ u32 keysrc;
+
+ if (keydata.hwrand) {
+ keysrc = KEYSRC_HWRAND;
+ retry = KEYSRC_HWRAND_RETRY;
+ } else {
+ keysrc = KEYSRC_SWRAND;
+ retry = 0;
+ }
kernel_fpu_begin();
@@ -78,13 +96,19 @@ static int load_keylocker(void)
"m"(keydata.value[1]),
"m"(keydata.value[2]));
- err = loadiwkey(keysrc);
+ do {
+ err = loadiwkey(keysrc);
+ retry--;
+ } while (err && retry >= 0);
asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;"
:: "m"(zeros));
kernel_fpu_end();
+ if (keydata.hwrand)
+ flush_keylocker_data();
+
return err;
}
@@ -138,6 +162,7 @@ void setup_keylocker(struct cpuinfo_x86 *c)
cr4_set_bits(X86_CR4_KEYLOCKER);
if (c == &boot_cpu_data) {
+ bool use_hwrand = check_random_trust_cpu();
u32 eax, ebx, ecx, edx;
cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx);
@@ -156,7 +181,9 @@ void setup_keylocker(struct cpuinfo_x86 *c)
goto disable;
}
- make_keylocker_data();
+ keyhwrand_available = (ecx & KEYLOCKER_CPUID_ECX_RAND);
+
+ make_keylocker_data(use_hwrand);
err = load_keylocker();
if (err) {
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 605969ed0f96..a3ece5968497 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -771,6 +771,12 @@ static int __init parse_trust_cpu(char *arg)
}
early_param("random.trust_cpu", parse_trust_cpu);
+bool check_random_trust_cpu(void)
+{
+ return trust_cpu;
+}
+EXPORT_SYMBOL(check_random_trust_cpu);
+
static bool crng_init_try_arch(struct crng_state *crng)
{
int i;
diff --git a/include/linux/random.h b/include/linux/random.h
index f45b8be3e3c4..f08f44988b13 100644
--- a/include/linux/random.h
+++ b/include/linux/random.h
@@ -158,4 +158,6 @@ static inline bool __init arch_get_random_long_early(unsigned long *v)
}
#endif
+extern bool check_random_trust_cpu(void);
+
#endif /* _LINUX_RANDOM_H */
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
` (10 preceding siblings ...)
2021-05-14 20:15 ` [RFC PATCH v2 11/11] x86/cpu: Support the hardware randomization option for Key Locker internal key Chang S. Bae
@ 2021-05-15 18:01 ` Andy Lutomirski
2021-05-17 18:21 ` Bae, Chang Seok
11 siblings, 1 reply; 28+ messages in thread
From: Andy Lutomirski @ 2021-05-15 18:01 UTC (permalink / raw)
To: Chang S. Bae, tglx, mingo, bp, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto, linux-kernel
On 5/14/21 1:14 PM, Chang S. Bae wrote:
> Key Locker [1][2] is a new security feature available in new Intel CPUs to
> protect data encryption keys for the Advanced Encryption Standard
> algorithm. The protection limits the amount of time an AES key is exposed
> in memory by sealing a key and referencing it with new AES instructions.
>
> The new AES instruction set is a successor of Intel's AES-NI (AES New
> Instruction). Users may switch to the Key Locker version from crypto
> libraries. This series includes a new AES implementation for the Crypto
> API, which was validated through the crypto unit tests. The performance in
> the test cases was measured and found comparable to the AES-NI version.
>
> Key Locker introduces a (CPU-)internal key to encode AES keys. The kernel
> needs to load it and ensure it unchanged as long as CPUs are operational.
I have high-level questions:
What is the expected use case? My personal hypothesis, based on various
public Intel slides, is that the actual intended use case was internal
to the ME, and that KL was ported to end-user CPUs more or less
verbatim. I certainly understand how KL is valuable in a context where
a verified boot process installs some KL keys that are not subsequently
accessible outside the KL ISA, but Linux does not really work like this.
I'm wondering what people will use it for.
On a related note, does Intel plan to extend KL with ways to securely
load keys? (E.g. the ability to, in effect, LOADIWKEY from inside an
enclave? Key wrapping/unwrapping operations?) In other words, is
should we look at KL the way we look at MKTME, i.e. the foundation of
something neat but not necessarily very useful as is, or should we
expect that KL is in its more or less final form?
What is the expected interaction between a KL-using VM guest and the
host VMM? Will there be performance impacts (to context switching, for
example) if a guest enables KL, even if the guest does not subsequently
do anything with it? Should Linux actually enable KL if it detects that
it's a VM guest? Should Linux have use a specific keying method as a guest?
--Andy
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-15 18:01 ` [RFC PATCH v2 00/11] x86: Support Intel Key Locker Andy Lutomirski
@ 2021-05-17 18:21 ` Bae, Chang Seok
2021-05-17 18:45 ` Dan Williams
` (2 more replies)
0 siblings, 3 replies; 28+ messages in thread
From: Bae, Chang Seok @ 2021-05-17 18:21 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, X86 ML,
Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar, Ravi V,
Linux Crypto Mailing List, linux-kernel
On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
> On 5/14/21 1:14 PM, Chang S. Bae wrote:
>> Key Locker [1][2] is a new security feature available in new Intel CPUs to
>> protect data encryption keys for the Advanced Encryption Standard
>> algorithm. The protection limits the amount of time an AES key is exposed
>> in memory by sealing a key and referencing it with new AES instructions.
>>
>> The new AES instruction set is a successor of Intel's AES-NI (AES New
>> Instruction). Users may switch to the Key Locker version from crypto
>> libraries. This series includes a new AES implementation for the Crypto
>> API, which was validated through the crypto unit tests. The performance in
>> the test cases was measured and found comparable to the AES-NI version.
>>
>> Key Locker introduces a (CPU-)internal key to encode AES keys. The kernel
>> needs to load it and ensure it unchanged as long as CPUs are operational.
>
> I have high-level questions:
>
> What is the expected use case?
The wrapping key here is only used for new AES instructions.
I’m aware of their potential use cases for encrypting file system or disks.
> My personal hypothesis, based on various
> public Intel slides, is that the actual intended use case was internal
> to the ME, and that KL was ported to end-user CPUs more or less
> verbatim.
No, this is a separate one. The feature has nothing to do with the firmware
except that in some situations it merely helps to back up the key in its
state.
> I certainly understand how KL is valuable in a context where
> a verified boot process installs some KL keys that are not subsequently
> accessible outside the KL ISA, but Linux does not really work like this.
Do you mind elaborating on the concern? I try to understand any issue with
PATCH3 [1], specifically.
> I'm wondering what people will use it for.
Mentioned above.
> On a related note, does Intel plan to extend KL with ways to securely
> load keys? (E.g. the ability to, in effect, LOADIWKEY from inside an
> enclave? Key wrapping/unwrapping operations?) In other words, is
> should we look at KL the way we look at MKTME, i.e. the foundation of
> something neat but not necessarily very useful as is, or should we
> expect that KL is in its more or less final form?
All I have is pretty much in the spec. So, I think the latter is the case.
I don’t see anything about that LOADIWKEY inside an enclave in the spec. (A
relevant section is A.6.1 Key Locker Usage with TEE.)
> What is the expected interaction between a KL-using VM guest and the
> host VMM? Will there be performance impacts (to context switching, for
> example) if a guest enables KL, even if the guest does not subsequently
> do anything with it? Should Linux actually enable KL if it detects that
> it's a VM guest? Should Linux have use a specific keying method as a guest?
First of all, there is an RFC series for KVM [2].
Each CPU has one internal key state so it needs to reload it between guest and
host if both are enabled. The proposed approach enables it exclusively; expose
it to guests only when disabled in a host. Then, I guess a guest may enable it.
Thanks,
Chang
[1] https://lore.kernel.org/lkml/20210514201508.27967-4-chang.seok.bae@intel.com/
[2] https://lore.kernel.org/kvm/1611565580-47718-1-git-send-email-robert.hu@linux.intel.com/
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-17 18:21 ` Bae, Chang Seok
@ 2021-05-17 18:45 ` Dan Williams
2021-05-17 22:20 ` Bae, Chang Seok
2021-05-17 20:15 ` Sean Christopherson
2021-05-18 17:10 ` Andy Lutomirski
2 siblings, 1 reply; 28+ messages in thread
From: Dan Williams @ 2021-05-17 18:45 UTC (permalink / raw)
To: Bae, Chang Seok
Cc: Andy Lutomirski, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Hansen, Dave, Shankar, Ravi V,
Linux Crypto Mailing List, linux-kernel
On Mon, May 17, 2021 at 11:21 AM Bae, Chang Seok
<chang.seok.bae@intel.com> wrote:
>
> On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
> > On 5/14/21 1:14 PM, Chang S. Bae wrote:
> >> Key Locker [1][2] is a new security feature available in new Intel CPUs to
> >> protect data encryption keys for the Advanced Encryption Standard
> >> algorithm. The protection limits the amount of time an AES key is exposed
> >> in memory by sealing a key and referencing it with new AES instructions.
> >>
> >> The new AES instruction set is a successor of Intel's AES-NI (AES New
> >> Instruction). Users may switch to the Key Locker version from crypto
> >> libraries. This series includes a new AES implementation for the Crypto
> >> API, which was validated through the crypto unit tests. The performance in
> >> the test cases was measured and found comparable to the AES-NI version.
> >>
> >> Key Locker introduces a (CPU-)internal key to encode AES keys. The kernel
> >> needs to load it and ensure it unchanged as long as CPUs are operational.
> >
> > I have high-level questions:
> >
> > What is the expected use case?
>
> The wrapping key here is only used for new AES instructions.
>
> I’m aware of their potential use cases for encrypting file system or disks.
>
> > My personal hypothesis, based on various
> > public Intel slides, is that the actual intended use case was internal
> > to the ME, and that KL was ported to end-user CPUs more or less
> > verbatim.
>
> No, this is a separate one. The feature has nothing to do with the firmware
> except that in some situations it merely helps to back up the key in its
> state.
>
> > I certainly understand how KL is valuable in a context where
> > a verified boot process installs some KL keys that are not subsequently
> > accessible outside the KL ISA, but Linux does not really work like this.
>
> Do you mind elaborating on the concern? I try to understand any issue with
> PATCH3 [1], specifically.
If I understand Andy's concern it is the observation that the weakest
link in this facility is the initial key load. Yes, KL reduces
exposure after that event, but the key loading process is still
vulnerable. This question is similar to the concern between the Linux
"encrypted-keys" and "trusted-keys" interface. The trusted-keys
interface still has an attack window where the key is unwrapped in
kernel space to decrypt the sub-keys, but that exposure need not cross
the user-kernel boundary and can be time-limited to a given PCR state.
The encrypted-keys interface maintains the private-key material
outside the kernel where it has increased exposure. KL is effectively
"encrypted-keys" and Andy is questioning whether this makes KL similar
to the MKTME vs SGX / TDX situation.
>
> > I'm wondering what people will use it for.
>
> Mentioned above.
I don't think this answers Andy's question. There is a distinction
between what it can be used for and what people will deploy with it in
practice given the "encrypted-keys"-like exposure. Clarify the end
user benefit that motivates the kernel to carry this support.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-17 18:21 ` Bae, Chang Seok
2021-05-17 18:45 ` Dan Williams
@ 2021-05-17 20:15 ` Sean Christopherson
2021-05-18 17:10 ` Andy Lutomirski
2 siblings, 0 replies; 28+ messages in thread
From: Sean Christopherson @ 2021-05-17 20:15 UTC (permalink / raw)
To: Bae, Chang Seok
Cc: Andy Lutomirski, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On Mon, May 17, 2021, Bae, Chang Seok wrote:
> On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
> > What is the expected interaction between a KL-using VM guest and the
> > host VMM?
Messy. :-)
> > Will there be performance impacts (to context switching, for
> > example) if a guest enables KL, even if the guest does not subsequently
> > do anything with it?
Short answer, yes. But the proposed solution is to disallow KL in KVM guests if
KL is in use by the host. The problem is that, by design, the host can't restore
its key via LOADIWKEY because the whole point is to throw away the real key. To
restore its value, the host would need to use the platform backup/restore
mechanism, which is comically slow (tens of thousands of cycles).
If KL virtualization is mutually exclusive with use in the host, then IIRC the
context switching penalty is only paid by vCPUs that have executed LOADIWKEY, as
other tasks can safely run with a stale/bogus key.
> > Should Linux actually enable KL if it detects that it's a VM guest?
Probably not by default. It shouldn't even be considered unless the VMM is
trusted, as a malicious VMM can completely subvert KL. Even if the host is
trusted, it's not clear that the tradeoffs are a net win.
Practically speaking, VMMs have to either (a) save the real key in host memory
or (b) provide a single VM exclusive access to the underlying hardware.
For (a), that rules out using an ephemeral, random key, as using a truly random
key prevents the VMM from saving/restoring the real key. That means the guest
has to generate its own key, and the host has to also store the key in memory.
There are also potential performance and live migration implications. The only
benefit to using KL in the guest is that the real key is not stored in _guest_
accessible memory. So it probably reduces the attack surface, but on the other
hand the VMM may store the guest's master key in a known location, which might
make cross-VM attacks easier in some ways.
(b) is a fairly unlikely scenario, and certainly can't be assumed to be the
default scenario for a guest.
> > Should Linux have use a specific keying method as a guest?
Could you rephrase this question? I didn't follow.
> First of all, there is an RFC series for KVM [2].
That series also fails to address the use case question.
[*] https://lore.kernel.org/kvm/YGs07I%2FmKhDy3pxD@google.com/
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions
2021-05-14 20:15 ` [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Chang S. Bae
@ 2021-05-17 21:34 ` Eric Biggers
2021-05-17 22:20 ` Bae, Chang Seok
2021-05-18 16:57 ` Andy Lutomirski
1 sibling, 1 reply; 28+ messages in thread
From: Eric Biggers @ 2021-05-17 21:34 UTC (permalink / raw)
To: Chang S. Bae
Cc: tglx, mingo, bp, luto, x86, herbert, dan.j.williams, dave.hansen,
ravi.v.shankar, linux-crypto, linux-kernel
On Fri, May 14, 2021 at 01:15:07PM -0700, Chang S. Bae wrote:
>
> Included are methods for ECB, CBC, CTR, and XTS modes. They are not
> compatible with other implementations as referencing an encrypted form
> only.
Your code uses the standard algorithm names like cbc(aes), which implies that it
is compatible with the standard cbc(aes). So which is it -- compatible or not
compatible -- and if it isn't compatible, what is the expected use case?
- Eric
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-17 18:45 ` Dan Williams
@ 2021-05-17 22:20 ` Bae, Chang Seok
0 siblings, 0 replies; 28+ messages in thread
From: Bae, Chang Seok @ 2021-05-17 22:20 UTC (permalink / raw)
To: Williams, Dan J
Cc: Andy Lutomirski, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Hansen, Dave, Shankar, Ravi V,
Linux Crypto Mailing List, linux-kernel
On May 17, 2021, at 11:45, Dan Williams <dan.j.williams@intel.com> wrote:
> On Mon, May 17, 2021 at 11:21 AM Bae, Chang Seok
> <chang.seok.bae@intel.com> wrote:
>>
>> On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
>>>
>>>
>>> I certainly understand how KL is valuable in a context where
>>> a verified boot process installs some KL keys that are not subsequently
>>> accessible outside the KL ISA, but Linux does not really work like this.
>>
>> Do you mind elaborating on the concern? I try to understand any issue with
>> PATCH3 [1], specifically.
>
> If I understand Andy's concern it is the observation that the weakest
> link in this facility is the initial key load. Yes, KL reduces
> exposure after that event, but the key loading process is still
> vulnerable. This question is similar to the concern between the Linux
> "encrypted-keys" and "trusted-keys" interface. The trusted-keys
> interface still has an attack window where the key is unwrapped in
> kernel space to decrypt the sub-keys, but that exposure need not cross
> the user-kernel boundary and can be time-limited to a given PCR state.
> The encrypted-keys interface maintains the private-key material
> outside the kernel where it has increased exposure. KL is effectively
> "encrypted-keys" and Andy is questioning whether this makes KL similar
> to the MKTME vs SGX / TDX situation.
I don’t fully grasp the MKTME vs SGX/TDX background, but LOADIWKEY provides a
hardware randomization option for the initial load. Then, the internal key is
unknown. Nonetheless, if one does not trust this randomization and decides
not to use it, then perhaps unavoidable is the key in memory sometime during
boot-time.
I think Dan just gave an example here, but FWIW, these “encrypted-keys” and
“trusted-keys” are for the kernel keyring service. I wish to clarify the
keyring service itself is not intended usage here. Instead, this series is
intended to focus on the kernel Crypto API, as this technology protects AES
keys during data transformation time.
>>> I'm wondering what people will use it for.
>>
>> Mentioned above.
>
> I don't think this answers Andy's question. There is a distinction
> between what it can be used for and what people will deploy with it in
> practice given the "encrypted-keys"-like exposure. Clarify the end
> user benefit that motivates the kernel to carry this support.
The end-user of this series will benefit from key protection at data
transformation time and also be provided with matched performance as AES-NI
does.
Thanks,
Chang
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions
2021-05-17 21:34 ` Eric Biggers
@ 2021-05-17 22:20 ` Bae, Chang Seok
2021-05-17 23:33 ` Eric Biggers
0 siblings, 1 reply; 28+ messages in thread
From: Bae, Chang Seok @ 2021-05-17 22:20 UTC (permalink / raw)
To: Eric Biggers
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Andy Lutomirski,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On May 17, 2021, at 14:34, Eric Biggers <ebiggers@kernel.org> wrote:
> On Fri, May 14, 2021 at 01:15:07PM -0700, Chang S. Bae wrote:
>> Included are methods for ECB, CBC, CTR, and XTS modes. They are not
>> compatible with other implementations as referencing an encrypted form
>> only.
>
> Your code uses the standard algorithm names like cbc(aes), which implies that it
> is compatible with the standard cbc(aes). So which is it -- compatible or not
> compatible -- and if it isn't compatible, what is the expected use case?
Yes, it provides AES-CBC functionality. Well, it was intended to avoid mixed
use of functions -- setkey(), decrypt(), and encrypt() -- from others.
Perhaps, rewrite this as:
Each method should not be used along with other implementations'. E.g., KL’s
setkey() output can’t be used to the input to the encrypt() method of AES-NI or
generic implementation.
Thanks,
Chang
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions
2021-05-17 22:20 ` Bae, Chang Seok
@ 2021-05-17 23:33 ` Eric Biggers
0 siblings, 0 replies; 28+ messages in thread
From: Eric Biggers @ 2021-05-17 23:33 UTC (permalink / raw)
To: Bae, Chang Seok
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Andy Lutomirski,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On Mon, May 17, 2021 at 10:20:44PM +0000, Bae, Chang Seok wrote:
> On May 17, 2021, at 14:34, Eric Biggers <ebiggers@kernel.org> wrote:
> > On Fri, May 14, 2021 at 01:15:07PM -0700, Chang S. Bae wrote:
> >> Included are methods for ECB, CBC, CTR, and XTS modes. They are not
> >> compatible with other implementations as referencing an encrypted form
> >> only.
> >
> > Your code uses the standard algorithm names like cbc(aes), which implies that it
> > is compatible with the standard cbc(aes). So which is it -- compatible or not
> > compatible -- and if it isn't compatible, what is the expected use case?
>
> Yes, it provides AES-CBC functionality. Well, it was intended to avoid mixed
> use of functions -- setkey(), decrypt(), and encrypt() -- from others.
> Perhaps, rewrite this as:
>
> Each method should not be used along with other implementations'. E.g., KL’s
> setkey() output can’t be used to the input to the encrypt() method of AES-NI or
> generic implementation.
>
Sure. But that is just the implementation, so not really as interesting as what
the user sees. I think you need to do a better job explaining what this looks
like from a user's perspective. It sounds like the answer is "it looks the
same" -- right? What is the benefit, exactly? (Please be more specific than
"it protects the AES keys".)
- Eric
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions
2021-05-14 20:15 ` [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Chang S. Bae
2021-05-17 21:34 ` Eric Biggers
@ 2021-05-18 16:57 ` Andy Lutomirski
1 sibling, 0 replies; 28+ messages in thread
From: Andy Lutomirski @ 2021-05-18 16:57 UTC (permalink / raw)
To: Chang S. Bae, tglx, mingo, bp, x86, herbert
Cc: dan.j.williams, dave.hansen, ravi.v.shankar, linux-crypto, linux-kernel
On 5/14/21 1:15 PM, Chang S. Bae wrote:
> Key Locker (KL) is Intel's new security feature that protects the AES key
> at the time of data transformation. New AES SIMD instructions -- as a
> successor of Intel's AES-NI -- are provided to encode an AES key and
> reference it for the AES algorithm.
>
> New instructions support 128/256-bit keys. While it is not desirable to
> receive any 192-bit key, AES-NI instructions are taken to serve this size.
>
> New instructions are operational in both 32-/64-bit modes.
>
> Included are methods for ECB, CBC, CTR, and XTS modes. They are not
> compatible with other implementations as referencing an encrypted form
> only.
This entire concept is severely problematic. The nineties are over --
in 2021, a responsible author of a crypto library will understand,
document, and preferably prove what security properties it is supposed
to have.
Even assuming that Key Locker is used properly and that the wrapping key
is never compromised, the security properties of Key Locker-ified
AES-CTR are weak at best. In particular, all the usual caveats of CTR
apply, and KL does nothing to mitigate it. Any attacker who recovers,
directly or by capture of a plaintext-ciphertext pair, enc_k(0),
enc_k(1), etc can encrypt and forge any message involving those counter
values.
In-kernel support for an AES Key Locker mode needs to document precisely
what that mode accomplishes and when it should and should not be used.
At least AES-XTS *probably* has the property that at attacker who gets
the contents of main memory once can't decrypt future disk images
encrypted against the same key. AES-CTR can't even achieve that.
So far, the only compelling use case for KL that I've seen is full disk
encryption. I think it would be nice for the entire process of doing
that to be worked out.
--Andy
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-17 18:21 ` Bae, Chang Seok
2021-05-17 18:45 ` Dan Williams
2021-05-17 20:15 ` Sean Christopherson
@ 2021-05-18 17:10 ` Andy Lutomirski
2021-05-18 17:52 ` Sean Christopherson
2021-12-06 21:48 ` Bae, Chang Seok
2 siblings, 2 replies; 28+ messages in thread
From: Andy Lutomirski @ 2021-05-18 17:10 UTC (permalink / raw)
To: Bae, Chang Seok
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, X86 ML,
Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar, Ravi V,
Linux Crypto Mailing List, linux-kernel
On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
> On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
>> On 5/14/21 1:14 PM, Chang S. Bae wrote:
>>> Key Locker [1][2] is a new security feature available in new Intel CPUs to
>>> protect data encryption keys for the Advanced Encryption Standard
>>> algorithm. The protection limits the amount of time an AES key is exposed
>>> in memory by sealing a key and referencing it with new AES instructions.
>>>
>>> The new AES instruction set is a successor of Intel's AES-NI (AES New
>>> Instruction). Users may switch to the Key Locker version from crypto
>>> libraries. This series includes a new AES implementation for the Crypto
>>> API, which was validated through the crypto unit tests. The performance in
>>> the test cases was measured and found comparable to the AES-NI version.
>>>
>>> Key Locker introduces a (CPU-)internal key to encode AES keys. The kernel
>>> needs to load it and ensure it unchanged as long as CPUs are operational.
>>
>> I have high-level questions:
>>
>> What is the expected use case?
>
> The wrapping key here is only used for new AES instructions.
>
> I’m aware of their potential use cases for encrypting file system or disks.
I would like to understand what people are actually going to do with
this. Give me a user story or two, please. If it turns out to be
useless, I would rather not merge it.
>
>> I certainly understand how KL is valuable in a context where
>> a verified boot process installs some KL keys that are not subsequently
>> accessible outside the KL ISA, but Linux does not really work like this.
>
> Do you mind elaborating on the concern? I try to understand any issue with
> PATCH3 [1], specifically.
My concern has nothing to do with your patches per se.
I want to understand the entire workflow that makes Key Locker safer
than not using Key Locker. Something like:
Step 1: Computer is powered on.
Step 2: Boot loader loads Linux
Step 3: Linux does such-and-such
Step 4: Attacker compromises the computer in the following way
and an explanation of why this is realistic and how Key Locker helps
would be nice.
>> What is the expected interaction between a KL-using VM guest and the
>> host VMM? Will there be performance impacts (to context switching, for
>> example) if a guest enables KL, even if the guest does not subsequently
>> do anything with it? Should Linux actually enable KL if it detects that
>> it's a VM guest? Should Linux have use a specific keying method as a guest?
>
> First of all, there is an RFC series for KVM [2].
>
> Each CPU has one internal key state so it needs to reload it between guest and
> host if both are enabled. The proposed approach enables it exclusively; expose
> it to guests only when disabled in a host. Then, I guess a guest may enable it.
I read that series. This is not a good solution.
I can think of at least a few reasonable ways that a host and a guest
can cooperate to, potentially, make KL useful.
a) Host knows that the guest will never migrate, and guest delegates
IWKEY management to the host. The host generates a random key and does
not permit the guest to use LOADIWKEY. The guest shares the random key
with the host. Of course, this means that a host key handle that leaks
to a guest can be used within the guest.
b) Host may migrate the guest. Guest delegates IWKEY management to the
host, and the host generates and remembers a key for the guest. On
migration, the host forwards the key to the new host. The host can
still internally any type of key, but context switches may be quite slow.
c) Guest wants to manage its own non-random key. Host lets it and
context switches it.
d) Guest does not need KL and leaves CR4.KL clear. Host does whatever
it wants with no overhead.
All of these have tradeoffs.
My current thought is that, if Linux is going to support Key Locker,
then this all needs to be explicitly controlled. On initial boot, Linux
should not initialize Key Locker. Upon explicit administrator request
(via sysfs?), Linux will initialize Key Locker in the mode requested by
the administrator. Modes could include:
native_random_key: Use a random key per the ISA.
native_kernel_key_remember: Use a random key but load it as a non-random
key. Remember the key in kernel memory and use it for S3 resume, etc.
native_kernel_key_backup: Use a random key, put it in the backup
storage, and forget it. Use the backup for resume, etc.
native_kernel_key_norestore: Use a random key. The key is lost on any
power transition that forgets the key. Backup is not used.
paravirt_any: Ask the hypervisor to handle keying. Any mechanism is
acceptable.
paravirt_random: Ask the hypervisor for a random key. Only succeeds if
we get an actual random key.
Does this make sense?
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-18 17:10 ` Andy Lutomirski
@ 2021-05-18 17:52 ` Sean Christopherson
2021-05-19 23:26 ` Andy Lutomirski
2021-12-06 21:48 ` Bae, Chang Seok
1 sibling, 1 reply; 28+ messages in thread
From: Sean Christopherson @ 2021-05-18 17:52 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Bae, Chang Seok, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On Tue, May 18, 2021, Andy Lutomirski wrote:
> On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
> > First of all, there is an RFC series for KVM [2].
> >
> > Each CPU has one internal key state so it needs to reload it between guest and
> > host if both are enabled. The proposed approach enables it exclusively; expose
> > it to guests only when disabled in a host. Then, I guess a guest may enable it.
>
> I read that series. This is not a good solution.
>
> I can think of at least a few reasonable ways that a host and a guest
> can cooperate to, potentially, make KL useful.
>
> a) Host knows that the guest will never migrate, and guest delegates
> IWKEY management to the host. The host generates a random key and does
> not permit the guest to use LOADIWKEY. The guest shares the random key
> with the host. Of course, this means that a host key handle that leaks
> to a guest can be used within the guest.
If the guest and host share a random key, then they also share the key handle.
And that handle+key would also need to be shared across all guests. I doubt this
option is acceptable on the security front.
Using multiple random keys is a non-starter because they can't be restored via
LOADIWKEY.
Using multiple software-defined keys will have moderate overhead because of the
possibility of using KL from soft IRQ context, i.e. KVM would have to do
LOADIWKEY on every VM-Enter _and_ VM-Exit. It sounds like LOADIWKEY has latency
similar to WRMSR, so it's not a deal-breaker, but the added latency on top of the
restrictions on how the host can use KL certainly lessen the appeal.
> b) Host may migrate the guest. Guest delegates IWKEY management to the
> host, and the host generates and remembers a key for the guest. On
> migration, the host forwards the key to the new host. The host can
> still internally any type of key, but context switches may be quite slow.
Migrating is sketchy because the IWKEY has to be exposed to host userspace.
But, I think the migration aspect is a secondary discussion.
> c) Guest wants to manage its own non-random key. Host lets it and
> context switches it.
This is essentially a variant of (b). In both cases, the host has full control
over the guest's key.
> d) Guest does not need KL and leaves CR4.KL clear. Host does whatever
> it wants with no overhead.
>
> All of these have tradeoffs.
>
> My current thought is that, if Linux is going to support Key Locker,
> then this all needs to be explicitly controlled. On initial boot, Linux
> should not initialize Key Locker. Upon explicit administrator request
> (via sysfs?), Linux will initialize Key Locker in the mode requested by
> the administrator.
Deferring KL usage to post-boot can work, but KVM shouldn't be allowed to expose
KL to a guest until KL has been explicitly configured in the host. If KVM can
spawn KL guests before the host is configured, the sysfs knob would have to deal
with the case where the desired configuration is incompatible with exposing KL
to a guest.
> Modes could include:
>
> native_random_key: Use a random key per the ISA.
>
> native_kernel_key_remember: Use a random key but load it as a non-random
> key. Remember the key in kernel memory and use it for S3 resume, etc.
What would be the motivation for this mode? It largely defeats the value
proposition of KL, no?
> native_kernel_key_backup: Use a random key, put it in the backup
> storage, and forget it. Use the backup for resume, etc.
>
> native_kernel_key_norestore: Use a random key. The key is lost on any
> power transition that forgets the key. Backup is not used.
>
> paravirt_any: Ask the hypervisor to handle keying. Any mechanism is
> acceptable.
>
> paravirt_random: Ask the hypervisor for a random key. Only succeeds if
> we get an actual random key.
AFAIK, there's no way for the guest to verify that it got a truly random key.
Hell, the guest can't even easily verify that KL is even supported. The host
can lie about CPUID and CR4.KL, and intercept all KL instructions via #UD by
running the guest with CR4.KL=0.
I also don't see any reason to define a paravirt interface for a truly random
key. Using a random key all but requires a single guest to have exclusive access
to KL, and in that case the host can simply expose KL to only that guest.
> Does this make sense?
I really want to use see concrete guest use cases before we start adding paravirt
interfaces.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-18 17:52 ` Sean Christopherson
@ 2021-05-19 23:26 ` Andy Lutomirski
2021-05-19 23:34 ` Sean Christopherson
0 siblings, 1 reply; 28+ messages in thread
From: Andy Lutomirski @ 2021-05-19 23:26 UTC (permalink / raw)
To: Sean Christopherson
Cc: Bae, Chang Seok, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On 5/18/21 10:52 AM, Sean Christopherson wrote:
> On Tue, May 18, 2021, Andy Lutomirski wrote:
>> On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
>>> First of all, there is an RFC series for KVM [2].
>>>
>>> Each CPU has one internal key state so it needs to reload it between guest and
>>> host if both are enabled. The proposed approach enables it exclusively; expose
>>> it to guests only when disabled in a host. Then, I guess a guest may enable it.
>>
>> I read that series. This is not a good solution.
>>
>> I can think of at least a few reasonable ways that a host and a guest
>> can cooperate to, potentially, make KL useful.
>>
>> a) Host knows that the guest will never migrate, and guest delegates
>> IWKEY management to the host. The host generates a random key and does
>> not permit the guest to use LOADIWKEY. The guest shares the random key
>> with the host. Of course, this means that a host key handle that leaks
>> to a guest can be used within the guest.
>
> If the guest and host share a random key, then they also share the key handle.
> And that handle+key would also need to be shared across all guests. I doubt this
> option is acceptable on the security front.
>
Indeed. Oddly, SGX has the exact same problem for any scenario in which
SGX is used for HSM-like functionality, and people still use SGX.
However, I suspect that there will be use cases in which exactly one VM
is permitted to use KL. Qubes might want that (any Qubes people around?)
> Using multiple random keys is a non-starter because they can't be restored via
> LOADIWKEY.
>
> Using multiple software-defined keys will have moderate overhead because of the
> possibility of using KL from soft IRQ context, i.e. KVM would have to do
> LOADIWKEY on every VM-Enter _and_ VM-Exit. It sounds like LOADIWKEY has latency
> similar to WRMSR, so it's not a deal-breaker, but the added latency on top of the
> restrictions on how the host can use KL certainly lessen the appeal.
Indeed. This stinks.
>
>> b) Host may migrate the guest. Guest delegates IWKEY management to the
>> host, and the host generates and remembers a key for the guest. On
>> migration, the host forwards the key to the new host. The host can
>> still internally any type of key, but context switches may be quite slow.
>
> Migrating is sketchy because the IWKEY has to be exposed to host userspace.
> But, I think the migration aspect is a secondary discussion.
>
>> c) Guest wants to manage its own non-random key. Host lets it and
>> context switches it.
>
> This is essentially a variant of (b). In both cases, the host has full control
> over the guest's key.
>
>> d) Guest does not need KL and leaves CR4.KL clear. Host does whatever
>> it wants with no overhead.
>>
>> All of these have tradeoffs.
>>
>> My current thought is that, if Linux is going to support Key Locker,
>> then this all needs to be explicitly controlled. On initial boot, Linux
>> should not initialize Key Locker. Upon explicit administrator request
>> (via sysfs?), Linux will initialize Key Locker in the mode requested by
>> the administrator.
>
> Deferring KL usage to post-boot can work, but KVM shouldn't be allowed to expose
> KL to a guest until KL has been explicitly configured in the host. If KVM can
> spawn KL guests before the host is configured, the sysfs knob would have to deal
> with the case where the desired configuration is incompatible with exposing KL
> to a guest.
There could be a host configuration "guest_only", perhaps.
>
>> Modes could include:
>>
>> native_random_key: Use a random key per the ISA.
>>
>> native_kernel_key_remember: Use a random key but load it as a non-random
>> key. Remember the key in kernel memory and use it for S3 resume, etc.
>
> What would be the motivation for this mode? It largely defeats the value
> proposition of KL, no?
It lets userspace use KL with some degree of security.
>
>> native_kernel_key_backup: Use a random key, put it in the backup
>> storage, and forget it. Use the backup for resume, etc.
>>
>> native_kernel_key_norestore: Use a random key. The key is lost on any
>> power transition that forgets the key. Backup is not used.
>>
>> paravirt_any: Ask the hypervisor to handle keying. Any mechanism is
>> acceptable.
>>
>> paravirt_random: Ask the hypervisor for a random key. Only succeeds if
>> we get an actual random key.
>
> AFAIK, there's no way for the guest to verify that it got a truly random key.
> Hell, the guest can't even easily verify that KL is even supported. The host
> can lie about CPUID and CR4.KL, and intercept all KL instructions via #UD by
> running the guest with CR4.KL=0.
The guest can use TDX. Oh wait, TDX doesn't support KL.
That being said, a host attack on the guest of this sort would be quite
slow.
>
> I also don't see any reason to define a paravirt interface for a truly random
> key. Using a random key all but requires a single guest to have exclusive access
> to KL, and in that case the host can simply expose KL to only that guest.
>
>> Does this make sense?
>
> I really want to use see concrete guest use cases before we start adding paravirt
> interfaces.
>
I want to see concrete guest use cases before we start adding *any*
guest support. And this cuts both ways -- I think that, until the guest
use cases are at least somewhat worked out, Linux should certainly not
initialize KL by default on boot if the CPUID hypervisor bit is set.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-19 23:26 ` Andy Lutomirski
@ 2021-05-19 23:34 ` Sean Christopherson
2021-05-20 0:00 ` Sean Christopherson
0 siblings, 1 reply; 28+ messages in thread
From: Sean Christopherson @ 2021-05-19 23:34 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Bae, Chang Seok, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On Wed, May 19, 2021, Andy Lutomirski wrote:
> On 5/18/21 10:52 AM, Sean Christopherson wrote:
> > On Tue, May 18, 2021, Andy Lutomirski wrote:
> >> On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
> >>> First of all, there is an RFC series for KVM [2].
> >>>
> >>> Each CPU has one internal key state so it needs to reload it between guest and
> >>> host if both are enabled. The proposed approach enables it exclusively; expose
> >>> it to guests only when disabled in a host. Then, I guess a guest may enable it.
> >>
> >> I read that series. This is not a good solution.
> >>
> >> I can think of at least a few reasonable ways that a host and a guest
> >> can cooperate to, potentially, make KL useful.
> >>
> >> a) Host knows that the guest will never migrate, and guest delegates
> >> IWKEY management to the host. The host generates a random key and does
> >> not permit the guest to use LOADIWKEY. The guest shares the random key
> >> with the host. Of course, this means that a host key handle that leaks
> >> to a guest can be used within the guest.
> >
> > If the guest and host share a random key, then they also share the key handle.
> > And that handle+key would also need to be shared across all guests. I doubt this
> > option is acceptable on the security front.
> >
>
> Indeed. Oddly, SGX has the exact same problem for any scenario in which
> SGX is used for HSM-like functionality, and people still use SGX.
The entire PRM/EPC shares a single key, but SGX doesn't rely on encryption to
isolate enclaves from other software, including other enclaves. E.g. Intel could
ship a CPU with the EPC backed entirely by on-die cache and avoid hardware
encryption entirely.
> However, I suspect that there will be use cases in which exactly one VM
> is permitted to use KL. Qubes might want that (any Qubes people around?)
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-19 23:34 ` Sean Christopherson
@ 2021-05-20 0:00 ` Sean Christopherson
0 siblings, 0 replies; 28+ messages in thread
From: Sean Christopherson @ 2021-05-20 0:00 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Bae, Chang Seok, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
X86 ML, Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar,
Ravi V, Linux Crypto Mailing List, linux-kernel
On Wed, May 19, 2021, Sean Christopherson wrote:
> On Wed, May 19, 2021, Andy Lutomirski wrote:
> > On 5/18/21 10:52 AM, Sean Christopherson wrote:
> > > On Tue, May 18, 2021, Andy Lutomirski wrote:
> > >> On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
> > >>> First of all, there is an RFC series for KVM [2].
> > >>>
> > >>> Each CPU has one internal key state so it needs to reload it between guest and
> > >>> host if both are enabled. The proposed approach enables it exclusively; expose
> > >>> it to guests only when disabled in a host. Then, I guess a guest may enable it.
> > >>
> > >> I read that series. This is not a good solution.
> > >>
> > >> I can think of at least a few reasonable ways that a host and a guest
> > >> can cooperate to, potentially, make KL useful.
> > >>
> > >> a) Host knows that the guest will never migrate, and guest delegates
> > >> IWKEY management to the host. The host generates a random key and does
> > >> not permit the guest to use LOADIWKEY. The guest shares the random key
> > >> with the host. Of course, this means that a host key handle that leaks
> > >> to a guest can be used within the guest.
> > >
> > > If the guest and host share a random key, then they also share the key handle.
> > > And that handle+key would also need to be shared across all guests. I doubt this
> > > option is acceptable on the security front.
> > >
> >
> > Indeed. Oddly, SGX has the exact same problem for any scenario in which
> > SGX is used for HSM-like functionality, and people still use SGX.
>
> The entire PRM/EPC shares a single key, but SGX doesn't rely on encryption to
> isolate enclaves from other software, including other enclaves. E.g. Intel could
> ship a CPU with the EPC backed entirely by on-die cache and avoid hardware
> encryption entirely.
Ha! I belatedly see your point: in the end, virtualized KL would also rely on a
trusted entity to isolate its sensitive data via paging-like mechanisms.
The difference in my mind is that encryption is a means to an end for SGX,
whereas hiding the key is the entire point of KL. E.g. the guest is already
relying on the VMM to isolate its code and data, adding KL doesn't change that.
Sharing an IWKEY across multiple guests would add intra-VM protection, at the
cost of making cross-VM attacks easier to some degree.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states
2021-05-14 20:15 ` [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states Chang S. Bae
@ 2021-05-24 14:21 ` Rafael J. Wysocki
0 siblings, 0 replies; 28+ messages in thread
From: Rafael J. Wysocki @ 2021-05-24 14:21 UTC (permalink / raw)
To: Chang S. Bae
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Andy Lutomirski,
the arch/x86 maintainers, Herbert Xu, Dan Williams, Dave Hansen,
Ravi V. Shankar, Linux Crypto Mailing List,
Linux Kernel Mailing List, Linux PM
On Fri, May 14, 2021 at 10:20 PM Chang S. Bae <chang.seok.bae@intel.com> wrote:
>
> When the system state switches to these sleep states, the internal key gets
> reset. Since this system transition is transparent to userspace, the
> internal key needs to be restored properly.
>
> Key Locker provides a mechanism to back up the internal key in non-volatile
> memory. The kernel requests a backup right after the key loaded at
> boot-time and copies it later when the system wakes up.
>
> The backup during the S5 sleep state is not trusted. It is overwritten by a
> new key at the next boot.
>
> On a system with the S3/4 states, enable the feature only when the backup
> mechanism is supported.
>
> Disable the feature when the copy fails (or the backup corrupts). The
> shutdown is considered too noisy. A new key is considerable only when
> threads can be synchronously suspended.
>
> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
> Cc: x86@kernel.org
> Cc: linux-kernel@vger.kernel.org
> Cc: linux-pm@vger.kernel.org
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
> Changes from RFC v1:
> * Folded the warning message into the if condition check. (Rafael Wysocki)
> * Rebased on the changes of the previous patches.
> * Added error code for key restoration failures.
> * Moved the restore helper.
> * Added function descriptions.
> ---
> arch/x86/include/asm/keylocker.h | 3 +
> arch/x86/kernel/keylocker.c | 122 +++++++++++++++++++++++++++++--
> arch/x86/power/cpu.c | 2 +
> 3 files changed, 122 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
> index 870832f007ec..74b806346bee 100644
> --- a/arch/x86/include/asm/keylocker.h
> +++ b/arch/x86/include/asm/keylocker.h
> @@ -7,6 +7,7 @@
>
> #include <asm/processor.h>
> #include <linux/bits.h>
> +#include <asm/msr.h>
>
> #define KEYLOCKER_CPUID 0x019
> #define KEYLOCKER_CPUID_EAX_SUPERVISOR BIT(0)
> @@ -18,9 +19,11 @@
> #ifdef CONFIG_X86_KEYLOCKER
> void setup_keylocker(struct cpuinfo_x86 *c);
> void flush_keylocker_data(void);
> +void restore_keylocker(void);
> #else
> #define setup_keylocker(c) do { } while (0)
> #define flush_keylocker_data() do { } while (0)
> +#define restore_keylocker() do { } while (0)
> #endif
>
> #endif /*__ASSEMBLY__ */
> diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
> index d590815de508..0f60350944fa 100644
> --- a/arch/x86/kernel/keylocker.c
> +++ b/arch/x86/kernel/keylocker.c
> @@ -5,6 +5,8 @@
> */
>
> #include <linux/random.h>
> +#include <linux/acpi.h>
> +#include <linux/delay.h>
>
> #include <asm/cacheflush.h>
> #include <asm/fpu/api.h>
> @@ -12,10 +14,13 @@
> #include <asm/keylocker.h>
> #include <asm/tlbflush.h>
>
> +static bool keybackup_available;
> +
> /* Internal (Wrapping) Key size fits in three 128-bit registers. */
> #define KEYSIZE_128BIT 3
>
> static struct _keydata {
> + bool valid;
> struct reg_128_bit value[KEYSIZE_128BIT];
> } keydata;
>
> @@ -30,6 +35,8 @@ static void make_keylocker_data(void)
>
> for (i = 0; i < KEYSIZE_128BIT; i++)
> get_random_bytes(&keydata.value[i], sizeof(struct reg_128_bit));
> +
> + keydata.valid = true;
> }
>
> /**
> @@ -47,6 +54,8 @@ void flush_keylocker_data(void)
>
> memset(keyaddr, 0, size);
> clflush_cache_range(keyaddr, size);
> +
> + keydata.valid = false;
> }
>
> #define KEYSRC_SWRAND 0
> @@ -79,6 +88,40 @@ static int load_keylocker(void)
> return err;
> }
>
> +#define KEYRESTORE_RETRY 1
> +
> +/**
> + * copy_keylocker() - Copy the internal key from the backup.
> + *
> + * Request hardware to copy the key in non-volatile memory to the CPU state. Do this
> + * again if the copy fails. The key may not be ready when the precedent backup is
> + * still in progress.
> + *
> + * Returns: -EBUSY if the copy fails, -ENODEV if no backup is available, or 0 if
> + * successful.
> + */
> +static int copy_keylocker(void)
> +{
> + int i;
> +
> + if (!keybackup_available)
> + return -ENODEV;
> +
> + wrmsrl(MSR_IA32_COPY_PLATFORM_TO_LOCAL, 1);
> +
> + for (i = 0; i <= KEYRESTORE_RETRY; i++) {
> + u64 status;
> +
> + if (i)
> + udelay(1);
> +
> + rdmsrl(MSR_IA32_COPY_STATUS, status);
> + if (status & BIT(0))
> + return 0;
> + }
> + return -EBUSY;
> +}
> +
> /**
> * setup_keylocker() - Enable the feature if supported.
> * @c: A pointer to struct cpuinfo_x86
> @@ -104,13 +147,43 @@ void setup_keylocker(struct cpuinfo_x86 *c)
> goto disable;
> }
>
> + keybackup_available = (ebx & KEYLOCKER_CPUID_EBX_BACKUP);
> + /* Internal key backup is essential with S3/4 states. */
> + if (!keybackup_available &&
> + (acpi_sleep_state_supported(ACPI_STATE_S3) ||
> + acpi_sleep_state_supported(ACPI_STATE_S4))) {
> + pr_debug("x86/keylocker: no key backup support with possible S3/4.\n");
> + goto disable;
> + }
> +
> make_keylocker_data();
> - }
>
> - err = load_keylocker();
> - if (err) {
> - pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err);
> - goto disable;
> + err = load_keylocker();
> + if (err) {
> + pr_err_once("x86/keylocker: Failed to load internal key (rc: %d).\n", err);
> + goto disable;
> + }
> +
> + /* Back up the internal key in non-volatile memory if supported. */
> + if (keybackup_available)
> + wrmsrl(MSR_IA32_COPY_LOCAL_TO_PLATFORM, 1);
> + } else {
> +
> + /*
> + * Load the internal key directly when available in memory, which is only
> + * possible at boot-time.
> + *
> + * NB: When system wakes up, this path also recovers the internal key.
> + */
> + if (keydata.valid)
> + err = load_keylocker();
> + else
> + err = copy_keylocker();
> + if (err) {
> + pr_err_once("x86/keylocker: Fail to %s internal key (rc: %d).\n",
> + keydata.valid ? "load" : "copy", err);
> + goto disable;
> + }
> }
>
> pr_info_once("x86/keylocker: Enabled.\n");
> @@ -123,3 +196,42 @@ void setup_keylocker(struct cpuinfo_x86 *c)
> /* Make sure the feature disabled for kexec-reboot. */
> cr4_clear_bits(X86_CR4_KEYLOCKER);
> }
> +
> +/**
> + * restore_keylocker() - Restore the internal key.
> + *
> + * The boot CPU executes this while other CPUs restore it through the setup function.
> + *
> + * Returns: Nothing
> + */
> +void restore_keylocker(void)
> +{
> + u64 backup_status;
> + int err;
> +
> + if (!boot_cpu_has(X86_FEATURE_KEYLOCKER))
> + return;
> +
> + /*
> + * IA32_IWKEYBACKUP_STATUS MSR contains a bitmap that indicates an invalid backup if bit 0
> + * is set and a read (or write) error if bit 2 is set.
> + */
> + rdmsrl(MSR_IA32_IWKEYBACKUP_STATUS, backup_status);
> + if (WARN(!(backup_status & BIT(0)),
> + "x86/keylocker: Internal key backup access failed with %s.\n",
> + (backup_status & BIT(2)) ? "read error" : "invalid status"))
> + goto disable_out;
> +
> + err = copy_keylocker();
> + if (err) {
> + pr_err("x86/keylocker: Internal key restoration failed (rc: %d).\n", err);
> + goto disable_out;
> + }
> +
> + return;
> +
> +disable_out:
> + pr_info("x86/keylocker: Disabled with internal key restoration failure.\n");
> + setup_clear_cpu_cap(X86_FEATURE_KEYLOCKER);
> + cr4_clear_bits(X86_CR4_KEYLOCKER);
> +}
> diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
> index 3a070e7cdb8b..ace94f07701a 100644
> --- a/arch/x86/power/cpu.c
> +++ b/arch/x86/power/cpu.c
> @@ -25,6 +25,7 @@
> #include <asm/cpu.h>
> #include <asm/mmu_context.h>
> #include <asm/cpu_device_id.h>
> +#include <asm/keylocker.h>
>
> #ifdef CONFIG_X86_32
> __visible unsigned long saved_context_ebx;
> @@ -261,6 +262,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt)
> mtrr_bp_restore();
> perf_restore_debug_store();
> msr_restore_context(ctxt);
> + restore_keylocker();
>
> c = &cpu_data(smp_processor_id());
> if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL))
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [RFC PATCH v2 00/11] x86: Support Intel Key Locker
2021-05-18 17:10 ` Andy Lutomirski
2021-05-18 17:52 ` Sean Christopherson
@ 2021-12-06 21:48 ` Bae, Chang Seok
1 sibling, 0 replies; 28+ messages in thread
From: Bae, Chang Seok @ 2021-12-06 21:48 UTC (permalink / raw)
To: Lutomirski, Andy
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, X86 ML,
Herbert Xu, Williams, Dan J, Hansen, Dave, Shankar, Ravi V,
Linux Crypto Mailing List, linux-kernel
On May 18, 2021, at 10:10, Andy Lutomirski <luto@kernel.org> wrote:
> On 5/17/21 11:21 AM, Bae, Chang Seok wrote:
>> On May 15, 2021, at 11:01, Andy Lutomirski <luto@kernel.org> wrote:
>>>
>>> I have high-level questions:
>>>
>>> What is the expected use case?
>>
>> The wrapping key here is only used for new AES instructions.
>>
>> I’m aware of their potential use cases for encrypting file system or disks.
>
> I would like to understand what people are actually going to do with
> this. Give me a user story or two, please. If it turns out to be
> useless, I would rather not merge it.
Hi Andy,
V3 was posted here with both cover letter and code changes to address this:
https://lore.kernel.org/lkml/20211124200700.15888-1-chang.seok.bae@intel.com/
Appreciate, if you can comment on the use case at least.
Thanks,
Chang
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2021-12-06 21:48 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-14 20:14 [RFC PATCH v2 00/11] x86: Support Intel Key Locker Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 01/11] x86/cpufeature: Enumerate Key Locker feature Chang S. Bae
2021-05-14 20:14 ` [RFC PATCH v2 02/11] x86/insn: Add Key Locker instructions to the opcode map Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 03/11] x86/cpu: Load Key Locker internal key at boot-time Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 04/11] x86/msr-index: Add MSRs for Key Locker internal key Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 05/11] x86/power: Restore Key Locker internal key from the ACPI S3/4 sleep states Chang S. Bae
2021-05-24 14:21 ` Rafael J. Wysocki
2021-05-14 20:15 ` [RFC PATCH v2 06/11] x86/cpu: Add a config option and a chicken bit for Key Locker Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 07/11] selftests/x86: Test Key Locker internal key maintenance Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 08/11] crypto: x86/aes-ni - Improve error handling Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 09/11] crypto: x86/aes-ni - Refactor to prepare a new AES implementation Chang S. Bae
2021-05-14 20:15 ` [RFC PATCH v2 10/11] crypto: x86/aes-kl - Support AES algorithm using Key Locker instructions Chang S. Bae
2021-05-17 21:34 ` Eric Biggers
2021-05-17 22:20 ` Bae, Chang Seok
2021-05-17 23:33 ` Eric Biggers
2021-05-18 16:57 ` Andy Lutomirski
2021-05-14 20:15 ` [RFC PATCH v2 11/11] x86/cpu: Support the hardware randomization option for Key Locker internal key Chang S. Bae
2021-05-15 18:01 ` [RFC PATCH v2 00/11] x86: Support Intel Key Locker Andy Lutomirski
2021-05-17 18:21 ` Bae, Chang Seok
2021-05-17 18:45 ` Dan Williams
2021-05-17 22:20 ` Bae, Chang Seok
2021-05-17 20:15 ` Sean Christopherson
2021-05-18 17:10 ` Andy Lutomirski
2021-05-18 17:52 ` Sean Christopherson
2021-05-19 23:26 ` Andy Lutomirski
2021-05-19 23:34 ` Sean Christopherson
2021-05-20 0:00 ` Sean Christopherson
2021-12-06 21:48 ` Bae, Chang Seok
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).