[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

Seeking comments on the APIs supporting MKTME on future Intel platforms.

MKTME (Multi-Key Total Memory Encryption) is a technology supporting
memory encryption on upcoming Intel platforms. Whereas TME allows
encryption of the entire system memory using a single key, MKTME
allows mulitple encryption domains, each having their own key. While 
the main use case for the feature is virtual machine isolation, the
API needs the flexibility to work for a wide range of use cases.

This RFC presents the 2 API additions that enable userspace to:
 1) Create Encryption Keys: Kernel Key Service type "mktme"
 2) Use the Encryption Keys: system call encrypt_mprotect()

In order to share between: the Kernel Key Service, the new system call,
and the existing mm code, helper functions were created in arch/x86/mktme

This patchset is built upon Kirill Shutemov's patchset for the core MKTME
support. You can find that here:
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip


Alison Schofield (12):
  docs/x86: Document the Multi-Key Total Memory Encryption API
  mm: Generalize the mprotect implementation to support extensions
  syscall/x86: Wire up a new system call for memory encryption keys
  x86/mm: Add helper functions to manage memory encryption keys
  x86/mm: Add a helper function to set keyid bits in encrypted VMA's
  mm: Add the encrypt_mprotect() system call
  x86/mm: Add helper functions to track encrypted VMA's
  mm: Track VMA's in use for each memory encryption keyid
  mm: Restrict memory encryption to anonymous VMA's
  x86/pconfig: Program memory encryption keys on a system-wide basis
  keys/mktme: Add a new key service type for memory encryption keys
  keys/mktme: Do not revoke in use memory encryption keys

 Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
 arch/x86/Kconfig                       |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
 arch/x86/include/asm/mktme.h           |  21 +++
 arch/x86/mm/mktme.c                    | 141 ++++++++++++++
 fs/exec.c                              |   4 +-
 include/keys/mktme-type.h              |  28 +++
 include/linux/key.h                    |   2 +
 include/linux/mm.h                     |   9 +-
 include/linux/syscalls.h               |   2 +
 include/uapi/asm-generic/unistd.h      |   4 +-
 kernel/fork.c                          |   2 +
 kernel/sys_ni.c                        |   2 +
 mm/mmap.c                              |  12 ++
 mm/mprotect.c                          |  93 +++++++++-
 mm/nommu.c                             |   4 +
 security/keys/Kconfig                  |  11 ++
 security/keys/Makefile                 |   1 +
 security/keys/internal.h               |   6 +
 security/keys/keyctl.c                 |   7 +
 security/keys/mktme_keys.c             | 325 +++++++++++++++++++++++++++++++++
 23 files changed, 855 insertions(+), 17 deletions(-)
 create mode 100644 Documentation/x86/mktme-keys.txt
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

-- 
2.14.1

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

Seeking comments on the APIs supporting MKTME on future Intel platforms.

MKTME (Multi-Key Total Memory Encryption) is a technology supporting
memory encryption on upcoming Intel platforms. Whereas TME allows
encryption of the entire system memory using a single key, MKTME
allows mulitple encryption domains, each having their own key. While 
the main use case for the feature is virtual machine isolation, the
API needs the flexibility to work for a wide range of use cases.

This RFC presents the 2 API additions that enable userspace to:
 1) Create Encryption Keys: Kernel Key Service type "mktme"
 2) Use the Encryption Keys: system call encrypt_mprotect()

In order to share between: the Kernel Key Service, the new system call,
and the existing mm code, helper functions were created in arch/x86/mktme

This patchset is built upon Kirill Shutemov's patchset for the core MKTME
support. You can find that here:
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip


Alison Schofield (12):
  docs/x86: Document the Multi-Key Total Memory Encryption API
  mm: Generalize the mprotect implementation to support extensions
  syscall/x86: Wire up a new system call for memory encryption keys
  x86/mm: Add helper functions to manage memory encryption keys
  x86/mm: Add a helper function to set keyid bits in encrypted VMA's
  mm: Add the encrypt_mprotect() system call
  x86/mm: Add helper functions to track encrypted VMA's
  mm: Track VMA's in use for each memory encryption keyid
  mm: Restrict memory encryption to anonymous VMA's
  x86/pconfig: Program memory encryption keys on a system-wide basis
  keys/mktme: Add a new key service type for memory encryption keys
  keys/mktme: Do not revoke in use memory encryption keys

 Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
 arch/x86/Kconfig                       |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
 arch/x86/include/asm/mktme.h           |  21 +++
 arch/x86/mm/mktme.c                    | 141 ++++++++++++++
 fs/exec.c                              |   4 +-
 include/keys/mktme-type.h              |  28 +++
 include/linux/key.h                    |   2 +
 include/linux/mm.h                     |   9 +-
 include/linux/syscalls.h               |   2 +
 include/uapi/asm-generic/unistd.h      |   4 +-
 kernel/fork.c                          |   2 +
 kernel/sys_ni.c                        |   2 +
 mm/mmap.c                              |  12 ++
 mm/mprotect.c                          |  93 +++++++++-
 mm/nommu.c                             |   4 +
 security/keys/Kconfig                  |  11 ++
 security/keys/Makefile                 |   1 +
 security/keys/internal.h               |   6 +
 security/keys/keyctl.c                 |   7 +
 security/keys/mktme_keys.c             | 325 +++++++++++++++++++++++++++++++++
 23 files changed, 855 insertions(+), 17 deletions(-)
 create mode 100644 Documentation/x86/mktme-keys.txt
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

-- 
2.14.1

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

Seeking comments on the APIs supporting MKTME on future Intel platforms.

MKTME (Multi-Key Total Memory Encryption) is a technology supporting
memory encryption on upcoming Intel platforms. Whereas TME allows
encryption of the entire system memory using a single key, MKTME
allows mulitple encryption domains, each having their own key. While 
the main use case for the feature is virtual machine isolation, the
API needs the flexibility to work for a wide range of use cases.

This RFC presents the 2 API additions that enable userspace to:
 1) Create Encryption Keys: Kernel Key Service type "mktme"
 2) Use the Encryption Keys: system call encrypt_mprotect()

In order to share between: the Kernel Key Service, the new system call,
and the existing mm code, helper functions were created in arch/x86/mktme

This patchset is built upon Kirill Shutemov's patchset for the core MKTME
support. You can find that here:
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip


Alison Schofield (12):
  docs/x86: Document the Multi-Key Total Memory Encryption API
  mm: Generalize the mprotect implementation to support extensions
  syscall/x86: Wire up a new system call for memory encryption keys
  x86/mm: Add helper functions to manage memory encryption keys
  x86/mm: Add a helper function to set keyid bits in encrypted VMA's
  mm: Add the encrypt_mprotect() system call
  x86/mm: Add helper functions to track encrypted VMA's
  mm: Track VMA's in use for each memory encryption keyid
  mm: Restrict memory encryption to anonymous VMA's
  x86/pconfig: Program memory encryption keys on a system-wide basis
  keys/mktme: Add a new key service type for memory encryption keys
  keys/mktme: Do not revoke in use memory encryption keys

 Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
 arch/x86/Kconfig                       |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
 arch/x86/include/asm/mktme.h           |  21 +++
 arch/x86/mm/mktme.c                    | 141 ++++++++++++++
 fs/exec.c                              |   4 +-
 include/keys/mktme-type.h              |  28 +++
 include/linux/key.h                    |   2 +
 include/linux/mm.h                     |   9 +-
 include/linux/syscalls.h               |   2 +
 include/uapi/asm-generic/unistd.h      |   4 +-
 kernel/fork.c                          |   2 +
 kernel/sys_ni.c                        |   2 +
 mm/mmap.c                              |  12 ++
 mm/mprotect.c                          |  93 +++++++++-
 mm/nommu.c                             |   4 +
 security/keys/Kconfig                  |  11 ++
 security/keys/Makefile                 |   1 +
 security/keys/internal.h               |   6 +
 security/keys/keyctl.c                 |   7 +
 security/keys/mktme_keys.c             | 325 +++++++++++++++++++++++++++++++++
 23 files changed, 855 insertions(+), 17 deletions(-)
 create mode 100644 Documentation/x86/mktme-keys.txt
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

-- 
2.14.1

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

Document the API's used for MKTME on Intel platforms.
MKTME: Multi-KEY Total Memory Encryption

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 Documentation/x86/mktme-keys.txt | 153 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 153 insertions(+)
 create mode 100644 Documentation/x86/mktme-keys.txt

diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-keys.txt
new file mode 100644
index 000000000000..2dea7acd2a17
--- /dev/null
+++ b/Documentation/x86/mktme-keys.txt
@@ -0,0 +1,153 @@
+MKTME (Multi-Key Total Memory Encryption) is a technology that allows
+memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
+allows encryption of the entire system memory using a single key, MKTME
+allows multiple encryption domains, each having their own key. The main use
+case for the feature is virtual machine isolation. The API's introduced here
+are intended to offer flexibility to work in a wide range of uses.
+
+The externally available Intel Architecture Spec:
+https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
+
+============================  API Overview  ============================
+
+There are 2 MKTME specific API's that enable userspace to create and use
+the memory encryption keys:
+
+1) Kernel Key Service: MKTME Type
+
+   MKTME is a new key type added to the existing Kernel Key Services
+   to support the memory encryption keys. The MKTME service manages
+   the addition and removal of MKTME keys. It maps userspace keys
+   to hardware keyids and programs the hardware with user requested
+   encryption parameters.
+
+   o An understanding of the Kernel Key Service is required in order
+     to use the MKTME key type as it is a subset of that service.
+
+   o MKTME keys are a limited resource. There is a single pool of
+     MKTME keys for a system and that pool can be from 3 to 63 keys.
+     With that in mind, userspace may take advantage of the kernel
+     key services sharing and permissions model for userspace keys.
+     One key can be shared as long as each user has the permission
+     of "KEY_NEED_VIEW" to use it.
+
+   o MKTME key type uses capabilities to restrict the allocation
+     of keys. It only requires CAP_SYS_RESOURCE, but will accept
+     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
+
+   o The MKTME key service blocks kernel key service commands that
+     could lead to reprogramming of in use keys, or loss of keys from
+     the pool. This means MKTME does not allow a key to be invalidated,
+     unlinked, or timed out. These operations are blocked by MKTME as
+     it creates all keys with the internal flag KEY_FLAG_KEEP.
+
+   o MKTME does not support the keyctl option of UPDATE. Userspace
+     may change the programming of a key by revoking it and adding
+     a new key with the updated encryption options (or vice-versa).
+
+2) System Call: encrypt_mprotect()
+
+   MKTME encryption is requested by calling encrypt_mprotect(). The
+   caller passes the serial number to a previously allocated and
+   programmed encryption key. That handle was created with the MKTME
+   Key Service.
+
+   o The caller must have KEY_NEED_VIEW permission on the key
+
+   o The range of memory that is to be protected must be mapped as
+     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
+     fails with EINVAL.
+
+   o As an extension to the existing mprotect() system call,
+     encrypt_mprotect() supports the legacy mprotect behavior plus
+     the enabling of memory encryption. That means that in addition
+     to encrypting the memory, the protection flags will be updated
+     as requested in the call.
+
+   o Additional mprotect() calls to memory already protected with
+     MKTME will not alter the MKTME status.
+
+======================  Usage: MKTME Key Service  ======================
+
+MKTME is enabled on supported Intel platforms by selecting
+CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
+
+Allocating MKTME Keys via command line or system call:
+    keyctl add mktme name "[options]" ring
+
+    key_serial_t add_key(const char *type, const char *description,
+                         const void *payload, size_t plen,
+                         key_serial_t keyring);
+
+Revoking MKTME Keys via command line or system call::
+   keyctl revoke <key>
+
+   long keyctl(KEYCTL_REVOKE, key_serial_t key);
+
+Options Field Definition:
+    userkey=      ASCII HEX value encryption key. Defaults to a CPU
+		  generated key if a userkey is not defined here.
+
+    algorithm=    Encryption algorithm name as a string.
+		  Valid algorithm: "aes-xts-128"
+
+    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
+                  userkey...  (need to be clear here that this is being sent
+                  to the hardware - kernel not messing w it)
+
+    entropy=      ascii hex value entropy.
+                  This entropy will be used to generated the CPU key and
+		  the tweak key when CPU generated key is requested.
+
+Algorithm Dependencies:
+    AES-XTS 128 is the only supported algorithm.
+    There are only 2 ways that AES-XTS 128 may be used:
+
+    1) User specified encryption key
+	- The user specified encryption key must be exactly
+	  16 ASCII Hex bytes (128 bits).
+	- A tweak key must be specified and it must be exactly
+	  16 ASCII Hex bytes (128 bits).
+	- No entropy field is accepted.
+
+    2) CPU generated encryption key
+	- When no user specified encryption key is provided, the
+	  default encryption key will be CPU generated.
+	- User must specify 16 ASCII Hex bytes of entropy. This 
+	  entropy will be used by the CPU to generate both the
+	  encryption key and the tweak key.
+	- No entropy field is accepted.
+
+======================  Usage: encrypt_mprotect()  ======================
+
+System Call encrypt_mprotect()::
+
+    This system call is an extension of the existing mprotect() system
+    call. It requires the same parameters as legary mprotect() plus
+    one additional parameter, the keyid. Userspace must provide the
+    key serial number assigned through the kernel key service.
+
+    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
+
+======================  Usage: Sample Roundtrip  ======================
+
+Sample usage of MKTME Key Service API with encrypt_mprotect() API:
+
+  Add a key:
+        key = add_key(mktme, name, options, strlen(option), keyring);
+
+  Map memory:
+        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
+
+  Protect memory:
+        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
+
+  Use protected memory:
+        ................
+
+  Free memory:
+        ret = munmap(ptr, size);
+
+  Revoke key:
+        ret = keyctl(KEYCTL_REVOKE, key);
+
-- 
2.14.1

[RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

Today mprotect is implemented to support legacy mprotect behavior
plus an extension for memory protection keys. Make it more generic
so that it can support additional extensions in the future.

This is done is preparation for adding a new system call for memory
encyption keys. The intent is that the new encrypted mprotect will be
another extension to legacy mprotect.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 68dc476310c0..56e64ef7931e 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -35,6 +35,8 @@
 
 #include "internal.h"
 
+#define NO_PKEY  -1
+
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
 		int dirty_accountable, int prot_numa)
@@ -402,9 +404,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * pkey=-1 when doing a legacy mprotect()
+ * When pkey=NO_PKEY we get legacy mprotect behavior here.
  */
-static int do_mprotect_pkey(unsigned long start, size_t len,
+static int do_mprotect_ext(unsigned long start, size_t len,
 		unsigned long prot, int pkey)
 {
 	unsigned long nstart, end, tmp, reqprot;
@@ -528,7 +530,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_pkey(start, len, prot, -1);
+	return do_mprotect_ext(start, len, prot, NO_PKEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -536,7 +538,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_pkey(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
-- 
2.14.1

[RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

Today mprotect is implemented to support legacy mprotect behavior
plus an extension for memory protection keys. Make it more generic
so that it can support additional extensions in the future.

This is done is preparation for adding a new system call for memory
encyption keys. The intent is that the new encrypted mprotect will be
another extension to legacy mprotect.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 68dc476310c0..56e64ef7931e 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -35,6 +35,8 @@
 
 #include "internal.h"
 
+#define NO_PKEY  -1
+
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
 		int dirty_accountable, int prot_numa)
@@ -402,9 +404,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * pkey==-1 when doing a legacy mprotect()
+ * When pkey==NO_PKEY we get legacy mprotect behavior here.
  */
-static int do_mprotect_pkey(unsigned long start, size_t len,
+static int do_mprotect_ext(unsigned long start, size_t len,
 		unsigned long prot, int pkey)
 {
 	unsigned long nstart, end, tmp, reqprot;
@@ -528,7 +530,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_pkey(start, len, prot, -1);
+	return do_mprotect_ext(start, len, prot, NO_PKEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -536,7 +538,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_pkey(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
-- 
2.14.1

[RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

Today mprotect is implemented to support legacy mprotect behavior
plus an extension for memory protection keys. Make it more generic
so that it can support additional extensions in the future.

This is done is preparation for adding a new system call for memory
encyption keys. The intent is that the new encrypted mprotect will be
another extension to legacy mprotect.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 68dc476310c0..56e64ef7931e 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -35,6 +35,8 @@
 
 #include "internal.h"
 
+#define NO_PKEY  -1
+
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
 		int dirty_accountable, int prot_numa)
@@ -402,9 +404,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * pkey==-1 when doing a legacy mprotect()
+ * When pkey==NO_PKEY we get legacy mprotect behavior here.
  */
-static int do_mprotect_pkey(unsigned long start, size_t len,
+static int do_mprotect_ext(unsigned long start, size_t len,
 		unsigned long prot, int pkey)
 {
 	unsigned long nstart, end, tmp, reqprot;
@@ -528,7 +530,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_pkey(start, len, prot, -1);
+	return do_mprotect_ext(start, len, prot, NO_PKEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -536,7 +538,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_pkey(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
-- 
2.14.1

[RFC 03/12] syscall/x86: Wire up a new system call for memory encryption keys

[Alison Schofield]

encrypt_mprotect() is a new system call to support memory encryption.

It takes the same parameters as legacy mprotect, plus an additional
key serial number that is mapped to an encryption keyid.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl | 1 +
 include/linux/syscalls.h               | 2 ++
 include/uapi/asm-generic/unistd.h      | 4 +++-
 kernel/sys_ni.c                        | 2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..f41ad857d5c6 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,4 @@
 384	i386	arch_prctl		sys_arch_prctl			__ia32_compat_sys_arch_prctl
 385	i386	io_pgetevents		sys_io_pgetevents		__ia32_compat_sys_io_pgetevents
 386	i386	rseq			sys_rseq			__ia32_sys_rseq
+387	i386	encrypt_mprotect	sys_encrypt_mprotect		__ia32_sys_encrypt_mprotect
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..cf2decfa6119 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,7 @@
 332	common	statx			__x64_sys_statx
 333	common	io_pgetevents		__x64_sys_io_pgetevents
 334	common	rseq			__x64_sys_rseq
+335	common	encrypt_mprotect	__x64_sys_encrypt_mprotect
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 3ed377d0c46c..7dc0ed3a182e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -904,6 +904,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
 			  unsigned mask, struct statx __user *buffer);
 asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,
 			 int flags, uint32_t sig);
+asmlinkage long sys_encrypt_mprotect(unsigned long start, size_t len,
+				     unsigned long prot, int serial);
 
 /*
  * Architecture-specific system calls
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 42990676a55e..d2cb0af68160 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -734,9 +734,11 @@ __SYSCALL(__NR_pkey_free,     sys_pkey_free)
 __SYSCALL(__NR_statx,     sys_statx)
 #define __NR_io_pgetevents 292
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
+#define __NR_encrypt_mprotect 293
+__SYSCALL(__NR_encrypt_mprotect, sys_encrypt_mprotect)
 
 #undef __NR_syscalls
-#define __NR_syscalls 293
+#define __NR_syscalls 294
 
 /*
  * 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index df556175be50..1b48f709c265 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -336,6 +336,8 @@ COND_SYSCALL(pkey_mprotect);
 COND_SYSCALL(pkey_alloc);
 COND_SYSCALL(pkey_free);
 
+/* multi-key total memory encryption keys */
+COND_SYSCALL(encrypt_mprotect);
 
 /*
  * Architecture specific weak syscall entries.
-- 
2.14.1

[RFC 03/12] syscall/x86: Wire up a new system call for memory encryption keys

[Alison Schofield]

encrypt_mprotect() is a new system call to support memory encryption.

It takes the same parameters as legacy mprotect, plus an additional
key serial number that is mapped to an encryption keyid.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl | 1 +
 include/linux/syscalls.h               | 2 ++
 include/uapi/asm-generic/unistd.h      | 4 +++-
 kernel/sys_ni.c                        | 2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..f41ad857d5c6 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,4 @@
 384	i386	arch_prctl		sys_arch_prctl			__ia32_compat_sys_arch_prctl
 385	i386	io_pgetevents		sys_io_pgetevents		__ia32_compat_sys_io_pgetevents
 386	i386	rseq			sys_rseq			__ia32_sys_rseq
+387	i386	encrypt_mprotect	sys_encrypt_mprotect		__ia32_sys_encrypt_mprotect
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..cf2decfa6119 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,7 @@
 332	common	statx			__x64_sys_statx
 333	common	io_pgetevents		__x64_sys_io_pgetevents
 334	common	rseq			__x64_sys_rseq
+335	common	encrypt_mprotect	__x64_sys_encrypt_mprotect
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 3ed377d0c46c..7dc0ed3a182e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -904,6 +904,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
 			  unsigned mask, struct statx __user *buffer);
 asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,
 			 int flags, uint32_t sig);
+asmlinkage long sys_encrypt_mprotect(unsigned long start, size_t len,
+				     unsigned long prot, int serial);
 
 /*
  * Architecture-specific system calls
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 42990676a55e..d2cb0af68160 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -734,9 +734,11 @@ __SYSCALL(__NR_pkey_free,     sys_pkey_free)
 __SYSCALL(__NR_statx,     sys_statx)
 #define __NR_io_pgetevents 292
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
+#define __NR_encrypt_mprotect 293
+__SYSCALL(__NR_encrypt_mprotect, sys_encrypt_mprotect)
 
 #undef __NR_syscalls
-#define __NR_syscalls 293
+#define __NR_syscalls 294
 
 /*
  * 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index df556175be50..1b48f709c265 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -336,6 +336,8 @@ COND_SYSCALL(pkey_mprotect);
 COND_SYSCALL(pkey_alloc);
 COND_SYSCALL(pkey_free);
 
+/* multi-key total memory encryption keys */
+COND_SYSCALL(encrypt_mprotect);
 
 /*
  * Architecture specific weak syscall entries.
-- 
2.14.1

[RFC 03/12] syscall/x86: Wire up a new system call for memory encryption keys

[Alison Schofield]

encrypt_mprotect() is a new system call to support memory encryption.

It takes the same parameters as legacy mprotect, plus an additional
key serial number that is mapped to an encryption keyid.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl | 1 +
 include/linux/syscalls.h               | 2 ++
 include/uapi/asm-generic/unistd.h      | 4 +++-
 kernel/sys_ni.c                        | 2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..f41ad857d5c6 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,4 @@
 384	i386	arch_prctl		sys_arch_prctl			__ia32_compat_sys_arch_prctl
 385	i386	io_pgetevents		sys_io_pgetevents		__ia32_compat_sys_io_pgetevents
 386	i386	rseq			sys_rseq			__ia32_sys_rseq
+387	i386	encrypt_mprotect	sys_encrypt_mprotect		__ia32_sys_encrypt_mprotect
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..cf2decfa6119 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,7 @@
 332	common	statx			__x64_sys_statx
 333	common	io_pgetevents		__x64_sys_io_pgetevents
 334	common	rseq			__x64_sys_rseq
+335	common	encrypt_mprotect	__x64_sys_encrypt_mprotect
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 3ed377d0c46c..7dc0ed3a182e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -904,6 +904,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
 			  unsigned mask, struct statx __user *buffer);
 asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,
 			 int flags, uint32_t sig);
+asmlinkage long sys_encrypt_mprotect(unsigned long start, size_t len,
+				     unsigned long prot, int serial);
 
 /*
  * Architecture-specific system calls
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 42990676a55e..d2cb0af68160 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -734,9 +734,11 @@ __SYSCALL(__NR_pkey_free,     sys_pkey_free)
 __SYSCALL(__NR_statx,     sys_statx)
 #define __NR_io_pgetevents 292
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
+#define __NR_encrypt_mprotect 293
+__SYSCALL(__NR_encrypt_mprotect, sys_encrypt_mprotect)
 
 #undef __NR_syscalls
-#define __NR_syscalls 293
+#define __NR_syscalls 294
 
 /*
  * 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index df556175be50..1b48f709c265 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -336,6 +336,8 @@ COND_SYSCALL(pkey_mprotect);
 COND_SYSCALL(pkey_alloc);
 COND_SYSCALL(pkey_free);
 
+/* multi-key total memory encryption keys */
+COND_SYSCALL(encrypt_mprotect);
 
 /*
  * Architecture specific weak syscall entries.
-- 
2.14.1

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

Define a global mapping structure to track the mapping of userspace
keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
This data will be used for the memory encryption system call and the
kernel key service API.

Implement helper functions to access this mapping structure and make
them visible to the MKTME Kernel Key Service: security/keys/mktme_keys

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h | 11 ++++++
 arch/x86/mm/mktme.c          | 85 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index dbfbd955da98..f6acd551457f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Manage mappings between hardware keyids and userspace keys */
+extern int mktme_map_alloc(void);
+extern void mktme_map_free(void);
+extern void mktme_map_lock(void);
+extern void mktme_map_unlock(void);
+extern int mktme_map_get_free_keyid(void);
+extern void mktme_map_clear_keyid(int keyid);
+extern void mktme_map_set_keyid(int keyid, unsigned int serial);
+extern int mktme_map_keyid_from_serial(unsigned int serial);
+extern unsigned int mktme_map_serial_from_keyid(int keyid);
+
 extern struct page_ext_operations page_mktme_ops;
 
 #define page_keyid page_keyid
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 660caf6a5ce1..5246d8323359 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/*
+ * struct mktme_mapping and the mktme_map_* functions manage the mapping
+ * of userspace keys to hardware keyids in MKTME. They are used by the
+ * the encrypt_mprotect system call and the MKTME Key Service API.
+ */
+struct mktme_mapping {
+	struct mutex	lock;		/* protect this map & HW state */
+	unsigned int	mapped_keyids;
+	unsigned int	serial[];
+};
+
+struct mktme_mapping *mktme_map;
+
+static inline long mktme_map_size(void)
+{
+	long size = 0;
+
+	size += sizeof(mktme_map);
+	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
+	return size;
+}
+
+int mktme_map_alloc(void)
+{
+	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
+	if (!mktme_map)
+		return 0;
+	mutex_init(&mktme_map->lock);
+	return 1;
+}
+
+void mktme_map_free(void)
+{
+	kfree(mktme_map);
+}
+
+void mktme_map_lock(void)
+{
+	mutex_lock(&mktme_map->lock);
+}
+
+void mktme_map_unlock(void)
+{
+	mutex_unlock(&mktme_map->lock);
+}
+
+void mktme_map_set_keyid(int keyid, unsigned int serial)
+{
+	mktme_map->serial[keyid] = serial;
+	mktme_map->mapped_keyids++;
+}
+
+void mktme_map_clear_keyid(int keyid)
+{
+	mktme_map->serial[keyid] = 0;
+	mktme_map->mapped_keyids--;
+}
+
+unsigned int mktme_map_serial_from_keyid(int keyid)
+{
+	return mktme_map->serial[keyid];
+}
+
+int mktme_map_keyid_from_serial(unsigned int serial)
+{
+	int i;
+
+	for (i = 1; i < mktme_nr_keyids; i++)
+		if (mktme_map->serial[i] = serial)
+			return i;
+	return 0;
+}
+
+int mktme_map_get_free_keyid(void)
+{
+	int i;
+
+	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
+		for (i = 1; i < mktme_nr_keyids; i++)
+			if (mktme_map->serial[i] = 0)
+				return i;
+	}
+	return 0;
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
-- 
2.14.1

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

Define a global mapping structure to track the mapping of userspace
keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
This data will be used for the memory encryption system call and the
kernel key service API.

Implement helper functions to access this mapping structure and make
them visible to the MKTME Kernel Key Service: security/keys/mktme_keys

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h | 11 ++++++
 arch/x86/mm/mktme.c          | 85 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index dbfbd955da98..f6acd551457f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Manage mappings between hardware keyids and userspace keys */
+extern int mktme_map_alloc(void);
+extern void mktme_map_free(void);
+extern void mktme_map_lock(void);
+extern void mktme_map_unlock(void);
+extern int mktme_map_get_free_keyid(void);
+extern void mktme_map_clear_keyid(int keyid);
+extern void mktme_map_set_keyid(int keyid, unsigned int serial);
+extern int mktme_map_keyid_from_serial(unsigned int serial);
+extern unsigned int mktme_map_serial_from_keyid(int keyid);
+
 extern struct page_ext_operations page_mktme_ops;
 
 #define page_keyid page_keyid
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 660caf6a5ce1..5246d8323359 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/*
+ * struct mktme_mapping and the mktme_map_* functions manage the mapping
+ * of userspace keys to hardware keyids in MKTME. They are used by the
+ * the encrypt_mprotect system call and the MKTME Key Service API.
+ */
+struct mktme_mapping {
+	struct mutex	lock;		/* protect this map & HW state */
+	unsigned int	mapped_keyids;
+	unsigned int	serial[];
+};
+
+struct mktme_mapping *mktme_map;
+
+static inline long mktme_map_size(void)
+{
+	long size = 0;
+
+	size += sizeof(mktme_map);
+	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
+	return size;
+}
+
+int mktme_map_alloc(void)
+{
+	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
+	if (!mktme_map)
+		return 0;
+	mutex_init(&mktme_map->lock);
+	return 1;
+}
+
+void mktme_map_free(void)
+{
+	kfree(mktme_map);
+}
+
+void mktme_map_lock(void)
+{
+	mutex_lock(&mktme_map->lock);
+}
+
+void mktme_map_unlock(void)
+{
+	mutex_unlock(&mktme_map->lock);
+}
+
+void mktme_map_set_keyid(int keyid, unsigned int serial)
+{
+	mktme_map->serial[keyid] = serial;
+	mktme_map->mapped_keyids++;
+}
+
+void mktme_map_clear_keyid(int keyid)
+{
+	mktme_map->serial[keyid] = 0;
+	mktme_map->mapped_keyids--;
+}
+
+unsigned int mktme_map_serial_from_keyid(int keyid)
+{
+	return mktme_map->serial[keyid];
+}
+
+int mktme_map_keyid_from_serial(unsigned int serial)
+{
+	int i;
+
+	for (i = 1; i < mktme_nr_keyids; i++)
+		if (mktme_map->serial[i] == serial)
+			return i;
+	return 0;
+}
+
+int mktme_map_get_free_keyid(void)
+{
+	int i;
+
+	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
+		for (i = 1; i < mktme_nr_keyids; i++)
+			if (mktme_map->serial[i] == 0)
+				return i;
+	}
+	return 0;
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
-- 
2.14.1

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

Define a global mapping structure to track the mapping of userspace
keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
This data will be used for the memory encryption system call and the
kernel key service API.

Implement helper functions to access this mapping structure and make
them visible to the MKTME Kernel Key Service: security/keys/mktme_keys

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h | 11 ++++++
 arch/x86/mm/mktme.c          | 85 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index dbfbd955da98..f6acd551457f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Manage mappings between hardware keyids and userspace keys */
+extern int mktme_map_alloc(void);
+extern void mktme_map_free(void);
+extern void mktme_map_lock(void);
+extern void mktme_map_unlock(void);
+extern int mktme_map_get_free_keyid(void);
+extern void mktme_map_clear_keyid(int keyid);
+extern void mktme_map_set_keyid(int keyid, unsigned int serial);
+extern int mktme_map_keyid_from_serial(unsigned int serial);
+extern unsigned int mktme_map_serial_from_keyid(int keyid);
+
 extern struct page_ext_operations page_mktme_ops;
 
 #define page_keyid page_keyid
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 660caf6a5ce1..5246d8323359 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/*
+ * struct mktme_mapping and the mktme_map_* functions manage the mapping
+ * of userspace keys to hardware keyids in MKTME. They are used by the
+ * the encrypt_mprotect system call and the MKTME Key Service API.
+ */
+struct mktme_mapping {
+	struct mutex	lock;		/* protect this map & HW state */
+	unsigned int	mapped_keyids;
+	unsigned int	serial[];
+};
+
+struct mktme_mapping *mktme_map;
+
+static inline long mktme_map_size(void)
+{
+	long size = 0;
+
+	size += sizeof(mktme_map);
+	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
+	return size;
+}
+
+int mktme_map_alloc(void)
+{
+	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
+	if (!mktme_map)
+		return 0;
+	mutex_init(&mktme_map->lock);
+	return 1;
+}
+
+void mktme_map_free(void)
+{
+	kfree(mktme_map);
+}
+
+void mktme_map_lock(void)
+{
+	mutex_lock(&mktme_map->lock);
+}
+
+void mktme_map_unlock(void)
+{
+	mutex_unlock(&mktme_map->lock);
+}
+
+void mktme_map_set_keyid(int keyid, unsigned int serial)
+{
+	mktme_map->serial[keyid] = serial;
+	mktme_map->mapped_keyids++;
+}
+
+void mktme_map_clear_keyid(int keyid)
+{
+	mktme_map->serial[keyid] = 0;
+	mktme_map->mapped_keyids--;
+}
+
+unsigned int mktme_map_serial_from_keyid(int keyid)
+{
+	return mktme_map->serial[keyid];
+}
+
+int mktme_map_keyid_from_serial(unsigned int serial)
+{
+	int i;
+
+	for (i = 1; i < mktme_nr_keyids; i++)
+		if (mktme_map->serial[i] == serial)
+			return i;
+	return 0;
+}
+
+int mktme_map_get_free_keyid(void)
+{
+	int i;
+
+	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
+		for (i = 1; i < mktme_nr_keyids; i++)
+			if (mktme_map->serial[i] == 0)
+				return i;
+	}
+	return 0;
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
-- 
2.14.1

[RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Alison Schofield]

Store the memory encryption keyid in the upper bits of vm_page_prot
that match position of keyid, bits 51:46, in a PTE.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  3 +++
 arch/x86/mm/mktme.c          | 15 +++++++++++++++
 include/linux/mm.h           |  4 ++++
 3 files changed, 22 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index f6acd551457f..b707f800b68f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,9 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Set the encryption keyid bits in a VMA */
+extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5246d8323359..5ee7f37e9cd0 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,21 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/* Set the encryption keyid bits in a VMA */
+void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
+{
+	int oldkeyid = vma_keyid(vma);
+	pgprotval_t newprot;
+
+	if (newkeyid = oldkeyid)
+		return;
+
+	newprot = pgprot_val(vma->vm_page_prot);
+	newprot &= ~mktme_keyid_mask;
+	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
+	vma->vm_page_prot = __pgprot(newprot);
+}
+
 /*
  * struct mktme_mapping and the mktme_map_* functions manage the mapping
  * of userspace keys to hardware keyids in MKTME. They are used by the
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a4ce26aa0b65..ac85c0805761 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2799,5 +2799,9 @@ void __init setup_nr_node_ids(void);
 static inline void setup_nr_node_ids(void) {}
 #endif
 
+#ifndef CONFIG_X86_INTEL_MKTME
+static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
+					int newkeyid) {}
+#endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Alison Schofield]

Store the memory encryption keyid in the upper bits of vm_page_prot
that match position of keyid, bits 51:46, in a PTE.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  3 +++
 arch/x86/mm/mktme.c          | 15 +++++++++++++++
 include/linux/mm.h           |  4 ++++
 3 files changed, 22 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index f6acd551457f..b707f800b68f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,9 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Set the encryption keyid bits in a VMA */
+extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5246d8323359..5ee7f37e9cd0 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,21 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/* Set the encryption keyid bits in a VMA */
+void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
+{
+	int oldkeyid = vma_keyid(vma);
+	pgprotval_t newprot;
+
+	if (newkeyid == oldkeyid)
+		return;
+
+	newprot = pgprot_val(vma->vm_page_prot);
+	newprot &= ~mktme_keyid_mask;
+	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
+	vma->vm_page_prot = __pgprot(newprot);
+}
+
 /*
  * struct mktme_mapping and the mktme_map_* functions manage the mapping
  * of userspace keys to hardware keyids in MKTME. They are used by the
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a4ce26aa0b65..ac85c0805761 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2799,5 +2799,9 @@ void __init setup_nr_node_ids(void);
 static inline void setup_nr_node_ids(void) {}
 #endif
 
+#ifndef CONFIG_X86_INTEL_MKTME
+static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
+					int newkeyid) {}
+#endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Alison Schofield]

Store the memory encryption keyid in the upper bits of vm_page_prot
that match position of keyid, bits 51:46, in a PTE.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  3 +++
 arch/x86/mm/mktme.c          | 15 +++++++++++++++
 include/linux/mm.h           |  4 ++++
 3 files changed, 22 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index f6acd551457f..b707f800b68f 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -13,6 +13,9 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Set the encryption keyid bits in a VMA */
+extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5246d8323359..5ee7f37e9cd0 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -63,6 +63,21 @@ int vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/* Set the encryption keyid bits in a VMA */
+void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
+{
+	int oldkeyid = vma_keyid(vma);
+	pgprotval_t newprot;
+
+	if (newkeyid == oldkeyid)
+		return;
+
+	newprot = pgprot_val(vma->vm_page_prot);
+	newprot &= ~mktme_keyid_mask;
+	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
+	vma->vm_page_prot = __pgprot(newprot);
+}
+
 /*
  * struct mktme_mapping and the mktme_map_* functions manage the mapping
  * of userspace keys to hardware keyids in MKTME. They are used by the
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a4ce26aa0b65..ac85c0805761 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2799,5 +2799,9 @@ void __init setup_nr_node_ids(void);
 static inline void setup_nr_node_ids(void) {}
 #endif
 
+#ifndef CONFIG_X86_INTEL_MKTME
+static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
+					int newkeyid) {}
+#endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 06/12] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

Implement memory encryption with a new system call that is an
extension of the legacy mprotect() system call.

In encrypt_mprotect the caller must pass a handle to a previously
allocated and programmed encryption key. Validate the key and store
the keyid bits in the vm_page_prot for each VMA in the protection
range.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 fs/exec.c           |  4 ++--
 include/linux/key.h |  2 ++
 include/linux/mm.h  |  3 ++-
 mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++-----
 4 files changed, 67 insertions(+), 9 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index a1a246062561..b681a413db9c 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
 	vm_flags |= mm->def_flags;
 	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
 
-	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
-			vm_flags);
+	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end, vm_flags,
+			     -1);
 	if (ret)
 		goto out_unlock;
 	BUG_ON(prev != vma);
diff --git a/include/linux/key.h b/include/linux/key.h
index e58ee10f6e58..fb8a7d5f6149 100644
--- a/include/linux/key.h
+++ b/include/linux/key.h
@@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key *key)
 
 extern void key_set_timeout(struct key *, unsigned);
 
+extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
+				 key_perm_t perm);
 /*
  * The permissions required on a key that we're looking up.
  */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index ac85c0805761..0f9422c7841e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct vm_area_struct *vma, unsigned long
 			      int dirty_accountable, int prot_numa);
 extern int mprotect_fixup(struct vm_area_struct *vma,
 			  struct vm_area_struct **pprev, unsigned long start,
-			  unsigned long end, unsigned long newflags);
+			  unsigned long end, unsigned long newflags,
+			  int newkeyid);
 
 /*
  * doesn't attempt to fault and will return short.
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 56e64ef7931e..6c2e1106525c 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -28,14 +28,17 @@
 #include <linux/ksm.h>
 #include <linux/uaccess.h>
 #include <linux/mm_inline.h>
+#include <linux/key.h>
 #include <asm/pgtable.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/tlbflush.h>
+#include <asm/mktme.h>
 
 #include "internal.h"
 
 #define NO_PKEY  -1
+#define NO_KEYID -1
 
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
@@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,
 
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
-	unsigned long start, unsigned long end, unsigned long newflags)
+	       unsigned long start, unsigned long end, unsigned long newflags,
+	       int newkeyid)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long oldflags = vma->vm_flags;
@@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	int error;
 	int dirty_accountable = 0;
 
+	/*
+	 * Flags match and Keyids match or we have NO_KEYID.
+	 * This _fixup is usually called from do_mprotect_ext() except
+	 * for one special case: caller fs/exec.c/setup_arg_pages()
+	 * In that case, newkeyid is passed as -1 (NO_KEYID).
+	 */
+	if (newflags == oldflags &&
+	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEYID)) {
+		*pprev = vma;
+		return 0;
+	}
+	/* Flags match and Keyid changes */
 	if (newflags == oldflags) {
+		mprotect_set_encrypt(vma, newkeyid);
 		*pprev = vma;
 		return 0;
 	}
+	/* Flags and Keyids both change, continue. */
 
 	/*
 	 * If we make a private mapping writable we increase our commit;
@@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	}
 
 success:
+	if (newkeyid != NO_KEYID)
+		mprotect_set_encrypt(vma, newkeyid);
 	/*
 	 * vm_flags and vm_page_prot are protected by the mmap_sem
 	 * held in write mode.
@@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * When pkey==NO_PKEY we get legacy mprotect behavior here.
+ * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
+ * for protection keys and memory encryption keys. These extensions are
+ * mutually exclusive and the behavior is:
+ *	(pkey==NO_PKEY && keyid==NO_KEYID) ==> legacy mprotect
+ *	(pkey is valid)  ==> legacy mprotect plus protection key extensions
+ *	(keyid is valid) ==> legacy mprotect plus encryption key extensions
  */
 static int do_mprotect_ext(unsigned long start, size_t len,
-		unsigned long prot, int pkey)
+			   unsigned long prot, int pkey, int keyid)
 {
 	unsigned long nstart, end, tmp, reqprot;
 	struct vm_area_struct *vma, *prev;
@@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 		tmp = vma->vm_end;
 		if (tmp > end)
 			tmp = end;
-		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
+		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
+				       keyid);
 		if (error)
 			goto out;
 		nstart = tmp;
@@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_ext(start, len, prot, NO_PKEY);
+	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_ext(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
@@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
 }
 
 #endif /* CONFIG_ARCH_HAS_PKEYS */
+
+#ifdef CONFIG_X86_INTEL_MKTME
+
+SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
+		unsigned long, prot, key_serial_t, serial)
+{
+	key_ref_t key_ref;
+	int ret, keyid;
+
+	/* TODO MKTME key service must be initialized */
+
+	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
+	if (IS_ERR(key_ref))
+		return PTR_ERR(key_ref);
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_serial(serial);
+	if (!keyid) {
+		mktme_map_unlock();
+		key_ref_put(key_ref);
+		return -EINVAL;
+	}
+	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
+	mktme_map_unlock();
+	key_ref_put(key_ref);
+	return ret;
+}
+
+#endif /* CONFIG_X86_INTEL_MKTME */
-- 
2.14.1

[RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's

[Alison Schofield]

In order to safely manage the usage of memory encryption keys, VMA's
using each keyid need to be tracked. This tracking allows the Kernel
Key Service to know when the keyid resource is actually in use, or
when it is idle and may be considered for reuse.

Define a global atomic encrypt_count array to track the number of VMA's
oustanding for each encryption keyid.

Implement helper functions to manipulate this encrypt_count array.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  7 +++++++
 arch/x86/mm/mktme.c          | 39 +++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h           |  2 ++
 3 files changed, 48 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index b707f800b68f..5f3fa0c39c1c 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -16,6 +16,13 @@ extern int mktme_keyid_shift;
 /* Set the encryption keyid bits in a VMA */
 extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
 
+/* Manage the references to outstanding VMA's per encryption key */
+extern int vma_alloc_encrypt_array(void);
+extern void vma_free_encrypt_array(void);
+extern int vma_read_encrypt_ref(int keyid);
+extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
+extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5ee7f37e9cd0..5690ef51a79a 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -163,6 +163,45 @@ int mktme_map_get_free_keyid(void)
 	return 0;
 }
 
+/*
+ *  Helper functions manage the encrypt_count[] array that tracks the
+ *  VMA's outstanding for each encryption keyid. The gets & puts are
+ *  used in core mm code that allocates and free's VMA's. The alloc,
+ *  free, and read functions are used by the MKTME key service to
+ *  manage key allocation and programming.
+ */
+atomic_t *encrypt_count;
+
+int vma_alloc_encrypt_array(void)
+{
+	encrypt_count = kcalloc(mktme_nr_keyids, sizeof(atomic_t), GFP_KERNEL);
+	if (!encrypt_count)
+		return -ENOMEM;
+	return 0;
+}
+
+void vma_free_encrypt_array(void)
+{
+	kfree(encrypt_count);
+}
+
+int vma_read_encrypt_ref(int keyid)
+{
+	return atomic_read(&encrypt_count[keyid]);
+}
+
+void vma_get_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_inc(&encrypt_count[vma_keyid(vma)]);
+}
+
+void vma_put_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_dec(&encrypt_count[vma_keyid(vma)]);
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0f9422c7841e..b217c699dbab 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2803,6 +2803,8 @@ static inline void setup_nr_node_ids(void) {}
 #ifndef CONFIG_X86_INTEL_MKTME
 static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
 					int newkeyid) {}
+static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
+static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
 #endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's

[Alison Schofield]

In order to safely manage the usage of memory encryption keys, VMA's
using each keyid need to be tracked. This tracking allows the Kernel
Key Service to know when the keyid resource is actually in use, or
when it is idle and may be considered for reuse.

Define a global atomic encrypt_count array to track the number of VMA's
oustanding for each encryption keyid.

Implement helper functions to manipulate this encrypt_count array.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  7 +++++++
 arch/x86/mm/mktme.c          | 39 +++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h           |  2 ++
 3 files changed, 48 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index b707f800b68f..5f3fa0c39c1c 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -16,6 +16,13 @@ extern int mktme_keyid_shift;
 /* Set the encryption keyid bits in a VMA */
 extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
 
+/* Manage the references to outstanding VMA's per encryption key */
+extern int vma_alloc_encrypt_array(void);
+extern void vma_free_encrypt_array(void);
+extern int vma_read_encrypt_ref(int keyid);
+extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
+extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5ee7f37e9cd0..5690ef51a79a 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -163,6 +163,45 @@ int mktme_map_get_free_keyid(void)
 	return 0;
 }
 
+/*
+ *  Helper functions manage the encrypt_count[] array that tracks the
+ *  VMA's outstanding for each encryption keyid. The gets & puts are
+ *  used in core mm code that allocates and free's VMA's. The alloc,
+ *  free, and read functions are used by the MKTME key service to
+ *  manage key allocation and programming.
+ */
+atomic_t *encrypt_count;
+
+int vma_alloc_encrypt_array(void)
+{
+	encrypt_count = kcalloc(mktme_nr_keyids, sizeof(atomic_t), GFP_KERNEL);
+	if (!encrypt_count)
+		return -ENOMEM;
+	return 0;
+}
+
+void vma_free_encrypt_array(void)
+{
+	kfree(encrypt_count);
+}
+
+int vma_read_encrypt_ref(int keyid)
+{
+	return atomic_read(&encrypt_count[keyid]);
+}
+
+void vma_get_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_inc(&encrypt_count[vma_keyid(vma)]);
+}
+
+void vma_put_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_dec(&encrypt_count[vma_keyid(vma)]);
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0f9422c7841e..b217c699dbab 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2803,6 +2803,8 @@ static inline void setup_nr_node_ids(void) {}
 #ifndef CONFIG_X86_INTEL_MKTME
 static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
 					int newkeyid) {}
+static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
+static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
 #endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's

[Alison Schofield]

In order to safely manage the usage of memory encryption keys, VMA's
using each keyid need to be tracked. This tracking allows the Kernel
Key Service to know when the keyid resource is actually in use, or
when it is idle and may be considered for reuse.

Define a global atomic encrypt_count array to track the number of VMA's
oustanding for each encryption keyid.

Implement helper functions to manipulate this encrypt_count array.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/mktme.h |  7 +++++++
 arch/x86/mm/mktme.c          | 39 +++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h           |  2 ++
 3 files changed, 48 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index b707f800b68f..5f3fa0c39c1c 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -16,6 +16,13 @@ extern int mktme_keyid_shift;
 /* Set the encryption keyid bits in a VMA */
 extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
 
+/* Manage the references to outstanding VMA's per encryption key */
+extern int vma_alloc_encrypt_array(void);
+extern void vma_free_encrypt_array(void);
+extern int vma_read_encrypt_ref(int keyid);
+extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
+extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
+
 /* Manage mappings between hardware keyids and userspace keys */
 extern int mktme_map_alloc(void);
 extern void mktme_map_free(void);
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5ee7f37e9cd0..5690ef51a79a 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -163,6 +163,45 @@ int mktme_map_get_free_keyid(void)
 	return 0;
 }
 
+/*
+ *  Helper functions manage the encrypt_count[] array that tracks the
+ *  VMA's outstanding for each encryption keyid. The gets & puts are
+ *  used in core mm code that allocates and free's VMA's. The alloc,
+ *  free, and read functions are used by the MKTME key service to
+ *  manage key allocation and programming.
+ */
+atomic_t *encrypt_count;
+
+int vma_alloc_encrypt_array(void)
+{
+	encrypt_count = kcalloc(mktme_nr_keyids, sizeof(atomic_t), GFP_KERNEL);
+	if (!encrypt_count)
+		return -ENOMEM;
+	return 0;
+}
+
+void vma_free_encrypt_array(void)
+{
+	kfree(encrypt_count);
+}
+
+int vma_read_encrypt_ref(int keyid)
+{
+	return atomic_read(&encrypt_count[keyid]);
+}
+
+void vma_get_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_inc(&encrypt_count[vma_keyid(vma)]);
+}
+
+void vma_put_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		atomic_dec(&encrypt_count[vma_keyid(vma)]);
+}
+
 void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
 	int i;
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0f9422c7841e..b217c699dbab 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2803,6 +2803,8 @@ static inline void setup_nr_node_ids(void) {}
 #ifndef CONFIG_X86_INTEL_MKTME
 static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
 					int newkeyid) {}
+static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
+static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
 #endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

Keep track of the VMA's oustanding for each memory encryption keyid.
The count is used by the MKTME (Multi-Key Total Memory Encryption)
Key Service to determine when it is safe to reprogram a hardware
encryption key.

Approach here is to do gets and puts on the encryption reference
wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
A couple of these locations will not be hit until cgroup support is
added. One of these locations should never hit, so use a VM_WARN_ON.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/mm/mktme.c |  2 ++
 kernel/fork.c       |  2 ++
 mm/mmap.c           | 12 ++++++++++++
 mm/nommu.c          |  4 ++++
 4 files changed, 20 insertions(+)

diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5690ef51a79a..8a7c326d4546 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
 	if (newkeyid = oldkeyid)
 		return;
 
+	vma_put_encrypt_ref(vma);
 	newprot = pgprot_val(vma->vm_page_prot);
 	newprot &= ~mktme_keyid_mask;
 	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
 	vma->vm_page_prot = __pgprot(newprot);
+	vma_get_encrypt_ref(vma);
 }
 
 /*
diff --git a/kernel/fork.c b/kernel/fork.c
index e5e7a220a124..2d0e507bde7c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		if (!tmp)
 			goto fail_nomem;
 		*tmp = *mpnt;
+		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
 		INIT_LIST_HEAD(&tmp->anon_vma_chain);
 		retval = vma_dup_policy(mpnt, tmp);
 		if (retval)
@@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 fail_nomem_anon_vma_fork:
 	mpol_put(vma_policy(tmp));
 fail_nomem_policy:
+	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
 	kmem_cache_free(vm_area_cachep, tmp);
 fail_nomem:
 	retval = -ENOMEM;
diff --git a/mm/mmap.c b/mm/mmap.c
index 4c604eb644b4..7390b8b69fd6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	mpol_put(vma_policy(vma));
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return next;
 }
@@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
 			anon_vma_merge(vma, next);
 		mm->map_count--;
 		mpol_put(vma_policy(next));
+		vma_put_encrypt_ref(next);
 		kmem_cache_free(vm_area_cachep, next);
 		/*
 		 * In mprotect's case 6 (see comments on vma_merge),
@@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 		goto unacct_error;
 	}
 
+	vma_get_encrypt_ref(vma);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
 	vma->vm_end = addr + len;
@@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 unmap_and_free_vma:
 	vma->vm_file = NULL;
 	fput(file);
+	vma_put_encrypt_ref(vma);
 
 	/* Undo any partial mapping done by a device driver. */
 	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
@@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
 	}
 
+	vma_get_encrypt_ref(new);
 	err = vma_dup_policy(vma, new);
 	if (err)
 		goto out_free_vma;
@@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
  out_free_mpol:
 	mpol_put(vma_policy(new));
  out_free_vma:
+	vma_put_encrypt_ref(new);
 	kmem_cache_free(vm_area_cachep, new);
 	return err;
 }
@@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
 		return -ENOMEM;
 	}
 
+	vma_get_encrypt_ref(vma);
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_ops = &anon_vm_ops;
@@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 		new_vma->vm_pgoff = pgoff;
 		if (vma_dup_policy(vma, new_vma))
 			goto out_free_vma;
+		vma_get_encrypt_ref(new_vma);
 		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
 		if (anon_vma_clone(new_vma, vma))
 			goto out_free_mempol;
@@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 
 out_free_mempol:
 	mpol_put(vma_policy(new_vma));
+	vma_put_encrypt_ref(new_vma);
 out_free_vma:
 	kmem_cache_free(vm_area_cachep, new_vma);
 out:
@@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
 	if (unlikely(vma = NULL))
 		return ERR_PTR(-ENOMEM);
 
+	/* Do not expect a memory encrypted vma here */
+	VM_WARN_ON(vma_keyid(vma));
+
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
diff --git a/mm/nommu.c b/mm/nommu.c
index 73f66e81cfb0..85f04c174638 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	put_nommu_region(vma->vm_region);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 }
 
@@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
 	if (!vma)
 		goto error_getting_vma;
 
+	vma_get_encrypt_ref(vma);
 	region->vm_usage = 1;
 	region->vm_flags = vm_flags;
 	region->vm_pgoff = pgoff;
@@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
 	kmem_cache_free(vm_region_jar, region);
 	if (vma->vm_file)
 		fput(vma->vm_file);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return ret;
 
@@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	*new = *vma;
 	*region = *vma->vm_region;
 	new->vm_region = region;
+	vma_get_encrypt_ref(new);
 
 	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
 
-- 
2.14.1

[RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

Keep track of the VMA's oustanding for each memory encryption keyid.
The count is used by the MKTME (Multi-Key Total Memory Encryption)
Key Service to determine when it is safe to reprogram a hardware
encryption key.

Approach here is to do gets and puts on the encryption reference
wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
A couple of these locations will not be hit until cgroup support is
added. One of these locations should never hit, so use a VM_WARN_ON.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/mm/mktme.c |  2 ++
 kernel/fork.c       |  2 ++
 mm/mmap.c           | 12 ++++++++++++
 mm/nommu.c          |  4 ++++
 4 files changed, 20 insertions(+)

diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5690ef51a79a..8a7c326d4546 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
 	if (newkeyid == oldkeyid)
 		return;
 
+	vma_put_encrypt_ref(vma);
 	newprot = pgprot_val(vma->vm_page_prot);
 	newprot &= ~mktme_keyid_mask;
 	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
 	vma->vm_page_prot = __pgprot(newprot);
+	vma_get_encrypt_ref(vma);
 }
 
 /*
diff --git a/kernel/fork.c b/kernel/fork.c
index e5e7a220a124..2d0e507bde7c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		if (!tmp)
 			goto fail_nomem;
 		*tmp = *mpnt;
+		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
 		INIT_LIST_HEAD(&tmp->anon_vma_chain);
 		retval = vma_dup_policy(mpnt, tmp);
 		if (retval)
@@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 fail_nomem_anon_vma_fork:
 	mpol_put(vma_policy(tmp));
 fail_nomem_policy:
+	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
 	kmem_cache_free(vm_area_cachep, tmp);
 fail_nomem:
 	retval = -ENOMEM;
diff --git a/mm/mmap.c b/mm/mmap.c
index 4c604eb644b4..7390b8b69fd6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	mpol_put(vma_policy(vma));
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return next;
 }
@@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
 			anon_vma_merge(vma, next);
 		mm->map_count--;
 		mpol_put(vma_policy(next));
+		vma_put_encrypt_ref(next);
 		kmem_cache_free(vm_area_cachep, next);
 		/*
 		 * In mprotect's case 6 (see comments on vma_merge),
@@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 		goto unacct_error;
 	}
 
+	vma_get_encrypt_ref(vma);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
 	vma->vm_end = addr + len;
@@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 unmap_and_free_vma:
 	vma->vm_file = NULL;
 	fput(file);
+	vma_put_encrypt_ref(vma);
 
 	/* Undo any partial mapping done by a device driver. */
 	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
@@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
 	}
 
+	vma_get_encrypt_ref(new);
 	err = vma_dup_policy(vma, new);
 	if (err)
 		goto out_free_vma;
@@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
  out_free_mpol:
 	mpol_put(vma_policy(new));
  out_free_vma:
+	vma_put_encrypt_ref(new);
 	kmem_cache_free(vm_area_cachep, new);
 	return err;
 }
@@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
 		return -ENOMEM;
 	}
 
+	vma_get_encrypt_ref(vma);
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_ops = &anon_vm_ops;
@@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 		new_vma->vm_pgoff = pgoff;
 		if (vma_dup_policy(vma, new_vma))
 			goto out_free_vma;
+		vma_get_encrypt_ref(new_vma);
 		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
 		if (anon_vma_clone(new_vma, vma))
 			goto out_free_mempol;
@@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 
 out_free_mempol:
 	mpol_put(vma_policy(new_vma));
+	vma_put_encrypt_ref(new_vma);
 out_free_vma:
 	kmem_cache_free(vm_area_cachep, new_vma);
 out:
@@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
 	if (unlikely(vma == NULL))
 		return ERR_PTR(-ENOMEM);
 
+	/* Do not expect a memory encrypted vma here */
+	VM_WARN_ON(vma_keyid(vma));
+
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
diff --git a/mm/nommu.c b/mm/nommu.c
index 73f66e81cfb0..85f04c174638 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	put_nommu_region(vma->vm_region);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 }
 
@@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
 	if (!vma)
 		goto error_getting_vma;
 
+	vma_get_encrypt_ref(vma);
 	region->vm_usage = 1;
 	region->vm_flags = vm_flags;
 	region->vm_pgoff = pgoff;
@@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
 	kmem_cache_free(vm_region_jar, region);
 	if (vma->vm_file)
 		fput(vma->vm_file);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return ret;
 
@@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	*new = *vma;
 	*region = *vma->vm_region;
 	new->vm_region = region;
+	vma_get_encrypt_ref(new);
 
 	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
 
-- 
2.14.1

[RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

Keep track of the VMA's oustanding for each memory encryption keyid.
The count is used by the MKTME (Multi-Key Total Memory Encryption)
Key Service to determine when it is safe to reprogram a hardware
encryption key.

Approach here is to do gets and puts on the encryption reference
wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
A couple of these locations will not be hit until cgroup support is
added. One of these locations should never hit, so use a VM_WARN_ON.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/mm/mktme.c |  2 ++
 kernel/fork.c       |  2 ++
 mm/mmap.c           | 12 ++++++++++++
 mm/nommu.c          |  4 ++++
 4 files changed, 20 insertions(+)

diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 5690ef51a79a..8a7c326d4546 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid)
 	if (newkeyid == oldkeyid)
 		return;
 
+	vma_put_encrypt_ref(vma);
 	newprot = pgprot_val(vma->vm_page_prot);
 	newprot &= ~mktme_keyid_mask;
 	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
 	vma->vm_page_prot = __pgprot(newprot);
+	vma_get_encrypt_ref(vma);
 }
 
 /*
diff --git a/kernel/fork.c b/kernel/fork.c
index e5e7a220a124..2d0e507bde7c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		if (!tmp)
 			goto fail_nomem;
 		*tmp = *mpnt;
+		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
 		INIT_LIST_HEAD(&tmp->anon_vma_chain);
 		retval = vma_dup_policy(mpnt, tmp);
 		if (retval)
@@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 fail_nomem_anon_vma_fork:
 	mpol_put(vma_policy(tmp));
 fail_nomem_policy:
+	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
 	kmem_cache_free(vm_area_cachep, tmp);
 fail_nomem:
 	retval = -ENOMEM;
diff --git a/mm/mmap.c b/mm/mmap.c
index 4c604eb644b4..7390b8b69fd6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	mpol_put(vma_policy(vma));
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return next;
 }
@@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
 			anon_vma_merge(vma, next);
 		mm->map_count--;
 		mpol_put(vma_policy(next));
+		vma_put_encrypt_ref(next);
 		kmem_cache_free(vm_area_cachep, next);
 		/*
 		 * In mprotect's case 6 (see comments on vma_merge),
@@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 		goto unacct_error;
 	}
 
+	vma_get_encrypt_ref(vma);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
 	vma->vm_end = addr + len;
@@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
 unmap_and_free_vma:
 	vma->vm_file = NULL;
 	fput(file);
+	vma_put_encrypt_ref(vma);
 
 	/* Undo any partial mapping done by a device driver. */
 	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
@@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
 	}
 
+	vma_get_encrypt_ref(new);
 	err = vma_dup_policy(vma, new);
 	if (err)
 		goto out_free_vma;
@@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
  out_free_mpol:
 	mpol_put(vma_policy(new));
  out_free_vma:
+	vma_put_encrypt_ref(new);
 	kmem_cache_free(vm_area_cachep, new);
 	return err;
 }
@@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
 		return -ENOMEM;
 	}
 
+	vma_get_encrypt_ref(vma);
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_ops = &anon_vm_ops;
@@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 		new_vma->vm_pgoff = pgoff;
 		if (vma_dup_policy(vma, new_vma))
 			goto out_free_vma;
+		vma_get_encrypt_ref(new_vma);
 		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
 		if (anon_vma_clone(new_vma, vma))
 			goto out_free_mempol;
@@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
 
 out_free_mempol:
 	mpol_put(vma_policy(new_vma));
+	vma_put_encrypt_ref(new_vma);
 out_free_vma:
 	kmem_cache_free(vm_area_cachep, new_vma);
 out:
@@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
 	if (unlikely(vma == NULL))
 		return ERR_PTR(-ENOMEM);
 
+	/* Do not expect a memory encrypted vma here */
+	VM_WARN_ON(vma_keyid(vma));
+
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma->vm_mm = mm;
 	vma->vm_start = addr;
diff --git a/mm/nommu.c b/mm/nommu.c
index 73f66e81cfb0..85f04c174638 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma)
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	put_nommu_region(vma->vm_region);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 }
 
@@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
 	if (!vma)
 		goto error_getting_vma;
 
+	vma_get_encrypt_ref(vma);
 	region->vm_usage = 1;
 	region->vm_flags = vm_flags;
 	region->vm_pgoff = pgoff;
@@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
 	kmem_cache_free(vm_region_jar, region);
 	if (vma->vm_file)
 		fput(vma->vm_file);
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 	return ret;
 
@@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	*new = *vma;
 	*region = *vma->vm_region;
 	new->vm_region = region;
+	vma_get_encrypt_ref(new);
 
 	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
 
-- 
2.14.1

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

Memory encryption is only supported for mappings that are ANONYMOUS.
Test the entire range of VMA's in an encrypt_mprotect() request to
make sure they all meet that requirement before encrypting any.

The encrypt_mprotect syscall will return -EINVAL and will not encrypt
any VMA's if this check fails.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 6c2e1106525c..3384b755aad1 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -311,6 +311,24 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,
 	return pages;
 }
 
+/*
+ * Encrypted mprotect is only supported on anonymous mappings.
+ * All VMA's in the requested range must be anonymous. If this
+ * test fails on any single VMA, the entire mprotect request fails.
+ */
+bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
+{
+	struct vm_area_struct *test_vma = vma;
+
+	do {
+		if (!vma_is_anonymous(test_vma))
+			return false;
+
+		test_vma = test_vma->vm_next;
+	} while (test_vma && test_vma->vm_start < end);
+	return true;
+}
+
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	       unsigned long start, unsigned long end, unsigned long newflags,
@@ -491,6 +509,10 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 				goto out;
 		}
 	}
+	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
+		error = -EINVAL;
+		goto out;
+	}
 	if (start > vma->vm_start)
 		prev = vma;
 
-- 
2.14.1

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

Memory encryption is only supported for mappings that are ANONYMOUS.
Test the entire range of VMA's in an encrypt_mprotect() request to
make sure they all meet that requirement before encrypting any.

The encrypt_mprotect syscall will return -EINVAL and will not encrypt
any VMA's if this check fails.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 6c2e1106525c..3384b755aad1 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -311,6 +311,24 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,
 	return pages;
 }
 
+/*
+ * Encrypted mprotect is only supported on anonymous mappings.
+ * All VMA's in the requested range must be anonymous. If this
+ * test fails on any single VMA, the entire mprotect request fails.
+ */
+bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
+{
+	struct vm_area_struct *test_vma = vma;
+
+	do {
+		if (!vma_is_anonymous(test_vma))
+			return false;
+
+		test_vma = test_vma->vm_next;
+	} while (test_vma && test_vma->vm_start < end);
+	return true;
+}
+
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	       unsigned long start, unsigned long end, unsigned long newflags,
@@ -491,6 +509,10 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 				goto out;
 		}
 	}
+	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
+		error = -EINVAL;
+		goto out;
+	}
 	if (start > vma->vm_start)
 		prev = vma;
 
-- 
2.14.1

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

Memory encryption is only supported for mappings that are ANONYMOUS.
Test the entire range of VMA's in an encrypt_mprotect() request to
make sure they all meet that requirement before encrypting any.

The encrypt_mprotect syscall will return -EINVAL and will not encrypt
any VMA's if this check fails.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 mm/mprotect.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index 6c2e1106525c..3384b755aad1 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -311,6 +311,24 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,
 	return pages;
 }
 
+/*
+ * Encrypted mprotect is only supported on anonymous mappings.
+ * All VMA's in the requested range must be anonymous. If this
+ * test fails on any single VMA, the entire mprotect request fails.
+ */
+bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
+{
+	struct vm_area_struct *test_vma = vma;
+
+	do {
+		if (!vma_is_anonymous(test_vma))
+			return false;
+
+		test_vma = test_vma->vm_next;
+	} while (test_vma && test_vma->vm_start < end);
+	return true;
+}
+
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	       unsigned long start, unsigned long end, unsigned long newflags,
@@ -491,6 +509,10 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 				goto out;
 		}
 	}
+	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
+		error = -EINVAL;
+		goto out;
+	}
 	if (start > vma->vm_start)
 		prev = vma;
 
-- 
2.14.1

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
as a system wide single pool of keys. The hardware, however, manages
the keys on a per physical package basis. Each physical package
maintains a key table that all CPU's in that package share.

In order to maintain the consistent, system wide view that the kernel
requires, program all physical packages during a key program request.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/intel_pconfig.h | 42 ++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/intel_pconfig.h b/arch/x86/include/asm/intel_pconfig.h
index 3cb002b1d0f9..d3bf0a297e89 100644
--- a/arch/x86/include/asm/intel_pconfig.h
+++ b/arch/x86/include/asm/intel_pconfig.h
@@ -3,6 +3,7 @@
 
 #include <asm/asm.h>
 #include <asm/processor.h>
+#include <linux/cpu.h>
 
 enum pconfig_target {
 	INVALID_TARGET	= 0,
@@ -47,19 +48,48 @@ struct mktme_key_program {
 	u8 key_field_2[64];
 } __packed __aligned(256);
 
-static inline int mktme_key_program(struct mktme_key_program *key_program)
+struct mktme_key_program_info {
+	struct mktme_key_program *key_program;
+	unsigned long status;
+};
+
+static void mktme_package_program(void *key_program_info)
 {
+	struct mktme_key_program_info *info = key_program_info;
 	unsigned long rax = MKTME_KEY_PROGRAM;
 
+	asm volatile(PCONFIG
+		: "=a" (rax), "=b" (info->key_program)
+		: "0" (rax), "1" (info->key_program)
+		: "memory", "cc");
+
+	if (rax != MKTME_PROG_SUCCESS)
+		WRITE_ONCE(info->status, rax);
+}
+
+/*
+ * MKTME keys are managed as a system-wide single pool of keys.
+ * In the hardware, each physical package maintains a separate key
+ * table. Program all physical packages with the same key info to
+ * maintain that system-wide kernel view.
+ */
+static inline int mktme_key_program(struct mktme_key_program *key_program,
+				    cpumask_var_t mktme_cpumask)
+{
+	struct mktme_key_program_info info = {
+		.key_program = key_program,
+		.status = MKTME_PROG_SUCCESS,
+	};
+
 	if (!pconfig_target_supported(MKTME_TARGET))
 		return -ENXIO;
 
-	asm volatile(PCONFIG
-		: "=a" (rax), "=b" (key_program)
-		: "0" (rax), "1" (key_program)
-		: "memory", "cc");
+	get_online_cpus();
+	on_each_cpu_mask(mktme_cpumask, mktme_package_program,
+			 &info, 1);
+	put_online_cpus();
 
-	return rax;
+	return info.status;
 }
 
 #endif	/* _ASM_X86_INTEL_PCONFIG_H */
-- 
2.14.1

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
as a system wide single pool of keys. The hardware, however, manages
the keys on a per physical package basis. Each physical package
maintains a key table that all CPU's in that package share.

In order to maintain the consistent, system wide view that the kernel
requires, program all physical packages during a key program request.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/intel_pconfig.h | 42 ++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/intel_pconfig.h b/arch/x86/include/asm/intel_pconfig.h
index 3cb002b1d0f9..d3bf0a297e89 100644
--- a/arch/x86/include/asm/intel_pconfig.h
+++ b/arch/x86/include/asm/intel_pconfig.h
@@ -3,6 +3,7 @@
 
 #include <asm/asm.h>
 #include <asm/processor.h>
+#include <linux/cpu.h>
 
 enum pconfig_target {
 	INVALID_TARGET	= 0,
@@ -47,19 +48,48 @@ struct mktme_key_program {
 	u8 key_field_2[64];
 } __packed __aligned(256);
 
-static inline int mktme_key_program(struct mktme_key_program *key_program)
+struct mktme_key_program_info {
+	struct mktme_key_program *key_program;
+	unsigned long status;
+};
+
+static void mktme_package_program(void *key_program_info)
 {
+	struct mktme_key_program_info *info = key_program_info;
 	unsigned long rax = MKTME_KEY_PROGRAM;
 
+	asm volatile(PCONFIG
+		: "=a" (rax), "=b" (info->key_program)
+		: "0" (rax), "1" (info->key_program)
+		: "memory", "cc");
+
+	if (rax != MKTME_PROG_SUCCESS)
+		WRITE_ONCE(info->status, rax);
+}
+
+/*
+ * MKTME keys are managed as a system-wide single pool of keys.
+ * In the hardware, each physical package maintains a separate key
+ * table. Program all physical packages with the same key info to
+ * maintain that system-wide kernel view.
+ */
+static inline int mktme_key_program(struct mktme_key_program *key_program,
+				    cpumask_var_t mktme_cpumask)
+{
+	struct mktme_key_program_info info = {
+		.key_program = key_program,
+		.status = MKTME_PROG_SUCCESS,
+	};
+
 	if (!pconfig_target_supported(MKTME_TARGET))
 		return -ENXIO;
 
-	asm volatile(PCONFIG
-		: "=a" (rax), "=b" (key_program)
-		: "0" (rax), "1" (key_program)
-		: "memory", "cc");
+	get_online_cpus();
+	on_each_cpu_mask(mktme_cpumask, mktme_package_program,
+			 &info, 1);
+	put_online_cpus();
 
-	return rax;
+	return info.status;
 }
 
 #endif	/* _ASM_X86_INTEL_PCONFIG_H */
-- 
2.14.1

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
as a system wide single pool of keys. The hardware, however, manages
the keys on a per physical package basis. Each physical package
maintains a key table that all CPU's in that package share.

In order to maintain the consistent, system wide view that the kernel
requires, program all physical packages during a key program request.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/include/asm/intel_pconfig.h | 42 ++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/intel_pconfig.h b/arch/x86/include/asm/intel_pconfig.h
index 3cb002b1d0f9..d3bf0a297e89 100644
--- a/arch/x86/include/asm/intel_pconfig.h
+++ b/arch/x86/include/asm/intel_pconfig.h
@@ -3,6 +3,7 @@
 
 #include <asm/asm.h>
 #include <asm/processor.h>
+#include <linux/cpu.h>
 
 enum pconfig_target {
 	INVALID_TARGET	= 0,
@@ -47,19 +48,48 @@ struct mktme_key_program {
 	u8 key_field_2[64];
 } __packed __aligned(256);
 
-static inline int mktme_key_program(struct mktme_key_program *key_program)
+struct mktme_key_program_info {
+	struct mktme_key_program *key_program;
+	unsigned long status;
+};
+
+static void mktme_package_program(void *key_program_info)
 {
+	struct mktme_key_program_info *info = key_program_info;
 	unsigned long rax = MKTME_KEY_PROGRAM;
 
+	asm volatile(PCONFIG
+		: "=a" (rax), "=b" (info->key_program)
+		: "0" (rax), "1" (info->key_program)
+		: "memory", "cc");
+
+	if (rax != MKTME_PROG_SUCCESS)
+		WRITE_ONCE(info->status, rax);
+}
+
+/*
+ * MKTME keys are managed as a system-wide single pool of keys.
+ * In the hardware, each physical package maintains a separate key
+ * table. Program all physical packages with the same key info to
+ * maintain that system-wide kernel view.
+ */
+static inline int mktme_key_program(struct mktme_key_program *key_program,
+				    cpumask_var_t mktme_cpumask)
+{
+	struct mktme_key_program_info info = {
+		.key_program = key_program,
+		.status = MKTME_PROG_SUCCESS,
+	};
+
 	if (!pconfig_target_supported(MKTME_TARGET))
 		return -ENXIO;
 
-	asm volatile(PCONFIG
-		: "=a" (rax), "=b" (key_program)
-		: "0" (rax), "1" (key_program)
-		: "memory", "cc");
+	get_online_cpus();
+	on_each_cpu_mask(mktme_cpumask, mktme_package_program,
+			 &info, 1);
+	put_online_cpus();
 
-	return rax;
+	return info.status;
 }
 
 #endif	/* _ASM_X86_INTEL_PCONFIG_H */
-- 
2.14.1

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) is a technology that allows
transparent memory encryption in upcoming Intel platforms. MKTME will
support mulitple encryption domains, each having their own key. The main
use case for the feature is virtual machine isolation. The API needs the
flexibility to work for a wide range of uses.

The MKTME key service type manages the addition and removal of the memory
encryption keys. It maps software keys to hardware keyids and programs
the hardware with the user requested encryption options.

The only supported encryption algorithm is AES-XTS 128.

The MKTME key service is half of the MKTME API level solution. It pairs
with a new memory encryption system call: encrypt_mprotect() that uses
the keys to encrypt memory.

See Documentation/x86/mktme-keys.txt

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/Kconfig           |   1 +
 include/keys/mktme-type.h  |  28 +++++
 security/keys/Kconfig      |  11 ++
 security/keys/Makefile     |   1 +
 security/keys/mktme_keys.c | 278 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 319 insertions(+)
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 023a22568c06..50d8aa6a58e9 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
 	bool "Intel Multi-Key Total Memory Encryption"
 	select DYNAMIC_PHYSICAL_MASK
 	select PAGE_EXTENSION
+	select MKTME_KEYS
 	depends on X86_64 && CPU_SUP_INTEL
 	---help---
 	  Say yes to enable support for Multi-Key Total Memory Encryption.
diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
new file mode 100644
index 000000000000..bebe74cb2b51
--- /dev/null
+++ b/include/keys/mktme-type.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Key service for Multi-KEY Total Memory Encryption
+ */
+
+#ifndef _KEYS_MKTME_TYPE_H
+#define _KEYS_MKTME_TYPE_H
+
+#include <linux/key.h>
+
+/*
+ * The AES-XTS 128 encryption algorithm requires 128 bits for each
+ * user supplied option: userkey=, tweak=, entropy=.
+ */
+#define MKTME_AES_XTS_SIZE	16
+
+enum mktme_alg {
+	MKTME_ALG_AES_XTS_128,
+};
+
+const char *const mktme_alg_names[] = {
+	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
+};
+
+extern struct key_type key_type_mktme;
+
+#endif /* _KEYS_MKTME_TYPE_H */
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index 6462e6654ccf..c36972113e67 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
 	 in the kernel.
 
 	 If you are unsure as to whether this is required, answer N.
+
+config MKTME_KEYS
+	bool "Multi-Key Total Memory Encryption Keys"
+	depends on KEYS && X86_INTEL_MKTME
+	help
+	  This option provides support for Multi-Key Total Memory
+	  Encryption (MKTME) on Intel platforms offering the feature.
+	  MKTME allows userspace to manage the hardware encryption
+	  keys through the kernel key services.
+
+	  If you are unsure as to whether this is required, answer N.
diff --git a/security/keys/Makefile b/security/keys/Makefile
index ef1581b337a3..2d9f9a82cb8a 100644
--- a/security/keys/Makefile
+++ b/security/keys/Makefile
@@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
 obj-$(CONFIG_BIG_KEYS) += big_key.o
 obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
 obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
+obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
new file mode 100644
index 000000000000..dcbce7194647
--- /dev/null
+++ b/security/keys/mktme_keys.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-3.0
+
+/* Documentation/x86/mktme-keys.txt */
+
+#include <linux/cred.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/key.h>
+#include <linux/key-type.h>
+#include <linux/init.h>
+#include <linux/parser.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <asm/intel_pconfig.h>
+#include <asm/mktme.h>
+#include <keys/mktme-type.h>
+#include <keys/user-type.h>
+
+#include "internal.h"
+
+struct kmem_cache *mktme_prog_cache;	/* hardware programming struct */
+cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program keys */
+
+static const char * const mktme_program_err[] = {
+	"KeyID was successfully programmed",	/* 0 */
+	"Invalid KeyID programming command",	/* 1 */
+	"Insufficient entropy",			/* 2 */
+	"KeyID not valid",			/* 3 */
+	"Invalid encryption algorithm chosen",	/* 4 */
+	"Failure to access key table",		/* 5 */
+};
+
+/* If a key is available, program and add the key to the software map. */
+static int mktme_program_key(key_serial_t serial,
+			     struct mktme_key_program *kprog)
+{
+	int keyid, ret;
+
+	keyid = mktme_map_get_free_keyid();
+	if (keyid = 0)
+		return -EDQUOT;
+
+	kprog->keyid = keyid;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret = MKTME_PROG_SUCCESS)
+		mktme_map_set_keyid(keyid, serial);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	return ret;
+}
+
+enum mktme_opt_id	{
+	OPT_ERROR = -1,
+	OPT_USERKEY,
+	OPT_TWEAK,
+	OPT_ENTROPY,
+	OPT_ALGORITHM,
+};
+
+static const match_table_t mktme_token = {
+	{OPT_USERKEY, "userkey=%s"},
+	{OPT_TWEAK, "tweak=%s"},
+	{OPT_ENTROPY, "entropy=%s"},
+	{OPT_ALGORITHM, "algorithm=%s"},
+	{OPT_ERROR, NULL}
+
+};
+
+/*
+ * Algorithm AES-XTS 128 is the only supported encryption algorithm.
+ * CPU Generated Key: requires user supplied entropy and accepts no
+ *		      other options.
+ * User Supplied Key: requires user supplied tweak key and accepts
+ *		      no other options.
+ */
+static int mktme_check_options(struct mktme_key_program *kprog,
+			       unsigned long token_mask)
+{
+	if (!token_mask)
+		return -EINVAL;
+
+	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
+
+	if (!test_bit(OPT_USERKEY, &token_mask)) {
+		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
+	}
+	if (test_bit(OPT_USERKEY, &token_mask)) {
+		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (!test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
+	}
+	return 0;
+}
+
+/*
+ * Parse the options and begin to fill in the key programming struct kprog.
+ * Check the lengths of incoming data and push data directly into kprog fields.
+ */
+static int mktme_get_options(char *options, struct mktme_key_program *kprog)
+{
+	int len = MKTME_AES_XTS_SIZE / 2;
+	substring_t args[MAX_OPT_ARGS];
+	unsigned long token_mask = 0;
+	enum mktme_alg alg;
+	char *p = options;
+	int ret, token;
+
+	while ((p = strsep(&options, " \t"))) {
+		if (*p = '\0' || *p = ' ' || *p = '\t')
+			continue;
+		token = match_token(p, mktme_token, args);
+		if (test_and_set_bit(token, &token_mask))
+			return -EINVAL;
+
+		switch (token) {
+		case OPT_USERKEY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TWEAK:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ENTROPY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			/* Applied to both CPU-generated data and tweak keys */
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ALGORITHM:
+			alg = match_string(mktme_alg_names,
+					   ARRAY_SIZE(mktme_alg_names),
+					   args[0].from);
+			if (alg != MKTME_ALG_AES_XTS_128)
+				return -EINVAL;
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+	return mktme_check_options(kprog, token_mask);
+}
+
+/* Key Service Command: Creates a software key and programs hardware */
+int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
+{
+	struct mktme_key_program *kprog = NULL;
+	size_t datalen = prep->datalen;
+	char *options;
+	int ret = 0;
+
+	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	if (datalen <= 0 || datalen > 1024 || !prep->data)
+		return -EINVAL;
+
+	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	options[datalen] = '\0';
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog) {
+		kzfree(options);
+		return -ENOMEM;
+	}
+	ret = mktme_get_options(options, kprog);
+	if (ret < 0)
+		goto out;
+
+	mktme_map_lock();
+	ret = mktme_program_key(key->serial, kprog);
+	mktme_map_unlock();
+out:
+	kzfree(options);
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+struct key_type key_type_mktme = {
+	.name = "mktme",
+	.instantiate = mktme_instantiate,
+	.describe = user_describe,
+};
+
+/*
+ * Build mktme_cpumask to include one cpu per physical package.
+ * The mask is used in mktme_key_program() when the hardware key
+ * table is programmed on a per package basis.
+ */
+static int mktme_build_cpumask(void)
+{
+	int online_cpu, mktme_cpu;
+	int online_pkgid, mktme_pkgid = -1;
+
+	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(online_cpu) {
+		online_pkgid = topology_physical_package_id(online_cpu);
+
+		for_each_cpu(mktme_cpu, mktme_cpumask) {
+			mktme_pkgid = topology_physical_package_id(mktme_cpu);
+			if (mktme_pkgid = online_pkgid)
+				break;
+		}
+		if (mktme_pkgid != online_pkgid)
+			cpumask_set_cpu(online_cpu, mktme_cpumask);
+	}
+	return 0;
+}
+
+/*
+ * Allocate the global key map structure based on the available keyids
+ * at boot time. Create a cache and a cpu_mask to use for programming
+ * the hardware. Initialize the encrypt_count array to track VMA's per
+ * keyid. Once all that succeeds, register the 'mktme' key type.
+ */
+static int __init init_mktme(void)
+{
+	int ret;
+
+	/* Verify keys are present */
+	if (!(mktme_nr_keyids > 0))
+		return -EINVAL;
+
+	if (!mktme_map_alloc())
+		return -ENOMEM;
+
+	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
+	if (!mktme_prog_cache)
+		goto free_map;
+
+	if (vma_alloc_encrypt_array() < 0)
+		goto free_cache;
+
+	if (mktme_build_cpumask() < 0)
+		goto free_array;
+
+	ret = register_key_type(&key_type_mktme);
+	if (!ret)
+		return ret;
+
+	free_cpumask_var(mktme_cpumask);
+free_array:
+	vma_free_encrypt_array();
+free_cache:
+	kmem_cache_destroy(mktme_prog_cache);
+free_map:
+	mktme_map_free();
+
+	return -ENOMEM;
+}
+
+late_initcall(init_mktme);
-- 
2.14.1

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) is a technology that allows
transparent memory encryption in upcoming Intel platforms. MKTME will
support mulitple encryption domains, each having their own key. The main
use case for the feature is virtual machine isolation. The API needs the
flexibility to work for a wide range of uses.

The MKTME key service type manages the addition and removal of the memory
encryption keys. It maps software keys to hardware keyids and programs
the hardware with the user requested encryption options.

The only supported encryption algorithm is AES-XTS 128.

The MKTME key service is half of the MKTME API level solution. It pairs
with a new memory encryption system call: encrypt_mprotect() that uses
the keys to encrypt memory.

See Documentation/x86/mktme-keys.txt

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/Kconfig           |   1 +
 include/keys/mktme-type.h  |  28 +++++
 security/keys/Kconfig      |  11 ++
 security/keys/Makefile     |   1 +
 security/keys/mktme_keys.c | 278 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 319 insertions(+)
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 023a22568c06..50d8aa6a58e9 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
 	bool "Intel Multi-Key Total Memory Encryption"
 	select DYNAMIC_PHYSICAL_MASK
 	select PAGE_EXTENSION
+	select MKTME_KEYS
 	depends on X86_64 && CPU_SUP_INTEL
 	---help---
 	  Say yes to enable support for Multi-Key Total Memory Encryption.
diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
new file mode 100644
index 000000000000..bebe74cb2b51
--- /dev/null
+++ b/include/keys/mktme-type.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Key service for Multi-KEY Total Memory Encryption
+ */
+
+#ifndef _KEYS_MKTME_TYPE_H
+#define _KEYS_MKTME_TYPE_H
+
+#include <linux/key.h>
+
+/*
+ * The AES-XTS 128 encryption algorithm requires 128 bits for each
+ * user supplied option: userkey=, tweak=, entropy=.
+ */
+#define MKTME_AES_XTS_SIZE	16
+
+enum mktme_alg {
+	MKTME_ALG_AES_XTS_128,
+};
+
+const char *const mktme_alg_names[] = {
+	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
+};
+
+extern struct key_type key_type_mktme;
+
+#endif /* _KEYS_MKTME_TYPE_H */
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index 6462e6654ccf..c36972113e67 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
 	 in the kernel.
 
 	 If you are unsure as to whether this is required, answer N.
+
+config MKTME_KEYS
+	bool "Multi-Key Total Memory Encryption Keys"
+	depends on KEYS && X86_INTEL_MKTME
+	help
+	  This option provides support for Multi-Key Total Memory
+	  Encryption (MKTME) on Intel platforms offering the feature.
+	  MKTME allows userspace to manage the hardware encryption
+	  keys through the kernel key services.
+
+	  If you are unsure as to whether this is required, answer N.
diff --git a/security/keys/Makefile b/security/keys/Makefile
index ef1581b337a3..2d9f9a82cb8a 100644
--- a/security/keys/Makefile
+++ b/security/keys/Makefile
@@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
 obj-$(CONFIG_BIG_KEYS) += big_key.o
 obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
 obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
+obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
new file mode 100644
index 000000000000..dcbce7194647
--- /dev/null
+++ b/security/keys/mktme_keys.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-3.0
+
+/* Documentation/x86/mktme-keys.txt */
+
+#include <linux/cred.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/key.h>
+#include <linux/key-type.h>
+#include <linux/init.h>
+#include <linux/parser.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <asm/intel_pconfig.h>
+#include <asm/mktme.h>
+#include <keys/mktme-type.h>
+#include <keys/user-type.h>
+
+#include "internal.h"
+
+struct kmem_cache *mktme_prog_cache;	/* hardware programming struct */
+cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program keys */
+
+static const char * const mktme_program_err[] = {
+	"KeyID was successfully programmed",	/* 0 */
+	"Invalid KeyID programming command",	/* 1 */
+	"Insufficient entropy",			/* 2 */
+	"KeyID not valid",			/* 3 */
+	"Invalid encryption algorithm chosen",	/* 4 */
+	"Failure to access key table",		/* 5 */
+};
+
+/* If a key is available, program and add the key to the software map. */
+static int mktme_program_key(key_serial_t serial,
+			     struct mktme_key_program *kprog)
+{
+	int keyid, ret;
+
+	keyid = mktme_map_get_free_keyid();
+	if (keyid == 0)
+		return -EDQUOT;
+
+	kprog->keyid = keyid;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret == MKTME_PROG_SUCCESS)
+		mktme_map_set_keyid(keyid, serial);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	return ret;
+}
+
+enum mktme_opt_id	{
+	OPT_ERROR = -1,
+	OPT_USERKEY,
+	OPT_TWEAK,
+	OPT_ENTROPY,
+	OPT_ALGORITHM,
+};
+
+static const match_table_t mktme_token = {
+	{OPT_USERKEY, "userkey=%s"},
+	{OPT_TWEAK, "tweak=%s"},
+	{OPT_ENTROPY, "entropy=%s"},
+	{OPT_ALGORITHM, "algorithm=%s"},
+	{OPT_ERROR, NULL}
+
+};
+
+/*
+ * Algorithm AES-XTS 128 is the only supported encryption algorithm.
+ * CPU Generated Key: requires user supplied entropy and accepts no
+ *		      other options.
+ * User Supplied Key: requires user supplied tweak key and accepts
+ *		      no other options.
+ */
+static int mktme_check_options(struct mktme_key_program *kprog,
+			       unsigned long token_mask)
+{
+	if (!token_mask)
+		return -EINVAL;
+
+	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
+
+	if (!test_bit(OPT_USERKEY, &token_mask)) {
+		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
+	}
+	if (test_bit(OPT_USERKEY, &token_mask)) {
+		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (!test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
+	}
+	return 0;
+}
+
+/*
+ * Parse the options and begin to fill in the key programming struct kprog.
+ * Check the lengths of incoming data and push data directly into kprog fields.
+ */
+static int mktme_get_options(char *options, struct mktme_key_program *kprog)
+{
+	int len = MKTME_AES_XTS_SIZE / 2;
+	substring_t args[MAX_OPT_ARGS];
+	unsigned long token_mask = 0;
+	enum mktme_alg alg;
+	char *p = options;
+	int ret, token;
+
+	while ((p = strsep(&options, " \t"))) {
+		if (*p == '\0' || *p == ' ' || *p == '\t')
+			continue;
+		token = match_token(p, mktme_token, args);
+		if (test_and_set_bit(token, &token_mask))
+			return -EINVAL;
+
+		switch (token) {
+		case OPT_USERKEY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TWEAK:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ENTROPY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			/* Applied to both CPU-generated data and tweak keys */
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ALGORITHM:
+			alg = match_string(mktme_alg_names,
+					   ARRAY_SIZE(mktme_alg_names),
+					   args[0].from);
+			if (alg != MKTME_ALG_AES_XTS_128)
+				return -EINVAL;
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+	return mktme_check_options(kprog, token_mask);
+}
+
+/* Key Service Command: Creates a software key and programs hardware */
+int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
+{
+	struct mktme_key_program *kprog = NULL;
+	size_t datalen = prep->datalen;
+	char *options;
+	int ret = 0;
+
+	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	if (datalen <= 0 || datalen > 1024 || !prep->data)
+		return -EINVAL;
+
+	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	options[datalen] = '\0';
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog) {
+		kzfree(options);
+		return -ENOMEM;
+	}
+	ret = mktme_get_options(options, kprog);
+	if (ret < 0)
+		goto out;
+
+	mktme_map_lock();
+	ret = mktme_program_key(key->serial, kprog);
+	mktme_map_unlock();
+out:
+	kzfree(options);
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+struct key_type key_type_mktme = {
+	.name = "mktme",
+	.instantiate = mktme_instantiate,
+	.describe = user_describe,
+};
+
+/*
+ * Build mktme_cpumask to include one cpu per physical package.
+ * The mask is used in mktme_key_program() when the hardware key
+ * table is programmed on a per package basis.
+ */
+static int mktme_build_cpumask(void)
+{
+	int online_cpu, mktme_cpu;
+	int online_pkgid, mktme_pkgid = -1;
+
+	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(online_cpu) {
+		online_pkgid = topology_physical_package_id(online_cpu);
+
+		for_each_cpu(mktme_cpu, mktme_cpumask) {
+			mktme_pkgid = topology_physical_package_id(mktme_cpu);
+			if (mktme_pkgid == online_pkgid)
+				break;
+		}
+		if (mktme_pkgid != online_pkgid)
+			cpumask_set_cpu(online_cpu, mktme_cpumask);
+	}
+	return 0;
+}
+
+/*
+ * Allocate the global key map structure based on the available keyids
+ * at boot time. Create a cache and a cpu_mask to use for programming
+ * the hardware. Initialize the encrypt_count array to track VMA's per
+ * keyid. Once all that succeeds, register the 'mktme' key type.
+ */
+static int __init init_mktme(void)
+{
+	int ret;
+
+	/* Verify keys are present */
+	if (!(mktme_nr_keyids > 0))
+		return -EINVAL;
+
+	if (!mktme_map_alloc())
+		return -ENOMEM;
+
+	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
+	if (!mktme_prog_cache)
+		goto free_map;
+
+	if (vma_alloc_encrypt_array() < 0)
+		goto free_cache;
+
+	if (mktme_build_cpumask() < 0)
+		goto free_array;
+
+	ret = register_key_type(&key_type_mktme);
+	if (!ret)
+		return ret;
+
+	free_cpumask_var(mktme_cpumask);
+free_array:
+	vma_free_encrypt_array();
+free_cache:
+	kmem_cache_destroy(mktme_prog_cache);
+free_map:
+	mktme_map_free();
+
+	return -ENOMEM;
+}
+
+late_initcall(init_mktme);
-- 
2.14.1

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) is a technology that allows
transparent memory encryption in upcoming Intel platforms. MKTME will
support mulitple encryption domains, each having their own key. The main
use case for the feature is virtual machine isolation. The API needs the
flexibility to work for a wide range of uses.

The MKTME key service type manages the addition and removal of the memory
encryption keys. It maps software keys to hardware keyids and programs
the hardware with the user requested encryption options.

The only supported encryption algorithm is AES-XTS 128.

The MKTME key service is half of the MKTME API level solution. It pairs
with a new memory encryption system call: encrypt_mprotect() that uses
the keys to encrypt memory.

See Documentation/x86/mktme-keys.txt

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 arch/x86/Kconfig           |   1 +
 include/keys/mktme-type.h  |  28 +++++
 security/keys/Kconfig      |  11 ++
 security/keys/Makefile     |   1 +
 security/keys/mktme_keys.c | 278 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 319 insertions(+)
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 023a22568c06..50d8aa6a58e9 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
 	bool "Intel Multi-Key Total Memory Encryption"
 	select DYNAMIC_PHYSICAL_MASK
 	select PAGE_EXTENSION
+	select MKTME_KEYS
 	depends on X86_64 && CPU_SUP_INTEL
 	---help---
 	  Say yes to enable support for Multi-Key Total Memory Encryption.
diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
new file mode 100644
index 000000000000..bebe74cb2b51
--- /dev/null
+++ b/include/keys/mktme-type.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Key service for Multi-KEY Total Memory Encryption
+ */
+
+#ifndef _KEYS_MKTME_TYPE_H
+#define _KEYS_MKTME_TYPE_H
+
+#include <linux/key.h>
+
+/*
+ * The AES-XTS 128 encryption algorithm requires 128 bits for each
+ * user supplied option: userkey=, tweak=, entropy=.
+ */
+#define MKTME_AES_XTS_SIZE	16
+
+enum mktme_alg {
+	MKTME_ALG_AES_XTS_128,
+};
+
+const char *const mktme_alg_names[] = {
+	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
+};
+
+extern struct key_type key_type_mktme;
+
+#endif /* _KEYS_MKTME_TYPE_H */
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index 6462e6654ccf..c36972113e67 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
 	 in the kernel.
 
 	 If you are unsure as to whether this is required, answer N.
+
+config MKTME_KEYS
+	bool "Multi-Key Total Memory Encryption Keys"
+	depends on KEYS && X86_INTEL_MKTME
+	help
+	  This option provides support for Multi-Key Total Memory
+	  Encryption (MKTME) on Intel platforms offering the feature.
+	  MKTME allows userspace to manage the hardware encryption
+	  keys through the kernel key services.
+
+	  If you are unsure as to whether this is required, answer N.
diff --git a/security/keys/Makefile b/security/keys/Makefile
index ef1581b337a3..2d9f9a82cb8a 100644
--- a/security/keys/Makefile
+++ b/security/keys/Makefile
@@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
 obj-$(CONFIG_BIG_KEYS) += big_key.o
 obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
 obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
+obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
new file mode 100644
index 000000000000..dcbce7194647
--- /dev/null
+++ b/security/keys/mktme_keys.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-3.0
+
+/* Documentation/x86/mktme-keys.txt */
+
+#include <linux/cred.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/key.h>
+#include <linux/key-type.h>
+#include <linux/init.h>
+#include <linux/parser.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <asm/intel_pconfig.h>
+#include <asm/mktme.h>
+#include <keys/mktme-type.h>
+#include <keys/user-type.h>
+
+#include "internal.h"
+
+struct kmem_cache *mktme_prog_cache;	/* hardware programming struct */
+cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program keys */
+
+static const char * const mktme_program_err[] = {
+	"KeyID was successfully programmed",	/* 0 */
+	"Invalid KeyID programming command",	/* 1 */
+	"Insufficient entropy",			/* 2 */
+	"KeyID not valid",			/* 3 */
+	"Invalid encryption algorithm chosen",	/* 4 */
+	"Failure to access key table",		/* 5 */
+};
+
+/* If a key is available, program and add the key to the software map. */
+static int mktme_program_key(key_serial_t serial,
+			     struct mktme_key_program *kprog)
+{
+	int keyid, ret;
+
+	keyid = mktme_map_get_free_keyid();
+	if (keyid == 0)
+		return -EDQUOT;
+
+	kprog->keyid = keyid;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret == MKTME_PROG_SUCCESS)
+		mktme_map_set_keyid(keyid, serial);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	return ret;
+}
+
+enum mktme_opt_id	{
+	OPT_ERROR = -1,
+	OPT_USERKEY,
+	OPT_TWEAK,
+	OPT_ENTROPY,
+	OPT_ALGORITHM,
+};
+
+static const match_table_t mktme_token = {
+	{OPT_USERKEY, "userkey=%s"},
+	{OPT_TWEAK, "tweak=%s"},
+	{OPT_ENTROPY, "entropy=%s"},
+	{OPT_ALGORITHM, "algorithm=%s"},
+	{OPT_ERROR, NULL}
+
+};
+
+/*
+ * Algorithm AES-XTS 128 is the only supported encryption algorithm.
+ * CPU Generated Key: requires user supplied entropy and accepts no
+ *		      other options.
+ * User Supplied Key: requires user supplied tweak key and accepts
+ *		      no other options.
+ */
+static int mktme_check_options(struct mktme_key_program *kprog,
+			       unsigned long token_mask)
+{
+	if (!token_mask)
+		return -EINVAL;
+
+	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
+
+	if (!test_bit(OPT_USERKEY, &token_mask)) {
+		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
+	}
+	if (test_bit(OPT_USERKEY, &token_mask)) {
+		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
+		    (!test_bit(OPT_TWEAK, &token_mask)))
+			return -EINVAL;
+
+		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
+	}
+	return 0;
+}
+
+/*
+ * Parse the options and begin to fill in the key programming struct kprog.
+ * Check the lengths of incoming data and push data directly into kprog fields.
+ */
+static int mktme_get_options(char *options, struct mktme_key_program *kprog)
+{
+	int len = MKTME_AES_XTS_SIZE / 2;
+	substring_t args[MAX_OPT_ARGS];
+	unsigned long token_mask = 0;
+	enum mktme_alg alg;
+	char *p = options;
+	int ret, token;
+
+	while ((p = strsep(&options, " \t"))) {
+		if (*p == '\0' || *p == ' ' || *p == '\t')
+			continue;
+		token = match_token(p, mktme_token, args);
+		if (test_and_set_bit(token, &token_mask))
+			return -EINVAL;
+
+		switch (token) {
+		case OPT_USERKEY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TWEAK:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ENTROPY:
+			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
+				return -EINVAL;
+			/* Applied to both CPU-generated data and tweak keys */
+			ret = hex2bin(kprog->key_field_1, args[0].from, len);
+			ret = hex2bin(kprog->key_field_2, args[0].from, len);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ALGORITHM:
+			alg = match_string(mktme_alg_names,
+					   ARRAY_SIZE(mktme_alg_names),
+					   args[0].from);
+			if (alg != MKTME_ALG_AES_XTS_128)
+				return -EINVAL;
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+	return mktme_check_options(kprog, token_mask);
+}
+
+/* Key Service Command: Creates a software key and programs hardware */
+int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
+{
+	struct mktme_key_program *kprog = NULL;
+	size_t datalen = prep->datalen;
+	char *options;
+	int ret = 0;
+
+	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	if (datalen <= 0 || datalen > 1024 || !prep->data)
+		return -EINVAL;
+
+	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	options[datalen] = '\0';
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog) {
+		kzfree(options);
+		return -ENOMEM;
+	}
+	ret = mktme_get_options(options, kprog);
+	if (ret < 0)
+		goto out;
+
+	mktme_map_lock();
+	ret = mktme_program_key(key->serial, kprog);
+	mktme_map_unlock();
+out:
+	kzfree(options);
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+struct key_type key_type_mktme = {
+	.name = "mktme",
+	.instantiate = mktme_instantiate,
+	.describe = user_describe,
+};
+
+/*
+ * Build mktme_cpumask to include one cpu per physical package.
+ * The mask is used in mktme_key_program() when the hardware key
+ * table is programmed on a per package basis.
+ */
+static int mktme_build_cpumask(void)
+{
+	int online_cpu, mktme_cpu;
+	int online_pkgid, mktme_pkgid = -1;
+
+	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(online_cpu) {
+		online_pkgid = topology_physical_package_id(online_cpu);
+
+		for_each_cpu(mktme_cpu, mktme_cpumask) {
+			mktme_pkgid = topology_physical_package_id(mktme_cpu);
+			if (mktme_pkgid == online_pkgid)
+				break;
+		}
+		if (mktme_pkgid != online_pkgid)
+			cpumask_set_cpu(online_cpu, mktme_cpumask);
+	}
+	return 0;
+}
+
+/*
+ * Allocate the global key map structure based on the available keyids
+ * at boot time. Create a cache and a cpu_mask to use for programming
+ * the hardware. Initialize the encrypt_count array to track VMA's per
+ * keyid. Once all that succeeds, register the 'mktme' key type.
+ */
+static int __init init_mktme(void)
+{
+	int ret;
+
+	/* Verify keys are present */
+	if (!(mktme_nr_keyids > 0))
+		return -EINVAL;
+
+	if (!mktme_map_alloc())
+		return -ENOMEM;
+
+	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
+	if (!mktme_prog_cache)
+		goto free_map;
+
+	if (vma_alloc_encrypt_array() < 0)
+		goto free_cache;
+
+	if (mktme_build_cpumask() < 0)
+		goto free_array;
+
+	ret = register_key_type(&key_type_mktme);
+	if (!ret)
+		return ret;
+
+	free_cpumask_var(mktme_cpumask);
+free_array:
+	vma_free_encrypt_array();
+free_cache:
+	kmem_cache_destroy(mktme_prog_cache);
+free_map:
+	mktme_map_free();
+
+	return -ENOMEM;
+}
+
+late_initcall(init_mktme);
-- 
2.14.1

[RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[Alison Schofield]

The MKTME key service maps userspace keys to hardware keyids. Those
keys are used in a new system call that encrypts memory. The keys
need to be tightly controlled. One example is that userspace keys
should not be revoked while the hardware keyid slot is still in use.

The KEY_FLAG_KEEP bit offers good control. The mktme service uses
that flag to prevent userspace keys from going away without proper
synchronization with the mktme service type.

The problem is that we need a safe and synchronous way to revoke keys.
The way .revoke methods function now, the key service type is called late
in the revoke process for cleanup after the fact. The mktme key service
has no means to consider and perhaps reject the revoke request.

This proposal inserts the MKTME revoke call earlier into the existing
keyctl <revoke> path. If it is safe to revoke the key, MKTME key service
will turn off KEY_FLAG_KEEP and let the revoke continue and succeed.
Otherwise, not safe, KEY_FLAG_KEEP stays on, which causes the normal
path of revoke to fail.

For the MKTME Key Service, a revoke may be done safely when there are
no outstanding memory mappings encrypted with the key being revoked.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 security/keys/internal.h   |  6 ++++++
 security/keys/keyctl.c     |  7 +++++++
 security/keys/mktme_keys.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 60 insertions(+)

diff --git a/security/keys/internal.h b/security/keys/internal.h
index 9f8208dc0e55..9fb871522efe 100644
--- a/security/keys/internal.h
+++ b/security/keys/internal.h
@@ -316,4 +316,10 @@ static inline void key_check(const struct key *key)
 
 #endif
 
+#ifdef CONFIG_MKTME_KEYS
+extern void mktme_revoke_key(struct key *key);
+#else
+static inline void mktme_revoke_key(struct key *key) {}
+#endif /* CONFIG_MKTME_KEYS */
+
 #endif /* _INTERNAL_H */
diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c
index 1ffe60bb2845..86d2596ff275 100644
--- a/security/keys/keyctl.c
+++ b/security/keys/keyctl.c
@@ -363,6 +363,9 @@ long keyctl_update_key(key_serial_t id,
  * and any links to the key will be automatically garbage collected after a
  * certain amount of time (/proc/sys/kernel/keys/gc_delay).
  *
+ * The MKTME key service type checks if a memory encryption key is in use
+ * before allowing a revoke to proceed.
+ *
  * Keys with KEY_FLAG_KEEP set should not be revoked.
  *
  * If successful, 0 is returned.
@@ -387,6 +390,10 @@ long keyctl_revoke_key(key_serial_t id)
 
 	key = key_ref_to_ptr(key_ref);
 	ret = 0;
+
+	if (strcmp(key->type->name, "mktme") = 0)
+		mktme_revoke_key(key);
+
 	if (test_bit(KEY_FLAG_KEEP, &key->flags))
 		ret = -EPERM;
 	else
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index dcbce7194647..c665be860538 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -31,6 +31,52 @@ static const char * const mktme_program_err[] = {
 	"Failure to access key table",		/* 5 */
 };
 
+static int mktme_clear_programmed_key(int keyid)
+{
+	struct mktme_key_program *kprog = NULL;
+	int ret;
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog)
+		return -ENOMEM;
+
+	kprog->keyid = keyid;
+	kprog->keyid_ctrl = MKTME_KEYID_CLEAR_KEY;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret = MKTME_PROG_SUCCESS)
+		mktme_map_clear_keyid(keyid);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+/*
+ * If the key is not in use, clear the hardware programming and
+ * allow the revoke to continue by clearing KEY_FLAG_KEEP.
+ */
+void mktme_revoke_key(struct key *key)
+{
+	int keyid, vma_count;
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_serial(key->serial);
+	if (keyid <= 0)
+		goto out;
+
+	vma_count = vma_read_encrypt_ref(keyid);
+	if (vma_count > 0) {
+		pr_debug("mktme not freeing keyid[%d] encrypt_count[%d]\n",
+			 keyid, vma_count);
+		goto out;
+	}
+	if (!mktme_clear_programmed_key(keyid))
+		clear_bit(KEY_FLAG_KEEP, &key->flags);
+out:
+	mktme_map_unlock();
+}
+
 /* If a key is available, program and add the key to the software map. */
 static int mktme_program_key(key_serial_t serial,
 			     struct mktme_key_program *kprog)
@@ -193,6 +239,7 @@ int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
 
 	mktme_map_lock();
 	ret = mktme_program_key(key->serial, kprog);
+	set_bit(KEY_FLAG_KEEP, &key->flags);
 	mktme_map_unlock();
 out:
 	kzfree(options);
-- 
2.14.1

[RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[Alison Schofield]

The MKTME key service maps userspace keys to hardware keyids. Those
keys are used in a new system call that encrypts memory. The keys
need to be tightly controlled. One example is that userspace keys
should not be revoked while the hardware keyid slot is still in use.

The KEY_FLAG_KEEP bit offers good control. The mktme service uses
that flag to prevent userspace keys from going away without proper
synchronization with the mktme service type.

The problem is that we need a safe and synchronous way to revoke keys.
The way .revoke methods function now, the key service type is called late
in the revoke process for cleanup after the fact. The mktme key service
has no means to consider and perhaps reject the revoke request.

This proposal inserts the MKTME revoke call earlier into the existing
keyctl <revoke> path. If it is safe to revoke the key, MKTME key service
will turn off KEY_FLAG_KEEP and let the revoke continue and succeed.
Otherwise, not safe, KEY_FLAG_KEEP stays on, which causes the normal
path of revoke to fail.

For the MKTME Key Service, a revoke may be done safely when there are
no outstanding memory mappings encrypted with the key being revoked.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 security/keys/internal.h   |  6 ++++++
 security/keys/keyctl.c     |  7 +++++++
 security/keys/mktme_keys.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 60 insertions(+)

diff --git a/security/keys/internal.h b/security/keys/internal.h
index 9f8208dc0e55..9fb871522efe 100644
--- a/security/keys/internal.h
+++ b/security/keys/internal.h
@@ -316,4 +316,10 @@ static inline void key_check(const struct key *key)
 
 #endif
 
+#ifdef CONFIG_MKTME_KEYS
+extern void mktme_revoke_key(struct key *key);
+#else
+static inline void mktme_revoke_key(struct key *key) {}
+#endif /* CONFIG_MKTME_KEYS */
+
 #endif /* _INTERNAL_H */
diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c
index 1ffe60bb2845..86d2596ff275 100644
--- a/security/keys/keyctl.c
+++ b/security/keys/keyctl.c
@@ -363,6 +363,9 @@ long keyctl_update_key(key_serial_t id,
  * and any links to the key will be automatically garbage collected after a
  * certain amount of time (/proc/sys/kernel/keys/gc_delay).
  *
+ * The MKTME key service type checks if a memory encryption key is in use
+ * before allowing a revoke to proceed.
+ *
  * Keys with KEY_FLAG_KEEP set should not be revoked.
  *
  * If successful, 0 is returned.
@@ -387,6 +390,10 @@ long keyctl_revoke_key(key_serial_t id)
 
 	key = key_ref_to_ptr(key_ref);
 	ret = 0;
+
+	if (strcmp(key->type->name, "mktme") == 0)
+		mktme_revoke_key(key);
+
 	if (test_bit(KEY_FLAG_KEEP, &key->flags))
 		ret = -EPERM;
 	else
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index dcbce7194647..c665be860538 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -31,6 +31,52 @@ static const char * const mktme_program_err[] = {
 	"Failure to access key table",		/* 5 */
 };
 
+static int mktme_clear_programmed_key(int keyid)
+{
+	struct mktme_key_program *kprog = NULL;
+	int ret;
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog)
+		return -ENOMEM;
+
+	kprog->keyid = keyid;
+	kprog->keyid_ctrl = MKTME_KEYID_CLEAR_KEY;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret == MKTME_PROG_SUCCESS)
+		mktme_map_clear_keyid(keyid);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+/*
+ * If the key is not in use, clear the hardware programming and
+ * allow the revoke to continue by clearing KEY_FLAG_KEEP.
+ */
+void mktme_revoke_key(struct key *key)
+{
+	int keyid, vma_count;
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_serial(key->serial);
+	if (keyid <= 0)
+		goto out;
+
+	vma_count = vma_read_encrypt_ref(keyid);
+	if (vma_count > 0) {
+		pr_debug("mktme not freeing keyid[%d] encrypt_count[%d]\n",
+			 keyid, vma_count);
+		goto out;
+	}
+	if (!mktme_clear_programmed_key(keyid))
+		clear_bit(KEY_FLAG_KEEP, &key->flags);
+out:
+	mktme_map_unlock();
+}
+
 /* If a key is available, program and add the key to the software map. */
 static int mktme_program_key(key_serial_t serial,
 			     struct mktme_key_program *kprog)
@@ -193,6 +239,7 @@ int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
 
 	mktme_map_lock();
 	ret = mktme_program_key(key->serial, kprog);
+	set_bit(KEY_FLAG_KEEP, &key->flags);
 	mktme_map_unlock();
 out:
 	kzfree(options);
-- 
2.14.1

[RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[Alison Schofield]

The MKTME key service maps userspace keys to hardware keyids. Those
keys are used in a new system call that encrypts memory. The keys
need to be tightly controlled. One example is that userspace keys
should not be revoked while the hardware keyid slot is still in use.

The KEY_FLAG_KEEP bit offers good control. The mktme service uses
that flag to prevent userspace keys from going away without proper
synchronization with the mktme service type.

The problem is that we need a safe and synchronous way to revoke keys.
The way .revoke methods function now, the key service type is called late
in the revoke process for cleanup after the fact. The mktme key service
has no means to consider and perhaps reject the revoke request.

This proposal inserts the MKTME revoke call earlier into the existing
keyctl <revoke> path. If it is safe to revoke the key, MKTME key service
will turn off KEY_FLAG_KEEP and let the revoke continue and succeed.
Otherwise, not safe, KEY_FLAG_KEEP stays on, which causes the normal
path of revoke to fail.

For the MKTME Key Service, a revoke may be done safely when there are
no outstanding memory mappings encrypted with the key being revoked.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 security/keys/internal.h   |  6 ++++++
 security/keys/keyctl.c     |  7 +++++++
 security/keys/mktme_keys.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 60 insertions(+)

diff --git a/security/keys/internal.h b/security/keys/internal.h
index 9f8208dc0e55..9fb871522efe 100644
--- a/security/keys/internal.h
+++ b/security/keys/internal.h
@@ -316,4 +316,10 @@ static inline void key_check(const struct key *key)
 
 #endif
 
+#ifdef CONFIG_MKTME_KEYS
+extern void mktme_revoke_key(struct key *key);
+#else
+static inline void mktme_revoke_key(struct key *key) {}
+#endif /* CONFIG_MKTME_KEYS */
+
 #endif /* _INTERNAL_H */
diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c
index 1ffe60bb2845..86d2596ff275 100644
--- a/security/keys/keyctl.c
+++ b/security/keys/keyctl.c
@@ -363,6 +363,9 @@ long keyctl_update_key(key_serial_t id,
  * and any links to the key will be automatically garbage collected after a
  * certain amount of time (/proc/sys/kernel/keys/gc_delay).
  *
+ * The MKTME key service type checks if a memory encryption key is in use
+ * before allowing a revoke to proceed.
+ *
  * Keys with KEY_FLAG_KEEP set should not be revoked.
  *
  * If successful, 0 is returned.
@@ -387,6 +390,10 @@ long keyctl_revoke_key(key_serial_t id)
 
 	key = key_ref_to_ptr(key_ref);
 	ret = 0;
+
+	if (strcmp(key->type->name, "mktme") == 0)
+		mktme_revoke_key(key);
+
 	if (test_bit(KEY_FLAG_KEEP, &key->flags))
 		ret = -EPERM;
 	else
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index dcbce7194647..c665be860538 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -31,6 +31,52 @@ static const char * const mktme_program_err[] = {
 	"Failure to access key table",		/* 5 */
 };
 
+static int mktme_clear_programmed_key(int keyid)
+{
+	struct mktme_key_program *kprog = NULL;
+	int ret;
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog)
+		return -ENOMEM;
+
+	kprog->keyid = keyid;
+	kprog->keyid_ctrl = MKTME_KEYID_CLEAR_KEY;
+	ret = mktme_key_program(kprog, mktme_cpumask);
+	if (ret == MKTME_PROG_SUCCESS)
+		mktme_map_clear_keyid(keyid);
+	else
+		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
+
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+/*
+ * If the key is not in use, clear the hardware programming and
+ * allow the revoke to continue by clearing KEY_FLAG_KEEP.
+ */
+void mktme_revoke_key(struct key *key)
+{
+	int keyid, vma_count;
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_serial(key->serial);
+	if (keyid <= 0)
+		goto out;
+
+	vma_count = vma_read_encrypt_ref(keyid);
+	if (vma_count > 0) {
+		pr_debug("mktme not freeing keyid[%d] encrypt_count[%d]\n",
+			 keyid, vma_count);
+		goto out;
+	}
+	if (!mktme_clear_programmed_key(keyid))
+		clear_bit(KEY_FLAG_KEEP, &key->flags);
+out:
+	mktme_map_unlock();
+}
+
 /* If a key is available, program and add the key to the software map. */
 static int mktme_program_key(key_serial_t serial,
 			     struct mktme_key_program *kprog)
@@ -193,6 +239,7 @@ int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
 
 	mktme_map_lock();
 	ret = mktme_program_key(key->serial, kprog);
+	set_bit(KEY_FLAG_KEEP, &key->flags);
 	mktme_map_unlock();
 out:
 	kzfree(options);
-- 
2.14.1

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Randy Dunlap]

On 09/07/2018 03:34 PM, Alison Schofield wrote:
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Hi,
A few comments below...

> ---
>  Documentation/x86/mktme-keys.txt | 153 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> +allows encryption of the entire system memory using a single key, MKTME
> +allows multiple encryption domains, each having their own key. The main use
> +case for the feature is virtual machine isolation. The API's introduced here
> +are intended to offer flexibility to work in a wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
> +
> +==============  API Overview  ==============
> +
> +There are 2 MKTME specific API's that enable userspace to create and use
> +the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.

reference available?

> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key

                                                             key.

> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.
> +
> +===========  Usage: MKTME Key Service  ===========
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)

                                                          with it)

> +
> +    entropy=      ascii hex value entropy.

                     ASCII

> +                  This entropy will be used to generated the CPU key and

                                               to generate the

> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This 

above line ends with a space.  Please drop it.

> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.
> +
> +===========  Usage: encrypt_mprotect()  ===========
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus

                                                legacy

> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +===========  Usage: Sample Roundtrip  ===========
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);

                                               strlen(options),

> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> 


-- 
~Randy

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Randy Dunlap]

On 09/07/2018 03:34 PM, Alison Schofield wrote:
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Hi,
A few comments below...

> ---
>  Documentation/x86/mktme-keys.txt | 153 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> +allows encryption of the entire system memory using a single key, MKTME
> +allows multiple encryption domains, each having their own key. The main use
> +case for the feature is virtual machine isolation. The API's introduced here
> +are intended to offer flexibility to work in a wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
> +
> +============================  API Overview  ============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and use
> +the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.

reference available?

> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key

                                                             key.

> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.
> +
> +======================  Usage: MKTME Key Service  ======================
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)

                                                          with it)

> +
> +    entropy=      ascii hex value entropy.

                     ASCII

> +                  This entropy will be used to generated the CPU key and

                                               to generate the

> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This 

above line ends with a space.  Please drop it.

> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.
> +
> +======================  Usage: encrypt_mprotect()  ======================
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus

                                                legacy

> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +======================  Usage: Sample Roundtrip  ======================
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);

                                               strlen(options),

> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> 


-- 
~Randy

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Randy Dunlap]

On 09/07/2018 03:34 PM, Alison Schofield wrote:
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Hi,
A few comments below...

> ---
>  Documentation/x86/mktme-keys.txt | 153 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> +allows encryption of the entire system memory using a single key, MKTME
> +allows multiple encryption domains, each having their own key. The main use
> +case for the feature is virtual machine isolation. The API's introduced here
> +are intended to offer flexibility to work in a wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
> +
> +============================  API Overview  ============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and use
> +the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.

reference available?

> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key

                                                             key.

> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.
> +
> +======================  Usage: MKTME Key Service  ======================
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)

                                                          with it)

> +
> +    entropy=      ascii hex value entropy.

                     ASCII

> +                  This entropy will be used to generated the CPU key and

                                               to generate the

> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This 

above line ends with a space.  Please drop it.

> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.
> +
> +======================  Usage: encrypt_mprotect()  ======================
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus

                                                legacy

> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +======================  Usage: Sample Roundtrip  ======================
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);

                                               strlen(options),

> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> 


-- 
~Randy

RE: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> owner@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:23 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)
> 
> Seeking comments on the APIs supporting MKTME on future Intel platforms.
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> memory encryption on upcoming Intel platforms. Whereas TME allows
> encryption of the entire system memory using a single key, MKTME allows
> mulitple encryption domains, each having their own key. While the main use
> case for the feature is virtual machine isolation, the API needs the flexibility to
> work for a wide range of use cases.
> 
> This RFC presents the 2 API additions that enable userspace to:
>  1) Create Encryption Keys: Kernel Key Service type "mktme"
>  2) Use the Encryption Keys: system call encrypt_mprotect()
> 
> In order to share between: the Kernel Key Service, the new system call, and the
> existing mm code, helper functions were created in arch/x86/mktme

IMHO, we can separate this series into 2 parts, as you did above, and send out them separately. The reason is, in general I think adding new MKTME type to key retention services is not that related to memory management code, namely the encrypt_mprotect() API part.

So if we split the two parts and send them out separately, the first part can be reviewed by keyring and security guys, without involving mm guys, and the encrypt_mprotect() part can be more reviewed more by mm guys. 

And since encrypt_mprotect() is a new syscall, you may need to add more lists for the review, ie, linux-api, and maybe linux-kernel as well.

Thanks,
-Kai

> 
> This patchset is built upon Kirill Shutemov's patchset for the core MKTME
> support. You can find that here:
> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip
> 
> 
> Alison Schofield (12):
>   docs/x86: Document the Multi-Key Total Memory Encryption API
>   mm: Generalize the mprotect implementation to support extensions
>   syscall/x86: Wire up a new system call for memory encryption keys
>   x86/mm: Add helper functions to manage memory encryption keys
>   x86/mm: Add a helper function to set keyid bits in encrypted VMA's
>   mm: Add the encrypt_mprotect() system call
>   x86/mm: Add helper functions to track encrypted VMA's
>   mm: Track VMA's in use for each memory encryption keyid
>   mm: Restrict memory encryption to anonymous VMA's
>   x86/pconfig: Program memory encryption keys on a system-wide basis
>   keys/mktme: Add a new key service type for memory encryption keys
>   keys/mktme: Do not revoke in use memory encryption keys
> 
>  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
>  arch/x86/Kconfig                       |   1 +
>  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
>  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
>  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
>  arch/x86/include/asm/mktme.h           |  21 +++
>  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
>  fs/exec.c                              |   4 +-
>  include/keys/mktme-type.h              |  28 +++
>  include/linux/key.h                    |   2 +
>  include/linux/mm.h                     |   9 +-
>  include/linux/syscalls.h               |   2 +
>  include/uapi/asm-generic/unistd.h      |   4 +-
>  kernel/fork.c                          |   2 +
>  kernel/sys_ni.c                        |   2 +
>  mm/mmap.c                              |  12 ++
>  mm/mprotect.c                          |  93 +++++++++-
>  mm/nommu.c                             |   4 +
>  security/keys/Kconfig                  |  11 ++
>  security/keys/Makefile                 |   1 +
>  security/keys/internal.h               |   6 +
>  security/keys/keyctl.c                 |   7 +
>  security/keys/mktme_keys.c             | 325
> +++++++++++++++++++++++++++++++++
>  23 files changed, 855 insertions(+), 17 deletions(-)  create mode 100644
> Documentation/x86/mktme-keys.txt  create mode 100644 include/keys/mktme-
> type.h  create mode 100644 security/keys/mktme_keys.c
> 
> --
> 2.14.1

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> owner at vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:23 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)
> 
> Seeking comments on the APIs supporting MKTME on future Intel platforms.
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> memory encryption on upcoming Intel platforms. Whereas TME allows
> encryption of the entire system memory using a single key, MKTME allows
> mulitple encryption domains, each having their own key. While the main use
> case for the feature is virtual machine isolation, the API needs the flexibility to
> work for a wide range of use cases.
> 
> This RFC presents the 2 API additions that enable userspace to:
>  1) Create Encryption Keys: Kernel Key Service type "mktme"
>  2) Use the Encryption Keys: system call encrypt_mprotect()
> 
> In order to share between: the Kernel Key Service, the new system call, and the
> existing mm code, helper functions were created in arch/x86/mktme

IMHO, we can separate this series into 2 parts, as you did above, and send out them separately. The reason is, in general I think adding new MKTME type to key retention services is not that related to memory management code, namely the encrypt_mprotect() API part.

So if we split the two parts and send them out separately, the first part can be reviewed by keyring and security guys, without involving mm guys, and the encrypt_mprotect() part can be more reviewed more by mm guys. 

And since encrypt_mprotect() is a new syscall, you may need to add more lists for the review, ie, linux-api, and maybe linux-kernel as well.

Thanks,
-Kai

> 
> This patchset is built upon Kirill Shutemov's patchset for the core MKTME
> support. You can find that here:
> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip
> 
> 
> Alison Schofield (12):
>   docs/x86: Document the Multi-Key Total Memory Encryption API
>   mm: Generalize the mprotect implementation to support extensions
>   syscall/x86: Wire up a new system call for memory encryption keys
>   x86/mm: Add helper functions to manage memory encryption keys
>   x86/mm: Add a helper function to set keyid bits in encrypted VMA's
>   mm: Add the encrypt_mprotect() system call
>   x86/mm: Add helper functions to track encrypted VMA's
>   mm: Track VMA's in use for each memory encryption keyid
>   mm: Restrict memory encryption to anonymous VMA's
>   x86/pconfig: Program memory encryption keys on a system-wide basis
>   keys/mktme: Add a new key service type for memory encryption keys
>   keys/mktme: Do not revoke in use memory encryption keys
> 
>  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
>  arch/x86/Kconfig                       |   1 +
>  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
>  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
>  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
>  arch/x86/include/asm/mktme.h           |  21 +++
>  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
>  fs/exec.c                              |   4 +-
>  include/keys/mktme-type.h              |  28 +++
>  include/linux/key.h                    |   2 +
>  include/linux/mm.h                     |   9 +-
>  include/linux/syscalls.h               |   2 +
>  include/uapi/asm-generic/unistd.h      |   4 +-
>  kernel/fork.c                          |   2 +
>  kernel/sys_ni.c                        |   2 +
>  mm/mmap.c                              |  12 ++
>  mm/mprotect.c                          |  93 +++++++++-
>  mm/nommu.c                             |   4 +
>  security/keys/Kconfig                  |  11 ++
>  security/keys/Makefile                 |   1 +
>  security/keys/internal.h               |   6 +
>  security/keys/keyctl.c                 |   7 +
>  security/keys/mktme_keys.c             | 325
> +++++++++++++++++++++++++++++++++
>  23 files changed, 855 insertions(+), 17 deletions(-)  create mode 100644
> Documentation/x86/mktme-keys.txt  create mode 100644 include/keys/mktme-
> type.h  create mode 100644 security/keys/mktme_keys.c
> 
> --
> 2.14.1

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:34 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  Documentation/x86/mktme-keys.txt | 153
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory
> +Encryption) allows encryption of the entire system memory using a
> +single key, MKTME allows multiple encryption domains, each having their
> +own key. The main use case for the feature is virtual machine
> +isolation. The API's introduced here are intended to offer flexibility to work in a
> wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> +Total-Memory-Encryption-Spec.pdf
> +
> +==============  API Overview
> +==============
> +
> +There are 2 MKTME specific API's that enable userspace to create and
> +use the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.
> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.

Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key
> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.

I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

> +
> +===========  Usage: MKTME Key Service
> +===========
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)
> +
> +    entropy=      ascii hex value entropy.
> +                  This entropy will be used to generated the CPU key and
> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This
> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.

This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.

Actually, simple 'XOR' is used to generate the final key:

case KEYID_SET_KEY_RANDOM:
	......
	(* Mix user supplied entropy to the data key and tweak key *)
	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];

So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.

In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.

Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Thanks,
-Kai
> +
> +===========  Usage: encrypt_mprotect()
> +===========
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus
> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +===========  Usage: Sample Roundtrip
> ===========
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);
> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> --
> 2.14.1

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:34 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  Documentation/x86/mktme-keys.txt | 153
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory
> +Encryption) allows encryption of the entire system memory using a
> +single key, MKTME allows multiple encryption domains, each having their
> +own key. The main use case for the feature is virtual machine
> +isolation. The API's introduced here are intended to offer flexibility to work in a
> wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> +Total-Memory-Encryption-Spec.pdf
> +
> +============================  API Overview
> +============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and
> +use the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.
> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.

Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key
> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.

I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

> +
> +======================  Usage: MKTME Key Service
> +======================
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)
> +
> +    entropy=      ascii hex value entropy.
> +                  This entropy will be used to generated the CPU key and
> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This
> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.

This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.

Actually, simple 'XOR' is used to generate the final key:

case KEYID_SET_KEY_RANDOM:
	......
	(* Mix user supplied entropy to the data key and tweak key *)
	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];

So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.

In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.

Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Thanks,
-Kai
> +
> +======================  Usage: encrypt_mprotect()
> +======================
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus
> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +======================  Usage: Sample Roundtrip
> ======================
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);
> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> --
> 2.14.1

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:34 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  Documentation/x86/mktme-keys.txt | 153
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory
> +Encryption) allows encryption of the entire system memory using a
> +single key, MKTME allows multiple encryption domains, each having their
> +own key. The main use case for the feature is virtual machine
> +isolation. The API's introduced here are intended to offer flexibility to work in a
> wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> +Total-Memory-Encryption-Spec.pdf
> +
> +============================  API Overview
> +============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and
> +use the memory encryption keys:
> +
> +1) Kernel Key Service: MKTME Type
> +
> +   MKTME is a new key type added to the existing Kernel Key Services
> +   to support the memory encryption keys. The MKTME service manages
> +   the addition and removal of MKTME keys. It maps userspace keys
> +   to hardware keyids and programs the hardware with user requested
> +   encryption parameters.
> +
> +   o An understanding of the Kernel Key Service is required in order
> +     to use the MKTME key type as it is a subset of that service.
> +
> +   o MKTME keys are a limited resource. There is a single pool of
> +     MKTME keys for a system and that pool can be from 3 to 63 keys.

Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

> +     With that in mind, userspace may take advantage of the kernel
> +     key services sharing and permissions model for userspace keys.
> +     One key can be shared as long as each user has the permission
> +     of "KEY_NEED_VIEW" to use it.
> +
> +   o MKTME key type uses capabilities to restrict the allocation
> +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> +
> +   o The MKTME key service blocks kernel key service commands that
> +     could lead to reprogramming of in use keys, or loss of keys from
> +     the pool. This means MKTME does not allow a key to be invalidated,
> +     unlinked, or timed out. These operations are blocked by MKTME as
> +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> +
> +   o MKTME does not support the keyctl option of UPDATE. Userspace
> +     may change the programming of a key by revoking it and adding
> +     a new key with the updated encryption options (or vice-versa).
> +
> +2) System Call: encrypt_mprotect()
> +
> +   MKTME encryption is requested by calling encrypt_mprotect(). The
> +   caller passes the serial number to a previously allocated and
> +   programmed encryption key. That handle was created with the MKTME
> +   Key Service.
> +
> +   o The caller must have KEY_NEED_VIEW permission on the key
> +
> +   o The range of memory that is to be protected must be mapped as
> +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> +     fails with EINVAL.
> +
> +   o As an extension to the existing mprotect() system call,
> +     encrypt_mprotect() supports the legacy mprotect behavior plus
> +     the enabling of memory encryption. That means that in addition
> +     to encrypting the memory, the protection flags will be updated
> +     as requested in the call.
> +
> +   o Additional mprotect() calls to memory already protected with
> +     MKTME will not alter the MKTME status.

I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

> +
> +======================  Usage: MKTME Key Service
> +======================
> +
> +MKTME is enabled on supported Intel platforms by selecting
> +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> +
> +Allocating MKTME Keys via command line or system call:
> +    keyctl add mktme name "[options]" ring
> +
> +    key_serial_t add_key(const char *type, const char *description,
> +                         const void *payload, size_t plen,
> +                         key_serial_t keyring);
> +
> +Revoking MKTME Keys via command line or system call::
> +   keyctl revoke <key>
> +
> +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> +
> +Options Field Definition:
> +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> +		  generated key if a userkey is not defined here.
> +
> +    algorithm=    Encryption algorithm name as a string.
> +		  Valid algorithm: "aes-xts-128"
> +
> +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> +                  userkey...  (need to be clear here that this is being sent
> +                  to the hardware - kernel not messing w it)
> +
> +    entropy=      ascii hex value entropy.
> +                  This entropy will be used to generated the CPU key and
> +		  the tweak key when CPU generated key is requested.
> +
> +Algorithm Dependencies:
> +    AES-XTS 128 is the only supported algorithm.
> +    There are only 2 ways that AES-XTS 128 may be used:
> +
> +    1) User specified encryption key
> +	- The user specified encryption key must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- A tweak key must be specified and it must be exactly
> +	  16 ASCII Hex bytes (128 bits).
> +	- No entropy field is accepted.
> +
> +    2) CPU generated encryption key
> +	- When no user specified encryption key is provided, the
> +	  default encryption key will be CPU generated.
> +	- User must specify 16 ASCII Hex bytes of entropy. This
> +	  entropy will be used by the CPU to generate both the
> +	  encryption key and the tweak key.
> +	- No entropy field is accepted.

This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.

Actually, simple 'XOR' is used to generate the final key:

case KEYID_SET_KEY_RANDOM:
	......
	(* Mix user supplied entropy to the data key and tweak key *)
	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];

So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.

In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.

Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Thanks,
-Kai
> +
> +======================  Usage: encrypt_mprotect()
> +======================
> +
> +System Call encrypt_mprotect()::
> +
> +    This system call is an extension of the existing mprotect() system
> +    call. It requires the same parameters as legary mprotect() plus
> +    one additional parameter, the keyid. Userspace must provide the
> +    key serial number assigned through the kernel key service.
> +
> +    int encrypt_mprotect(void *addr, size_t len, int prot, int keyid);
> +
> +======================  Usage: Sample Roundtrip
> ======================
> +
> +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> +
> +  Add a key:
> +        key = add_key(mktme, name, options, strlen(option), keyring);
> +
> +  Map memory:
> +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory:
> +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> +
> +  Use protected memory:
> +        ................
> +
> +  Free memory:
> +        ret = munmap(ptr, size);
> +
> +  Revoke key:
> +        ret = keyctl(KEYCTL_REVOKE, key);
> +
> --
> 2.14.1

RE: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Huang, Kai]

> -----Original Message-----
> From: Schofield, Alison
> Sent: Saturday, September 8, 2018 10:38 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 10/12] x86/pconfig: Program memory encryption keys on a
> system-wide basis
> 
> The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys as a
> system wide single pool of keys. The hardware, however, manages the keys on a
> per physical package basis. Each physical package maintains a key table that all
> CPU's in that package share.
> 
> In order to maintain the consistent, system wide view that the kernel requires,
> program all physical packages during a key program request.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/intel_pconfig.h | 42
> ++++++++++++++++++++++++++++++------
>  1 file changed, 36 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/include/asm/intel_pconfig.h
> b/arch/x86/include/asm/intel_pconfig.h
> index 3cb002b1d0f9..d3bf0a297e89 100644
> --- a/arch/x86/include/asm/intel_pconfig.h
> +++ b/arch/x86/include/asm/intel_pconfig.h
> @@ -3,6 +3,7 @@
> 
>  #include <asm/asm.h>
>  #include <asm/processor.h>
> +#include <linux/cpu.h>
> 
>  enum pconfig_target {
>  	INVALID_TARGET	= 0,
> @@ -47,19 +48,48 @@ struct mktme_key_program {
>  	u8 key_field_2[64];
>  } __packed __aligned(256);
> 
> -static inline int mktme_key_program(struct mktme_key_program
> *key_program)
> +struct mktme_key_program_info {
> +	struct mktme_key_program *key_program;
> +	unsigned long status;
> +};
> +
> +static void mktme_package_program(void *key_program_info)
>  {
> +	struct mktme_key_program_info *info = key_program_info;
>  	unsigned long rax = MKTME_KEY_PROGRAM;
> 
> +	asm volatile(PCONFIG
> +		: "=a" (rax), "=b" (info->key_program)

Why do we need "=b" (info->key_program)? To me PCONFIG only reads from rbx, but won't write to it.

> +		: "0" (rax), "1" (info->key_program)
> +		: "memory", "cc");
> +
> +	if (rax != MKTME_PROG_SUCCESS)
> +		WRITE_ONCE(info->status, rax);
> +}
> +
> +/*
> + * MKTME keys are managed as a system-wide single pool of keys.
> + * In the hardware, each physical package maintains a separate key
> + * table. Program all physical packages with the same key info to
> + * maintain that system-wide kernel view.
> + */
> +static inline int mktme_key_program(struct mktme_key_program
> *key_program,
> +				    cpumask_var_t mktme_cpumask)
> +{
> +	struct mktme_key_program_info info = {
> +		.key_program = key_program,
> +		.status = MKTME_PROG_SUCCESS,
> +	};
> +
>  	if (!pconfig_target_supported(MKTME_TARGET))
>  		return -ENXIO;
> 
> -	asm volatile(PCONFIG
> -		: "=a" (rax), "=b" (key_program)
> -		: "0" (rax), "1" (key_program)
> -		: "memory", "cc");
> +	get_online_cpus();
> +	on_each_cpu_mask(mktme_cpumask, mktme_package_program,
> +			 &info, 1);
> +	put_online_cpus();

What is the value of 'mktme_cpumask'? Does it only contain one core for each package? 

Since we are using 'on_each_cpu_mask', I think  we should make sure only one core is set for each node in 'mktme_cpumask'. Otherwise we have to deal with 'DEVICE_BUSY' case, since if one core is already in middle of PCONFIG, the other PCONFIGs on the same node would get 'DEVICE_BUSY' error, but this doesn't mean PCONFIG has failed on that node.

Thanks,
-Kai
> 
> -	return rax;
> +	return info.status;
>  }
> 
>  #endif	/* _ASM_X86_INTEL_PCONFIG_H */
> --
> 2.14.1

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Huang, Kai]

> -----Original Message-----
> From: Schofield, Alison
> Sent: Saturday, September 8, 2018 10:38 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 10/12] x86/pconfig: Program memory encryption keys on a
> system-wide basis
> 
> The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys as a
> system wide single pool of keys. The hardware, however, manages the keys on a
> per physical package basis. Each physical package maintains a key table that all
> CPU's in that package share.
> 
> In order to maintain the consistent, system wide view that the kernel requires,
> program all physical packages during a key program request.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/intel_pconfig.h | 42
> ++++++++++++++++++++++++++++++------
>  1 file changed, 36 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/include/asm/intel_pconfig.h
> b/arch/x86/include/asm/intel_pconfig.h
> index 3cb002b1d0f9..d3bf0a297e89 100644
> --- a/arch/x86/include/asm/intel_pconfig.h
> +++ b/arch/x86/include/asm/intel_pconfig.h
> @@ -3,6 +3,7 @@
> 
>  #include <asm/asm.h>
>  #include <asm/processor.h>
> +#include <linux/cpu.h>
> 
>  enum pconfig_target {
>  	INVALID_TARGET	= 0,
> @@ -47,19 +48,48 @@ struct mktme_key_program {
>  	u8 key_field_2[64];
>  } __packed __aligned(256);
> 
> -static inline int mktme_key_program(struct mktme_key_program
> *key_program)
> +struct mktme_key_program_info {
> +	struct mktme_key_program *key_program;
> +	unsigned long status;
> +};
> +
> +static void mktme_package_program(void *key_program_info)
>  {
> +	struct mktme_key_program_info *info = key_program_info;
>  	unsigned long rax = MKTME_KEY_PROGRAM;
> 
> +	asm volatile(PCONFIG
> +		: "=a" (rax), "=b" (info->key_program)

Why do we need "=b" (info->key_program)? To me PCONFIG only reads from rbx, but won't write to it.

> +		: "0" (rax), "1" (info->key_program)
> +		: "memory", "cc");
> +
> +	if (rax != MKTME_PROG_SUCCESS)
> +		WRITE_ONCE(info->status, rax);
> +}
> +
> +/*
> + * MKTME keys are managed as a system-wide single pool of keys.
> + * In the hardware, each physical package maintains a separate key
> + * table. Program all physical packages with the same key info to
> + * maintain that system-wide kernel view.
> + */
> +static inline int mktme_key_program(struct mktme_key_program
> *key_program,
> +				    cpumask_var_t mktme_cpumask)
> +{
> +	struct mktme_key_program_info info = {
> +		.key_program = key_program,
> +		.status = MKTME_PROG_SUCCESS,
> +	};
> +
>  	if (!pconfig_target_supported(MKTME_TARGET))
>  		return -ENXIO;
> 
> -	asm volatile(PCONFIG
> -		: "=a" (rax), "=b" (key_program)
> -		: "0" (rax), "1" (key_program)
> -		: "memory", "cc");
> +	get_online_cpus();
> +	on_each_cpu_mask(mktme_cpumask, mktme_package_program,
> +			 &info, 1);
> +	put_online_cpus();

What is the value of 'mktme_cpumask'? Does it only contain one core for each package? 

Since we are using 'on_each_cpu_mask', I think  we should make sure only one core is set for each node in 'mktme_cpumask'. Otherwise we have to deal with 'DEVICE_BUSY' case, since if one core is already in middle of PCONFIG, the other PCONFIGs on the same node would get 'DEVICE_BUSY' error, but this doesn't mean PCONFIG has failed on that node.

Thanks,
-Kai
> 
> -	return rax;
> +	return info.status;
>  }
> 
>  #endif	/* _ASM_X86_INTEL_PCONFIG_H */
> --
> 2.14.1

RE: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> security-module@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:36 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> Define a global mapping structure to track the mapping of userspace keys to
> hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> This data will be used for the memory encryption system call and the kernel key
> service API.
> 
> Implement helper functions to access this mapping structure and make them
> visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h | 11 ++++++
>  arch/x86/mm/mktme.c          | 85
> ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 96 insertions(+)

Maybe it's better to put those changes to include/keys/mktme-type.h, and security/keys/mktme_key.c? It seems you don't have to involve linux-mm and x86 guys by doing so?

Thanks,
-Kai
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbfbd955da98..f6acd551457f 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> mktme_nr_keyids;  extern int mktme_keyid_shift;
> 
> +/* Manage mappings between hardware keyids and userspace keys */ extern
> +int mktme_map_alloc(void); extern void mktme_map_free(void); extern
> +void mktme_map_lock(void); extern void mktme_map_unlock(void); extern
> +int mktme_map_get_free_keyid(void); extern void
> +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> +keyid, unsigned int serial); extern int
> +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> +mktme_map_serial_from_keyid(int keyid);
> +
>  extern struct page_ext_operations page_mktme_ops;
> 
>  #define page_keyid page_keyid
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> 660caf6a5ce1..5246d8323359 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> 
> +/*
> + * struct mktme_mapping and the mktme_map_* functions manage the
> +mapping
> + * of userspace keys to hardware keyids in MKTME. They are used by the
> + * the encrypt_mprotect system call and the MKTME Key Service API.
> + */
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW state */
> +	unsigned int	mapped_keyids;
> +	unsigned int	serial[];
> +};
> +
> +struct mktme_mapping *mktme_map;
> +
> +static inline long mktme_map_size(void) {
> +	long size = 0;
> +
> +	size += sizeof(mktme_map);
> +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> +	return size;
> +}
> +
> +int mktme_map_alloc(void)
> +{
> +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> +	if (!mktme_map)
> +		return 0;
> +	mutex_init(&mktme_map->lock);
> +	return 1;
> +}
> +
> +void mktme_map_free(void)
> +{
> +	kfree(mktme_map);
> +}
> +
> +void mktme_map_lock(void)
> +{
> +	mutex_lock(&mktme_map->lock);
> +}
> +
> +void mktme_map_unlock(void)
> +{
> +	mutex_unlock(&mktme_map->lock);
> +}
> +
> +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}
> +
> +void mktme_map_clear_keyid(int keyid)
> +{
> +	mktme_map->serial[keyid] = 0;
> +	mktme_map->mapped_keyids--;
> +}
> +
> +unsigned int mktme_map_serial_from_keyid(int keyid) {
> +	return mktme_map->serial[keyid];
> +}
> +
> +int mktme_map_keyid_from_serial(unsigned int serial) {
> +	int i;
> +
> +	for (i = 1; i < mktme_nr_keyids; i++)
> +		if (mktme_map->serial[i] = serial)
> +			return i;
> +	return 0;
> +}
> +
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i < mktme_nr_keyids; i++)
> +			if (mktme_map->serial[i] = 0)
> +				return i;
> +	}
> +	return 0;
> +}
> +
>  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
>  	int i;
> --
> 2.14.1

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module at vger.kernel.org [mailto:owner-linux-
> security-module at vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:36 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> Define a global mapping structure to track the mapping of userspace keys to
> hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> This data will be used for the memory encryption system call and the kernel key
> service API.
> 
> Implement helper functions to access this mapping structure and make them
> visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h | 11 ++++++
>  arch/x86/mm/mktme.c          | 85
> ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 96 insertions(+)

Maybe it's better to put those changes to include/keys/mktme-type.h, and security/keys/mktme_key.c? It seems you don't have to involve linux-mm and x86 guys by doing so?

Thanks,
-Kai
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbfbd955da98..f6acd551457f 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> mktme_nr_keyids;  extern int mktme_keyid_shift;
> 
> +/* Manage mappings between hardware keyids and userspace keys */ extern
> +int mktme_map_alloc(void); extern void mktme_map_free(void); extern
> +void mktme_map_lock(void); extern void mktme_map_unlock(void); extern
> +int mktme_map_get_free_keyid(void); extern void
> +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> +keyid, unsigned int serial); extern int
> +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> +mktme_map_serial_from_keyid(int keyid);
> +
>  extern struct page_ext_operations page_mktme_ops;
> 
>  #define page_keyid page_keyid
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> 660caf6a5ce1..5246d8323359 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> 
> +/*
> + * struct mktme_mapping and the mktme_map_* functions manage the
> +mapping
> + * of userspace keys to hardware keyids in MKTME. They are used by the
> + * the encrypt_mprotect system call and the MKTME Key Service API.
> + */
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW state */
> +	unsigned int	mapped_keyids;
> +	unsigned int	serial[];
> +};
> +
> +struct mktme_mapping *mktme_map;
> +
> +static inline long mktme_map_size(void) {
> +	long size = 0;
> +
> +	size += sizeof(mktme_map);
> +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> +	return size;
> +}
> +
> +int mktme_map_alloc(void)
> +{
> +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> +	if (!mktme_map)
> +		return 0;
> +	mutex_init(&mktme_map->lock);
> +	return 1;
> +}
> +
> +void mktme_map_free(void)
> +{
> +	kfree(mktme_map);
> +}
> +
> +void mktme_map_lock(void)
> +{
> +	mutex_lock(&mktme_map->lock);
> +}
> +
> +void mktme_map_unlock(void)
> +{
> +	mutex_unlock(&mktme_map->lock);
> +}
> +
> +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}
> +
> +void mktme_map_clear_keyid(int keyid)
> +{
> +	mktme_map->serial[keyid] = 0;
> +	mktme_map->mapped_keyids--;
> +}
> +
> +unsigned int mktme_map_serial_from_keyid(int keyid) {
> +	return mktme_map->serial[keyid];
> +}
> +
> +int mktme_map_keyid_from_serial(unsigned int serial) {
> +	int i;
> +
> +	for (i = 1; i < mktme_nr_keyids; i++)
> +		if (mktme_map->serial[i] == serial)
> +			return i;
> +	return 0;
> +}
> +
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i < mktme_nr_keyids; i++)
> +			if (mktme_map->serial[i] == 0)
> +				return i;
> +	}
> +	return 0;
> +}
> +
>  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
>  	int i;
> --
> 2.14.1

RE: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> security-module@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:36 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> Define a global mapping structure to track the mapping of userspace keys to
> hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> This data will be used for the memory encryption system call and the kernel key
> service API.
> 
> Implement helper functions to access this mapping structure and make them
> visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h | 11 ++++++
>  arch/x86/mm/mktme.c          | 85
> ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 96 insertions(+)

Maybe it's better to put those changes to include/keys/mktme-type.h, and security/keys/mktme_key.c? It seems you don't have to involve linux-mm and x86 guys by doing so?

Thanks,
-Kai
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbfbd955da98..f6acd551457f 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> mktme_nr_keyids;  extern int mktme_keyid_shift;
> 
> +/* Manage mappings between hardware keyids and userspace keys */ extern
> +int mktme_map_alloc(void); extern void mktme_map_free(void); extern
> +void mktme_map_lock(void); extern void mktme_map_unlock(void); extern
> +int mktme_map_get_free_keyid(void); extern void
> +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> +keyid, unsigned int serial); extern int
> +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> +mktme_map_serial_from_keyid(int keyid);
> +
>  extern struct page_ext_operations page_mktme_ops;
> 
>  #define page_keyid page_keyid
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> 660caf6a5ce1..5246d8323359 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> 
> +/*
> + * struct mktme_mapping and the mktme_map_* functions manage the
> +mapping
> + * of userspace keys to hardware keyids in MKTME. They are used by the
> + * the encrypt_mprotect system call and the MKTME Key Service API.
> + */
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW state */
> +	unsigned int	mapped_keyids;
> +	unsigned int	serial[];
> +};
> +
> +struct mktme_mapping *mktme_map;
> +
> +static inline long mktme_map_size(void) {
> +	long size = 0;
> +
> +	size += sizeof(mktme_map);
> +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> +	return size;
> +}
> +
> +int mktme_map_alloc(void)
> +{
> +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> +	if (!mktme_map)
> +		return 0;
> +	mutex_init(&mktme_map->lock);
> +	return 1;
> +}
> +
> +void mktme_map_free(void)
> +{
> +	kfree(mktme_map);
> +}
> +
> +void mktme_map_lock(void)
> +{
> +	mutex_lock(&mktme_map->lock);
> +}
> +
> +void mktme_map_unlock(void)
> +{
> +	mutex_unlock(&mktme_map->lock);
> +}
> +
> +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}
> +
> +void mktme_map_clear_keyid(int keyid)
> +{
> +	mktme_map->serial[keyid] = 0;
> +	mktme_map->mapped_keyids--;
> +}
> +
> +unsigned int mktme_map_serial_from_keyid(int keyid) {
> +	return mktme_map->serial[keyid];
> +}
> +
> +int mktme_map_keyid_from_serial(unsigned int serial) {
> +	int i;
> +
> +	for (i = 1; i < mktme_nr_keyids; i++)
> +		if (mktme_map->serial[i] == serial)
> +			return i;
> +	return 0;
> +}
> +
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i < mktme_nr_keyids; i++)
> +			if (mktme_map->serial[i] == 0)
> +				return i;
> +	}
> +	return 0;
> +}
> +
>  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
>  	int i;
> --
> 2.14.1

RE: [RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> owner@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:37 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's
> 
> In order to safely manage the usage of memory encryption keys, VMA's using
> each keyid need to be tracked. This tracking allows the Kernel Key Service to
> know when the keyid resource is actually in use, or when it is idle and may be
> considered for reuse.
> 
> Define a global atomic encrypt_count array to track the number of VMA's
> oustanding for each encryption keyid.
> 
> Implement helper functions to manipulate this encrypt_count array.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h |  7 +++++++
>  arch/x86/mm/mktme.c          | 39
> +++++++++++++++++++++++++++++++++++++++
>  include/linux/mm.h           |  2 ++
>  3 files changed, 48 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index b707f800b68f..5f3fa0c39c1c 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -16,6 +16,13 @@ extern int mktme_keyid_shift;
>  /* Set the encryption keyid bits in a VMA */  extern void
> mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
> 
> +/* Manage the references to outstanding VMA's per encryption key */
> +extern int vma_alloc_encrypt_array(void); extern void
> +vma_free_encrypt_array(void); extern int vma_read_encrypt_ref(int
> +keyid); extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
> +extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
> +
>  /* Manage mappings between hardware keyids and userspace keys */  extern
> int mktme_map_alloc(void);  extern void mktme_map_free(void); diff --git
> a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> 5ee7f37e9cd0..5690ef51a79a 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -163,6 +163,45 @@ int mktme_map_get_free_keyid(void)
>  	return 0;
>  }
> 
> +/*
> + *  Helper functions manage the encrypt_count[] array that tracks the
> + *  VMA's outstanding for each encryption keyid. The gets & puts are
> + *  used in core mm code that allocates and free's VMA's. The alloc,
> + *  free, and read functions are used by the MKTME key service to
> + *  manage key allocation and programming.
> + */
> +atomic_t *encrypt_count;
> +
> +int vma_alloc_encrypt_array(void)
> +{
> +	encrypt_count = kcalloc(mktme_nr_keyids, sizeof(atomic_t),
> GFP_KERNEL);
> +	if (!encrypt_count)
> +		return -ENOMEM;
> +	return 0;
> +}
> +
> +void vma_free_encrypt_array(void)
> +{
> +	kfree(encrypt_count);
> +}
> +
> +int vma_read_encrypt_ref(int keyid)
> +{
> +	return atomic_read(&encrypt_count[keyid]);
> +}

I think it's better to move above to security/keys/mktme_keys.c w/ appropriate renaming.  

Thanks,
-Kai
> +
> +void vma_get_encrypt_ref(struct vm_area_struct *vma) {
> +	if (vma_keyid(vma))
> +		atomic_inc(&encrypt_count[vma_keyid(vma)]);
> +}
> +
> +void vma_put_encrypt_ref(struct vm_area_struct *vma) {
> +	if (vma_keyid(vma))
> +		atomic_dec(&encrypt_count[vma_keyid(vma)]);
> +}
> +
>  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
>  	int i;
> diff --git a/include/linux/mm.h b/include/linux/mm.h index
> 0f9422c7841e..b217c699dbab 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2803,6 +2803,8 @@ static inline void setup_nr_node_ids(void) {}  #ifndef
> CONFIG_X86_INTEL_MKTME  static inline void mprotect_set_encrypt(struct
> vm_area_struct *vma,
>  					int newkeyid) {}
> +static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
> +static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
>  #endif /* CONFIG_X86_INTEL_MKTME */
>  #endif /* __KERNEL__ */
>  #endif /* _LINUX_MM_H */
> --
> 2.14.1

[RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> owner at vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:37 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 07/12] x86/mm: Add helper functions to track encrypted VMA's
> 
> In order to safely manage the usage of memory encryption keys, VMA's using
> each keyid need to be tracked. This tracking allows the Kernel Key Service to
> know when the keyid resource is actually in use, or when it is idle and may be
> considered for reuse.
> 
> Define a global atomic encrypt_count array to track the number of VMA's
> oustanding for each encryption keyid.
> 
> Implement helper functions to manipulate this encrypt_count array.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h |  7 +++++++
>  arch/x86/mm/mktme.c          | 39
> +++++++++++++++++++++++++++++++++++++++
>  include/linux/mm.h           |  2 ++
>  3 files changed, 48 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index b707f800b68f..5f3fa0c39c1c 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -16,6 +16,13 @@ extern int mktme_keyid_shift;
>  /* Set the encryption keyid bits in a VMA */  extern void
> mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid);
> 
> +/* Manage the references to outstanding VMA's per encryption key */
> +extern int vma_alloc_encrypt_array(void); extern void
> +vma_free_encrypt_array(void); extern int vma_read_encrypt_ref(int
> +keyid); extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
> +extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
> +
>  /* Manage mappings between hardware keyids and userspace keys */  extern
> int mktme_map_alloc(void);  extern void mktme_map_free(void); diff --git
> a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> 5ee7f37e9cd0..5690ef51a79a 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -163,6 +163,45 @@ int mktme_map_get_free_keyid(void)
>  	return 0;
>  }
> 
> +/*
> + *  Helper functions manage the encrypt_count[] array that tracks the
> + *  VMA's outstanding for each encryption keyid. The gets & puts are
> + *  used in core mm code that allocates and free's VMA's. The alloc,
> + *  free, and read functions are used by the MKTME key service to
> + *  manage key allocation and programming.
> + */
> +atomic_t *encrypt_count;
> +
> +int vma_alloc_encrypt_array(void)
> +{
> +	encrypt_count = kcalloc(mktme_nr_keyids, sizeof(atomic_t),
> GFP_KERNEL);
> +	if (!encrypt_count)
> +		return -ENOMEM;
> +	return 0;
> +}
> +
> +void vma_free_encrypt_array(void)
> +{
> +	kfree(encrypt_count);
> +}
> +
> +int vma_read_encrypt_ref(int keyid)
> +{
> +	return atomic_read(&encrypt_count[keyid]);
> +}

I think it's better to move above to security/keys/mktme_keys.c w/ appropriate renaming.  

Thanks,
-Kai
> +
> +void vma_get_encrypt_ref(struct vm_area_struct *vma) {
> +	if (vma_keyid(vma))
> +		atomic_inc(&encrypt_count[vma_keyid(vma)]);
> +}
> +
> +void vma_put_encrypt_ref(struct vm_area_struct *vma) {
> +	if (vma_keyid(vma))
> +		atomic_dec(&encrypt_count[vma_keyid(vma)]);
> +}
> +
>  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
>  	int i;
> diff --git a/include/linux/mm.h b/include/linux/mm.h index
> 0f9422c7841e..b217c699dbab 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2803,6 +2803,8 @@ static inline void setup_nr_node_ids(void) {}  #ifndef
> CONFIG_X86_INTEL_MKTME  static inline void mprotect_set_encrypt(struct
> vm_area_struct *vma,
>  					int newkeyid) {}
> +static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
> +static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
>  #endif /* CONFIG_X86_INTEL_MKTME */
>  #endif /* __KERNEL__ */
>  #endif /* _LINUX_MM_H */
> --
> 2.14.1

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> owner@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:39 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> transparent memory encryption in upcoming Intel platforms. MKTME will
> support mulitple encryption domains, each having their own key. The main use
> case for the feature is virtual machine isolation. The API needs the flexibility to
> work for a wide range of uses.
> 
> The MKTME key service type manages the addition and removal of the memory
> encryption keys. It maps software keys to hardware keyids and programs the
> hardware with the user requested encryption options.
> 
> The only supported encryption algorithm is AES-XTS 128.
> 
> The MKTME key service is half of the MKTME API level solution. It pairs with a
> new memory encryption system call: encrypt_mprotect() that uses the keys to
> encrypt memory.
> 
> See Documentation/x86/mktme-keys.txt
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/Kconfig           |   1 +
>  include/keys/mktme-type.h  |  28 +++++
>  security/keys/Kconfig      |  11 ++
>  security/keys/Makefile     |   1 +
>  security/keys/mktme_keys.c | 278
> +++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 319 insertions(+)
>  create mode 100644 include/keys/mktme-type.h  create mode 100644
> security/keys/mktme_keys.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index
> 023a22568c06..50d8aa6a58e9 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
>  	bool "Intel Multi-Key Total Memory Encryption"
>  	select DYNAMIC_PHYSICAL_MASK
>  	select PAGE_EXTENSION
> +	select MKTME_KEYS
>  	depends on X86_64 && CPU_SUP_INTEL
>  	---help---
>  	  Say yes to enable support for Multi-Key Total Memory Encryption.
> diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h new file
> mode 100644 index 000000000000..bebe74cb2b51
> --- /dev/null
> +++ b/include/keys/mktme-type.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Key service for Multi-KEY Total Memory Encryption  */
> +
> +#ifndef _KEYS_MKTME_TYPE_H
> +#define _KEYS_MKTME_TYPE_H
> +
> +#include <linux/key.h>
> +
> +/*
> + * The AES-XTS 128 encryption algorithm requires 128 bits for each
> + * user supplied option: userkey=, tweak=, entropy=.
> + */
> +#define MKTME_AES_XTS_SIZE	16
> +
> +enum mktme_alg {
> +	MKTME_ALG_AES_XTS_128,
> +};
> +
> +const char *const mktme_alg_names[] = {
> +	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
> +};
> +
> +extern struct key_type key_type_mktme;
> +
> +#endif /* _KEYS_MKTME_TYPE_H */
> diff --git a/security/keys/Kconfig b/security/keys/Kconfig index
> 6462e6654ccf..c36972113e67 100644
> --- a/security/keys/Kconfig
> +++ b/security/keys/Kconfig
> @@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
>  	 in the kernel.
> 
>  	 If you are unsure as to whether this is required, answer N.
> +
> +config MKTME_KEYS
> +	bool "Multi-Key Total Memory Encryption Keys"
> +	depends on KEYS && X86_INTEL_MKTME
> +	help
> +	  This option provides support for Multi-Key Total Memory
> +	  Encryption (MKTME) on Intel platforms offering the feature.
> +	  MKTME allows userspace to manage the hardware encryption
> +	  keys through the kernel key services.
> +
> +	  If you are unsure as to whether this is required, answer N.
> diff --git a/security/keys/Makefile b/security/keys/Makefile index
> ef1581b337a3..2d9f9a82cb8a 100644
> --- a/security/keys/Makefile
> +++ b/security/keys/Makefile
> @@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
>  obj-$(CONFIG_BIG_KEYS) += big_key.o
>  obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
>  obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
> +obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
> diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c new file
> mode 100644 index 000000000000..dcbce7194647
> --- /dev/null
> +++ b/security/keys/mktme_keys.c
> @@ -0,0 +1,278 @@
> +// SPDX-License-Identifier: GPL-3.0
> +
> +/* Documentation/x86/mktme-keys.txt */
> +
> +#include <linux/cred.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/key.h>
> +#include <linux/key-type.h>
> +#include <linux/init.h>
> +#include <linux/parser.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include <asm/intel_pconfig.h>
> +#include <asm/mktme.h>
> +#include <keys/mktme-type.h>
> +#include <keys/user-type.h>
> +
> +#include "internal.h"
> +
> +struct kmem_cache *mktme_prog_cache;	/* hardware programming
> struct */
> +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> keys */

Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)

But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

> +
> +static const char * const mktme_program_err[] = {
> +	"KeyID was successfully programmed",	/* 0 */
> +	"Invalid KeyID programming command",	/* 1 */
> +	"Insufficient entropy",			/* 2 */
> +	"KeyID not valid",			/* 3 */
> +	"Invalid encryption algorithm chosen",	/* 4 */
> +	"Failure to access key table",		/* 5 */
> +};
> +
> +/* If a key is available, program and add the key to the software map.
> +*/ static int mktme_program_key(key_serial_t serial,
> +			     struct mktme_key_program *kprog) {
> +	int keyid, ret;
> +
> +	keyid = mktme_map_get_free_keyid();
> +	if (keyid = 0)
> +		return -EDQUOT;
> +
> +	kprog->keyid = keyid;
> +	ret = mktme_key_program(kprog, mktme_cpumask);
> +	if (ret = MKTME_PROG_SUCCESS)
> +		mktme_map_set_keyid(keyid, serial);
> +	else
> +		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
> +
> +	return ret;
> +}
> +
> +enum mktme_opt_id	{
> +	OPT_ERROR = -1,
> +	OPT_USERKEY,
> +	OPT_TWEAK,
> +	OPT_ENTROPY,
> +	OPT_ALGORITHM,
> +};
> +
> +static const match_table_t mktme_token = {
> +	{OPT_USERKEY, "userkey=%s"},
> +	{OPT_TWEAK, "tweak=%s"},
> +	{OPT_ENTROPY, "entropy=%s"},
> +	{OPT_ALGORITHM, "algorithm=%s"},
> +	{OPT_ERROR, NULL}
> +
> +};
> +
> +/*
> + * Algorithm AES-XTS 128 is the only supported encryption algorithm.
> + * CPU Generated Key: requires user supplied entropy and accepts no
> + *		      other options.
> + * User Supplied Key: requires user supplied tweak key and accepts
> + *		      no other options.
> + */
> +static int mktme_check_options(struct mktme_key_program *kprog,
> +			       unsigned long token_mask)
> +{
> +	if (!token_mask)
> +		return -EINVAL;
> +
> +	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
> +
> +	if (!test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
> +	}
> +	if (test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (!test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Parse the options and begin to fill in the key programming struct kprog.
> + * Check the lengths of incoming data and push data directly into kprog fields.
> + */
> +static int mktme_get_options(char *options, struct mktme_key_program
> +*kprog) {
> +	int len = MKTME_AES_XTS_SIZE / 2;
> +	substring_t args[MAX_OPT_ARGS];
> +	unsigned long token_mask = 0;
> +	enum mktme_alg alg;
> +	char *p = options;
> +	int ret, token;
> +
> +	while ((p = strsep(&options, " \t"))) {
> +		if (*p = '\0' || *p = ' ' || *p = '\t')
> +			continue;
> +		token = match_token(p, mktme_token, args);
> +		if (test_and_set_bit(token, &token_mask))
> +			return -EINVAL;
> +
> +		switch (token) {
> +		case OPT_USERKEY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TWEAK:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_ENTROPY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			/* Applied to both CPU-generated data and tweak keys
> */
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;

I replied w/ some comments in patch 1 (Document part). Do you have any particular reason to introduce OPT_ENTROPY, while we can simply use OPT_USERKEY and OPT_TWEAK?

Actually I think it might be better that we disallow or ignore OPT_USERKEY, OPT_TWEAK (or OPT_ENTROPY in your patch). Please see my reply to your patch 1.

> +
> +		case OPT_ALGORITHM:
> +			alg = match_string(mktme_alg_names,
> +					   ARRAY_SIZE(mktme_alg_names),
> +					   args[0].from);
> +			if (alg != MKTME_ALG_AES_XTS_128)
> +				return -EINVAL;
> +			break;
> +
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +	return mktme_check_options(kprog, token_mask); }
> +
> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload
> +*prep) {
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;
> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}
> +
> +struct key_type key_type_mktme = {
> +	.name = "mktme",
> +	.instantiate = mktme_instantiate,
> +	.describe = user_describe,
> +};
> +
> +/*
> + * Build mktme_cpumask to include one cpu per physical package.
> + * The mask is used in mktme_key_program() when the hardware key
> + * table is programmed on a per package basis.
> + */
> +static int mktme_build_cpumask(void)
> +{
> +	int online_cpu, mktme_cpu;
> +	int online_pkgid, mktme_pkgid = -1;
> +
> +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	for_each_online_cpu(online_cpu) {
> +		online_pkgid = topology_physical_package_id(online_cpu);
> +
> +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> +			mktme_pkgid > topology_physical_package_id(mktme_cpu);
> +			if (mktme_pkgid = online_pkgid)
> +				break;
> +		}
> +		if (mktme_pkgid != online_pkgid)
> +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> +	}

Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

> +	return 0;
> +}
> +
> +/*
> + * Allocate the global key map structure based on the available keyids
> + * at boot time. Create a cache and a cpu_mask to use for programming
> + * the hardware. Initialize the encrypt_count array to track VMA's per
> + * keyid. Once all that succeeds, register the 'mktme' key type.
> + */
> +static int __init init_mktme(void)
> +{
> +	int ret;
> +
> +	/* Verify keys are present */
> +	if (!(mktme_nr_keyids > 0))
> +		return -EINVAL;
> +
> +	if (!mktme_map_alloc())
> +		return -ENOMEM;
> +
> +	mktme_prog_cache = KMEM_CACHE(mktme_key_program,
> SLAB_PANIC);
> +	if (!mktme_prog_cache)
> +		goto free_map;
> +
> +	if (vma_alloc_encrypt_array() < 0)
> +		goto free_cache;

I think it's better to move 'vma_alloc_encrypt_array' part to this patch. Please see my reply to your patch 7.

I also think we should avoid adding new staff to arch/x86/include/asm/mktme.h since it is included by some basic page table manipulation header files. Adding mm related structure to asm/mktme.h sometimes may fail to compile kernel in my experience.

Thanks,
-Kai
> +
> +	if (mktme_build_cpumask() < 0)
> +		goto free_array;
> +
> +	ret = register_key_type(&key_type_mktme);
> +	if (!ret)
> +		return ret;
> +
> +	free_cpumask_var(mktme_cpumask);
> +free_array:
> +	vma_free_encrypt_array();
> +free_cache:
> +	kmem_cache_destroy(mktme_prog_cache);
> +free_map:
> +	mktme_map_free();
> +
> +	return -ENOMEM;
> +}
> +
> +late_initcall(init_mktme);
> --
> 2.14.1

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> owner at vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:39 AM
> To: dhowells at redhat.com; tglx at linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> transparent memory encryption in upcoming Intel platforms. MKTME will
> support mulitple encryption domains, each having their own key. The main use
> case for the feature is virtual machine isolation. The API needs the flexibility to
> work for a wide range of uses.
> 
> The MKTME key service type manages the addition and removal of the memory
> encryption keys. It maps software keys to hardware keyids and programs the
> hardware with the user requested encryption options.
> 
> The only supported encryption algorithm is AES-XTS 128.
> 
> The MKTME key service is half of the MKTME API level solution. It pairs with a
> new memory encryption system call: encrypt_mprotect() that uses the keys to
> encrypt memory.
> 
> See Documentation/x86/mktme-keys.txt
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/Kconfig           |   1 +
>  include/keys/mktme-type.h  |  28 +++++
>  security/keys/Kconfig      |  11 ++
>  security/keys/Makefile     |   1 +
>  security/keys/mktme_keys.c | 278
> +++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 319 insertions(+)
>  create mode 100644 include/keys/mktme-type.h  create mode 100644
> security/keys/mktme_keys.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index
> 023a22568c06..50d8aa6a58e9 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
>  	bool "Intel Multi-Key Total Memory Encryption"
>  	select DYNAMIC_PHYSICAL_MASK
>  	select PAGE_EXTENSION
> +	select MKTME_KEYS
>  	depends on X86_64 && CPU_SUP_INTEL
>  	---help---
>  	  Say yes to enable support for Multi-Key Total Memory Encryption.
> diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h new file
> mode 100644 index 000000000000..bebe74cb2b51
> --- /dev/null
> +++ b/include/keys/mktme-type.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Key service for Multi-KEY Total Memory Encryption  */
> +
> +#ifndef _KEYS_MKTME_TYPE_H
> +#define _KEYS_MKTME_TYPE_H
> +
> +#include <linux/key.h>
> +
> +/*
> + * The AES-XTS 128 encryption algorithm requires 128 bits for each
> + * user supplied option: userkey=, tweak=, entropy=.
> + */
> +#define MKTME_AES_XTS_SIZE	16
> +
> +enum mktme_alg {
> +	MKTME_ALG_AES_XTS_128,
> +};
> +
> +const char *const mktme_alg_names[] = {
> +	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
> +};
> +
> +extern struct key_type key_type_mktme;
> +
> +#endif /* _KEYS_MKTME_TYPE_H */
> diff --git a/security/keys/Kconfig b/security/keys/Kconfig index
> 6462e6654ccf..c36972113e67 100644
> --- a/security/keys/Kconfig
> +++ b/security/keys/Kconfig
> @@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
>  	 in the kernel.
> 
>  	 If you are unsure as to whether this is required, answer N.
> +
> +config MKTME_KEYS
> +	bool "Multi-Key Total Memory Encryption Keys"
> +	depends on KEYS && X86_INTEL_MKTME
> +	help
> +	  This option provides support for Multi-Key Total Memory
> +	  Encryption (MKTME) on Intel platforms offering the feature.
> +	  MKTME allows userspace to manage the hardware encryption
> +	  keys through the kernel key services.
> +
> +	  If you are unsure as to whether this is required, answer N.
> diff --git a/security/keys/Makefile b/security/keys/Makefile index
> ef1581b337a3..2d9f9a82cb8a 100644
> --- a/security/keys/Makefile
> +++ b/security/keys/Makefile
> @@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
>  obj-$(CONFIG_BIG_KEYS) += big_key.o
>  obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
>  obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
> +obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
> diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c new file
> mode 100644 index 000000000000..dcbce7194647
> --- /dev/null
> +++ b/security/keys/mktme_keys.c
> @@ -0,0 +1,278 @@
> +// SPDX-License-Identifier: GPL-3.0
> +
> +/* Documentation/x86/mktme-keys.txt */
> +
> +#include <linux/cred.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/key.h>
> +#include <linux/key-type.h>
> +#include <linux/init.h>
> +#include <linux/parser.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include <asm/intel_pconfig.h>
> +#include <asm/mktme.h>
> +#include <keys/mktme-type.h>
> +#include <keys/user-type.h>
> +
> +#include "internal.h"
> +
> +struct kmem_cache *mktme_prog_cache;	/* hardware programming
> struct */
> +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> keys */

Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)

But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

> +
> +static const char * const mktme_program_err[] = {
> +	"KeyID was successfully programmed",	/* 0 */
> +	"Invalid KeyID programming command",	/* 1 */
> +	"Insufficient entropy",			/* 2 */
> +	"KeyID not valid",			/* 3 */
> +	"Invalid encryption algorithm chosen",	/* 4 */
> +	"Failure to access key table",		/* 5 */
> +};
> +
> +/* If a key is available, program and add the key to the software map.
> +*/ static int mktme_program_key(key_serial_t serial,
> +			     struct mktme_key_program *kprog) {
> +	int keyid, ret;
> +
> +	keyid = mktme_map_get_free_keyid();
> +	if (keyid == 0)
> +		return -EDQUOT;
> +
> +	kprog->keyid = keyid;
> +	ret = mktme_key_program(kprog, mktme_cpumask);
> +	if (ret == MKTME_PROG_SUCCESS)
> +		mktme_map_set_keyid(keyid, serial);
> +	else
> +		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
> +
> +	return ret;
> +}
> +
> +enum mktme_opt_id	{
> +	OPT_ERROR = -1,
> +	OPT_USERKEY,
> +	OPT_TWEAK,
> +	OPT_ENTROPY,
> +	OPT_ALGORITHM,
> +};
> +
> +static const match_table_t mktme_token = {
> +	{OPT_USERKEY, "userkey=%s"},
> +	{OPT_TWEAK, "tweak=%s"},
> +	{OPT_ENTROPY, "entropy=%s"},
> +	{OPT_ALGORITHM, "algorithm=%s"},
> +	{OPT_ERROR, NULL}
> +
> +};
> +
> +/*
> + * Algorithm AES-XTS 128 is the only supported encryption algorithm.
> + * CPU Generated Key: requires user supplied entropy and accepts no
> + *		      other options.
> + * User Supplied Key: requires user supplied tweak key and accepts
> + *		      no other options.
> + */
> +static int mktme_check_options(struct mktme_key_program *kprog,
> +			       unsigned long token_mask)
> +{
> +	if (!token_mask)
> +		return -EINVAL;
> +
> +	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
> +
> +	if (!test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
> +	}
> +	if (test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (!test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Parse the options and begin to fill in the key programming struct kprog.
> + * Check the lengths of incoming data and push data directly into kprog fields.
> + */
> +static int mktme_get_options(char *options, struct mktme_key_program
> +*kprog) {
> +	int len = MKTME_AES_XTS_SIZE / 2;
> +	substring_t args[MAX_OPT_ARGS];
> +	unsigned long token_mask = 0;
> +	enum mktme_alg alg;
> +	char *p = options;
> +	int ret, token;
> +
> +	while ((p = strsep(&options, " \t"))) {
> +		if (*p == '\0' || *p == ' ' || *p == '\t')
> +			continue;
> +		token = match_token(p, mktme_token, args);
> +		if (test_and_set_bit(token, &token_mask))
> +			return -EINVAL;
> +
> +		switch (token) {
> +		case OPT_USERKEY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TWEAK:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_ENTROPY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			/* Applied to both CPU-generated data and tweak keys
> */
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;

I replied w/ some comments in patch 1 (Document part). Do you have any particular reason to introduce OPT_ENTROPY, while we can simply use OPT_USERKEY and OPT_TWEAK?

Actually I think it might be better that we disallow or ignore OPT_USERKEY, OPT_TWEAK (or OPT_ENTROPY in your patch). Please see my reply to your patch 1.

> +
> +		case OPT_ALGORITHM:
> +			alg = match_string(mktme_alg_names,
> +					   ARRAY_SIZE(mktme_alg_names),
> +					   args[0].from);
> +			if (alg != MKTME_ALG_AES_XTS_128)
> +				return -EINVAL;
> +			break;
> +
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +	return mktme_check_options(kprog, token_mask); }
> +
> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload
> +*prep) {
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;
> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}
> +
> +struct key_type key_type_mktme = {
> +	.name = "mktme",
> +	.instantiate = mktme_instantiate,
> +	.describe = user_describe,
> +};
> +
> +/*
> + * Build mktme_cpumask to include one cpu per physical package.
> + * The mask is used in mktme_key_program() when the hardware key
> + * table is programmed on a per package basis.
> + */
> +static int mktme_build_cpumask(void)
> +{
> +	int online_cpu, mktme_cpu;
> +	int online_pkgid, mktme_pkgid = -1;
> +
> +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	for_each_online_cpu(online_cpu) {
> +		online_pkgid = topology_physical_package_id(online_cpu);
> +
> +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> +			mktme_pkgid =
> topology_physical_package_id(mktme_cpu);
> +			if (mktme_pkgid == online_pkgid)
> +				break;
> +		}
> +		if (mktme_pkgid != online_pkgid)
> +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> +	}

Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

> +	return 0;
> +}
> +
> +/*
> + * Allocate the global key map structure based on the available keyids
> + * at boot time. Create a cache and a cpu_mask to use for programming
> + * the hardware. Initialize the encrypt_count array to track VMA's per
> + * keyid. Once all that succeeds, register the 'mktme' key type.
> + */
> +static int __init init_mktme(void)
> +{
> +	int ret;
> +
> +	/* Verify keys are present */
> +	if (!(mktme_nr_keyids > 0))
> +		return -EINVAL;
> +
> +	if (!mktme_map_alloc())
> +		return -ENOMEM;
> +
> +	mktme_prog_cache = KMEM_CACHE(mktme_key_program,
> SLAB_PANIC);
> +	if (!mktme_prog_cache)
> +		goto free_map;
> +
> +	if (vma_alloc_encrypt_array() < 0)
> +		goto free_cache;

I think it's better to move 'vma_alloc_encrypt_array' part to this patch. Please see my reply to your patch 7.

I also think we should avoid adding new staff to arch/x86/include/asm/mktme.h since it is included by some basic page table manipulation header files. Adding mm related structure to asm/mktme.h sometimes may fail to compile kernel in my experience.

Thanks,
-Kai
> +
> +	if (mktme_build_cpumask() < 0)
> +		goto free_array;
> +
> +	ret = register_key_type(&key_type_mktme);
> +	if (!ret)
> +		return ret;
> +
> +	free_cpumask_var(mktme_cpumask);
> +free_array:
> +	vma_free_encrypt_array();
> +free_cache:
> +	kmem_cache_destroy(mktme_prog_cache);
> +free_map:
> +	mktme_map_free();
> +
> +	return -ENOMEM;
> +}
> +
> +late_initcall(init_mktme);
> --
> 2.14.1

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> owner@vger.kernel.org] On Behalf Of Alison Schofield
> Sent: Saturday, September 8, 2018 10:39 AM
> To: dhowells@redhat.com; tglx@linutronix.de
> Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> transparent memory encryption in upcoming Intel platforms. MKTME will
> support mulitple encryption domains, each having their own key. The main use
> case for the feature is virtual machine isolation. The API needs the flexibility to
> work for a wide range of uses.
> 
> The MKTME key service type manages the addition and removal of the memory
> encryption keys. It maps software keys to hardware keyids and programs the
> hardware with the user requested encryption options.
> 
> The only supported encryption algorithm is AES-XTS 128.
> 
> The MKTME key service is half of the MKTME API level solution. It pairs with a
> new memory encryption system call: encrypt_mprotect() that uses the keys to
> encrypt memory.
> 
> See Documentation/x86/mktme-keys.txt
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/Kconfig           |   1 +
>  include/keys/mktme-type.h  |  28 +++++
>  security/keys/Kconfig      |  11 ++
>  security/keys/Makefile     |   1 +
>  security/keys/mktme_keys.c | 278
> +++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 319 insertions(+)
>  create mode 100644 include/keys/mktme-type.h  create mode 100644
> security/keys/mktme_keys.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index
> 023a22568c06..50d8aa6a58e9 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1527,6 +1527,7 @@ config X86_INTEL_MKTME
>  	bool "Intel Multi-Key Total Memory Encryption"
>  	select DYNAMIC_PHYSICAL_MASK
>  	select PAGE_EXTENSION
> +	select MKTME_KEYS
>  	depends on X86_64 && CPU_SUP_INTEL
>  	---help---
>  	  Say yes to enable support for Multi-Key Total Memory Encryption.
> diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h new file
> mode 100644 index 000000000000..bebe74cb2b51
> --- /dev/null
> +++ b/include/keys/mktme-type.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Key service for Multi-KEY Total Memory Encryption  */
> +
> +#ifndef _KEYS_MKTME_TYPE_H
> +#define _KEYS_MKTME_TYPE_H
> +
> +#include <linux/key.h>
> +
> +/*
> + * The AES-XTS 128 encryption algorithm requires 128 bits for each
> + * user supplied option: userkey=, tweak=, entropy=.
> + */
> +#define MKTME_AES_XTS_SIZE	16
> +
> +enum mktme_alg {
> +	MKTME_ALG_AES_XTS_128,
> +};
> +
> +const char *const mktme_alg_names[] = {
> +	[MKTME_ALG_AES_XTS_128]	= "aes_xts_128",
> +};
> +
> +extern struct key_type key_type_mktme;
> +
> +#endif /* _KEYS_MKTME_TYPE_H */
> diff --git a/security/keys/Kconfig b/security/keys/Kconfig index
> 6462e6654ccf..c36972113e67 100644
> --- a/security/keys/Kconfig
> +++ b/security/keys/Kconfig
> @@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
>  	 in the kernel.
> 
>  	 If you are unsure as to whether this is required, answer N.
> +
> +config MKTME_KEYS
> +	bool "Multi-Key Total Memory Encryption Keys"
> +	depends on KEYS && X86_INTEL_MKTME
> +	help
> +	  This option provides support for Multi-Key Total Memory
> +	  Encryption (MKTME) on Intel platforms offering the feature.
> +	  MKTME allows userspace to manage the hardware encryption
> +	  keys through the kernel key services.
> +
> +	  If you are unsure as to whether this is required, answer N.
> diff --git a/security/keys/Makefile b/security/keys/Makefile index
> ef1581b337a3..2d9f9a82cb8a 100644
> --- a/security/keys/Makefile
> +++ b/security/keys/Makefile
> @@ -29,3 +29,4 @@ obj-$(CONFIG_KEY_DH_OPERATIONS) += dh.o
>  obj-$(CONFIG_BIG_KEYS) += big_key.o
>  obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
>  obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
> +obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
> diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c new file
> mode 100644 index 000000000000..dcbce7194647
> --- /dev/null
> +++ b/security/keys/mktme_keys.c
> @@ -0,0 +1,278 @@
> +// SPDX-License-Identifier: GPL-3.0
> +
> +/* Documentation/x86/mktme-keys.txt */
> +
> +#include <linux/cred.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/key.h>
> +#include <linux/key-type.h>
> +#include <linux/init.h>
> +#include <linux/parser.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include <asm/intel_pconfig.h>
> +#include <asm/mktme.h>
> +#include <keys/mktme-type.h>
> +#include <keys/user-type.h>
> +
> +#include "internal.h"
> +
> +struct kmem_cache *mktme_prog_cache;	/* hardware programming
> struct */
> +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> keys */

Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)

But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

> +
> +static const char * const mktme_program_err[] = {
> +	"KeyID was successfully programmed",	/* 0 */
> +	"Invalid KeyID programming command",	/* 1 */
> +	"Insufficient entropy",			/* 2 */
> +	"KeyID not valid",			/* 3 */
> +	"Invalid encryption algorithm chosen",	/* 4 */
> +	"Failure to access key table",		/* 5 */
> +};
> +
> +/* If a key is available, program and add the key to the software map.
> +*/ static int mktme_program_key(key_serial_t serial,
> +			     struct mktme_key_program *kprog) {
> +	int keyid, ret;
> +
> +	keyid = mktme_map_get_free_keyid();
> +	if (keyid == 0)
> +		return -EDQUOT;
> +
> +	kprog->keyid = keyid;
> +	ret = mktme_key_program(kprog, mktme_cpumask);
> +	if (ret == MKTME_PROG_SUCCESS)
> +		mktme_map_set_keyid(keyid, serial);
> +	else
> +		pr_debug("mktme: %s [%d]\n", mktme_program_err[ret], ret);
> +
> +	return ret;
> +}
> +
> +enum mktme_opt_id	{
> +	OPT_ERROR = -1,
> +	OPT_USERKEY,
> +	OPT_TWEAK,
> +	OPT_ENTROPY,
> +	OPT_ALGORITHM,
> +};
> +
> +static const match_table_t mktme_token = {
> +	{OPT_USERKEY, "userkey=%s"},
> +	{OPT_TWEAK, "tweak=%s"},
> +	{OPT_ENTROPY, "entropy=%s"},
> +	{OPT_ALGORITHM, "algorithm=%s"},
> +	{OPT_ERROR, NULL}
> +
> +};
> +
> +/*
> + * Algorithm AES-XTS 128 is the only supported encryption algorithm.
> + * CPU Generated Key: requires user supplied entropy and accepts no
> + *		      other options.
> + * User Supplied Key: requires user supplied tweak key and accepts
> + *		      no other options.
> + */
> +static int mktme_check_options(struct mktme_key_program *kprog,
> +			       unsigned long token_mask)
> +{
> +	if (!token_mask)
> +		return -EINVAL;
> +
> +	kprog->keyid_ctrl |= MKTME_AES_XTS_128;
> +
> +	if (!test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((!test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
> +	}
> +	if (test_bit(OPT_USERKEY, &token_mask)) {
> +		if ((test_bit(OPT_ENTROPY, &token_mask)) ||
> +		    (!test_bit(OPT_TWEAK, &token_mask)))
> +			return -EINVAL;
> +
> +		kprog->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Parse the options and begin to fill in the key programming struct kprog.
> + * Check the lengths of incoming data and push data directly into kprog fields.
> + */
> +static int mktme_get_options(char *options, struct mktme_key_program
> +*kprog) {
> +	int len = MKTME_AES_XTS_SIZE / 2;
> +	substring_t args[MAX_OPT_ARGS];
> +	unsigned long token_mask = 0;
> +	enum mktme_alg alg;
> +	char *p = options;
> +	int ret, token;
> +
> +	while ((p = strsep(&options, " \t"))) {
> +		if (*p == '\0' || *p == ' ' || *p == '\t')
> +			continue;
> +		token = match_token(p, mktme_token, args);
> +		if (test_and_set_bit(token, &token_mask))
> +			return -EINVAL;
> +
> +		switch (token) {
> +		case OPT_USERKEY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TWEAK:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_ENTROPY:
> +			if (strlen(args[0].from) != MKTME_AES_XTS_SIZE)
> +				return -EINVAL;
> +			/* Applied to both CPU-generated data and tweak keys
> */
> +			ret = hex2bin(kprog->key_field_1, args[0].from, len);
> +			ret = hex2bin(kprog->key_field_2, args[0].from, len);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;

I replied w/ some comments in patch 1 (Document part). Do you have any particular reason to introduce OPT_ENTROPY, while we can simply use OPT_USERKEY and OPT_TWEAK?

Actually I think it might be better that we disallow or ignore OPT_USERKEY, OPT_TWEAK (or OPT_ENTROPY in your patch). Please see my reply to your patch 1.

> +
> +		case OPT_ALGORITHM:
> +			alg = match_string(mktme_alg_names,
> +					   ARRAY_SIZE(mktme_alg_names),
> +					   args[0].from);
> +			if (alg != MKTME_ALG_AES_XTS_128)
> +				return -EINVAL;
> +			break;
> +
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +	return mktme_check_options(kprog, token_mask); }
> +
> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload
> +*prep) {
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;
> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}
> +
> +struct key_type key_type_mktme = {
> +	.name = "mktme",
> +	.instantiate = mktme_instantiate,
> +	.describe = user_describe,
> +};
> +
> +/*
> + * Build mktme_cpumask to include one cpu per physical package.
> + * The mask is used in mktme_key_program() when the hardware key
> + * table is programmed on a per package basis.
> + */
> +static int mktme_build_cpumask(void)
> +{
> +	int online_cpu, mktme_cpu;
> +	int online_pkgid, mktme_pkgid = -1;
> +
> +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	for_each_online_cpu(online_cpu) {
> +		online_pkgid = topology_physical_package_id(online_cpu);
> +
> +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> +			mktme_pkgid =
> topology_physical_package_id(mktme_cpu);
> +			if (mktme_pkgid == online_pkgid)
> +				break;
> +		}
> +		if (mktme_pkgid != online_pkgid)
> +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> +	}

Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

> +	return 0;
> +}
> +
> +/*
> + * Allocate the global key map structure based on the available keyids
> + * at boot time. Create a cache and a cpu_mask to use for programming
> + * the hardware. Initialize the encrypt_count array to track VMA's per
> + * keyid. Once all that succeeds, register the 'mktme' key type.
> + */
> +static int __init init_mktme(void)
> +{
> +	int ret;
> +
> +	/* Verify keys are present */
> +	if (!(mktme_nr_keyids > 0))
> +		return -EINVAL;
> +
> +	if (!mktme_map_alloc())
> +		return -ENOMEM;
> +
> +	mktme_prog_cache = KMEM_CACHE(mktme_key_program,
> SLAB_PANIC);
> +	if (!mktme_prog_cache)
> +		goto free_map;
> +
> +	if (vma_alloc_encrypt_array() < 0)
> +		goto free_cache;

I think it's better to move 'vma_alloc_encrypt_array' part to this patch. Please see my reply to your patch 7.

I also think we should avoid adding new staff to arch/x86/include/asm/mktme.h since it is included by some basic page table manipulation header files. Adding mm related structure to asm/mktme.h sometimes may fail to compile kernel in my experience.

Thanks,
-Kai
> +
> +	if (mktme_build_cpumask() < 0)
> +		goto free_array;
> +
> +	ret = register_key_type(&key_type_mktme);
> +	if (!ret)
> +		return ret;
> +
> +	free_cpumask_var(mktme_cpumask);
> +free_array:
> +	vma_free_encrypt_array();
> +free_cache:
> +	kmem_cache_destroy(mktme_prog_cache);
> +free_map:
> +	mktme_map_free();
> +
> +	return -ENOMEM;
> +}
> +
> +late_initcall(init_mktme);
> --
> 2.14.1

Re: [RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> Today mprotect is implemented to support legacy mprotect behavior
> plus an extension for memory protection keys. Make it more generic
> so that it can support additional extensions in the future.
> 
> This is done is preparation for adding a new system call for memory
> encyption keys. The intent is that the new encrypted mprotect will be
> another extension to legacy mprotect.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  mm/mprotect.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 68dc476310c0..56e64ef7931e 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -35,6 +35,8 @@
>  
>  #include "internal.h"
>  
> +#define NO_PKEY  -1

This commit does not make anything more generic but it does take
away a magic number. The code change is senseful. The commit
message is nonsense.

PS. Please use @linux.intel.com for LKML.

/Jarkko

[RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> Today mprotect is implemented to support legacy mprotect behavior
> plus an extension for memory protection keys. Make it more generic
> so that it can support additional extensions in the future.
> 
> This is done is preparation for adding a new system call for memory
> encyption keys. The intent is that the new encrypted mprotect will be
> another extension to legacy mprotect.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  mm/mprotect.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 68dc476310c0..56e64ef7931e 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -35,6 +35,8 @@
>  
>  #include "internal.h"
>  
> +#define NO_PKEY  -1

This commit does not make anything more generic but it does take
away a magic number. The code change is senseful. The commit
message is nonsense.

PS. Please use @linux.intel.com for LKML.

/Jarkko

Re: [RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> Today mprotect is implemented to support legacy mprotect behavior
> plus an extension for memory protection keys. Make it more generic
> so that it can support additional extensions in the future.
> 
> This is done is preparation for adding a new system call for memory
> encyption keys. The intent is that the new encrypted mprotect will be
> another extension to legacy mprotect.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  mm/mprotect.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 68dc476310c0..56e64ef7931e 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -35,6 +35,8 @@
>  
>  #include "internal.h"
>  
> +#define NO_PKEY  -1

This commit does not make anything more generic but it does take
away a magic number. The code change is senseful. The commit
message is nonsense.

PS. Please use @linux.intel.com for LKML.

/Jarkko

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTA5LTA3IGF0IDE1OjIzIC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBTZWVraW5nIGNvbW1lbnRzIG9uIHRoZSBBUElzIHN1cHBvcnRpbmcgTUtUTUUgb24gZnV0
dXJlIEludGVsIHBsYXRmb3Jtcy4NCj4gDQo+IE1LVE1FIChNdWx0aS1LZXkgVG90YWwgTWVtb3J5
IEVuY3J5cHRpb24pIGlzIGEgdGVjaG5vbG9neSBzdXBwb3J0aW5nDQo+IG1lbW9yeSBlbmNyeXB0
aW9uIG9uIHVwY29taW5nIEludGVsIHBsYXRmb3Jtcy4gV2hlcmVhcyBUTUUgYWxsb3dzDQo+IGVu
Y3J5cHRpb24gb2YgdGhlIGVudGlyZSBzeXN0ZW0gbWVtb3J5IHVzaW5nIGEgc2luZ2xlIGtleSwg
TUtUTUUNCj4gYWxsb3dzIG11bGl0cGxlIGVuY3J5cHRpb24gZG9tYWlucywgZWFjaCBoYXZpbmcg
dGhlaXIgb3duIGtleS4gV2hpbGUgDQo+IHRoZSBtYWluIHVzZSBjYXNlIGZvciB0aGUgZmVhdHVy
ZSBpcyB2aXJ0dWFsIG1hY2hpbmUgaXNvbGF0aW9uLCB0aGUNCj4gQVBJIG5lZWRzIHRoZSBmbGV4
aWJpbGl0eSB0byB3b3JrIGZvciBhIHdpZGUgcmFuZ2Ugb2YgdXNlIGNhc2VzLg0KDQpJcyBpdCBh
IGNvbW1vbiBrbm93bGVkZ2Ugd2hhdCBUTUUgaXM/IEkga2luZCBvZiBkcm9wcGVkIG91dCBvZiB0
aGUgd2Fnb24NCmluIHRoZSAybmQgc2VudGVuY2Ugb2YgdGhpcyBwYXJhZ3JhcGguDQoNCi9KYXJr
a28

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:23 -0700, Alison Schofield wrote:
> Seeking comments on the APIs supporting MKTME on future Intel platforms.
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> memory encryption on upcoming Intel platforms. Whereas TME allows
> encryption of the entire system memory using a single key, MKTME
> allows mulitple encryption domains, each having their own key. While 
> the main use case for the feature is virtual machine isolation, the
> API needs the flexibility to work for a wide range of use cases.

Is it a common knowledge what TME is? I kind of dropped out of the wagon
in the 2nd sentence of this paragraph.

/Jarkko

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:23 -0700, Alison Schofield wrote:
> Seeking comments on the APIs supporting MKTME on future Intel platforms.
> 
> MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> memory encryption on upcoming Intel platforms. Whereas TME allows
> encryption of the entire system memory using a single key, MKTME
> allows mulitple encryption domains, each having their own key. While 
> the main use case for the feature is virtual machine isolation, the
> API needs the flexibility to work for a wide range of use cases.

Is it a common knowledge what TME is? I kind of dropped out of the wagon
in the 2nd sentence of this paragraph.

/Jarkko

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTA5LTA3IGF0IDE1OjM0IC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBEb2N1bWVudCB0aGUgQVBJJ3MgdXNlZCBmb3IgTUtUTUUgb24gSW50ZWwgcGxhdGZvcm1z
Lg0KPiBNS1RNRTogTXVsdGktS0VZIFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uDQo+IA0KPiBTaWdu
ZWQtb2ZmLWJ5OiBBbGlzb24gU2Nob2ZpZWxkIDxhbGlzb24uc2Nob2ZpZWxkQGludGVsLmNvbT4N
Cj4gLS0tDQo+ICBEb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS1rZXlzLnR4dCB8IDE1Mw0KPiArKysr
KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysNCj4gIDEgZmlsZSBjaGFuZ2VkLCAx
NTMgaW5zZXJ0aW9ucygrKQ0KPiAgY3JlYXRlIG1vZGUgMTAwNjQ0IERvY3VtZW50YXRpb24veDg2
L21rdG1lLWtleXMudHh0DQo+IA0KPiBkaWZmIC0tZ2l0IGEvRG9jdW1lbnRhdGlvbi94ODYvbWt0
bWUta2V5cy50eHQgYi9Eb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS0NCj4ga2V5cy50eHQNCj4gbmV3
IGZpbGUgbW9kZSAxMDA2NDQNCj4gaW5kZXggMDAwMDAwMDAwMDAwLi4yZGVhN2FjZDJhMTcNCj4g
LS0tIC9kZXYvbnVsbA0KPiArKysgYi9Eb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS1rZXlzLnR4dA0K
PiBAQCAtMCwwICsxLDE1MyBAQA0KPiArTUtUTUUgKE11bHRpLUtleSBUb3RhbCBNZW1vcnkgRW5j
cnlwdGlvbikgaXMgYSB0ZWNobm9sb2d5IHRoYXQgYWxsb3dzDQo+ICttZW1vcnkgZW5jcnlwdGlv
biBvbiBJbnRlbCBwbGF0Zm9ybXMuIFdoZXJlYXMgVE1FIChUb3RhbCBNZW1vcnkgRW5jcnlwdGlv
bikNCj4gK2FsbG93cyBlbmNyeXB0aW9uIG9mIHRoZSBlbnRpcmUgc3lzdGVtIG1lbW9yeSB1c2lu
ZyBhIHNpbmdsZSBrZXksIE1LVE1FDQo+ICthbGxvd3MgbXVsdGlwbGUgZW5jcnlwdGlvbiBkb21h
aW5zLCBlYWNoIGhhdmluZyB0aGVpciBvd24ga2V5LiBUaGUgbWFpbiB1c2UNCj4gK2Nhc2UgZm9y
IHRoZSBmZWF0dXJlIGlzIHZpcnR1YWwgbWFjaGluZSBpc29sYXRpb24uIFRoZSBBUEkncyBpbnRy
b2R1Y2VkIGhlcmUNCj4gK2FyZSBpbnRlbmRlZCB0byBvZmZlciBmbGV4aWJpbGl0eSB0byB3b3Jr
IGluIGEgd2lkZSByYW5nZSBvZiB1c2VzLg0KPiArDQo+ICtUaGUgZXh0ZXJuYWxseSBhdmFpbGFi
bGUgSW50ZWwgQXJjaGl0ZWN0dXJlIFNwZWM6DQo+ICtodHRwczovL3NvZnR3YXJlLmludGVsLmNv
bS9zaXRlcy9kZWZhdWx0L2ZpbGVzL21hbmFnZWQvYTUvMTYvTXVsdGktS2V5LVRvdGFsLQ0KPiBN
ZW1vcnktRW5jcnlwdGlvbi1TcGVjLnBkZg0KPiArDQo+ICs9PT09PT09PT09PT09PT09PT09PT09
PT09PT09ICBBUEkgT3ZlcnZpZXcgID09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCj4gKw0K
PiArVGhlcmUgYXJlIDIgTUtUTUUgc3BlY2lmaWMgQVBJJ3MgdGhhdCBlbmFibGUgdXNlcnNwYWNl
IHRvIGNyZWF0ZSBhbmQgdXNlDQo+ICt0aGUgbWVtb3J5IGVuY3J5cHRpb24ga2V5czoNCg0KVGhp
cyBpcyBsaWtlIHNheWluZyB0aGF0IHRoZXkgYXJlIGRpZmZlcmVudCBBUElzIHRvIGRvIHNlbWFu
dGljYWxseSB0aGUNCnNhbWUgZXhhY3QgdGhpbmcuIElzIHRoYXQgc28/DQoNCi9KYXJra28

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  Documentation/x86/mktme-keys.txt | 153
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> +allows encryption of the entire system memory using a single key, MKTME
> +allows multiple encryption domains, each having their own key. The main use
> +case for the feature is virtual machine isolation. The API's introduced here
> +are intended to offer flexibility to work in a wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-
> Memory-Encryption-Spec.pdf
> +
> +============================  API Overview  ============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and use
> +the memory encryption keys:

This is like saying that they are different APIs to do semantically the
same exact thing. Is that so?

/Jarkko

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> Document the API's used for MKTME on Intel platforms.
> MKTME: Multi-KEY Total Memory Encryption
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  Documentation/x86/mktme-keys.txt | 153
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 153 insertions(+)
>  create mode 100644 Documentation/x86/mktme-keys.txt
> 
> diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> keys.txt
> new file mode 100644
> index 000000000000..2dea7acd2a17
> --- /dev/null
> +++ b/Documentation/x86/mktme-keys.txt
> @@ -0,0 +1,153 @@
> +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> +allows encryption of the entire system memory using a single key, MKTME
> +allows multiple encryption domains, each having their own key. The main use
> +case for the feature is virtual machine isolation. The API's introduced here
> +are intended to offer flexibility to work in a wide range of uses.
> +
> +The externally available Intel Architecture Spec:
> +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-
> Memory-Encryption-Spec.pdf
> +
> +============================  API Overview  ============================
> +
> +There are 2 MKTME specific API's that enable userspace to create and use
> +the memory encryption keys:

This is like saying that they are different APIs to do semantically the
same exact thing. Is that so?

/Jarkko

Re: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Define a global mapping structure to track the mapping of userspace
> keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> This data will be used for the memory encryption system call and the
> kernel key service API.

This is also for earlier patches. I kind of feel uncomfortable because
I can neither TME nor MKTME how they reside in the chipset/board. Are
they inside the CPU package? Or is it perhaps a separate chip outside
the CPU? I would guess not but the problem is that I have to guess in
the first place.

> Implement helper functions to access this mapping structure and make
> them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/include/asm/mktme.h | 11 ++++++
>  arch/x86/mm/mktme.c          | 85
> ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 96 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbfbd955da98..f6acd551457f 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;
>  extern int mktme_nr_keyids;
>  extern int mktme_keyid_shift;
>  
> +/* Manage mappings between hardware keyids and userspace keys */
> +extern int mktme_map_alloc(void);
> +extern void mktme_map_free(void);
> +extern void mktme_map_lock(void);
> +extern void mktme_map_unlock(void);
> +extern int mktme_map_get_free_keyid(void);
> +extern void mktme_map_clear_keyid(int keyid);
> +extern void mktme_map_set_keyid(int keyid, unsigned int serial);
> +extern int mktme_map_keyid_from_serial(unsigned int serial);
> +extern unsigned int mktme_map_serial_from_keyid(int keyid);
> +
>  extern struct page_ext_operations page_mktme_ops;
>  
>  #define page_keyid page_keyid
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 660caf6a5ce1..5246d8323359 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
>  }
>  
> +/*
> + * struct mktme_mapping and the mktme_map_* functions manage the mapping
> + * of userspace keys to hardware keyids in MKTME. They are used by the
> + * the encrypt_mprotect system call and the MKTME Key Service API.
> + */
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW
> state */
> +	unsigned int	mapped_keyids;
> +	unsigned int	serial[];
> +};
> +
> +struct mktme_mapping *mktme_map;
> +
> +static inline long mktme_map_size(void)
> +{
> +	long size = 0;
> +
> +	size += sizeof(mktme_map);
> +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> +	return size;
> +}
> +
> +int mktme_map_alloc(void)
> +{
> +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> +	if (!mktme_map)
> +		return 0;
> +	mutex_init(&mktme_map->lock);
> +	return 1;
> +}
> +
> +void mktme_map_free(void)
> +{
> +	kfree(mktme_map);
> +}
> +
> +void mktme_map_lock(void)
> +{
> +	mutex_lock(&mktme_map->lock);
> +}
> +
> +void mktme_map_unlock(void)
> +{
> +	mutex_unlock(&mktme_map->lock);
> +}
> +
> +void mktme_map_set_keyid(int keyid, unsigned int serial)
> +{
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}
> +
> +void mktme_map_clear_keyid(int keyid)
> +{
> +	mktme_map->serial[keyid] = 0;
> +	mktme_map->mapped_keyids--;
> +}
> +
> +unsigned int mktme_map_serial_from_keyid(int keyid)
> +{
> +	return mktme_map->serial[keyid];
> +}
> +
> +int mktme_map_keyid_from_serial(unsigned int serial)
> +{
> +	int i;
> +
> +	for (i = 1; i < mktme_nr_keyids; i++)
> +		if (mktme_map->serial[i] == serial)
> +			return i;
> +	return 0;
> +}
> +
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i < mktme_nr_keyids; i++)
> +			if (mktme_map->serial[i] == 0)
> +				return i;
> +	}
> +	return 0;
> +}


The kdoc header is missing from all above.

/Jarkko

Re: [RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTA5LTA3IGF0IDE1OjM2IC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBTdG9yZSB0aGUgbWVtb3J5IGVuY3J5cHRpb24ga2V5aWQgaW4gdGhlIHVwcGVyIGJpdHMg
b2Ygdm1fcGFnZV9wcm90DQo+IHRoYXQgbWF0Y2ggcG9zaXRpb24gb2Yga2V5aWQsIGJpdHMgNTE6
NDYsIGluIGEgUFRFLg0KDQpXb3VsZCBub3QgZG8gYmFkIHRvIGV4cGxhaW4gdGhlIGNvbnRleHQg
YSBiaXQgaGVyZS4gQXQgbGVhc3QgSSBkbyBub3QNCmtub3cgd2h5IHlvdSBlbmRlZCB1cCB0byB0
aGlzIGJpdCByYW5nZS4NCg0KL0phcmtrbw0K

[RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Store the memory encryption keyid in the upper bits of vm_page_prot
> that match position of keyid, bits 51:46, in a PTE.

Would not do bad to explain the context a bit here. At least I do not
know why you ended up to this bit range.

/Jarkko

Re: [RFC 05/12] x86/mm: Add a helper function to set keyid bits in encrypted VMA's

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Store the memory encryption keyid in the upper bits of vm_page_prot
> that match position of keyid, bits 51:46, in a PTE.

Would not do bad to explain the context a bit here. At least I do not
know why you ended up to this bit range.

/Jarkko

Re: [RFC 06/12] mm: Add the encrypt_mprotect() system call

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Implement memory encryption with a new system call that is an
> extension of the legacy mprotect() system call.
> 
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed encryption key. Validate the key and store
> the keyid bits in the vm_page_prot for each VMA in the protection
> range.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  fs/exec.c           |  4 ++--
>  include/linux/key.h |  2 ++
>  include/linux/mm.h  |  3 ++-
>  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> ---
>  4 files changed, 67 insertions(+), 9 deletions(-)
> 
> diff --git a/fs/exec.c b/fs/exec.c
> index a1a246062561..b681a413db9c 100644
> --- a/fs/exec.c
> +++ b/fs/exec.c
> @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
>  	vm_flags |= mm->def_flags;
>  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
>  
> -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> -			vm_flags);
> +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> vm_flags,
> +			     -1);

Why you pass a magic number here when you went the trouble having
a named constant?

>  	if (ret)
>  		goto out_unlock;
>  	BUG_ON(prev != vma);
> diff --git a/include/linux/key.h b/include/linux/key.h
> index e58ee10f6e58..fb8a7d5f6149 100644
> --- a/include/linux/key.h
> +++ b/include/linux/key.h
> @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> *key)
>  
>  extern void key_set_timeout(struct key *, unsigned);
>  
> +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> +				 key_perm_t perm);
>  /*
>   * The permissions required on a key that we're looking up.
>   */
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index ac85c0805761..0f9422c7841e 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> vm_area_struct *vma, unsigned long
>  			      int dirty_accountable, int prot_numa);
>  extern int mprotect_fixup(struct vm_area_struct *vma,
>  			  struct vm_area_struct **pprev, unsigned long start,
> -			  unsigned long end, unsigned long newflags);
> +			  unsigned long end, unsigned long newflags,
> +			  int newkeyid);
>  
>  /*
>   * doesn't attempt to fault and will return short.
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 56e64ef7931e..6c2e1106525c 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -28,14 +28,17 @@
>  #include <linux/ksm.h>
>  #include <linux/uaccess.h>
>  #include <linux/mm_inline.h>
> +#include <linux/key.h>
>  #include <asm/pgtable.h>
>  #include <asm/cacheflush.h>
>  #include <asm/mmu_context.h>
>  #include <asm/tlbflush.h>
> +#include <asm/mktme.h>
>  
>  #include "internal.h"
>  
>  #define NO_PKEY  -1
> +#define NO_KEYID -1

Should have only single named constant IMHO. This ambiguity
is worse than some reasonable constant name for both cases.
Maybe NO_KEYID would be adequate?

>  
>  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>  		unsigned long addr, unsigned long end, pgprot_t newprot,
> @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> *vma, unsigned long start,
>  
>  int
>  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> -	unsigned long start, unsigned long end, unsigned long newflags)
> +	       unsigned long start, unsigned long end, unsigned long
> newflags,
> +	       int newkeyid)
>  {
>  	struct mm_struct *mm = vma->vm_mm;
>  	unsigned long oldflags = vma->vm_flags;
> @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	int error;
>  	int dirty_accountable = 0;
>  
> +	/*
> +	 * Flags match and Keyids match or we have NO_KEYID.
> +	 * This _fixup is usually called from do_mprotect_ext() except
> +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> +	 */
> +	if (newflags = oldflags &&
> +	    (newkeyid = vma_keyid(vma) || newkeyid = NO_KEYID)) {
> +		*pprev = vma;
> +		return 0;
> +	}
> +	/* Flags match and Keyid changes */
>  	if (newflags = oldflags) {
> +		mprotect_set_encrypt(vma, newkeyid);
>  		*pprev = vma;
>  		return 0;
>  	}
> +	/* Flags and Keyids both change, continue. */
>  
>  	/*
>  	 * If we make a private mapping writable we increase our commit;
> @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	}
>  
>  success:
> +	if (newkeyid != NO_KEYID)
> +		mprotect_set_encrypt(vma, newkeyid);
>  	/*
>  	 * vm_flags and vm_page_prot are protected by the mmap_sem
>  	 * held in write mode.
> @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  }
>  
>  /*
> - * When pkey=NO_PKEY we get legacy mprotect behavior here.
> + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> + * for protection keys and memory encryption keys. These extensions are
> + * mutually exclusive and the behavior is:
> + *	(pkey=NO_PKEY && keyid=NO_KEYID) => legacy mprotect
> + *	(pkey is valid)  => legacy mprotect plus protection key extensions
> + *	(keyid is valid) => legacy mprotect plus encryption key extensions
>   */

The header does not follow

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

>  static int do_mprotect_ext(unsigned long start, size_t len,
> -		unsigned long prot, int pkey)
> +			   unsigned long prot, int pkey, int keyid)
>  {
>  	unsigned long nstart, end, tmp, reqprot;
>  	struct vm_area_struct *vma, *prev;
> @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  		tmp = vma->vm_end;
>  		if (tmp > end)
>  			tmp = end;
> -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> +				       keyid);
>  		if (error)
>  			goto out;
>  		nstart = tmp;
> @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot)
>  {
> -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PKEYS
> @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> len,
>  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot, int, pkey)
>  {
> -	return do_mprotect_ext(start, len, prot, pkey);
> +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
>  }
>  
>  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
>  }
>  
>  #endif /* CONFIG_ARCH_HAS_PKEYS */
> +
> +#ifdef CONFIG_X86_INTEL_MKTME
> +
> +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> +		unsigned long, prot, key_serial_t, serial)
> +{
> +	key_ref_t key_ref;
> +	int ret, keyid;
> +
> +	/* TODO MKTME key service must be initialized */
> +
> +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> +	if (IS_ERR(key_ref))
> +		return PTR_ERR(key_ref);
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_keyid_from_serial(serial);
> +	if (!keyid) {
> +		mktme_map_unlock();
> +		key_ref_put(key_ref);
> +		return -EINVAL;
> +	}
> +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> +	mktme_map_unlock();
> +	key_ref_put(key_ref);
> +	return ret;
> +}
> +
> +#endif /* CONFIG_X86_INTEL_MKTME */

/Jarkko

[RFC 06/12] mm: Add the encrypt_mprotect() system call

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Implement memory encryption with a new system call that is an
> extension of the legacy mprotect() system call.
> 
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed encryption key. Validate the key and store
> the keyid bits in the vm_page_prot for each VMA in the protection
> range.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  fs/exec.c           |  4 ++--
>  include/linux/key.h |  2 ++
>  include/linux/mm.h  |  3 ++-
>  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> ---
>  4 files changed, 67 insertions(+), 9 deletions(-)
> 
> diff --git a/fs/exec.c b/fs/exec.c
> index a1a246062561..b681a413db9c 100644
> --- a/fs/exec.c
> +++ b/fs/exec.c
> @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
>  	vm_flags |= mm->def_flags;
>  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
>  
> -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> -			vm_flags);
> +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> vm_flags,
> +			     -1);

Why you pass a magic number here when you went the trouble having
a named constant?

>  	if (ret)
>  		goto out_unlock;
>  	BUG_ON(prev != vma);
> diff --git a/include/linux/key.h b/include/linux/key.h
> index e58ee10f6e58..fb8a7d5f6149 100644
> --- a/include/linux/key.h
> +++ b/include/linux/key.h
> @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> *key)
>  
>  extern void key_set_timeout(struct key *, unsigned);
>  
> +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> +				 key_perm_t perm);
>  /*
>   * The permissions required on a key that we're looking up.
>   */
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index ac85c0805761..0f9422c7841e 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> vm_area_struct *vma, unsigned long
>  			      int dirty_accountable, int prot_numa);
>  extern int mprotect_fixup(struct vm_area_struct *vma,
>  			  struct vm_area_struct **pprev, unsigned long start,
> -			  unsigned long end, unsigned long newflags);
> +			  unsigned long end, unsigned long newflags,
> +			  int newkeyid);
>  
>  /*
>   * doesn't attempt to fault and will return short.
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 56e64ef7931e..6c2e1106525c 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -28,14 +28,17 @@
>  #include <linux/ksm.h>
>  #include <linux/uaccess.h>
>  #include <linux/mm_inline.h>
> +#include <linux/key.h>
>  #include <asm/pgtable.h>
>  #include <asm/cacheflush.h>
>  #include <asm/mmu_context.h>
>  #include <asm/tlbflush.h>
> +#include <asm/mktme.h>
>  
>  #include "internal.h"
>  
>  #define NO_PKEY  -1
> +#define NO_KEYID -1

Should have only single named constant IMHO. This ambiguity
is worse than some reasonable constant name for both cases.
Maybe NO_KEYID would be adequate?

>  
>  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>  		unsigned long addr, unsigned long end, pgprot_t newprot,
> @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> *vma, unsigned long start,
>  
>  int
>  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> -	unsigned long start, unsigned long end, unsigned long newflags)
> +	       unsigned long start, unsigned long end, unsigned long
> newflags,
> +	       int newkeyid)
>  {
>  	struct mm_struct *mm = vma->vm_mm;
>  	unsigned long oldflags = vma->vm_flags;
> @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	int error;
>  	int dirty_accountable = 0;
>  
> +	/*
> +	 * Flags match and Keyids match or we have NO_KEYID.
> +	 * This _fixup is usually called from do_mprotect_ext() except
> +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> +	 */
> +	if (newflags == oldflags &&
> +	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEYID)) {
> +		*pprev = vma;
> +		return 0;
> +	}
> +	/* Flags match and Keyid changes */
>  	if (newflags == oldflags) {
> +		mprotect_set_encrypt(vma, newkeyid);
>  		*pprev = vma;
>  		return 0;
>  	}
> +	/* Flags and Keyids both change, continue. */
>  
>  	/*
>  	 * If we make a private mapping writable we increase our commit;
> @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	}
>  
>  success:
> +	if (newkeyid != NO_KEYID)
> +		mprotect_set_encrypt(vma, newkeyid);
>  	/*
>  	 * vm_flags and vm_page_prot are protected by the mmap_sem
>  	 * held in write mode.
> @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  }
>  
>  /*
> - * When pkey==NO_PKEY we get legacy mprotect behavior here.
> + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> + * for protection keys and memory encryption keys. These extensions are
> + * mutually exclusive and the behavior is:
> + *	(pkey==NO_PKEY && keyid==NO_KEYID) ==> legacy mprotect
> + *	(pkey is valid)  ==> legacy mprotect plus protection key extensions
> + *	(keyid is valid) ==> legacy mprotect plus encryption key extensions
>   */

The header does not follow

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

>  static int do_mprotect_ext(unsigned long start, size_t len,
> -		unsigned long prot, int pkey)
> +			   unsigned long prot, int pkey, int keyid)
>  {
>  	unsigned long nstart, end, tmp, reqprot;
>  	struct vm_area_struct *vma, *prev;
> @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  		tmp = vma->vm_end;
>  		if (tmp > end)
>  			tmp = end;
> -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> +				       keyid);
>  		if (error)
>  			goto out;
>  		nstart = tmp;
> @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot)
>  {
> -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PKEYS
> @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> len,
>  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot, int, pkey)
>  {
> -	return do_mprotect_ext(start, len, prot, pkey);
> +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
>  }
>  
>  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
>  }
>  
>  #endif /* CONFIG_ARCH_HAS_PKEYS */
> +
> +#ifdef CONFIG_X86_INTEL_MKTME
> +
> +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> +		unsigned long, prot, key_serial_t, serial)
> +{
> +	key_ref_t key_ref;
> +	int ret, keyid;
> +
> +	/* TODO MKTME key service must be initialized */
> +
> +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> +	if (IS_ERR(key_ref))
> +		return PTR_ERR(key_ref);
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_keyid_from_serial(serial);
> +	if (!keyid) {
> +		mktme_map_unlock();
> +		key_ref_put(key_ref);
> +		return -EINVAL;
> +	}
> +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> +	mktme_map_unlock();
> +	key_ref_put(key_ref);
> +	return ret;
> +}
> +
> +#endif /* CONFIG_X86_INTEL_MKTME */

/Jarkko

Re: [RFC 06/12] mm: Add the encrypt_mprotect() system call

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> Implement memory encryption with a new system call that is an
> extension of the legacy mprotect() system call.
> 
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed encryption key. Validate the key and store
> the keyid bits in the vm_page_prot for each VMA in the protection
> range.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  fs/exec.c           |  4 ++--
>  include/linux/key.h |  2 ++
>  include/linux/mm.h  |  3 ++-
>  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> ---
>  4 files changed, 67 insertions(+), 9 deletions(-)
> 
> diff --git a/fs/exec.c b/fs/exec.c
> index a1a246062561..b681a413db9c 100644
> --- a/fs/exec.c
> +++ b/fs/exec.c
> @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
>  	vm_flags |= mm->def_flags;
>  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
>  
> -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> -			vm_flags);
> +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> vm_flags,
> +			     -1);

Why you pass a magic number here when you went the trouble having
a named constant?

>  	if (ret)
>  		goto out_unlock;
>  	BUG_ON(prev != vma);
> diff --git a/include/linux/key.h b/include/linux/key.h
> index e58ee10f6e58..fb8a7d5f6149 100644
> --- a/include/linux/key.h
> +++ b/include/linux/key.h
> @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> *key)
>  
>  extern void key_set_timeout(struct key *, unsigned);
>  
> +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> +				 key_perm_t perm);
>  /*
>   * The permissions required on a key that we're looking up.
>   */
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index ac85c0805761..0f9422c7841e 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> vm_area_struct *vma, unsigned long
>  			      int dirty_accountable, int prot_numa);
>  extern int mprotect_fixup(struct vm_area_struct *vma,
>  			  struct vm_area_struct **pprev, unsigned long start,
> -			  unsigned long end, unsigned long newflags);
> +			  unsigned long end, unsigned long newflags,
> +			  int newkeyid);
>  
>  /*
>   * doesn't attempt to fault and will return short.
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 56e64ef7931e..6c2e1106525c 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -28,14 +28,17 @@
>  #include <linux/ksm.h>
>  #include <linux/uaccess.h>
>  #include <linux/mm_inline.h>
> +#include <linux/key.h>
>  #include <asm/pgtable.h>
>  #include <asm/cacheflush.h>
>  #include <asm/mmu_context.h>
>  #include <asm/tlbflush.h>
> +#include <asm/mktme.h>
>  
>  #include "internal.h"
>  
>  #define NO_PKEY  -1
> +#define NO_KEYID -1

Should have only single named constant IMHO. This ambiguity
is worse than some reasonable constant name for both cases.
Maybe NO_KEYID would be adequate?

>  
>  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>  		unsigned long addr, unsigned long end, pgprot_t newprot,
> @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> *vma, unsigned long start,
>  
>  int
>  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> -	unsigned long start, unsigned long end, unsigned long newflags)
> +	       unsigned long start, unsigned long end, unsigned long
> newflags,
> +	       int newkeyid)
>  {
>  	struct mm_struct *mm = vma->vm_mm;
>  	unsigned long oldflags = vma->vm_flags;
> @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	int error;
>  	int dirty_accountable = 0;
>  
> +	/*
> +	 * Flags match and Keyids match or we have NO_KEYID.
> +	 * This _fixup is usually called from do_mprotect_ext() except
> +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> +	 */
> +	if (newflags == oldflags &&
> +	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEYID)) {
> +		*pprev = vma;
> +		return 0;
> +	}
> +	/* Flags match and Keyid changes */
>  	if (newflags == oldflags) {
> +		mprotect_set_encrypt(vma, newkeyid);
>  		*pprev = vma;
>  		return 0;
>  	}
> +	/* Flags and Keyids both change, continue. */
>  
>  	/*
>  	 * If we make a private mapping writable we increase our commit;
> @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  	}
>  
>  success:
> +	if (newkeyid != NO_KEYID)
> +		mprotect_set_encrypt(vma, newkeyid);
>  	/*
>  	 * vm_flags and vm_page_prot are protected by the mmap_sem
>  	 * held in write mode.
> @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  }
>  
>  /*
> - * When pkey==NO_PKEY we get legacy mprotect behavior here.
> + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> + * for protection keys and memory encryption keys. These extensions are
> + * mutually exclusive and the behavior is:
> + *	(pkey==NO_PKEY && keyid==NO_KEYID) ==> legacy mprotect
> + *	(pkey is valid)  ==> legacy mprotect plus protection key extensions
> + *	(keyid is valid) ==> legacy mprotect plus encryption key extensions
>   */

The header does not follow

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

>  static int do_mprotect_ext(unsigned long start, size_t len,
> -		unsigned long prot, int pkey)
> +			   unsigned long prot, int pkey, int keyid)
>  {
>  	unsigned long nstart, end, tmp, reqprot;
>  	struct vm_area_struct *vma, *prev;
> @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  		tmp = vma->vm_end;
>  		if (tmp > end)
>  			tmp = end;
> -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> +				       keyid);
>  		if (error)
>  			goto out;
>  		nstart = tmp;
> @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot)
>  {
> -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PKEYS
> @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> len,
>  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot, int, pkey)
>  {
> -	return do_mprotect_ext(start, len, prot, pkey);
> +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
>  }
>  
>  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
>  }
>  
>  #endif /* CONFIG_ARCH_HAS_PKEYS */
> +
> +#ifdef CONFIG_X86_INTEL_MKTME
> +
> +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> +		unsigned long, prot, key_serial_t, serial)
> +{
> +	key_ref_t key_ref;
> +	int ret, keyid;
> +
> +	/* TODO MKTME key service must be initialized */
> +
> +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> +	if (IS_ERR(key_ref))
> +		return PTR_ERR(key_ref);
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_keyid_from_serial(serial);
> +	if (!keyid) {
> +		mktme_map_unlock();
> +		key_ref_put(key_ref);
> +		return -EINVAL;
> +	}
> +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> +	mktme_map_unlock();
> +	key_ref_put(key_ref);
> +	return ret;
> +}
> +
> +#endif /* CONFIG_X86_INTEL_MKTME */

/Jarkko

Re: [RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> Keep track of the VMA's oustanding for each memory encryption keyid.
> The count is used by the MKTME (Multi-Key Total Memory Encryption)
> Key Service to determine when it is safe to reprogram a hardware
> encryption key.

Maybe a stupid question but why they are tracked and what do you 
mean by tracking?

/Jarkko

> Approach here is to do gets and puts on the encryption reference
> wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> A couple of these locations will not be hit until cgroup support is
> added. One of these locations should never hit, so use a VM_WARN_ON.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/mm/mktme.c |  2 ++
>  kernel/fork.c       |  2 ++
>  mm/mmap.c           | 12 ++++++++++++
>  mm/nommu.c          |  4 ++++
>  4 files changed, 20 insertions(+)
> 
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 5690ef51a79a..8a7c326d4546 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int
> newkeyid)
>  	if (newkeyid = oldkeyid)
>  		return;
>  
> +	vma_put_encrypt_ref(vma);
>  	newprot = pgprot_val(vma->vm_page_prot);
>  	newprot &= ~mktme_keyid_mask;
>  	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
>  	vma->vm_page_prot = __pgprot(newprot);
> +	vma_get_encrypt_ref(vma);
>  }
>  
>  /*
> diff --git a/kernel/fork.c b/kernel/fork.c
> index e5e7a220a124..2d0e507bde7c 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  		if (!tmp)
>  			goto fail_nomem;
>  		*tmp = *mpnt;
> +		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
>  		INIT_LIST_HEAD(&tmp->anon_vma_chain);
>  		retval = vma_dup_policy(mpnt, tmp);
>  		if (retval)
> @@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  fail_nomem_anon_vma_fork:
>  	mpol_put(vma_policy(tmp));
>  fail_nomem_policy:
> +	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
>  	kmem_cache_free(vm_area_cachep, tmp);
>  fail_nomem:
>  	retval = -ENOMEM;
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 4c604eb644b4..7390b8b69fd6 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	mpol_put(vma_policy(vma));
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return next;
>  }
> @@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long
> start,
>  			anon_vma_merge(vma, next);
>  		mm->map_count--;
>  		mpol_put(vma_policy(next));
> +		vma_put_encrypt_ref(next);
>  		kmem_cache_free(vm_area_cachep, next);
>  		/*
>  		 * In mprotect's case 6 (see comments on vma_merge),
> @@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  		goto unacct_error;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
>  	vma->vm_end = addr + len;
> @@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  unmap_and_free_vma:
>  	vma->vm_file = NULL;
>  	fput(file);
> +	vma_put_encrypt_ref(vma);
>  
>  	/* Undo any partial mapping done by a device driver. */
>  	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
> @@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
>  	}
>  
> +	vma_get_encrypt_ref(new);
>  	err = vma_dup_policy(vma, new);
>  	if (err)
>  		goto out_free_vma;
> @@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>   out_free_mpol:
>  	mpol_put(vma_policy(new));
>   out_free_vma:
> +	vma_put_encrypt_ref(new);
>  	kmem_cache_free(vm_area_cachep, new);
>  	return err;
>  }
> @@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned
> long len, unsigned long fla
>  		return -ENOMEM;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_ops = &anon_vm_ops;
> @@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  		new_vma->vm_pgoff = pgoff;
>  		if (vma_dup_policy(vma, new_vma))
>  			goto out_free_vma;
> +		vma_get_encrypt_ref(new_vma);
>  		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
>  		if (anon_vma_clone(new_vma, vma))
>  			goto out_free_mempol;
> @@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  
>  out_free_mempol:
>  	mpol_put(vma_policy(new_vma));
> +	vma_put_encrypt_ref(new_vma);
>  out_free_vma:
>  	kmem_cache_free(vm_area_cachep, new_vma);
>  out:
> @@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
>  	if (unlikely(vma = NULL))
>  		return ERR_PTR(-ENOMEM);
>  
> +	/* Do not expect a memory encrypted vma here */
> +	VM_WARN_ON(vma_keyid(vma));
> +
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 73f66e81cfb0..85f04c174638 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	put_nommu_region(vma->vm_region);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  }
>  
> @@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
>  	if (!vma)
>  		goto error_getting_vma;
>  
> +	vma_get_encrypt_ref(vma);
>  	region->vm_usage = 1;
>  	region->vm_flags = vm_flags;
>  	region->vm_pgoff = pgoff;
> @@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
>  	kmem_cache_free(vm_region_jar, region);
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return ret;
>  
> @@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  	*new = *vma;
>  	*region = *vma->vm_region;
>  	new->vm_region = region;
> +	vma_get_encrypt_ref(new);
>  
>  	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
>  

[RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> Keep track of the VMA's oustanding for each memory encryption keyid.
> The count is used by the MKTME (Multi-Key Total Memory Encryption)
> Key Service to determine when it is safe to reprogram a hardware
> encryption key.

Maybe a stupid question but why they are tracked and what do you 
mean by tracking?

/Jarkko

> Approach here is to do gets and puts on the encryption reference
> wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> A couple of these locations will not be hit until cgroup support is
> added. One of these locations should never hit, so use a VM_WARN_ON.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/mm/mktme.c |  2 ++
>  kernel/fork.c       |  2 ++
>  mm/mmap.c           | 12 ++++++++++++
>  mm/nommu.c          |  4 ++++
>  4 files changed, 20 insertions(+)
> 
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 5690ef51a79a..8a7c326d4546 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int
> newkeyid)
>  	if (newkeyid == oldkeyid)
>  		return;
>  
> +	vma_put_encrypt_ref(vma);
>  	newprot = pgprot_val(vma->vm_page_prot);
>  	newprot &= ~mktme_keyid_mask;
>  	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
>  	vma->vm_page_prot = __pgprot(newprot);
> +	vma_get_encrypt_ref(vma);
>  }
>  
>  /*
> diff --git a/kernel/fork.c b/kernel/fork.c
> index e5e7a220a124..2d0e507bde7c 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  		if (!tmp)
>  			goto fail_nomem;
>  		*tmp = *mpnt;
> +		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
>  		INIT_LIST_HEAD(&tmp->anon_vma_chain);
>  		retval = vma_dup_policy(mpnt, tmp);
>  		if (retval)
> @@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  fail_nomem_anon_vma_fork:
>  	mpol_put(vma_policy(tmp));
>  fail_nomem_policy:
> +	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
>  	kmem_cache_free(vm_area_cachep, tmp);
>  fail_nomem:
>  	retval = -ENOMEM;
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 4c604eb644b4..7390b8b69fd6 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	mpol_put(vma_policy(vma));
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return next;
>  }
> @@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long
> start,
>  			anon_vma_merge(vma, next);
>  		mm->map_count--;
>  		mpol_put(vma_policy(next));
> +		vma_put_encrypt_ref(next);
>  		kmem_cache_free(vm_area_cachep, next);
>  		/*
>  		 * In mprotect's case 6 (see comments on vma_merge),
> @@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  		goto unacct_error;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
>  	vma->vm_end = addr + len;
> @@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  unmap_and_free_vma:
>  	vma->vm_file = NULL;
>  	fput(file);
> +	vma_put_encrypt_ref(vma);
>  
>  	/* Undo any partial mapping done by a device driver. */
>  	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
> @@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
>  	}
>  
> +	vma_get_encrypt_ref(new);
>  	err = vma_dup_policy(vma, new);
>  	if (err)
>  		goto out_free_vma;
> @@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>   out_free_mpol:
>  	mpol_put(vma_policy(new));
>   out_free_vma:
> +	vma_put_encrypt_ref(new);
>  	kmem_cache_free(vm_area_cachep, new);
>  	return err;
>  }
> @@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned
> long len, unsigned long fla
>  		return -ENOMEM;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_ops = &anon_vm_ops;
> @@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  		new_vma->vm_pgoff = pgoff;
>  		if (vma_dup_policy(vma, new_vma))
>  			goto out_free_vma;
> +		vma_get_encrypt_ref(new_vma);
>  		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
>  		if (anon_vma_clone(new_vma, vma))
>  			goto out_free_mempol;
> @@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  
>  out_free_mempol:
>  	mpol_put(vma_policy(new_vma));
> +	vma_put_encrypt_ref(new_vma);
>  out_free_vma:
>  	kmem_cache_free(vm_area_cachep, new_vma);
>  out:
> @@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
>  	if (unlikely(vma == NULL))
>  		return ERR_PTR(-ENOMEM);
>  
> +	/* Do not expect a memory encrypted vma here */
> +	VM_WARN_ON(vma_keyid(vma));
> +
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 73f66e81cfb0..85f04c174638 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	put_nommu_region(vma->vm_region);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  }
>  
> @@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
>  	if (!vma)
>  		goto error_getting_vma;
>  
> +	vma_get_encrypt_ref(vma);
>  	region->vm_usage = 1;
>  	region->vm_flags = vm_flags;
>  	region->vm_pgoff = pgoff;
> @@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
>  	kmem_cache_free(vm_region_jar, region);
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return ret;
>  
> @@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  	*new = *vma;
>  	*region = *vma->vm_region;
>  	new->vm_region = region;
> +	vma_get_encrypt_ref(new);
>  
>  	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
>  

Re: [RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Jarkko Sakkinen]

On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> Keep track of the VMA's oustanding for each memory encryption keyid.
> The count is used by the MKTME (Multi-Key Total Memory Encryption)
> Key Service to determine when it is safe to reprogram a hardware
> encryption key.

Maybe a stupid question but why they are tracked and what do you 
mean by tracking?

/Jarkko

> Approach here is to do gets and puts on the encryption reference
> wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> A couple of these locations will not be hit until cgroup support is
> added. One of these locations should never hit, so use a VM_WARN_ON.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  arch/x86/mm/mktme.c |  2 ++
>  kernel/fork.c       |  2 ++
>  mm/mmap.c           | 12 ++++++++++++
>  mm/nommu.c          |  4 ++++
>  4 files changed, 20 insertions(+)
> 
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 5690ef51a79a..8a7c326d4546 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -72,10 +72,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int
> newkeyid)
>  	if (newkeyid == oldkeyid)
>  		return;
>  
> +	vma_put_encrypt_ref(vma);
>  	newprot = pgprot_val(vma->vm_page_prot);
>  	newprot &= ~mktme_keyid_mask;
>  	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
>  	vma->vm_page_prot = __pgprot(newprot);
> +	vma_get_encrypt_ref(vma);
>  }
>  
>  /*
> diff --git a/kernel/fork.c b/kernel/fork.c
> index e5e7a220a124..2d0e507bde7c 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -459,6 +459,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  		if (!tmp)
>  			goto fail_nomem;
>  		*tmp = *mpnt;
> +		vma_get_encrypt_ref(tmp);	/* Track encrypted vma's */
>  		INIT_LIST_HEAD(&tmp->anon_vma_chain);
>  		retval = vma_dup_policy(mpnt, tmp);
>  		if (retval)
> @@ -539,6 +540,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>  fail_nomem_anon_vma_fork:
>  	mpol_put(vma_policy(tmp));
>  fail_nomem_policy:
> +	vma_put_encrypt_ref(tmp);		/* Track encrypted vma's */
>  	kmem_cache_free(vm_area_cachep, tmp);
>  fail_nomem:
>  	retval = -ENOMEM;
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 4c604eb644b4..7390b8b69fd6 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -182,6 +182,7 @@ static struct vm_area_struct *remove_vma(struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	mpol_put(vma_policy(vma));
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return next;
>  }
> @@ -913,6 +914,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long
> start,
>  			anon_vma_merge(vma, next);
>  		mm->map_count--;
>  		mpol_put(vma_policy(next));
> +		vma_put_encrypt_ref(next);
>  		kmem_cache_free(vm_area_cachep, next);
>  		/*
>  		 * In mprotect's case 6 (see comments on vma_merge),
> @@ -1744,6 +1746,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  		goto unacct_error;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
>  	vma->vm_end = addr + len;
> @@ -1839,6 +1842,7 @@ unsigned long mmap_region(struct file *file, unsigned
> long addr,
>  unmap_and_free_vma:
>  	vma->vm_file = NULL;
>  	fput(file);
> +	vma_put_encrypt_ref(vma);
>  
>  	/* Undo any partial mapping done by a device driver. */
>  	unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
> @@ -2653,6 +2657,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  		new->vm_pgoff += ((addr - vma->vm_start) >> PAGE_SHIFT);
>  	}
>  
> +	vma_get_encrypt_ref(new);
>  	err = vma_dup_policy(vma, new);
>  	if (err)
>  		goto out_free_vma;
> @@ -2686,6 +2691,7 @@ int __split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>   out_free_mpol:
>  	mpol_put(vma_policy(new));
>   out_free_vma:
> +	vma_put_encrypt_ref(new);
>  	kmem_cache_free(vm_area_cachep, new);
>  	return err;
>  }
> @@ -3007,6 +3013,7 @@ static int do_brk_flags(unsigned long addr, unsigned
> long len, unsigned long fla
>  		return -ENOMEM;
>  	}
>  
> +	vma_get_encrypt_ref(vma);
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_ops = &anon_vm_ops;
> @@ -3229,6 +3236,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  		new_vma->vm_pgoff = pgoff;
>  		if (vma_dup_policy(vma, new_vma))
>  			goto out_free_vma;
> +		vma_get_encrypt_ref(new_vma);
>  		INIT_LIST_HEAD(&new_vma->anon_vma_chain);
>  		if (anon_vma_clone(new_vma, vma))
>  			goto out_free_mempol;
> @@ -3243,6 +3251,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct
> **vmap,
>  
>  out_free_mempol:
>  	mpol_put(vma_policy(new_vma));
> +	vma_put_encrypt_ref(new_vma);
>  out_free_vma:
>  	kmem_cache_free(vm_area_cachep, new_vma);
>  out:
> @@ -3372,6 +3381,9 @@ static struct vm_area_struct *__install_special_mapping(
>  	if (unlikely(vma == NULL))
>  		return ERR_PTR(-ENOMEM);
>  
> +	/* Do not expect a memory encrypted vma here */
> +	VM_WARN_ON(vma_keyid(vma));
> +
>  	INIT_LIST_HEAD(&vma->anon_vma_chain);
>  	vma->vm_mm = mm;
>  	vma->vm_start = addr;
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 73f66e81cfb0..85f04c174638 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -769,6 +769,7 @@ static void delete_vma(struct mm_struct *mm, struct
> vm_area_struct *vma)
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	put_nommu_region(vma->vm_region);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  }
>  
> @@ -1215,6 +1216,7 @@ unsigned long do_mmap(struct file *file,
>  	if (!vma)
>  		goto error_getting_vma;
>  
> +	vma_get_encrypt_ref(vma);
>  	region->vm_usage = 1;
>  	region->vm_flags = vm_flags;
>  	region->vm_pgoff = pgoff;
> @@ -1375,6 +1377,7 @@ unsigned long do_mmap(struct file *file,
>  	kmem_cache_free(vm_region_jar, region);
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
> +	vma_put_encrypt_ref(vma);
>  	kmem_cache_free(vm_area_cachep, vma);
>  	return ret;
>  
> @@ -1486,6 +1489,7 @@ int split_vma(struct mm_struct *mm, struct
> vm_area_struct *vma,
>  	*new = *vma;
>  	*region = *vma->vm_region;
>  	new->vm_region = region;
> +	vma_get_encrypt_ref(new);
>  
>  	npages = (addr - vma->vm_start) >> PAGE_SHIFT;
>  

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTA5LTA3IGF0IDE1OjM3IC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBNZW1vcnkgZW5jcnlwdGlvbiBpcyBvbmx5IHN1cHBvcnRlZCBmb3IgbWFwcGluZ3MgdGhh
dCBhcmUgQU5PTllNT1VTLg0KPiBUZXN0IHRoZSBlbnRpcmUgcmFuZ2Ugb2YgVk1BJ3MgaW4gYW4g
ZW5jcnlwdF9tcHJvdGVjdCgpIHJlcXVlc3QgdG8NCj4gbWFrZSBzdXJlIHRoZXkgYWxsIG1lZXQg
dGhhdCByZXF1aXJlbWVudCBiZWZvcmUgZW5jcnlwdGluZyBhbnkuDQo+IA0KPiBUaGUgZW5jcnlw
dF9tcHJvdGVjdCBzeXNjYWxsIHdpbGwgcmV0dXJuIC1FSU5WQUwgYW5kIHdpbGwgbm90IGVuY3J5
cHQNCj4gYW55IFZNQSdzIGlmIHRoaXMgY2hlY2sgZmFpbHMuDQo+IA0KPiBTaWduZWQtb2ZmLWJ5
OiBBbGlzb24gU2Nob2ZpZWxkIDxhbGlzb24uc2Nob2ZpZWxkQGludGVsLmNvbT4NCj4gLS0tDQo+
ICBtbS9tcHJvdGVjdC5jIHwgMjIgKysrKysrKysrKysrKysrKysrKysrKw0KPiAgMSBmaWxlIGNo
YW5nZWQsIDIyIGluc2VydGlvbnMoKykNCj4gDQo+IGRpZmYgLS1naXQgYS9tbS9tcHJvdGVjdC5j
IGIvbW0vbXByb3RlY3QuYw0KPiBpbmRleCA2YzJlMTEwNjUyNWMuLjMzODRiNzU1YWFkMSAxMDA2
NDQNCj4gLS0tIGEvbW0vbXByb3RlY3QuYw0KPiArKysgYi9tbS9tcHJvdGVjdC5jDQo+IEBAIC0z
MTEsNiArMzExLDI0IEBAIHVuc2lnbmVkIGxvbmcgY2hhbmdlX3Byb3RlY3Rpb24oc3RydWN0IHZt
X2FyZWFfc3RydWN0DQo+ICp2bWEsIHVuc2lnbmVkIGxvbmcgc3RhcnQsDQo+ICAJcmV0dXJuIHBh
Z2VzOw0KPiAgfQ0KPiAgDQo+ICsvKg0KPiArICogRW5jcnlwdGVkIG1wcm90ZWN0IGlzIG9ubHkg
c3VwcG9ydGVkIG9uIGFub255bW91cyBtYXBwaW5ncy4NCj4gKyAqIEFsbCBWTUEncyBpbiB0aGUg
cmVxdWVzdGVkIHJhbmdlIG11c3QgYmUgYW5vbnltb3VzLiBJZiB0aGlzDQo+ICsgKiB0ZXN0IGZh
aWxzIG9uIGFueSBzaW5nbGUgVk1BLCB0aGUgZW50aXJlIG1wcm90ZWN0IHJlcXVlc3QgZmFpbHMu
DQo+ICsgKi8NCg0Ka2RvYw0KDQo+ICtib29sIG1lbV9zdXBwb3J0c19lbmNyeXB0aW9uKHN0cnVj
dCB2bV9hcmVhX3N0cnVjdCAqdm1hLCB1bnNpZ25lZCBsb25nIGVuZCkNCj4gK3sNCj4gKwlzdHJ1
Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnRlc3Rfdm1hID0gdm1hOw0KPiArDQo+ICsJZG8gew0KPiArCQlp
ZiAoIXZtYV9pc19hbm9ueW1vdXModGVzdF92bWEpKQ0KPiArCQkJcmV0dXJuIGZhbHNlOw0KPiAr
DQo+ICsJCXRlc3Rfdm1hID0gdGVzdF92bWEtPnZtX25leHQ7DQo+ICsJfSB3aGlsZSAodGVzdF92
bWEgJiYgdGVzdF92bWEtPnZtX3N0YXJ0IDwgZW5kKTsNCj4gKwlyZXR1cm4gdHJ1ZTsNCj4gK30N
Cj4gKw0KPiAgaW50DQo+ICBtcHJvdGVjdF9maXh1cChzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnZt
YSwgc3RydWN0IHZtX2FyZWFfc3RydWN0ICoqcHByZXYsDQo+ICAJICAgICAgIHVuc2lnbmVkIGxv
bmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgZW5kLCB1bnNpZ25lZCBsb25nDQo+IG5ld2ZsYWdzLA0K
PiBAQCAtNDkxLDYgKzUwOSwxMCBAQCBzdGF0aWMgaW50IGRvX21wcm90ZWN0X2V4dCh1bnNpZ25l
ZCBsb25nIHN0YXJ0LCBzaXplX3QNCj4gbGVuLA0KPiAgCQkJCWdvdG8gb3V0Ow0KPiAgCQl9DQo+
ICAJfQ0KPiArCWlmIChrZXlpZCA+IDAgJiYgIW1lbV9zdXBwb3J0c19lbmNyeXB0aW9uKHZtYSwg
ZW5kKSkgew0KPiArCQllcnJvciA9IC1FSU5WQUw7DQo+ICsJCWdvdG8gb3V0Ow0KPiArCX0NCj4g
IAlpZiAoc3RhcnQgPiB2bWEtPnZtX3N0YXJ0KQ0KPiAgCQlwcmV2ID0gdm1hOw0KPiAgDQoNCi9K
YXJra28

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> Memory encryption is only supported for mappings that are ANONYMOUS.
> Test the entire range of VMA's in an encrypt_mprotect() request to
> make sure they all meet that requirement before encrypting any.
> 
> The encrypt_mprotect syscall will return -EINVAL and will not encrypt
> any VMA's if this check fails.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  mm/mprotect.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 6c2e1106525c..3384b755aad1 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -311,6 +311,24 @@ unsigned long change_protection(struct vm_area_struct
> *vma, unsigned long start,
>  	return pages;
>  }
>  
> +/*
> + * Encrypted mprotect is only supported on anonymous mappings.
> + * All VMA's in the requested range must be anonymous. If this
> + * test fails on any single VMA, the entire mprotect request fails.
> + */

kdoc

> +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> +{
> +	struct vm_area_struct *test_vma = vma;
> +
> +	do {
> +		if (!vma_is_anonymous(test_vma))
> +			return false;
> +
> +		test_vma = test_vma->vm_next;
> +	} while (test_vma && test_vma->vm_start < end);
> +	return true;
> +}
> +
>  int
>  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
>  	       unsigned long start, unsigned long end, unsigned long
> newflags,
> @@ -491,6 +509,10 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  				goto out;
>  		}
>  	}
> +	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
> +		error = -EINVAL;
> +		goto out;
> +	}
>  	if (start > vma->vm_start)
>  		prev = vma;
>  

/Jarkko

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> Memory encryption is only supported for mappings that are ANONYMOUS.
> Test the entire range of VMA's in an encrypt_mprotect() request to
> make sure they all meet that requirement before encrypting any.
> 
> The encrypt_mprotect syscall will return -EINVAL and will not encrypt
> any VMA's if this check fails.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  mm/mprotect.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index 6c2e1106525c..3384b755aad1 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -311,6 +311,24 @@ unsigned long change_protection(struct vm_area_struct
> *vma, unsigned long start,
>  	return pages;
>  }
>  
> +/*
> + * Encrypted mprotect is only supported on anonymous mappings.
> + * All VMA's in the requested range must be anonymous. If this
> + * test fails on any single VMA, the entire mprotect request fails.
> + */

kdoc

> +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> +{
> +	struct vm_area_struct *test_vma = vma;
> +
> +	do {
> +		if (!vma_is_anonymous(test_vma))
> +			return false;
> +
> +		test_vma = test_vma->vm_next;
> +	} while (test_vma && test_vma->vm_start < end);
> +	return true;
> +}
> +
>  int
>  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
>  	       unsigned long start, unsigned long end, unsigned long
> newflags,
> @@ -491,6 +509,10 @@ static int do_mprotect_ext(unsigned long start, size_t
> len,
>  				goto out;
>  		}
>  	}
> +	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
> +		error = -EINVAL;
> +		goto out;
> +	}
>  	if (start > vma->vm_start)
>  		prev = vma;
>  

/Jarkko

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTA5LTA3IGF0IDE1OjM4IC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBUaGUga2VybmVsIG1hbmFnZXMgdGhlIE1LVE1FIChNdWx0aS1LZXkgVG90YWwgTWVtb3J5
IEVuY3J5cHRpb24pIEtleXMNCj4gYXMgYSBzeXN0ZW0gd2lkZSBzaW5nbGUgcG9vbCBvZiBrZXlz
LiBUaGUgaGFyZHdhcmUsIGhvd2V2ZXIsIG1hbmFnZXMNCj4gdGhlIGtleXMgb24gYSBwZXIgcGh5
c2ljYWwgcGFja2FnZSBiYXNpcy4gRWFjaCBwaHlzaWNhbCBwYWNrYWdlDQo+IG1haW50YWlucyBh
IGtleSB0YWJsZSB0aGF0IGFsbCBDUFUncyBpbiB0aGF0IHBhY2thZ2Ugc2hhcmUuDQo+IA0KPiBJ
biBvcmRlciB0byBtYWludGFpbiB0aGUgY29uc2lzdGVudCwgc3lzdGVtIHdpZGUgdmlldyB0aGF0
IHRoZSBrZXJuZWwNCj4gcmVxdWlyZXMsIHByb2dyYW0gYWxsIHBoeXNpY2FsIHBhY2thZ2VzIGR1
cmluZyBhIGtleSBwcm9ncmFtIHJlcXVlc3QuDQo+IA0KPiBTaWduZWQtb2ZmLWJ5OiBBbGlzb24g
U2Nob2ZpZWxkIDxhbGlzb24uc2Nob2ZpZWxkQGludGVsLmNvbT4NCg0KSnVzdCBraW5kIG9mIGNo
ZWNraW5nIHRoYXQgYXJlIHlvdSB0YWxraW5nIGFib3V0IG11bHRpcGxlIGNvcmVzIGluDQphIHNp
bmdsZSBwYWNrYWdlIG9yIHJlYWxseSBtdWx0aXBsZSBwYWNrYWdlcz8NCg0KL0phcmtrbw=

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> as a system wide single pool of keys. The hardware, however, manages
> the keys on a per physical package basis. Each physical package
> maintains a key table that all CPU's in that package share.
> 
> In order to maintain the consistent, system wide view that the kernel
> requires, program all physical packages during a key program request.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Just kind of checking that are you talking about multiple cores in
a single package or really multiple packages?

/Jarkko

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Sakkinen, Jarkko]

On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> as a system wide single pool of keys. The hardware, however, manages
> the keys on a per physical package basis. Each physical package
> maintains a key table that all CPU's in that package share.
> 
> In order to maintain the consistent, system wide view that the kernel
> requires, program all physical packages during a key program request.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Just kind of checking that are you talking about multiple cores in
a single package or really multiple packages?

/Jarkko

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>> +/*
>> + * Encrypted mprotect is only supported on anonymous mappings.
>> + * All VMA's in the requested range must be anonymous. If this
>> + * test fails on any single VMA, the entire mprotect request fails.
>> + */
> kdoc

kdoc what?  You want this comment in kdoc format?  Why?

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>> +/*
>> + * Encrypted mprotect is only supported on anonymous mappings.
>> + * All VMA's in the requested range must be anonymous. If this
>> + * test fails on any single VMA, the entire mprotect request fails.
>> + */
> kdoc

kdoc what?  You want this comment in kdoc format?  Why?

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>> +/*
>> + * Encrypted mprotect is only supported on anonymous mappings.
>> + * All VMA's in the requested range must be anonymous. If this
>> + * test fails on any single VMA, the entire mprotect request fails.
>> + */
> kdoc

kdoc what?  You want this comment in kdoc format?  Why?

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:10:19PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> > owner@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:23 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)
> > 
> > Seeking comments on the APIs supporting MKTME on future Intel platforms.
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> > memory encryption on upcoming Intel platforms. Whereas TME allows
> > encryption of the entire system memory using a single key, MKTME allows
> > mulitple encryption domains, each having their own key. While the main use
> > case for the feature is virtual machine isolation, the API needs the flexibility to
> > work for a wide range of use cases.
> > 
> > This RFC presents the 2 API additions that enable userspace to:
> >  1) Create Encryption Keys: Kernel Key Service type "mktme"
> >  2) Use the Encryption Keys: system call encrypt_mprotect()
> > 
> > In order to share between: the Kernel Key Service, the new system call, and the
> > existing mm code, helper functions were created in arch/x86/mktme
> 
> IMHO, we can separate this series into 2 parts, as you did above, and send out them separately. The reason is, in general I think adding new MKTME type to key retention services is not that related to memory management code, namely the encrypt_mprotect() API part.
> 
> So if we split the two parts and send them out separately, the first part can be reviewed by keyring and security guys, without involving mm guys, and the encrypt_mprotect() part can be more reviewed more by mm guys. 
> 

Kai,

That was the direction I had in mind at the onset: the MKTME key service
would be one patch(set) and the MKTME encrypt_mprotect() system call would
be delivered in another patch(set).

That separation falls apart when the shared structures and functions are
introduced. That 'mktme_map' (maps userspace keys to hardware keyid slots),
and the 'encrypt_count' array (counts vma's outstanding for each key) need
to be shared by both pieces. These mktme special shared structures and the
functions that operate on them are all defined in arch/x86/mm/mktme.h,.c.
From there they can be shared with the security/keys/mktme_keys.c

Once I made that separation, I stuck with it. Those structures, and any
functions that manipulate those structures live in arch/x86/mm/mktme.h,c

You noted that some of the functions that operate on the encrypt_count
might not need to be over in arch/x86/mm/mktme.c because they are not used
in the mm code. That is true.  But, then I'd be splitting up the definition
of the struct and the funcs that operate on it. So, I stuck with keeping it
all together in the arch specific mktme files.

Having said all the above, I do welcome other ideas on how to better organize
the code. 

Back to your request- to split it into smaller patchsets might look something
like:
1) the MKTME API helpers
2) the MKTME Key Service
3) the MKTME syscall encrypt_mprotect()

I'm not clear that would make anyones review life easier, than picking
the same pieces out of the greater patchset.

Suggestions welcome,
Alison

> And since encrypt_mprotect() is a new syscall, you may need to add more lists for the review, ie, linux-api, and maybe linux-kernel as well.

Got it. Will include theses in v1.

> 
> Thanks,
> -Kai
> 
> > 
> > This patchset is built upon Kirill Shutemov's patchset for the core MKTME
> > support. You can find that here:
> > git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip
> > 
> > 
> > Alison Schofield (12):
> >   docs/x86: Document the Multi-Key Total Memory Encryption API
> >   mm: Generalize the mprotect implementation to support extensions
> >   syscall/x86: Wire up a new system call for memory encryption keys
> >   x86/mm: Add helper functions to manage memory encryption keys
> >   x86/mm: Add a helper function to set keyid bits in encrypted VMA's
> >   mm: Add the encrypt_mprotect() system call
> >   x86/mm: Add helper functions to track encrypted VMA's
> >   mm: Track VMA's in use for each memory encryption keyid
> >   mm: Restrict memory encryption to anonymous VMA's
> >   x86/pconfig: Program memory encryption keys on a system-wide basis
> >   keys/mktme: Add a new key service type for memory encryption keys
> >   keys/mktme: Do not revoke in use memory encryption keys
> > 
> >  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
> >  arch/x86/Kconfig                       |   1 +
> >  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
> >  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
> >  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
> >  arch/x86/include/asm/mktme.h           |  21 +++
> >  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
> >  fs/exec.c                              |   4 +-
> >  include/keys/mktme-type.h              |  28 +++
> >  include/linux/key.h                    |   2 +
> >  include/linux/mm.h                     |   9 +-
> >  include/linux/syscalls.h               |   2 +
> >  include/uapi/asm-generic/unistd.h      |   4 +-
> >  kernel/fork.c                          |   2 +
> >  kernel/sys_ni.c                        |   2 +
> >  mm/mmap.c                              |  12 ++
> >  mm/mprotect.c                          |  93 +++++++++-
> >  mm/nommu.c                             |   4 +
> >  security/keys/Kconfig                  |  11 ++
> >  security/keys/Makefile                 |   1 +
> >  security/keys/internal.h               |   6 +
> >  security/keys/keyctl.c                 |   7 +
> >  security/keys/mktme_keys.c             | 325
> > +++++++++++++++++++++++++++++++++
> >  23 files changed, 855 insertions(+), 17 deletions(-)  create mode 100644
> > Documentation/x86/mktme-keys.txt  create mode 100644 include/keys/mktme-
> > type.h  create mode 100644 security/keys/mktme_keys.c
> > 
> > --
> > 2.14.1
> 

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:10:19PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> > owner at vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:23 AM
> > To: dhowells at redhat.com; tglx at linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> > x86 at kernel.org; linux-mm at kvack.org
> > Subject: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)
> > 
> > Seeking comments on the APIs supporting MKTME on future Intel platforms.
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> > memory encryption on upcoming Intel platforms. Whereas TME allows
> > encryption of the entire system memory using a single key, MKTME allows
> > mulitple encryption domains, each having their own key. While the main use
> > case for the feature is virtual machine isolation, the API needs the flexibility to
> > work for a wide range of use cases.
> > 
> > This RFC presents the 2 API additions that enable userspace to:
> >  1) Create Encryption Keys: Kernel Key Service type "mktme"
> >  2) Use the Encryption Keys: system call encrypt_mprotect()
> > 
> > In order to share between: the Kernel Key Service, the new system call, and the
> > existing mm code, helper functions were created in arch/x86/mktme
> 
> IMHO, we can separate this series into 2 parts, as you did above, and send out them separately. The reason is, in general I think adding new MKTME type to key retention services is not that related to memory management code, namely the encrypt_mprotect() API part.
> 
> So if we split the two parts and send them out separately, the first part can be reviewed by keyring and security guys, without involving mm guys, and the encrypt_mprotect() part can be more reviewed more by mm guys. 
> 

Kai,

That was the direction I had in mind at the onset: the MKTME key service
would be one patch(set) and the MKTME encrypt_mprotect() system call would
be delivered in another patch(set).

That separation falls apart when the shared structures and functions are
introduced. That 'mktme_map' (maps userspace keys to hardware keyid slots),
and the 'encrypt_count' array (counts vma's outstanding for each key) need
to be shared by both pieces. These mktme special shared structures and the
functions that operate on them are all defined in arch/x86/mm/mktme.h,.c.
>From there they can be shared with the security/keys/mktme_keys.c

Once I made that separation, I stuck with it. Those structures, and any
functions that manipulate those structures live in arch/x86/mm/mktme.h,c

You noted that some of the functions that operate on the encrypt_count
might not need to be over in arch/x86/mm/mktme.c because they are not used
in the mm code. That is true.  But, then I'd be splitting up the definition
of the struct and the funcs that operate on it. So, I stuck with keeping it
all together in the arch specific mktme files.

Having said all the above, I do welcome other ideas on how to better organize
the code. 

Back to your request- to split it into smaller patchsets might look something
like:
1) the MKTME API helpers
2) the MKTME Key Service
3) the MKTME syscall encrypt_mprotect()

I'm not clear that would make anyones review life easier, than picking
the same pieces out of the greater patchset.

Suggestions welcome,
Alison

> And since encrypt_mprotect() is a new syscall, you may need to add more lists for the review, ie, linux-api, and maybe linux-kernel as well.

Got it. Will include theses in v1.

> 
> Thanks,
> -Kai
> 
> > 
> > This patchset is built upon Kirill Shutemov's patchset for the core MKTME
> > support. You can find that here:
> > git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip
> > 
> > 
> > Alison Schofield (12):
> >   docs/x86: Document the Multi-Key Total Memory Encryption API
> >   mm: Generalize the mprotect implementation to support extensions
> >   syscall/x86: Wire up a new system call for memory encryption keys
> >   x86/mm: Add helper functions to manage memory encryption keys
> >   x86/mm: Add a helper function to set keyid bits in encrypted VMA's
> >   mm: Add the encrypt_mprotect() system call
> >   x86/mm: Add helper functions to track encrypted VMA's
> >   mm: Track VMA's in use for each memory encryption keyid
> >   mm: Restrict memory encryption to anonymous VMA's
> >   x86/pconfig: Program memory encryption keys on a system-wide basis
> >   keys/mktme: Add a new key service type for memory encryption keys
> >   keys/mktme: Do not revoke in use memory encryption keys
> > 
> >  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
> >  arch/x86/Kconfig                       |   1 +
> >  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
> >  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
> >  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
> >  arch/x86/include/asm/mktme.h           |  21 +++
> >  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
> >  fs/exec.c                              |   4 +-
> >  include/keys/mktme-type.h              |  28 +++
> >  include/linux/key.h                    |   2 +
> >  include/linux/mm.h                     |   9 +-
> >  include/linux/syscalls.h               |   2 +
> >  include/uapi/asm-generic/unistd.h      |   4 +-
> >  kernel/fork.c                          |   2 +
> >  kernel/sys_ni.c                        |   2 +
> >  mm/mmap.c                              |  12 ++
> >  mm/mprotect.c                          |  93 +++++++++-
> >  mm/nommu.c                             |   4 +
> >  security/keys/Kconfig                  |  11 ++
> >  security/keys/Makefile                 |   1 +
> >  security/keys/internal.h               |   6 +
> >  security/keys/keyctl.c                 |   7 +
> >  security/keys/mktme_keys.c             | 325
> > +++++++++++++++++++++++++++++++++
> >  23 files changed, 855 insertions(+), 17 deletions(-)  create mode 100644
> > Documentation/x86/mktme-keys.txt  create mode 100644 include/keys/mktme-
> > type.h  create mode 100644 security/keys/mktme_keys.c
> > 
> > --
> > 2.14.1
> 

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:10:19PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> > owner@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:23 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)
> > 
> > Seeking comments on the APIs supporting MKTME on future Intel platforms.
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology supporting
> > memory encryption on upcoming Intel platforms. Whereas TME allows
> > encryption of the entire system memory using a single key, MKTME allows
> > mulitple encryption domains, each having their own key. While the main use
> > case for the feature is virtual machine isolation, the API needs the flexibility to
> > work for a wide range of use cases.
> > 
> > This RFC presents the 2 API additions that enable userspace to:
> >  1) Create Encryption Keys: Kernel Key Service type "mktme"
> >  2) Use the Encryption Keys: system call encrypt_mprotect()
> > 
> > In order to share between: the Kernel Key Service, the new system call, and the
> > existing mm code, helper functions were created in arch/x86/mktme
> 
> IMHO, we can separate this series into 2 parts, as you did above, and send out them separately. The reason is, in general I think adding new MKTME type to key retention services is not that related to memory management code, namely the encrypt_mprotect() API part.
> 
> So if we split the two parts and send them out separately, the first part can be reviewed by keyring and security guys, without involving mm guys, and the encrypt_mprotect() part can be more reviewed more by mm guys. 
> 

Kai,

That was the direction I had in mind at the onset: the MKTME key service
would be one patch(set) and the MKTME encrypt_mprotect() system call would
be delivered in another patch(set).

That separation falls apart when the shared structures and functions are
introduced. That 'mktme_map' (maps userspace keys to hardware keyid slots),
and the 'encrypt_count' array (counts vma's outstanding for each key) need
to be shared by both pieces. These mktme special shared structures and the
functions that operate on them are all defined in arch/x86/mm/mktme.h,.c.
>From there they can be shared with the security/keys/mktme_keys.c

Once I made that separation, I stuck with it. Those structures, and any
functions that manipulate those structures live in arch/x86/mm/mktme.h,c

You noted that some of the functions that operate on the encrypt_count
might not need to be over in arch/x86/mm/mktme.c because they are not used
in the mm code. That is true.  But, then I'd be splitting up the definition
of the struct and the funcs that operate on it. So, I stuck with keeping it
all together in the arch specific mktme files.

Having said all the above, I do welcome other ideas on how to better organize
the code. 

Back to your request- to split it into smaller patchsets might look something
like:
1) the MKTME API helpers
2) the MKTME Key Service
3) the MKTME syscall encrypt_mprotect()

I'm not clear that would make anyones review life easier, than picking
the same pieces out of the greater patchset.

Suggestions welcome,
Alison

> And since encrypt_mprotect() is a new syscall, you may need to add more lists for the review, ie, linux-api, and maybe linux-kernel as well.

Got it. Will include theses in v1.

> 
> Thanks,
> -Kai
> 
> > 
> > This patchset is built upon Kirill Shutemov's patchset for the core MKTME
> > support. You can find that here:
> > git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git mktme/wip
> > 
> > 
> > Alison Schofield (12):
> >   docs/x86: Document the Multi-Key Total Memory Encryption API
> >   mm: Generalize the mprotect implementation to support extensions
> >   syscall/x86: Wire up a new system call for memory encryption keys
> >   x86/mm: Add helper functions to manage memory encryption keys
> >   x86/mm: Add a helper function to set keyid bits in encrypted VMA's
> >   mm: Add the encrypt_mprotect() system call
> >   x86/mm: Add helper functions to track encrypted VMA's
> >   mm: Track VMA's in use for each memory encryption keyid
> >   mm: Restrict memory encryption to anonymous VMA's
> >   x86/pconfig: Program memory encryption keys on a system-wide basis
> >   keys/mktme: Add a new key service type for memory encryption keys
> >   keys/mktme: Do not revoke in use memory encryption keys
> > 
> >  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
> >  arch/x86/Kconfig                       |   1 +
> >  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
> >  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
> >  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
> >  arch/x86/include/asm/mktme.h           |  21 +++
> >  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
> >  fs/exec.c                              |   4 +-
> >  include/keys/mktme-type.h              |  28 +++
> >  include/linux/key.h                    |   2 +
> >  include/linux/mm.h                     |   9 +-
> >  include/linux/syscalls.h               |   2 +
> >  include/uapi/asm-generic/unistd.h      |   4 +-
> >  kernel/fork.c                          |   2 +
> >  kernel/sys_ni.c                        |   2 +
> >  mm/mmap.c                              |  12 ++
> >  mm/mprotect.c                          |  93 +++++++++-
> >  mm/nommu.c                             |   4 +
> >  security/keys/Kconfig                  |  11 ++
> >  security/keys/Makefile                 |   1 +
> >  security/keys/internal.h               |   6 +
> >  security/keys/keyctl.c                 |   7 +
> >  security/keys/mktme_keys.c             | 325
> > +++++++++++++++++++++++++++++++++
> >  23 files changed, 855 insertions(+), 17 deletions(-)  create mode 100644
> > Documentation/x86/mktme-keys.txt  create mode 100644 include/keys/mktme-
> > type.h  create mode 100644 security/keys/mktme_keys.c
> > 
> > --
> > 2.14.1
> 

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
> >> +/*
> >> + * Encrypted mprotect is only supported on anonymous mappings.
> >> + * All VMA's in the requested range must be anonymous. If this
> >> + * test fails on any single VMA, the entire mprotect request fails.
> >> + */
> > kdoc
> 
> kdoc what?  You want this comment in kdoc format?  Why?

If there is a header comment for a function anyway, why wouldn't you
put it to kdoc-format?

/Jarkko

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
> >> +/*
> >> + * Encrypted mprotect is only supported on anonymous mappings.
> >> + * All VMA's in the requested range must be anonymous. If this
> >> + * test fails on any single VMA, the entire mprotect request fails.
> >> + */
> > kdoc
> 
> kdoc what?  You want this comment in kdoc format?  Why?

If there is a header comment for a function anyway, why wouldn't you
put it to kdoc-format?

/Jarkko

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
> >> +/*
> >> + * Encrypted mprotect is only supported on anonymous mappings.
> >> + * All VMA's in the requested range must be anonymous. If this
> >> + * test fails on any single VMA, the entire mprotect request fails.
> >> + */
> > kdoc
> 
> kdoc what?  You want this comment in kdoc format?  Why?

If there is a header comment for a function anyway, why wouldn't you
put it to kdoc-format?

/Jarkko

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 02:07 PM, Jarkko Sakkinen wrote:
> On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
>> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>>>> +/*
>>>> + * Encrypted mprotect is only supported on anonymous mappings.
>>>> + * All VMA's in the requested range must be anonymous. If this
>>>> + * test fails on any single VMA, the entire mprotect request fails.
>>>> + */
>>> kdoc
>> kdoc what?  You want this comment in kdoc format?  Why?
> If there is a header comment for a function anyway, why wouldn't you
> put it to kdoc-format?

Because this is a shorter and more concise way to document the function.
 kdoc isn't universally the best thing to do for function comments.

[RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 02:07 PM, Jarkko Sakkinen wrote:
> On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
>> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>>>> +/*
>>>> + * Encrypted mprotect is only supported on anonymous mappings.
>>>> + * All VMA's in the requested range must be anonymous. If this
>>>> + * test fails on any single VMA, the entire mprotect request fails.
>>>> + */
>>> kdoc
>> kdoc what?  You want this comment in kdoc format?  Why?
> If there is a header comment for a function anyway, why wouldn't you
> put it to kdoc-format?

Because this is a shorter and more concise way to document the function.
 kdoc isn't universally the best thing to do for function comments.

Re: [RFC 09/12] mm: Restrict memory encryption to anonymous VMA's

[Dave Hansen]

On 09/10/2018 02:07 PM, Jarkko Sakkinen wrote:
> On Mon, Sep 10, 2018 at 11:57:49AM -0700, Dave Hansen wrote:
>> On 09/10/2018 11:21 AM, Sakkinen, Jarkko wrote:
>>>> +/*
>>>> + * Encrypted mprotect is only supported on anonymous mappings.
>>>> + * All VMA's in the requested range must be anonymous. If this
>>>> + * test fails on any single VMA, the entire mprotect request fails.
>>>> + */
>>> kdoc
>> kdoc what?  You want this comment in kdoc format?  Why?
> If there is a header comment for a function anyway, why wouldn't you
> put it to kdoc-format?

Because this is a shorter and more concise way to document the function.
 kdoc isn't universally the best thing to do for function comments.

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> > owner@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:39 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> > encryption keys
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > transparent memory encryption in upcoming Intel platforms. MKTME will
> > support mulitple encryption domains, each having their own key. The main use
> > case for the feature is virtual machine isolation. The API needs the flexibility to
> > work for a wide range of uses.
> > 
> > The MKTME key service type manages the addition and removal of the memory
> > encryption keys. It maps software keys to hardware keyids and programs the
> > hardware with the user requested encryption options.
> > 
> > The only supported encryption algorithm is AES-XTS 128.
> > 
> > The MKTME key service is half of the MKTME API level solution. It pairs with a
> > new memory encryption system call: encrypt_mprotect() that uses the keys to
> > encrypt memory.
> > 


Kai -
Splitting out responses by subject...

> > +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> > keys */
> 
> Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)
> 
> But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

I'll try that out and see how it works.

> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid > > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid = online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Sure - I'll look at those. 

Thanks!
Alison

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> > owner at vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:39 AM
> > To: dhowells at redhat.com; tglx at linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> > x86 at kernel.org; linux-mm at kvack.org
> > Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> > encryption keys
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > transparent memory encryption in upcoming Intel platforms. MKTME will
> > support mulitple encryption domains, each having their own key. The main use
> > case for the feature is virtual machine isolation. The API needs the flexibility to
> > work for a wide range of uses.
> > 
> > The MKTME key service type manages the addition and removal of the memory
> > encryption keys. It maps software keys to hardware keyids and programs the
> > hardware with the user requested encryption options.
> > 
> > The only supported encryption algorithm is AES-XTS 128.
> > 
> > The MKTME key service is half of the MKTME API level solution. It pairs with a
> > new memory encryption system call: encrypt_mprotect() that uses the keys to
> > encrypt memory.
> > 


Kai -
Splitting out responses by subject...

> > +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> > keys */
> 
> Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)
> 
> But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

I'll try that out and see how it works.

> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid =
> > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid == online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Sure - I'll look at those. 

Thanks!
Alison

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> > owner@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:39 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 11/12] keys/mktme: Add a new key service type for memory
> > encryption keys
> > 
> > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > transparent memory encryption in upcoming Intel platforms. MKTME will
> > support mulitple encryption domains, each having their own key. The main use
> > case for the feature is virtual machine isolation. The API needs the flexibility to
> > work for a wide range of uses.
> > 
> > The MKTME key service type manages the addition and removal of the memory
> > encryption keys. It maps software keys to hardware keyids and programs the
> > hardware with the user requested encryption options.
> > 
> > The only supported encryption algorithm is AES-XTS 128.
> > 
> > The MKTME key service is half of the MKTME API level solution. It pairs with a
> > new memory encryption system call: encrypt_mprotect() that uses the keys to
> > encrypt memory.
> > 


Kai -
Splitting out responses by subject...

> > +cpumask_var_t mktme_cpumask;		/* one cpu per pkg to program
> > keys */
> 
> Oh the 'mktme_cpumask' is here. Sorry I didn't notice when replying to your patch 10. :)
> 
> But I think you can just move what you did in patch 10 here and leave intel_pconfig.h unchanged. It's much clearer. 

I'll try that out and see how it works.

> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid =
> > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid == online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Sure - I'll look at those. 

Thanks!
Alison

RE: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> security-module@vger.kernel.org] On Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 2:57 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> tglx@linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> 
> > -----Original Message-----
> > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > security-module@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:36 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> >
> > Define a global mapping structure to track the mapping of userspace
> > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > This data will be used for the memory encryption system call and the
> > kernel key service API.
> >
> > Implement helper functions to access this mapping structure and make
> > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/include/asm/mktme.h | 11 ++++++
> >  arch/x86/mm/mktme.c          | 85
> > ++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 96 insertions(+)
> 
> Maybe it's better to put those changes to include/keys/mktme-type.h, and
> security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> x86 guys by doing so?
> 
> Thanks,
> -Kai
> >
> > diff --git a/arch/x86/include/asm/mktme.h
> > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > --- a/arch/x86/include/asm/mktme.h
> > +++ b/arch/x86/include/asm/mktme.h
> > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > mktme_nr_keyids;  extern int mktme_keyid_shift;
> >
> > +/* Manage mappings between hardware keyids and userspace keys */
> > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > +extern int mktme_map_get_free_keyid(void); extern void
> > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > +keyid, unsigned int serial); extern int
> > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > +mktme_map_serial_from_keyid(int keyid);
> > +
> >  extern struct page_ext_operations page_mktme_ops;
> >
> >  #define page_keyid page_keyid
> > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > 660caf6a5ce1..5246d8323359 100644
> > --- a/arch/x86/mm/mktme.c
> > +++ b/arch/x86/mm/mktme.c
> > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> >
> > +/*
> > + * struct mktme_mapping and the mktme_map_* functions manage the
> > +mapping
> > + * of userspace keys to hardware keyids in MKTME. They are used by
> > +the
> > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > + */
> > +struct mktme_mapping {
> > +	struct mutex	lock;		/* protect this map & HW state */
> > +	unsigned int	mapped_keyids;
> > +	unsigned int	serial[];
> > +};

Sorry one more comment that I missed yesterday:

I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 

Thanks,
-Kai

> > +
> > +struct mktme_mapping *mktme_map;
> > +
> > +static inline long mktme_map_size(void) {
> > +	long size = 0;
> > +
> > +	size += sizeof(mktme_map);
> > +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> > +	return size;
> > +}
> > +
> > +int mktme_map_alloc(void)
> > +{
> > +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> > +	if (!mktme_map)
> > +		return 0;
> > +	mutex_init(&mktme_map->lock);
> > +	return 1;
> > +}
> > +
> > +void mktme_map_free(void)
> > +{
> > +	kfree(mktme_map);
> > +}
> > +
> > +void mktme_map_lock(void)
> > +{
> > +	mutex_lock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_unlock(void)
> > +{
> > +	mutex_unlock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> > +	mktme_map->serial[keyid] = serial;
> > +	mktme_map->mapped_keyids++;
> > +}
> > +
> > +void mktme_map_clear_keyid(int keyid) {
> > +	mktme_map->serial[keyid] = 0;
> > +	mktme_map->mapped_keyids--;
> > +}
> > +
> > +unsigned int mktme_map_serial_from_keyid(int keyid) {
> > +	return mktme_map->serial[keyid];
> > +}
> > +
> > +int mktme_map_keyid_from_serial(unsigned int serial) {
> > +	int i;
> > +
> > +	for (i = 1; i < mktme_nr_keyids; i++)
> > +		if (mktme_map->serial[i] = serial)
> > +			return i;
> > +	return 0;
> > +}
> > +
> > +int mktme_map_get_free_keyid(void)
> > +{
> > +	int i;
> > +
> > +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> > +		for (i = 1; i < mktme_nr_keyids; i++)
> > +			if (mktme_map->serial[i] = 0)
> > +				return i;
> > +	}
> > +	return 0;
> > +}
> > +
> >  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
> >  	int i;
> > --
> > 2.14.1

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module at vger.kernel.org [mailto:owner-linux-
> security-module at vger.kernel.org] On Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 2:57 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells at redhat.com;
> tglx at linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> 
> > -----Original Message-----
> > From: owner-linux-security-module at vger.kernel.org [mailto:owner-linux-
> > security-module at vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:36 AM
> > To: dhowells at redhat.com; tglx at linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> >
> > Define a global mapping structure to track the mapping of userspace
> > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > This data will be used for the memory encryption system call and the
> > kernel key service API.
> >
> > Implement helper functions to access this mapping structure and make
> > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/include/asm/mktme.h | 11 ++++++
> >  arch/x86/mm/mktme.c          | 85
> > ++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 96 insertions(+)
> 
> Maybe it's better to put those changes to include/keys/mktme-type.h, and
> security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> x86 guys by doing so?
> 
> Thanks,
> -Kai
> >
> > diff --git a/arch/x86/include/asm/mktme.h
> > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > --- a/arch/x86/include/asm/mktme.h
> > +++ b/arch/x86/include/asm/mktme.h
> > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > mktme_nr_keyids;  extern int mktme_keyid_shift;
> >
> > +/* Manage mappings between hardware keyids and userspace keys */
> > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > +extern int mktme_map_get_free_keyid(void); extern void
> > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > +keyid, unsigned int serial); extern int
> > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > +mktme_map_serial_from_keyid(int keyid);
> > +
> >  extern struct page_ext_operations page_mktme_ops;
> >
> >  #define page_keyid page_keyid
> > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > 660caf6a5ce1..5246d8323359 100644
> > --- a/arch/x86/mm/mktme.c
> > +++ b/arch/x86/mm/mktme.c
> > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> >
> > +/*
> > + * struct mktme_mapping and the mktme_map_* functions manage the
> > +mapping
> > + * of userspace keys to hardware keyids in MKTME. They are used by
> > +the
> > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > + */
> > +struct mktme_mapping {
> > +	struct mutex	lock;		/* protect this map & HW state */
> > +	unsigned int	mapped_keyids;
> > +	unsigned int	serial[];
> > +};

Sorry one more comment that I missed yesterday:

I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 

Thanks,
-Kai

> > +
> > +struct mktme_mapping *mktme_map;
> > +
> > +static inline long mktme_map_size(void) {
> > +	long size = 0;
> > +
> > +	size += sizeof(mktme_map);
> > +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> > +	return size;
> > +}
> > +
> > +int mktme_map_alloc(void)
> > +{
> > +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> > +	if (!mktme_map)
> > +		return 0;
> > +	mutex_init(&mktme_map->lock);
> > +	return 1;
> > +}
> > +
> > +void mktme_map_free(void)
> > +{
> > +	kfree(mktme_map);
> > +}
> > +
> > +void mktme_map_lock(void)
> > +{
> > +	mutex_lock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_unlock(void)
> > +{
> > +	mutex_unlock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> > +	mktme_map->serial[keyid] = serial;
> > +	mktme_map->mapped_keyids++;
> > +}
> > +
> > +void mktme_map_clear_keyid(int keyid) {
> > +	mktme_map->serial[keyid] = 0;
> > +	mktme_map->mapped_keyids--;
> > +}
> > +
> > +unsigned int mktme_map_serial_from_keyid(int keyid) {
> > +	return mktme_map->serial[keyid];
> > +}
> > +
> > +int mktme_map_keyid_from_serial(unsigned int serial) {
> > +	int i;
> > +
> > +	for (i = 1; i < mktme_nr_keyids; i++)
> > +		if (mktme_map->serial[i] == serial)
> > +			return i;
> > +	return 0;
> > +}
> > +
> > +int mktme_map_get_free_keyid(void)
> > +{
> > +	int i;
> > +
> > +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> > +		for (i = 1; i < mktme_nr_keyids; i++)
> > +			if (mktme_map->serial[i] == 0)
> > +				return i;
> > +	}
> > +	return 0;
> > +}
> > +
> >  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
> >  	int i;
> > --
> > 2.14.1

RE: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> security-module@vger.kernel.org] On Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 2:57 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> tglx@linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> encryption keys
> 
> 
> > -----Original Message-----
> > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > security-module@vger.kernel.org] On Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:36 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> >
> > Define a global mapping structure to track the mapping of userspace
> > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > This data will be used for the memory encryption system call and the
> > kernel key service API.
> >
> > Implement helper functions to access this mapping structure and make
> > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/include/asm/mktme.h | 11 ++++++
> >  arch/x86/mm/mktme.c          | 85
> > ++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 96 insertions(+)
> 
> Maybe it's better to put those changes to include/keys/mktme-type.h, and
> security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> x86 guys by doing so?
> 
> Thanks,
> -Kai
> >
> > diff --git a/arch/x86/include/asm/mktme.h
> > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > --- a/arch/x86/include/asm/mktme.h
> > +++ b/arch/x86/include/asm/mktme.h
> > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > mktme_nr_keyids;  extern int mktme_keyid_shift;
> >
> > +/* Manage mappings between hardware keyids and userspace keys */
> > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > +extern int mktme_map_get_free_keyid(void); extern void
> > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > +keyid, unsigned int serial); extern int
> > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > +mktme_map_serial_from_keyid(int keyid);
> > +
> >  extern struct page_ext_operations page_mktme_ops;
> >
> >  #define page_keyid page_keyid
> > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > 660caf6a5ce1..5246d8323359 100644
> > --- a/arch/x86/mm/mktme.c
> > +++ b/arch/x86/mm/mktme.c
> > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> >
> > +/*
> > + * struct mktme_mapping and the mktme_map_* functions manage the
> > +mapping
> > + * of userspace keys to hardware keyids in MKTME. They are used by
> > +the
> > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > + */
> > +struct mktme_mapping {
> > +	struct mutex	lock;		/* protect this map & HW state */
> > +	unsigned int	mapped_keyids;
> > +	unsigned int	serial[];
> > +};

Sorry one more comment that I missed yesterday:

I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 

Thanks,
-Kai

> > +
> > +struct mktme_mapping *mktme_map;
> > +
> > +static inline long mktme_map_size(void) {
> > +	long size = 0;
> > +
> > +	size += sizeof(mktme_map);
> > +	size += sizeof(mktme_map->serial[0]) * mktme_nr_keyids;
> > +	return size;
> > +}
> > +
> > +int mktme_map_alloc(void)
> > +{
> > +	mktme_map = kzalloc(mktme_map_size(), GFP_KERNEL);
> > +	if (!mktme_map)
> > +		return 0;
> > +	mutex_init(&mktme_map->lock);
> > +	return 1;
> > +}
> > +
> > +void mktme_map_free(void)
> > +{
> > +	kfree(mktme_map);
> > +}
> > +
> > +void mktme_map_lock(void)
> > +{
> > +	mutex_lock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_unlock(void)
> > +{
> > +	mutex_unlock(&mktme_map->lock);
> > +}
> > +
> > +void mktme_map_set_keyid(int keyid, unsigned int serial) {
> > +	mktme_map->serial[keyid] = serial;
> > +	mktme_map->mapped_keyids++;
> > +}
> > +
> > +void mktme_map_clear_keyid(int keyid) {
> > +	mktme_map->serial[keyid] = 0;
> > +	mktme_map->mapped_keyids--;
> > +}
> > +
> > +unsigned int mktme_map_serial_from_keyid(int keyid) {
> > +	return mktme_map->serial[keyid];
> > +}
> > +
> > +int mktme_map_keyid_from_serial(unsigned int serial) {
> > +	int i;
> > +
> > +	for (i = 1; i < mktme_nr_keyids; i++)
> > +		if (mktme_map->serial[i] == serial)
> > +			return i;
> > +	return 0;
> > +}
> > +
> > +int mktme_map_get_free_keyid(void)
> > +{
> > +	int i;
> > +
> > +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> > +		for (i = 1; i < mktme_nr_keyids; i++)
> > +			if (mktme_map->serial[i] == 0)
> > +				return i;
> > +	}
> > +	return 0;
> > +}
> > +
> >  void prep_encrypted_page(struct page *page, int order, int keyid, bool zero)  {
> >  	int i;
> > --
> > 2.14.1

Re: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

On Mon, Sep 10, 2018 at 04:37:01PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > security-module@vger.kernel.org] On Behalf Of Huang, Kai
> > Sent: Monday, September 10, 2018 2:57 PM
> > To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> > tglx@linutronix.de
> > Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> > 
> > 
> > > -----Original Message-----
> > > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > > security-module@vger.kernel.org] On Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:36 AM
> > > To: dhowells@redhat.com; tglx@linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > > encryption keys
> > >
> > > Define a global mapping structure to track the mapping of userspace
> > > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > > This data will be used for the memory encryption system call and the
> > > kernel key service API.
> > >
> > > Implement helper functions to access this mapping structure and make
> > > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  arch/x86/include/asm/mktme.h | 11 ++++++
> > >  arch/x86/mm/mktme.c          | 85
> > > ++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 96 insertions(+)
> > 
> > Maybe it's better to put those changes to include/keys/mktme-type.h, and
> > security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> > x86 guys by doing so?
> > 
> > Thanks,
> > -Kai
> > >
> > > diff --git a/arch/x86/include/asm/mktme.h
> > > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > > --- a/arch/x86/include/asm/mktme.h
> > > +++ b/arch/x86/include/asm/mktme.h
> > > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > > mktme_nr_keyids;  extern int mktme_keyid_shift;
> > >
> > > +/* Manage mappings between hardware keyids and userspace keys */
> > > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > > +extern int mktme_map_get_free_keyid(void); extern void
> > > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > > +keyid, unsigned int serial); extern int
> > > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > > +mktme_map_serial_from_keyid(int keyid);
> > > +
> > >  extern struct page_ext_operations page_mktme_ops;
> > >
> > >  #define page_keyid page_keyid
> > > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > > 660caf6a5ce1..5246d8323359 100644
> > > --- a/arch/x86/mm/mktme.c
> > > +++ b/arch/x86/mm/mktme.c
> > > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> > >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> > >
> > > +/*
> > > + * struct mktme_mapping and the mktme_map_* functions manage the
> > > +mapping
> > > + * of userspace keys to hardware keyids in MKTME. They are used by
> > > +the
> > > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > > + */
> > > +struct mktme_mapping {
> > > +	struct mutex	lock;		/* protect this map & HW state */
> > > +	unsigned int	mapped_keyids;
> > > +	unsigned int	serial[];
> > > +};
> 
> Sorry one more comment that I missed yesterday:
> 
> I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 
> 
> Thanks,
> -Kai

I agree! It's not an oversight, but rather a header file include nightmare.
I can look at it again.

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

On Mon, Sep 10, 2018 at 04:37:01PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-security-module at vger.kernel.org [mailto:owner-linux-
> > security-module at vger.kernel.org] On Behalf Of Huang, Kai
> > Sent: Monday, September 10, 2018 2:57 PM
> > To: Schofield, Alison <alison.schofield@intel.com>; dhowells at redhat.com;
> > tglx at linutronix.de
> > Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> > Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> > 
> > 
> > > -----Original Message-----
> > > From: owner-linux-security-module at vger.kernel.org [mailto:owner-linux-
> > > security-module at vger.kernel.org] On Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:36 AM
> > > To: dhowells at redhat.com; tglx at linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> > > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > > encryption keys
> > >
> > > Define a global mapping structure to track the mapping of userspace
> > > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > > This data will be used for the memory encryption system call and the
> > > kernel key service API.
> > >
> > > Implement helper functions to access this mapping structure and make
> > > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  arch/x86/include/asm/mktme.h | 11 ++++++
> > >  arch/x86/mm/mktme.c          | 85
> > > ++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 96 insertions(+)
> > 
> > Maybe it's better to put those changes to include/keys/mktme-type.h, and
> > security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> > x86 guys by doing so?
> > 
> > Thanks,
> > -Kai
> > >
> > > diff --git a/arch/x86/include/asm/mktme.h
> > > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > > --- a/arch/x86/include/asm/mktme.h
> > > +++ b/arch/x86/include/asm/mktme.h
> > > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > > mktme_nr_keyids;  extern int mktme_keyid_shift;
> > >
> > > +/* Manage mappings between hardware keyids and userspace keys */
> > > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > > +extern int mktme_map_get_free_keyid(void); extern void
> > > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > > +keyid, unsigned int serial); extern int
> > > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > > +mktme_map_serial_from_keyid(int keyid);
> > > +
> > >  extern struct page_ext_operations page_mktme_ops;
> > >
> > >  #define page_keyid page_keyid
> > > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > > 660caf6a5ce1..5246d8323359 100644
> > > --- a/arch/x86/mm/mktme.c
> > > +++ b/arch/x86/mm/mktme.c
> > > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> > >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> > >
> > > +/*
> > > + * struct mktme_mapping and the mktme_map_* functions manage the
> > > +mapping
> > > + * of userspace keys to hardware keyids in MKTME. They are used by
> > > +the
> > > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > > + */
> > > +struct mktme_mapping {
> > > +	struct mutex	lock;		/* protect this map & HW state */
> > > +	unsigned int	mapped_keyids;
> > > +	unsigned int	serial[];
> > > +};
> 
> Sorry one more comment that I missed yesterday:
> 
> I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 
> 
> Thanks,
> -Kai

I agree! It's not an oversight, but rather a header file include nightmare.
I can look at it again.

Re: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[Alison Schofield]

On Mon, Sep 10, 2018 at 04:37:01PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > security-module@vger.kernel.org] On Behalf Of Huang, Kai
> > Sent: Monday, September 10, 2018 2:57 PM
> > To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> > tglx@linutronix.de
> > Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: RE: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > encryption keys
> > 
> > 
> > > -----Original Message-----
> > > From: owner-linux-security-module@vger.kernel.org [mailto:owner-linux-
> > > security-module@vger.kernel.org] On Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:36 AM
> > > To: dhowells@redhat.com; tglx@linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > > Subject: [RFC 04/12] x86/mm: Add helper functions to manage memory
> > > encryption keys
> > >
> > > Define a global mapping structure to track the mapping of userspace
> > > keys to hardware keyids in MKTME (Multi-Key Total Memory Encryption).
> > > This data will be used for the memory encryption system call and the
> > > kernel key service API.
> > >
> > > Implement helper functions to access this mapping structure and make
> > > them visible to the MKTME Kernel Key Service: security/keys/mktme_keys
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  arch/x86/include/asm/mktme.h | 11 ++++++
> > >  arch/x86/mm/mktme.c          | 85
> > > ++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 96 insertions(+)
> > 
> > Maybe it's better to put those changes to include/keys/mktme-type.h, and
> > security/keys/mktme_key.c? It seems you don't have to involve linux-mm and
> > x86 guys by doing so?
> > 
> > Thanks,
> > -Kai
> > >
> > > diff --git a/arch/x86/include/asm/mktme.h
> > > b/arch/x86/include/asm/mktme.h index dbfbd955da98..f6acd551457f 100644
> > > --- a/arch/x86/include/asm/mktme.h
> > > +++ b/arch/x86/include/asm/mktme.h
> > > @@ -13,6 +13,17 @@ extern phys_addr_t mktme_keyid_mask;  extern int
> > > mktme_nr_keyids;  extern int mktme_keyid_shift;
> > >
> > > +/* Manage mappings between hardware keyids and userspace keys */
> > > +extern int mktme_map_alloc(void); extern void mktme_map_free(void);
> > > +extern void mktme_map_lock(void); extern void mktme_map_unlock(void);
> > > +extern int mktme_map_get_free_keyid(void); extern void
> > > +mktme_map_clear_keyid(int keyid); extern void mktme_map_set_keyid(int
> > > +keyid, unsigned int serial); extern int
> > > +mktme_map_keyid_from_serial(unsigned int serial); extern unsigned int
> > > +mktme_map_serial_from_keyid(int keyid);
> > > +
> > >  extern struct page_ext_operations page_mktme_ops;
> > >
> > >  #define page_keyid page_keyid
> > > diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c index
> > > 660caf6a5ce1..5246d8323359 100644
> > > --- a/arch/x86/mm/mktme.c
> > > +++ b/arch/x86/mm/mktme.c
> > > @@ -63,6 +63,91 @@ int vma_keyid(struct vm_area_struct *vma)
> > >  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;  }
> > >
> > > +/*
> > > + * struct mktme_mapping and the mktme_map_* functions manage the
> > > +mapping
> > > + * of userspace keys to hardware keyids in MKTME. They are used by
> > > +the
> > > + * the encrypt_mprotect system call and the MKTME Key Service API.
> > > + */
> > > +struct mktme_mapping {
> > > +	struct mutex	lock;		/* protect this map & HW state */
> > > +	unsigned int	mapped_keyids;
> > > +	unsigned int	serial[];
> > > +};
> 
> Sorry one more comment that I missed yesterday:
> 
> I think 'key_serial_t' should be used  as type of serial throughout this patch, but not 'unsigned int'. 
> 
> Thanks,
> -Kai

I agree! It's not an oversight, but rather a header file include nightmare.
I can look at it again.

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having their
> > +own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +==============  API Overview
> > +==============
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

My understanding is that low level SKU's could have as few as 3 bits 
available to hold the keyid, and that the max is 6 bits, hence 64.
I probably don't need to be stating that level of detail here, but
rather just iterate the important point that the resource is limited!

> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

I can do that.
Also, I do know I need man page for that too.
> 
> > +
> > +===========  Usage: MKTME Key Service
> > +===========
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
             ^^^^^^^ should be tweak

> 
> This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Kai,
I think my typo above, threw you off. We have the same understanding of
the key fields.

Is this the structure you are suggesting?

	Options

	key_type=	"user" or "CPU"

	key=		If key_type = user
				key= is the data key
			If key_type = CPU
				key= is not required
				if key= is present
					it is entropy to be mixed with
					CPU generated data key

	tweak=		If key_type = user
				tweak= is the tweak key
			If key_type = CPU
				tweak= is not required
				if tweak= is present
					it is entropy to be mixed with
					CPU generated tweak key


Alison
> 
> Thanks,
> -Kai

........snip...........

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells at redhat.com; tglx at linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> > x86 at kernel.org; linux-mm at kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having their
> > +own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview
> > +============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

My understanding is that low level SKU's could have as few as 3 bits 
available to hold the keyid, and that the max is 6 bits, hence 64.
I probably don't need to be stating that level of detail here, but
rather just iterate the important point that the resource is limited!

> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

I can do that.
Also, I do know I need man page for that too.
> 
> > +
> > +======================  Usage: MKTME Key Service
> > +======================
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
             ^^^^^^^ should be tweak

> 
> This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Kai,
I think my typo above, threw you off. We have the same understanding of
the key fields.

Is this the structure you are suggesting?

	Options

	key_type=	"user" or "CPU"

	key=		If key_type == user
				key= is the data key
			If key_type == CPU
				key= is not required
				if key= is present
					it is entropy to be mixed with
					CPU generated data key

	tweak=		If key_type == user
				tweak= is the tweak key
			If key_type == CPU
				tweak= is not required
				if tweak= is present
					it is entropy to be mixed with
					CPU generated tweak key


Alison
> 
> Thanks,
> -Kai

........snip...........

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> 
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having their
> > +own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview
> > +============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID, although in the first generation server we only support 6-bit keyID, which is 63 key/keyIDs (excluding keyID 0, which is TME's keyID).

My understanding is that low level SKU's could have as few as 3 bits 
available to hold the keyid, and that the max is 6 bits, hence 64.
I probably don't need to be stating that level of detail here, but
rather just iterate the important point that the resource is limited!

> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts can be reviewed easier.

I can do that.
Also, I do know I need man page for that too.
> 
> > +
> > +======================  Usage: MKTME Key Service
> > +======================
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
             ^^^^^^^ should be tweak

> 
> This is not true. The spec says in CPU generated random mode, both 'key' and 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey' and 'tweak' part for random key mode, since if we allow user to specify both entropies, and if user passes value with all 1, we are effectively making the key and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0, ignoring user input.

Kai,
I think my typo above, threw you off. We have the same understanding of
the key fields.

Is this the structure you are suggesting?

	Options

	key_type=	"user" or "CPU"

	key=		If key_type == user
				key= is the data key
			If key_type == CPU
				key= is not required
				if key= is present
					it is entropy to be mixed with
					CPU generated data key

	tweak=		If key_type == user
				tweak= is the tweak key
			If key_type == CPU
				tweak= is not required
				if tweak= is present
					it is entropy to be mixed with
					CPU generated tweak key


Alison
> 
> Thanks,
> -Kai

........snip...........

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 1:28 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> tglx@linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> Subject: RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> 
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> >
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> >
> > diff --git a/Documentation/x86/mktme-keys.txt
> > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having
> > +their own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer
> > +flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Ke
> > +y-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +==============  API Overview
> > +==============
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts
> can be reviewed easier.
> 
> > +
> > +===========  Usage: MKTME Key Service
> > +===========
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
> 
> This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both
> 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey'
> and 'tweak' part for random key mode, since if we allow user to specify both
> entropies, and if user passes value with all 1, we are effectively making the key
> and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0,
> ignoring user input.

Hmm.. I just realized that I confused 'OR' and 'XOR', so the last two paragraphs doesn't make sense. :)

But still I think we can simply use "userkey" and "tweak". "entropy" is not needed IMO.

Thanks,
-Kai
> 
> Thanks,
> -Kai
> > +
> > +===========  Usage: encrypt_mprotect()
> > +===========
> > +
> > +System Call encrypt_mprotect()::
> > +
> > +    This system call is an extension of the existing mprotect() system
> > +    call. It requires the same parameters as legary mprotect() plus
> > +    one additional parameter, the keyid. Userspace must provide the
> > +    key serial number assigned through the kernel key service.
> > +
> > +    int encrypt_mprotect(void *addr, size_t len, int prot, int
> > + keyid);
> > +
> > +===========  Usage: Sample Roundtrip
> > ===========
> > +
> > +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> > +
> > +  Add a key:
> > +        key = add_key(mktme, name, options, strlen(option), keyring);
> > +
> > +  Map memory:
> > +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1,
> > + 0);
> > +
> > +  Protect memory:
> > +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> > +
> > +  Use protected memory:
> > +        ................
> > +
> > +  Free memory:
> > +        ret = munmap(ptr, size);
> > +
> > +  Revoke key:
> > +        ret = keyctl(KEYCTL_REVOKE, key);
> > +
> > --
> > 2.14.1

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 1:28 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells at redhat.com;
> tglx at linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> Subject: RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> 
> > -----Original Message-----
> > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells at redhat.com; tglx at linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-mm at kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> >
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> >
> > diff --git a/Documentation/x86/mktme-keys.txt
> > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having
> > +their own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer
> > +flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Ke
> > +y-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview
> > +============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts
> can be reviewed easier.
> 
> > +
> > +======================  Usage: MKTME Key Service
> > +======================
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
> 
> This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both
> 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey'
> and 'tweak' part for random key mode, since if we allow user to specify both
> entropies, and if user passes value with all 1, we are effectively making the key
> and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0,
> ignoring user input.

Hmm.. I just realized that I confused 'OR' and 'XOR', so the last two paragraphs doesn't make sense. :)

But still I think we can simply use "userkey" and "tweak". "entropy" is not needed IMO.

Thanks,
-Kai
> 
> Thanks,
> -Kai
> > +
> > +======================  Usage: encrypt_mprotect()
> > +======================
> > +
> > +System Call encrypt_mprotect()::
> > +
> > +    This system call is an extension of the existing mprotect() system
> > +    call. It requires the same parameters as legary mprotect() plus
> > +    one additional parameter, the keyid. Userspace must provide the
> > +    key serial number assigned through the kernel key service.
> > +
> > +    int encrypt_mprotect(void *addr, size_t len, int prot, int
> > + keyid);
> > +
> > +======================  Usage: Sample Roundtrip
> > ======================
> > +
> > +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> > +
> > +  Add a key:
> > +        key = add_key(mktme, name, options, strlen(option), keyring);
> > +
> > +  Map memory:
> > +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1,
> > + 0);
> > +
> > +  Protect memory:
> > +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> > +
> > +  Use protected memory:
> > +        ................
> > +
> > +  Free memory:
> > +        ret = munmap(ptr, size);
> > +
> > +  Revoke key:
> > +        ret = keyctl(KEYCTL_REVOKE, key);
> > +
> > --
> > 2.14.1

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Huang, Kai
> Sent: Monday, September 10, 2018 1:28 PM
> To: Schofield, Alison <alison.schofield@intel.com>; dhowells@redhat.com;
> tglx@linutronix.de
> Cc: Nakajima, Jun <jun.nakajima@intel.com>; Shutemov, Kirill
> <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> Subject: RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> 
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Saturday, September 8, 2018 10:34 AM
> > To: dhowells@redhat.com; tglx@linutronix.de
> > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill
> > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-mm@kvack.org
> > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> >
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> >
> > diff --git a/Documentation/x86/mktme-keys.txt
> > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory
> > +Encryption) allows encryption of the entire system memory using a
> > +single key, MKTME allows multiple encryption domains, each having
> > +their own key. The main use case for the feature is virtual machine
> > +isolation. The API's introduced here are intended to offer
> > +flexibility to work in a
> > wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Ke
> > +y-
> > +Total-Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview
> > +============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and
> > +use the memory encryption keys:
> > +
> > +1) Kernel Key Service: MKTME Type
> > +
> > +   MKTME is a new key type added to the existing Kernel Key Services
> > +   to support the memory encryption keys. The MKTME service manages
> > +   the addition and removal of MKTME keys. It maps userspace keys
> > +   to hardware keyids and programs the hardware with user requested
> > +   encryption parameters.
> > +
> > +   o An understanding of the Kernel Key Service is required in order
> > +     to use the MKTME key type as it is a subset of that service.
> > +
> > +   o MKTME keys are a limited resource. There is a single pool of
> > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> 
> Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> > +     With that in mind, userspace may take advantage of the kernel
> > +     key services sharing and permissions model for userspace keys.
> > +     One key can be shared as long as each user has the permission
> > +     of "KEY_NEED_VIEW" to use it.
> > +
> > +   o MKTME key type uses capabilities to restrict the allocation
> > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > +
> > +   o The MKTME key service blocks kernel key service commands that
> > +     could lead to reprogramming of in use keys, or loss of keys from
> > +     the pool. This means MKTME does not allow a key to be invalidated,
> > +     unlinked, or timed out. These operations are blocked by MKTME as
> > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > +
> > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > +     may change the programming of a key by revoking it and adding
> > +     a new key with the updated encryption options (or vice-versa).
> > +
> > +2) System Call: encrypt_mprotect()
> > +
> > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > +   caller passes the serial number to a previously allocated and
> > +   programmed encryption key. That handle was created with the MKTME
> > +   Key Service.
> > +
> > +   o The caller must have KEY_NEED_VIEW permission on the key
> > +
> > +   o The range of memory that is to be protected must be mapped as
> > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > +     fails with EINVAL.
> > +
> > +   o As an extension to the existing mprotect() system call,
> > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > +     the enabling of memory encryption. That means that in addition
> > +     to encrypting the memory, the protection flags will be updated
> > +     as requested in the call.
> > +
> > +   o Additional mprotect() calls to memory already protected with
> > +     MKTME will not alter the MKTME status.
> 
> I think it's better to separate encrypt_mprotect() into another doc so both parts
> can be reviewed easier.
> 
> > +
> > +======================  Usage: MKTME Key Service
> > +======================
> > +
> > +MKTME is enabled on supported Intel platforms by selecting
> > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > +
> > +Allocating MKTME Keys via command line or system call:
> > +    keyctl add mktme name "[options]" ring
> > +
> > +    key_serial_t add_key(const char *type, const char *description,
> > +                         const void *payload, size_t plen,
> > +                         key_serial_t keyring);
> > +
> > +Revoking MKTME Keys via command line or system call::
> > +   keyctl revoke <key>
> > +
> > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > +
> > +Options Field Definition:
> > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > +		  generated key if a userkey is not defined here.
> > +
> > +    algorithm=    Encryption algorithm name as a string.
> > +		  Valid algorithm: "aes-xts-128"
> > +
> > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > +                  userkey...  (need to be clear here that this is being sent
> > +                  to the hardware - kernel not messing w it)
> > +
> > +    entropy=      ascii hex value entropy.
> > +                  This entropy will be used to generated the CPU key and
> > +		  the tweak key when CPU generated key is requested.
> > +
> > +Algorithm Dependencies:
> > +    AES-XTS 128 is the only supported algorithm.
> > +    There are only 2 ways that AES-XTS 128 may be used:
> > +
> > +    1) User specified encryption key
> > +	- The user specified encryption key must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- A tweak key must be specified and it must be exactly
> > +	  16 ASCII Hex bytes (128 bits).
> > +	- No entropy field is accepted.
> > +
> > +    2) CPU generated encryption key
> > +	- When no user specified encryption key is provided, the
> > +	  default encryption key will be CPU generated.
> > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > +	  entropy will be used by the CPU to generate both the
> > +	  encryption key and the tweak key.
> > +	- No entropy field is accepted.
> 
> This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> 
> Actually, simple 'XOR' is used to generate the final key:
> 
> case KEYID_SET_KEY_RANDOM:
> 	......
> 	(* Mix user supplied entropy to the data key and tweak key *)
> 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> 
> So I think we can either just remove 'entropy' parameter, since we can use both
> 'userkey' and 'tweak' even for random key mode.
> 
> In fact, which might be better IMHO, we can simply disallow or ignore 'userkey'
> and 'tweak' part for random key mode, since if we allow user to specify both
> entropies, and if user passes value with all 1, we are effectively making the key
> and tweak to be all 1, which is not random anymore.
> 
> Instead, kernel can generate random for both entropies, or we can simply uses 0,
> ignoring user input.

Hmm.. I just realized that I confused 'OR' and 'XOR', so the last two paragraphs doesn't make sense. :)

But still I think we can simply use "userkey" and "tweak". "entropy" is not needed IMO.

Thanks,
-Kai
> 
> Thanks,
> -Kai
> > +
> > +======================  Usage: encrypt_mprotect()
> > +======================
> > +
> > +System Call encrypt_mprotect()::
> > +
> > +    This system call is an extension of the existing mprotect() system
> > +    call. It requires the same parameters as legary mprotect() plus
> > +    one additional parameter, the keyid. Userspace must provide the
> > +    key serial number assigned through the kernel key service.
> > +
> > +    int encrypt_mprotect(void *addr, size_t len, int prot, int
> > + keyid);
> > +
> > +======================  Usage: Sample Roundtrip
> > ======================
> > +
> > +Sample usage of MKTME Key Service API with encrypt_mprotect() API:
> > +
> > +  Add a key:
> > +        key = add_key(mktme, name, options, strlen(option), keyring);
> > +
> > +  Map memory:
> > +        ptr = mmap(NULL, size, prot, MAP_ANONYMOUS|MAP_PRIVATE, -1,
> > + 0);
> > +
> > +  Protect memory:
> > +        ret = syscall(sys_encrypt_mprotect, ptr, size, prot, keyid);
> > +
> > +  Use protected memory:
> > +        ................
> > +
> > +  Free memory:
> > +        ret = munmap(ptr, size);
> > +
> > +  Revoke key:
> > +        ret = keyctl(KEYCTL_REVOKE, key);
> > +
> > --
> > 2.14.1

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 10:32:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> > +allows encryption of the entire system memory using a single key, MKTME
> > +allows multiple encryption domains, each having their own key. The main use
> > +case for the feature is virtual machine isolation. The API's introduced here
> > +are intended to offer flexibility to work in a wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-
> > Memory-Encryption-Spec.pdf
> > +
> > +==============  API Overview  ==============
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and use
> > +the memory encryption keys:
> 
> This is like saying that they are different APIs to do semantically the
> same exact thing. Is that so?

No. 
The API's used to create and use memory encryption keys are described below:

> 
> /Jarkko

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 10:32:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> > +allows encryption of the entire system memory using a single key, MKTME
> > +allows multiple encryption domains, each having their own key. The main use
> > +case for the feature is virtual machine isolation. The API's introduced here
> > +are intended to offer flexibility to work in a wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-
> > Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview  ============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and use
> > +the memory encryption keys:
> 
> This is like saying that they are different APIs to do semantically the
> same exact thing. Is that so?

No. 
The API's used to create and use memory encryption keys are described below:

> 
> /Jarkko

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 10:32:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Document the API's used for MKTME on Intel platforms.
> > MKTME: Multi-KEY Total Memory Encryption
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  Documentation/x86/mktme-keys.txt | 153
> > +++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 153 insertions(+)
> >  create mode 100644 Documentation/x86/mktme-keys.txt
> > 
> > diff --git a/Documentation/x86/mktme-keys.txt b/Documentation/x86/mktme-
> > keys.txt
> > new file mode 100644
> > index 000000000000..2dea7acd2a17
> > --- /dev/null
> > +++ b/Documentation/x86/mktme-keys.txt
> > @@ -0,0 +1,153 @@
> > +MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > +memory encryption on Intel platforms. Whereas TME (Total Memory Encryption)
> > +allows encryption of the entire system memory using a single key, MKTME
> > +allows multiple encryption domains, each having their own key. The main use
> > +case for the feature is virtual machine isolation. The API's introduced here
> > +are intended to offer flexibility to work in a wide range of uses.
> > +
> > +The externally available Intel Architecture Spec:
> > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-
> > Memory-Encryption-Spec.pdf
> > +
> > +============================  API Overview  ============================
> > +
> > +There are 2 MKTME specific API's that enable userspace to create and use
> > +the memory encryption keys:
> 
> This is like saying that they are different APIs to do semantically the
> same exact thing. Is that so?

No. 
The API's used to create and use memory encryption keys are described below:

> 
> /Jarkko

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Alison Schofield
> Sent: Tuesday, September 11, 2018 12:13 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> >
> > > -----Original Message-----
> > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > > Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:34 AM
> > > To: dhowells@redhat.com; tglx@linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-
> mm@kvack.org
> > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > Encryption API
> > >
> > > Document the API's used for MKTME on Intel platforms.
> > > MKTME: Multi-KEY Total Memory Encryption
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  Documentation/x86/mktme-keys.txt | 153
> > > +++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 153 insertions(+)
> > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > >
> > > diff --git a/Documentation/x86/mktme-keys.txt
> > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > 000000000000..2dea7acd2a17
> > > --- /dev/null
> > > +++ b/Documentation/x86/mktme-keys.txt
> > > @@ -0,0 +1,153 @@
> > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > +Memory
> > > +Encryption) allows encryption of the entire system memory using a
> > > +single key, MKTME allows multiple encryption domains, each having
> > > +their own key. The main use case for the feature is virtual machine
> > > +isolation. The API's introduced here are intended to offer
> > > +flexibility to work in a
> > > wide range of uses.
> > > +
> > > +The externally available Intel Architecture Spec:
> > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > +Key-
> > > +Total-Memory-Encryption-Spec.pdf
> > > +
> > > +==============  API Overview
> > > +==============
> > > +
> > > +There are 2 MKTME specific API's that enable userspace to create
> > > +and use the memory encryption keys:
> > > +
> > > +1) Kernel Key Service: MKTME Type
> > > +
> > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > +   to support the memory encryption keys. The MKTME service manages
> > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > +   to hardware keyids and programs the hardware with user requested
> > > +   encryption parameters.
> > > +
> > > +   o An understanding of the Kernel Key Service is required in order
> > > +     to use the MKTME key type as it is a subset of that service.
> > > +
> > > +   o MKTME keys are a limited resource. There is a single pool of
> > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> >
> > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> My understanding is that low level SKU's could have as few as 3 bits available to
> hold the keyid, and that the max is 6 bits, hence 64.
> I probably don't need to be stating that level of detail here, but rather just
> iterate the important point that the resource is limited!
> 
> >
> > > +     With that in mind, userspace may take advantage of the kernel
> > > +     key services sharing and permissions model for userspace keys.
> > > +     One key can be shared as long as each user has the permission
> > > +     of "KEY_NEED_VIEW" to use it.
> > > +
> > > +   o MKTME key type uses capabilities to restrict the allocation
> > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > +
> > > +   o The MKTME key service blocks kernel key service commands that
> > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > +
> > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > +     may change the programming of a key by revoking it and adding
> > > +     a new key with the updated encryption options (or vice-versa).
> > > +
> > > +2) System Call: encrypt_mprotect()
> > > +
> > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > +   caller passes the serial number to a previously allocated and
> > > +   programmed encryption key. That handle was created with the MKTME
> > > +   Key Service.
> > > +
> > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > +
> > > +   o The range of memory that is to be protected must be mapped as
> > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > +     fails with EINVAL.
> > > +
> > > +   o As an extension to the existing mprotect() system call,
> > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > +     the enabling of memory encryption. That means that in addition
> > > +     to encrypting the memory, the protection flags will be updated
> > > +     as requested in the call.
> > > +
> > > +   o Additional mprotect() calls to memory already protected with
> > > +     MKTME will not alter the MKTME status.
> >
> > I think it's better to separate encrypt_mprotect() into another doc so both
> parts can be reviewed easier.
> 
> I can do that.
> Also, I do know I need man page for that too.
> >
> > > +
> > > +===========  Usage: MKTME Key Service
> > > +===========
> > > +
> > > +MKTME is enabled on supported Intel platforms by selecting
> > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > +
> > > +Allocating MKTME Keys via command line or system call:
> > > +    keyctl add mktme name "[options]" ring
> > > +
> > > +    key_serial_t add_key(const char *type, const char *description,
> > > +                         const void *payload, size_t plen,
> > > +                         key_serial_t keyring);
> > > +
> > > +Revoking MKTME Keys via command line or system call::
> > > +   keyctl revoke <key>
> > > +
> > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > +
> > > +Options Field Definition:
> > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > +		  generated key if a userkey is not defined here.
> > > +
> > > +    algorithm=    Encryption algorithm name as a string.
> > > +		  Valid algorithm: "aes-xts-128"
> > > +
> > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > +                  userkey...  (need to be clear here that this is being sent
> > > +                  to the hardware - kernel not messing w it)
> > > +
> > > +    entropy=      ascii hex value entropy.
> > > +                  This entropy will be used to generated the CPU key and
> > > +		  the tweak key when CPU generated key is requested.
> > > +
> > > +Algorithm Dependencies:
> > > +    AES-XTS 128 is the only supported algorithm.
> > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > +
> > > +    1) User specified encryption key
> > > +	- The user specified encryption key must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- A tweak key must be specified and it must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- No entropy field is accepted.
> > > +
> > > +    2) CPU generated encryption key
> > > +	- When no user specified encryption key is provided, the
> > > +	  default encryption key will be CPU generated.
> > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > +	  entropy will be used by the CPU to generate both the
> > > +	  encryption key and the tweak key.
> > > +	- No entropy field is accepted.
>              ^^^^^^^ should be tweak
> 
> >
> > This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> >
> > Actually, simple 'XOR' is used to generate the final key:
> >
> > case KEYID_SET_KEY_RANDOM:
> > 	......
> > 	(* Mix user supplied entropy to the data key and tweak key *)
> > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> >
> > So I think we can either just remove 'entropy' parameter, since we can use
> both 'userkey' and 'tweak' even for random key mode.
> >
> > In fact, which might be better IMHO, we can simply disallow or ignore
> 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> both entropies, and if user passes value with all 1, we are effectively making the
> key and tweak to be all 1, which is not random anymore.
> >
> > Instead, kernel can generate random for both entropies, or we can simply uses
> 0, ignoring user input.
> 
> Kai,
> I think my typo above, threw you off. We have the same understanding of the
> key fields.
> 
> Is this the structure you are suggesting?
> 
> 	Options
> 
> 	key_type=	"user" or "CPU"
> 
> 	key=		If key_type = user
> 				key= is the data key
> 			If key_type = CPU
> 				key= is not required
> 				if key= is present
> 					it is entropy to be mixed with
> 					CPU generated data key
> 
> 	tweak=		If key_type = user
> 				tweak= is the tweak key
> 			If key_type = CPU
> 				tweak= is not required
> 				if tweak= is present
> 					it is entropy to be mixed with
> 					CPU generated tweak key

Exactly.

Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

Thanks,
-Kai
> 
> 
> Alison
> >
> > Thanks,
> > -Kai
> 
> ........snip...........

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> Behalf Of Alison Schofield
> Sent: Tuesday, September 11, 2018 12:13 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells at redhat.com; tglx at linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> >
> > > -----Original Message-----
> > > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> > > Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:34 AM
> > > To: dhowells at redhat.com; tglx at linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-
> mm at kvack.org
> > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > Encryption API
> > >
> > > Document the API's used for MKTME on Intel platforms.
> > > MKTME: Multi-KEY Total Memory Encryption
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  Documentation/x86/mktme-keys.txt | 153
> > > +++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 153 insertions(+)
> > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > >
> > > diff --git a/Documentation/x86/mktme-keys.txt
> > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > 000000000000..2dea7acd2a17
> > > --- /dev/null
> > > +++ b/Documentation/x86/mktme-keys.txt
> > > @@ -0,0 +1,153 @@
> > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > +Memory
> > > +Encryption) allows encryption of the entire system memory using a
> > > +single key, MKTME allows multiple encryption domains, each having
> > > +their own key. The main use case for the feature is virtual machine
> > > +isolation. The API's introduced here are intended to offer
> > > +flexibility to work in a
> > > wide range of uses.
> > > +
> > > +The externally available Intel Architecture Spec:
> > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > +Key-
> > > +Total-Memory-Encryption-Spec.pdf
> > > +
> > > +============================  API Overview
> > > +============================
> > > +
> > > +There are 2 MKTME specific API's that enable userspace to create
> > > +and use the memory encryption keys:
> > > +
> > > +1) Kernel Key Service: MKTME Type
> > > +
> > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > +   to support the memory encryption keys. The MKTME service manages
> > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > +   to hardware keyids and programs the hardware with user requested
> > > +   encryption parameters.
> > > +
> > > +   o An understanding of the Kernel Key Service is required in order
> > > +     to use the MKTME key type as it is a subset of that service.
> > > +
> > > +   o MKTME keys are a limited resource. There is a single pool of
> > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> >
> > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> My understanding is that low level SKU's could have as few as 3 bits available to
> hold the keyid, and that the max is 6 bits, hence 64.
> I probably don't need to be stating that level of detail here, but rather just
> iterate the important point that the resource is limited!
> 
> >
> > > +     With that in mind, userspace may take advantage of the kernel
> > > +     key services sharing and permissions model for userspace keys.
> > > +     One key can be shared as long as each user has the permission
> > > +     of "KEY_NEED_VIEW" to use it.
> > > +
> > > +   o MKTME key type uses capabilities to restrict the allocation
> > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > +
> > > +   o The MKTME key service blocks kernel key service commands that
> > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > +
> > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > +     may change the programming of a key by revoking it and adding
> > > +     a new key with the updated encryption options (or vice-versa).
> > > +
> > > +2) System Call: encrypt_mprotect()
> > > +
> > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > +   caller passes the serial number to a previously allocated and
> > > +   programmed encryption key. That handle was created with the MKTME
> > > +   Key Service.
> > > +
> > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > +
> > > +   o The range of memory that is to be protected must be mapped as
> > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > +     fails with EINVAL.
> > > +
> > > +   o As an extension to the existing mprotect() system call,
> > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > +     the enabling of memory encryption. That means that in addition
> > > +     to encrypting the memory, the protection flags will be updated
> > > +     as requested in the call.
> > > +
> > > +   o Additional mprotect() calls to memory already protected with
> > > +     MKTME will not alter the MKTME status.
> >
> > I think it's better to separate encrypt_mprotect() into another doc so both
> parts can be reviewed easier.
> 
> I can do that.
> Also, I do know I need man page for that too.
> >
> > > +
> > > +======================  Usage: MKTME Key Service
> > > +======================
> > > +
> > > +MKTME is enabled on supported Intel platforms by selecting
> > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > +
> > > +Allocating MKTME Keys via command line or system call:
> > > +    keyctl add mktme name "[options]" ring
> > > +
> > > +    key_serial_t add_key(const char *type, const char *description,
> > > +                         const void *payload, size_t plen,
> > > +                         key_serial_t keyring);
> > > +
> > > +Revoking MKTME Keys via command line or system call::
> > > +   keyctl revoke <key>
> > > +
> > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > +
> > > +Options Field Definition:
> > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > +		  generated key if a userkey is not defined here.
> > > +
> > > +    algorithm=    Encryption algorithm name as a string.
> > > +		  Valid algorithm: "aes-xts-128"
> > > +
> > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > +                  userkey...  (need to be clear here that this is being sent
> > > +                  to the hardware - kernel not messing w it)
> > > +
> > > +    entropy=      ascii hex value entropy.
> > > +                  This entropy will be used to generated the CPU key and
> > > +		  the tweak key when CPU generated key is requested.
> > > +
> > > +Algorithm Dependencies:
> > > +    AES-XTS 128 is the only supported algorithm.
> > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > +
> > > +    1) User specified encryption key
> > > +	- The user specified encryption key must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- A tweak key must be specified and it must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- No entropy field is accepted.
> > > +
> > > +    2) CPU generated encryption key
> > > +	- When no user specified encryption key is provided, the
> > > +	  default encryption key will be CPU generated.
> > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > +	  entropy will be used by the CPU to generate both the
> > > +	  encryption key and the tweak key.
> > > +	- No entropy field is accepted.
>              ^^^^^^^ should be tweak
> 
> >
> > This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> >
> > Actually, simple 'XOR' is used to generate the final key:
> >
> > case KEYID_SET_KEY_RANDOM:
> > 	......
> > 	(* Mix user supplied entropy to the data key and tweak key *)
> > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> >
> > So I think we can either just remove 'entropy' parameter, since we can use
> both 'userkey' and 'tweak' even for random key mode.
> >
> > In fact, which might be better IMHO, we can simply disallow or ignore
> 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> both entropies, and if user passes value with all 1, we are effectively making the
> key and tweak to be all 1, which is not random anymore.
> >
> > Instead, kernel can generate random for both entropies, or we can simply uses
> 0, ignoring user input.
> 
> Kai,
> I think my typo above, threw you off. We have the same understanding of the
> key fields.
> 
> Is this the structure you are suggesting?
> 
> 	Options
> 
> 	key_type=	"user" or "CPU"
> 
> 	key=		If key_type == user
> 				key= is the data key
> 			If key_type == CPU
> 				key= is not required
> 				if key= is present
> 					it is entropy to be mixed with
> 					CPU generated data key
> 
> 	tweak=		If key_type == user
> 				tweak= is the tweak key
> 			If key_type == CPU
> 				tweak= is not required
> 				if tweak= is present
> 					it is entropy to be mixed with
> 					CPU generated tweak key

Exactly.

Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

Thanks,
-Kai
> 
> 
> Alison
> >
> > Thanks,
> > -Kai
> 
> ........snip...........

RE: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

> -----Original Message-----
> From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> Behalf Of Alison Schofield
> Sent: Tuesday, September 11, 2018 12:13 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> Encryption API
> 
> On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> >
> > > -----Original Message-----
> > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > > Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:34 AM
> > > To: dhowells@redhat.com; tglx@linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-
> mm@kvack.org
> > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > Encryption API
> > >
> > > Document the API's used for MKTME on Intel platforms.
> > > MKTME: Multi-KEY Total Memory Encryption
> > >
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > ---
> > >  Documentation/x86/mktme-keys.txt | 153
> > > +++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 153 insertions(+)
> > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > >
> > > diff --git a/Documentation/x86/mktme-keys.txt
> > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > 000000000000..2dea7acd2a17
> > > --- /dev/null
> > > +++ b/Documentation/x86/mktme-keys.txt
> > > @@ -0,0 +1,153 @@
> > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > +Memory
> > > +Encryption) allows encryption of the entire system memory using a
> > > +single key, MKTME allows multiple encryption domains, each having
> > > +their own key. The main use case for the feature is virtual machine
> > > +isolation. The API's introduced here are intended to offer
> > > +flexibility to work in a
> > > wide range of uses.
> > > +
> > > +The externally available Intel Architecture Spec:
> > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > +Key-
> > > +Total-Memory-Encryption-Spec.pdf
> > > +
> > > +============================  API Overview
> > > +============================
> > > +
> > > +There are 2 MKTME specific API's that enable userspace to create
> > > +and use the memory encryption keys:
> > > +
> > > +1) Kernel Key Service: MKTME Type
> > > +
> > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > +   to support the memory encryption keys. The MKTME service manages
> > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > +   to hardware keyids and programs the hardware with user requested
> > > +   encryption parameters.
> > > +
> > > +   o An understanding of the Kernel Key Service is required in order
> > > +     to use the MKTME key type as it is a subset of that service.
> > > +
> > > +   o MKTME keys are a limited resource. There is a single pool of
> > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> >
> > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> although in the first generation server we only support 6-bit keyID, which is 63
> key/keyIDs (excluding keyID 0, which is TME's keyID).
> 
> My understanding is that low level SKU's could have as few as 3 bits available to
> hold the keyid, and that the max is 6 bits, hence 64.
> I probably don't need to be stating that level of detail here, but rather just
> iterate the important point that the resource is limited!
> 
> >
> > > +     With that in mind, userspace may take advantage of the kernel
> > > +     key services sharing and permissions model for userspace keys.
> > > +     One key can be shared as long as each user has the permission
> > > +     of "KEY_NEED_VIEW" to use it.
> > > +
> > > +   o MKTME key type uses capabilities to restrict the allocation
> > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > +
> > > +   o The MKTME key service blocks kernel key service commands that
> > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > +
> > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > +     may change the programming of a key by revoking it and adding
> > > +     a new key with the updated encryption options (or vice-versa).
> > > +
> > > +2) System Call: encrypt_mprotect()
> > > +
> > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > +   caller passes the serial number to a previously allocated and
> > > +   programmed encryption key. That handle was created with the MKTME
> > > +   Key Service.
> > > +
> > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > +
> > > +   o The range of memory that is to be protected must be mapped as
> > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > +     fails with EINVAL.
> > > +
> > > +   o As an extension to the existing mprotect() system call,
> > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > +     the enabling of memory encryption. That means that in addition
> > > +     to encrypting the memory, the protection flags will be updated
> > > +     as requested in the call.
> > > +
> > > +   o Additional mprotect() calls to memory already protected with
> > > +     MKTME will not alter the MKTME status.
> >
> > I think it's better to separate encrypt_mprotect() into another doc so both
> parts can be reviewed easier.
> 
> I can do that.
> Also, I do know I need man page for that too.
> >
> > > +
> > > +======================  Usage: MKTME Key Service
> > > +======================
> > > +
> > > +MKTME is enabled on supported Intel platforms by selecting
> > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > +
> > > +Allocating MKTME Keys via command line or system call:
> > > +    keyctl add mktme name "[options]" ring
> > > +
> > > +    key_serial_t add_key(const char *type, const char *description,
> > > +                         const void *payload, size_t plen,
> > > +                         key_serial_t keyring);
> > > +
> > > +Revoking MKTME Keys via command line or system call::
> > > +   keyctl revoke <key>
> > > +
> > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > +
> > > +Options Field Definition:
> > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > +		  generated key if a userkey is not defined here.
> > > +
> > > +    algorithm=    Encryption algorithm name as a string.
> > > +		  Valid algorithm: "aes-xts-128"
> > > +
> > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > +                  userkey...  (need to be clear here that this is being sent
> > > +                  to the hardware - kernel not messing w it)
> > > +
> > > +    entropy=      ascii hex value entropy.
> > > +                  This entropy will be used to generated the CPU key and
> > > +		  the tweak key when CPU generated key is requested.
> > > +
> > > +Algorithm Dependencies:
> > > +    AES-XTS 128 is the only supported algorithm.
> > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > +
> > > +    1) User specified encryption key
> > > +	- The user specified encryption key must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- A tweak key must be specified and it must be exactly
> > > +	  16 ASCII Hex bytes (128 bits).
> > > +	- No entropy field is accepted.
> > > +
> > > +    2) CPU generated encryption key
> > > +	- When no user specified encryption key is provided, the
> > > +	  default encryption key will be CPU generated.
> > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > +	  entropy will be used by the CPU to generate both the
> > > +	  encryption key and the tweak key.
> > > +	- No entropy field is accepted.
>              ^^^^^^^ should be tweak
> 
> >
> > This is not true. The spec says in CPU generated random mode, both 'key' and
> 'tweak' part are used to generate the final key and tweak respectively.
> >
> > Actually, simple 'XOR' is used to generate the final key:
> >
> > case KEYID_SET_KEY_RANDOM:
> > 	......
> > 	(* Mix user supplied entropy to the data key and tweak key *)
> > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> >
> > So I think we can either just remove 'entropy' parameter, since we can use
> both 'userkey' and 'tweak' even for random key mode.
> >
> > In fact, which might be better IMHO, we can simply disallow or ignore
> 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> both entropies, and if user passes value with all 1, we are effectively making the
> key and tweak to be all 1, which is not random anymore.
> >
> > Instead, kernel can generate random for both entropies, or we can simply uses
> 0, ignoring user input.
> 
> Kai,
> I think my typo above, threw you off. We have the same understanding of the
> key fields.
> 
> Is this the structure you are suggesting?
> 
> 	Options
> 
> 	key_type=	"user" or "CPU"
> 
> 	key=		If key_type == user
> 				key= is the data key
> 			If key_type == CPU
> 				key= is not required
> 				if key= is present
> 					it is entropy to be mixed with
> 					CPU generated data key
> 
> 	tweak=		If key_type == user
> 				tweak= is the tweak key
> 			If key_type == CPU
> 				tweak= is not required
> 				if tweak= is present
> 					it is entropy to be mixed with
> 					CPU generated tweak key

Exactly.

Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

Thanks,
-Kai
> 
> 
> Alison
> >
> > Thanks,
> > -Kai
> 
> ........snip...........

Re: [RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

On Mon, Sep 10, 2018 at 01:12:31PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Today mprotect is implemented to support legacy mprotect behavior
> > plus an extension for memory protection keys. Make it more generic
> > so that it can support additional extensions in the future.
> > 
> > This is done is preparation for adding a new system call for memory
> > encyption keys. The intent is that the new encrypted mprotect will be
> > another extension to legacy mprotect.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  mm/mprotect.c | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 68dc476310c0..56e64ef7931e 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -35,6 +35,8 @@
> >  
> >  #include "internal.h"
> >  
> > +#define NO_PKEY  -1
> 
> This commit does not make anything more generic but it does take
> away a magic number. The code change is senseful. The commit
> message is nonsense.

do_mprotect_ext() is intended to be the generic replacement for
do_mprotect_pkey() which was added for protection keys.

> 
> PS. Please use @linux.intel.com for LKML.
Is this a request to use your @linux.intel.com email address when I'm
posting to LKML's?

> 
> /Jarkko

[RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

On Mon, Sep 10, 2018 at 01:12:31PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Today mprotect is implemented to support legacy mprotect behavior
> > plus an extension for memory protection keys. Make it more generic
> > so that it can support additional extensions in the future.
> > 
> > This is done is preparation for adding a new system call for memory
> > encyption keys. The intent is that the new encrypted mprotect will be
> > another extension to legacy mprotect.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  mm/mprotect.c | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 68dc476310c0..56e64ef7931e 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -35,6 +35,8 @@
> >  
> >  #include "internal.h"
> >  
> > +#define NO_PKEY  -1
> 
> This commit does not make anything more generic but it does take
> away a magic number. The code change is senseful. The commit
> message is nonsense.

do_mprotect_ext() is intended to be the generic replacement for
do_mprotect_pkey() which was added for protection keys.

> 
> PS. Please use @linux.intel.com for LKML.
Is this a request to use your @linux.intel.com email address when I'm
posting to LKML's?

> 
> /Jarkko

Re: [RFC 02/12] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

On Mon, Sep 10, 2018 at 01:12:31PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:34 -0700, Alison Schofield wrote:
> > Today mprotect is implemented to support legacy mprotect behavior
> > plus an extension for memory protection keys. Make it more generic
> > so that it can support additional extensions in the future.
> > 
> > This is done is preparation for adding a new system call for memory
> > encyption keys. The intent is that the new encrypted mprotect will be
> > another extension to legacy mprotect.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  mm/mprotect.c | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 68dc476310c0..56e64ef7931e 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -35,6 +35,8 @@
> >  
> >  #include "internal.h"
> >  
> > +#define NO_PKEY  -1
> 
> This commit does not make anything more generic but it does take
> away a magic number. The code change is senseful. The commit
> message is nonsense.

do_mprotect_ext() is intended to be the generic replacement for
do_mprotect_pkey() which was added for protection keys.

> 
> PS. Please use @linux.intel.com for LKML.
Is this a request to use your @linux.intel.com email address when I'm
posting to LKML's?

> 
> /Jarkko

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 05:33:33PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Tuesday, September 11, 2018 12:13 PM
> > To: Huang, Kai <kai.huang@intel.com>
> > Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> > >
> > > > -----Original Message-----
> > > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > > > Behalf Of Alison Schofield
> > > > Sent: Saturday, September 8, 2018 10:34 AM
> > > > To: dhowells@redhat.com; tglx@linutronix.de
> > > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-
> > mm@kvack.org
> > > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > > Encryption API
> > > >
> > > > Document the API's used for MKTME on Intel platforms.
> > > > MKTME: Multi-KEY Total Memory Encryption
> > > >
> > > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > > ---
> > > >  Documentation/x86/mktme-keys.txt | 153
> > > > +++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 153 insertions(+)
> > > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > > >
> > > > diff --git a/Documentation/x86/mktme-keys.txt
> > > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > > 000000000000..2dea7acd2a17
> > > > --- /dev/null
> > > > +++ b/Documentation/x86/mktme-keys.txt
> > > > @@ -0,0 +1,153 @@
> > > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > > +Memory
> > > > +Encryption) allows encryption of the entire system memory using a
> > > > +single key, MKTME allows multiple encryption domains, each having
> > > > +their own key. The main use case for the feature is virtual machine
> > > > +isolation. The API's introduced here are intended to offer
> > > > +flexibility to work in a
> > > > wide range of uses.
> > > > +
> > > > +The externally available Intel Architecture Spec:
> > > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > > +Key-
> > > > +Total-Memory-Encryption-Spec.pdf
> > > > +
> > > > +==============  API Overview
> > > > +==============
> > > > +
> > > > +There are 2 MKTME specific API's that enable userspace to create
> > > > +and use the memory encryption keys:
> > > > +
> > > > +1) Kernel Key Service: MKTME Type
> > > > +
> > > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > > +   to support the memory encryption keys. The MKTME service manages
> > > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > > +   to hardware keyids and programs the hardware with user requested
> > > > +   encryption parameters.
> > > > +
> > > > +   o An understanding of the Kernel Key Service is required in order
> > > > +     to use the MKTME key type as it is a subset of that service.
> > > > +
> > > > +   o MKTME keys are a limited resource. There is a single pool of
> > > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> > >
> > > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> > although in the first generation server we only support 6-bit keyID, which is 63
> > key/keyIDs (excluding keyID 0, which is TME's keyID).
> > 
> > My understanding is that low level SKU's could have as few as 3 bits available to
> > hold the keyid, and that the max is 6 bits, hence 64.
> > I probably don't need to be stating that level of detail here, but rather just
> > iterate the important point that the resource is limited!
> > 
> > >
> > > > +     With that in mind, userspace may take advantage of the kernel
> > > > +     key services sharing and permissions model for userspace keys.
> > > > +     One key can be shared as long as each user has the permission
> > > > +     of "KEY_NEED_VIEW" to use it.
> > > > +
> > > > +   o MKTME key type uses capabilities to restrict the allocation
> > > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > > +
> > > > +   o The MKTME key service blocks kernel key service commands that
> > > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > > +
> > > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > > +     may change the programming of a key by revoking it and adding
> > > > +     a new key with the updated encryption options (or vice-versa).
> > > > +
> > > > +2) System Call: encrypt_mprotect()
> > > > +
> > > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > > +   caller passes the serial number to a previously allocated and
> > > > +   programmed encryption key. That handle was created with the MKTME
> > > > +   Key Service.
> > > > +
> > > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > > +
> > > > +   o The range of memory that is to be protected must be mapped as
> > > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > > +     fails with EINVAL.
> > > > +
> > > > +   o As an extension to the existing mprotect() system call,
> > > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > > +     the enabling of memory encryption. That means that in addition
> > > > +     to encrypting the memory, the protection flags will be updated
> > > > +     as requested in the call.
> > > > +
> > > > +   o Additional mprotect() calls to memory already protected with
> > > > +     MKTME will not alter the MKTME status.
> > >
> > > I think it's better to separate encrypt_mprotect() into another doc so both
> > parts can be reviewed easier.
> > 
> > I can do that.
> > Also, I do know I need man page for that too.
> > >
> > > > +
> > > > +===========  Usage: MKTME Key Service
> > > > +===========
> > > > +
> > > > +MKTME is enabled on supported Intel platforms by selecting
> > > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > > +
> > > > +Allocating MKTME Keys via command line or system call:
> > > > +    keyctl add mktme name "[options]" ring
> > > > +
> > > > +    key_serial_t add_key(const char *type, const char *description,
> > > > +                         const void *payload, size_t plen,
> > > > +                         key_serial_t keyring);
> > > > +
> > > > +Revoking MKTME Keys via command line or system call::
> > > > +   keyctl revoke <key>
> > > > +
> > > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > > +
> > > > +Options Field Definition:
> > > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > > +		  generated key if a userkey is not defined here.
> > > > +
> > > > +    algorithm=    Encryption algorithm name as a string.
> > > > +		  Valid algorithm: "aes-xts-128"
> > > > +
> > > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > > +                  userkey...  (need to be clear here that this is being sent
> > > > +                  to the hardware - kernel not messing w it)
> > > > +
> > > > +    entropy=      ascii hex value entropy.
> > > > +                  This entropy will be used to generated the CPU key and
> > > > +		  the tweak key when CPU generated key is requested.
> > > > +
> > > > +Algorithm Dependencies:
> > > > +    AES-XTS 128 is the only supported algorithm.
> > > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > > +
> > > > +    1) User specified encryption key
> > > > +	- The user specified encryption key must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- A tweak key must be specified and it must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- No entropy field is accepted.
> > > > +
> > > > +    2) CPU generated encryption key
> > > > +	- When no user specified encryption key is provided, the
> > > > +	  default encryption key will be CPU generated.
> > > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > > +	  entropy will be used by the CPU to generate both the
> > > > +	  encryption key and the tweak key.
> > > > +	- No entropy field is accepted.
> >              ^^^^^^^ should be tweak
> > 
> > >
> > > This is not true. The spec says in CPU generated random mode, both 'key' and
> > 'tweak' part are used to generate the final key and tweak respectively.
> > >
> > > Actually, simple 'XOR' is used to generate the final key:
> > >
> > > case KEYID_SET_KEY_RANDOM:
> > > 	......
> > > 	(* Mix user supplied entropy to the data key and tweak key *)
> > > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> > >
> > > So I think we can either just remove 'entropy' parameter, since we can use
> > both 'userkey' and 'tweak' even for random key mode.
> > >
> > > In fact, which might be better IMHO, we can simply disallow or ignore
> > 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> > both entropies, and if user passes value with all 1, we are effectively making the
> > key and tweak to be all 1, which is not random anymore.
> > >
> > > Instead, kernel can generate random for both entropies, or we can simply uses
> > 0, ignoring user input.
> > 
> > Kai,
> > I think my typo above, threw you off. We have the same understanding of the
> > key fields.
> > 
> > Is this the structure you are suggesting?
> > 
> > 	Options
> > 
> > 	key_type=	"user" or "CPU"
> > 
> > 	key=		If key_type = user
> > 				key= is the data key
> > 			If key_type = CPU
> > 				key= is not required
> > 				if key= is present
> > 					it is entropy to be mixed with
> > 					CPU generated data key
> > 
> > 	tweak=		If key_type = user
> > 				tweak= is the tweak key
> > 			If key_type = CPU
> > 				tweak= is not required
> > 				if tweak= is present
> > 					it is entropy to be mixed with
> > 					CPU generated tweak key
> 
> Exactly.
> 
> Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

A hardware key does get CLEAR'ed when the userspace key is revoked.
I don't think we identified any other user directed need to clear a key.

The no encryption option is currently considered not a requirement.
That means, although you see it in the Intel HW Spec, we don't have
use case that is driving us to implement it.

For other's info - no encryption would be an option where the key
tells the hardware not to do any encryption at all on this piece of memory.
All of memory not encrypted with these MKTME keys, is by default, encrypted
with the system level TME, Total Memory Encryption algorithm. (OK - not
really *all*, there is also a BIOS settable exclusion zone for TME)

> 
> Thanks,
> -Kai
> > 
> > 
> > Alison
> > >
> > > Thanks,
> > > -Kai
> > 
> > ........snip...........
> 

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 05:33:33PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Tuesday, September 11, 2018 12:13 PM
> > To: Huang, Kai <kai.huang@intel.com>
> > Cc: dhowells at redhat.com; tglx at linutronix.de; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> > x86 at kernel.org; linux-mm at kvack.org
> > Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> > >
> > > > -----Original Message-----
> > > > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org] On
> > > > Behalf Of Alison Schofield
> > > > Sent: Saturday, September 8, 2018 10:34 AM
> > > > To: dhowells at redhat.com; tglx at linutronix.de
> > > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.org;
> > > > keyrings at vger.kernel.org; linux-security-module at vger.kernel.org;
> > > > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-
> > mm at kvack.org
> > > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > > Encryption API
> > > >
> > > > Document the API's used for MKTME on Intel platforms.
> > > > MKTME: Multi-KEY Total Memory Encryption
> > > >
> > > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > > ---
> > > >  Documentation/x86/mktme-keys.txt | 153
> > > > +++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 153 insertions(+)
> > > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > > >
> > > > diff --git a/Documentation/x86/mktme-keys.txt
> > > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > > 000000000000..2dea7acd2a17
> > > > --- /dev/null
> > > > +++ b/Documentation/x86/mktme-keys.txt
> > > > @@ -0,0 +1,153 @@
> > > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > > +Memory
> > > > +Encryption) allows encryption of the entire system memory using a
> > > > +single key, MKTME allows multiple encryption domains, each having
> > > > +their own key. The main use case for the feature is virtual machine
> > > > +isolation. The API's introduced here are intended to offer
> > > > +flexibility to work in a
> > > > wide range of uses.
> > > > +
> > > > +The externally available Intel Architecture Spec:
> > > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > > +Key-
> > > > +Total-Memory-Encryption-Spec.pdf
> > > > +
> > > > +============================  API Overview
> > > > +============================
> > > > +
> > > > +There are 2 MKTME specific API's that enable userspace to create
> > > > +and use the memory encryption keys:
> > > > +
> > > > +1) Kernel Key Service: MKTME Type
> > > > +
> > > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > > +   to support the memory encryption keys. The MKTME service manages
> > > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > > +   to hardware keyids and programs the hardware with user requested
> > > > +   encryption parameters.
> > > > +
> > > > +   o An understanding of the Kernel Key Service is required in order
> > > > +     to use the MKTME key type as it is a subset of that service.
> > > > +
> > > > +   o MKTME keys are a limited resource. There is a single pool of
> > > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> > >
> > > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> > although in the first generation server we only support 6-bit keyID, which is 63
> > key/keyIDs (excluding keyID 0, which is TME's keyID).
> > 
> > My understanding is that low level SKU's could have as few as 3 bits available to
> > hold the keyid, and that the max is 6 bits, hence 64.
> > I probably don't need to be stating that level of detail here, but rather just
> > iterate the important point that the resource is limited!
> > 
> > >
> > > > +     With that in mind, userspace may take advantage of the kernel
> > > > +     key services sharing and permissions model for userspace keys.
> > > > +     One key can be shared as long as each user has the permission
> > > > +     of "KEY_NEED_VIEW" to use it.
> > > > +
> > > > +   o MKTME key type uses capabilities to restrict the allocation
> > > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > > +
> > > > +   o The MKTME key service blocks kernel key service commands that
> > > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > > +
> > > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > > +     may change the programming of a key by revoking it and adding
> > > > +     a new key with the updated encryption options (or vice-versa).
> > > > +
> > > > +2) System Call: encrypt_mprotect()
> > > > +
> > > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > > +   caller passes the serial number to a previously allocated and
> > > > +   programmed encryption key. That handle was created with the MKTME
> > > > +   Key Service.
> > > > +
> > > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > > +
> > > > +   o The range of memory that is to be protected must be mapped as
> > > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > > +     fails with EINVAL.
> > > > +
> > > > +   o As an extension to the existing mprotect() system call,
> > > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > > +     the enabling of memory encryption. That means that in addition
> > > > +     to encrypting the memory, the protection flags will be updated
> > > > +     as requested in the call.
> > > > +
> > > > +   o Additional mprotect() calls to memory already protected with
> > > > +     MKTME will not alter the MKTME status.
> > >
> > > I think it's better to separate encrypt_mprotect() into another doc so both
> > parts can be reviewed easier.
> > 
> > I can do that.
> > Also, I do know I need man page for that too.
> > >
> > > > +
> > > > +======================  Usage: MKTME Key Service
> > > > +======================
> > > > +
> > > > +MKTME is enabled on supported Intel platforms by selecting
> > > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > > +
> > > > +Allocating MKTME Keys via command line or system call:
> > > > +    keyctl add mktme name "[options]" ring
> > > > +
> > > > +    key_serial_t add_key(const char *type, const char *description,
> > > > +                         const void *payload, size_t plen,
> > > > +                         key_serial_t keyring);
> > > > +
> > > > +Revoking MKTME Keys via command line or system call::
> > > > +   keyctl revoke <key>
> > > > +
> > > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > > +
> > > > +Options Field Definition:
> > > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > > +		  generated key if a userkey is not defined here.
> > > > +
> > > > +    algorithm=    Encryption algorithm name as a string.
> > > > +		  Valid algorithm: "aes-xts-128"
> > > > +
> > > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > > +                  userkey...  (need to be clear here that this is being sent
> > > > +                  to the hardware - kernel not messing w it)
> > > > +
> > > > +    entropy=      ascii hex value entropy.
> > > > +                  This entropy will be used to generated the CPU key and
> > > > +		  the tweak key when CPU generated key is requested.
> > > > +
> > > > +Algorithm Dependencies:
> > > > +    AES-XTS 128 is the only supported algorithm.
> > > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > > +
> > > > +    1) User specified encryption key
> > > > +	- The user specified encryption key must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- A tweak key must be specified and it must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- No entropy field is accepted.
> > > > +
> > > > +    2) CPU generated encryption key
> > > > +	- When no user specified encryption key is provided, the
> > > > +	  default encryption key will be CPU generated.
> > > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > > +	  entropy will be used by the CPU to generate both the
> > > > +	  encryption key and the tweak key.
> > > > +	- No entropy field is accepted.
> >              ^^^^^^^ should be tweak
> > 
> > >
> > > This is not true. The spec says in CPU generated random mode, both 'key' and
> > 'tweak' part are used to generate the final key and tweak respectively.
> > >
> > > Actually, simple 'XOR' is used to generate the final key:
> > >
> > > case KEYID_SET_KEY_RANDOM:
> > > 	......
> > > 	(* Mix user supplied entropy to the data key and tweak key *)
> > > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> > >
> > > So I think we can either just remove 'entropy' parameter, since we can use
> > both 'userkey' and 'tweak' even for random key mode.
> > >
> > > In fact, which might be better IMHO, we can simply disallow or ignore
> > 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> > both entropies, and if user passes value with all 1, we are effectively making the
> > key and tweak to be all 1, which is not random anymore.
> > >
> > > Instead, kernel can generate random for both entropies, or we can simply uses
> > 0, ignoring user input.
> > 
> > Kai,
> > I think my typo above, threw you off. We have the same understanding of the
> > key fields.
> > 
> > Is this the structure you are suggesting?
> > 
> > 	Options
> > 
> > 	key_type=	"user" or "CPU"
> > 
> > 	key=		If key_type == user
> > 				key= is the data key
> > 			If key_type == CPU
> > 				key= is not required
> > 				if key= is present
> > 					it is entropy to be mixed with
> > 					CPU generated data key
> > 
> > 	tweak=		If key_type == user
> > 				tweak= is the tweak key
> > 			If key_type == CPU
> > 				tweak= is not required
> > 				if tweak= is present
> > 					it is entropy to be mixed with
> > 					CPU generated tweak key
> 
> Exactly.
> 
> Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

A hardware key does get CLEAR'ed when the userspace key is revoked.
I don't think we identified any other user directed need to clear a key.

The no encryption option is currently considered not a requirement.
That means, although you see it in the Intel HW Spec, we don't have
use case that is driving us to implement it.

For other's info - no encryption would be an option where the key
tells the hardware not to do any encryption at all on this piece of memory.
All of memory not encrypted with these MKTME keys, is by default, encrypted
with the system level TME, Total Memory Encryption algorithm. (OK - not
really *all*, there is also a BIOS settable exclusion zone for TME)

> 
> Thanks,
> -Kai
> > 
> > 
> > Alison
> > >
> > > Thanks,
> > > -Kai
> > 
> > ........snip...........
> 

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Alison Schofield]

On Mon, Sep 10, 2018 at 05:33:33PM -0700, Huang, Kai wrote:
> > -----Original Message-----
> > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > Behalf Of Alison Schofield
> > Sent: Tuesday, September 11, 2018 12:13 PM
> > To: Huang, Kai <kai.huang@intel.com>
> > Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> > x86@kernel.org; linux-mm@kvack.org
> > Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > Encryption API
> > 
> > On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> > >
> > > > -----Original Message-----
> > > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org] On
> > > > Behalf Of Alison Schofield
> > > > Sent: Saturday, September 8, 2018 10:34 AM
> > > > To: dhowells@redhat.com; tglx@linutronix.de
> > > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.com>;
> > > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.org;
> > > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.org;
> > > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-
> > mm@kvack.org
> > > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory
> > > > Encryption API
> > > >
> > > > Document the API's used for MKTME on Intel platforms.
> > > > MKTME: Multi-KEY Total Memory Encryption
> > > >
> > > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > > ---
> > > >  Documentation/x86/mktme-keys.txt | 153
> > > > +++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 153 insertions(+)
> > > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > > >
> > > > diff --git a/Documentation/x86/mktme-keys.txt
> > > > b/Documentation/x86/mktme- keys.txt new file mode 100644 index
> > > > 000000000000..2dea7acd2a17
> > > > --- /dev/null
> > > > +++ b/Documentation/x86/mktme-keys.txt
> > > > @@ -0,0 +1,153 @@
> > > > +MKTME (Multi-Key Total Memory Encryption) is a technology that
> > > > +allows memory encryption on Intel platforms. Whereas TME (Total
> > > > +Memory
> > > > +Encryption) allows encryption of the entire system memory using a
> > > > +single key, MKTME allows multiple encryption domains, each having
> > > > +their own key. The main use case for the feature is virtual machine
> > > > +isolation. The API's introduced here are intended to offer
> > > > +flexibility to work in a
> > > > wide range of uses.
> > > > +
> > > > +The externally available Intel Architecture Spec:
> > > > +https://software.intel.com/sites/default/files/managed/a5/16/Multi-
> > > > +Key-
> > > > +Total-Memory-Encryption-Spec.pdf
> > > > +
> > > > +============================  API Overview
> > > > +============================
> > > > +
> > > > +There are 2 MKTME specific API's that enable userspace to create
> > > > +and use the memory encryption keys:
> > > > +
> > > > +1) Kernel Key Service: MKTME Type
> > > > +
> > > > +   MKTME is a new key type added to the existing Kernel Key Services
> > > > +   to support the memory encryption keys. The MKTME service manages
> > > > +   the addition and removal of MKTME keys. It maps userspace keys
> > > > +   to hardware keyids and programs the hardware with user requested
> > > > +   encryption parameters.
> > > > +
> > > > +   o An understanding of the Kernel Key Service is required in order
> > > > +     to use the MKTME key type as it is a subset of that service.
> > > > +
> > > > +   o MKTME keys are a limited resource. There is a single pool of
> > > > +     MKTME keys for a system and that pool can be from 3 to 63 keys.
> > >
> > > Why 3 to 63 keys? Architecturally we are able to support up to 15-bit keyID,
> > although in the first generation server we only support 6-bit keyID, which is 63
> > key/keyIDs (excluding keyID 0, which is TME's keyID).
> > 
> > My understanding is that low level SKU's could have as few as 3 bits available to
> > hold the keyid, and that the max is 6 bits, hence 64.
> > I probably don't need to be stating that level of detail here, but rather just
> > iterate the important point that the resource is limited!
> > 
> > >
> > > > +     With that in mind, userspace may take advantage of the kernel
> > > > +     key services sharing and permissions model for userspace keys.
> > > > +     One key can be shared as long as each user has the permission
> > > > +     of "KEY_NEED_VIEW" to use it.
> > > > +
> > > > +   o MKTME key type uses capabilities to restrict the allocation
> > > > +     of keys. It only requires CAP_SYS_RESOURCE, but will accept
> > > > +     the broader capability of CAP_SYS_ADMIN.  See capabilities(7).
> > > > +
> > > > +   o The MKTME key service blocks kernel key service commands that
> > > > +     could lead to reprogramming of in use keys, or loss of keys from
> > > > +     the pool. This means MKTME does not allow a key to be invalidated,
> > > > +     unlinked, or timed out. These operations are blocked by MKTME as
> > > > +     it creates all keys with the internal flag KEY_FLAG_KEEP.
> > > > +
> > > > +   o MKTME does not support the keyctl option of UPDATE. Userspace
> > > > +     may change the programming of a key by revoking it and adding
> > > > +     a new key with the updated encryption options (or vice-versa).
> > > > +
> > > > +2) System Call: encrypt_mprotect()
> > > > +
> > > > +   MKTME encryption is requested by calling encrypt_mprotect(). The
> > > > +   caller passes the serial number to a previously allocated and
> > > > +   programmed encryption key. That handle was created with the MKTME
> > > > +   Key Service.
> > > > +
> > > > +   o The caller must have KEY_NEED_VIEW permission on the key
> > > > +
> > > > +   o The range of memory that is to be protected must be mapped as
> > > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect() request
> > > > +     fails with EINVAL.
> > > > +
> > > > +   o As an extension to the existing mprotect() system call,
> > > > +     encrypt_mprotect() supports the legacy mprotect behavior plus
> > > > +     the enabling of memory encryption. That means that in addition
> > > > +     to encrypting the memory, the protection flags will be updated
> > > > +     as requested in the call.
> > > > +
> > > > +   o Additional mprotect() calls to memory already protected with
> > > > +     MKTME will not alter the MKTME status.
> > >
> > > I think it's better to separate encrypt_mprotect() into another doc so both
> > parts can be reviewed easier.
> > 
> > I can do that.
> > Also, I do know I need man page for that too.
> > >
> > > > +
> > > > +======================  Usage: MKTME Key Service
> > > > +======================
> > > > +
> > > > +MKTME is enabled on supported Intel platforms by selecting
> > > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > > +
> > > > +Allocating MKTME Keys via command line or system call:
> > > > +    keyctl add mktme name "[options]" ring
> > > > +
> > > > +    key_serial_t add_key(const char *type, const char *description,
> > > > +                         const void *payload, size_t plen,
> > > > +                         key_serial_t keyring);
> > > > +
> > > > +Revoking MKTME Keys via command line or system call::
> > > > +   keyctl revoke <key>
> > > > +
> > > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > > +
> > > > +Options Field Definition:
> > > > +    userkey=      ASCII HEX value encryption key. Defaults to a CPU
> > > > +		  generated key if a userkey is not defined here.
> > > > +
> > > > +    algorithm=    Encryption algorithm name as a string.
> > > > +		  Valid algorithm: "aes-xts-128"
> > > > +
> > > > +    tweak=        ASCII HEX value tweak key. Tweak key will be added to the
> > > > +                  userkey...  (need to be clear here that this is being sent
> > > > +                  to the hardware - kernel not messing w it)
> > > > +
> > > > +    entropy=      ascii hex value entropy.
> > > > +                  This entropy will be used to generated the CPU key and
> > > > +		  the tweak key when CPU generated key is requested.
> > > > +
> > > > +Algorithm Dependencies:
> > > > +    AES-XTS 128 is the only supported algorithm.
> > > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > > +
> > > > +    1) User specified encryption key
> > > > +	- The user specified encryption key must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- A tweak key must be specified and it must be exactly
> > > > +	  16 ASCII Hex bytes (128 bits).
> > > > +	- No entropy field is accepted.
> > > > +
> > > > +    2) CPU generated encryption key
> > > > +	- When no user specified encryption key is provided, the
> > > > +	  default encryption key will be CPU generated.
> > > > +	- User must specify 16 ASCII Hex bytes of entropy. This
> > > > +	  entropy will be used by the CPU to generate both the
> > > > +	  encryption key and the tweak key.
> > > > +	- No entropy field is accepted.
> >              ^^^^^^^ should be tweak
> > 
> > >
> > > This is not true. The spec says in CPU generated random mode, both 'key' and
> > 'tweak' part are used to generate the final key and tweak respectively.
> > >
> > > Actually, simple 'XOR' is used to generate the final key:
> > >
> > > case KEYID_SET_KEY_RANDOM:
> > > 	......
> > > 	(* Mix user supplied entropy to the data key and tweak key *)
> > > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> > >
> > > So I think we can either just remove 'entropy' parameter, since we can use
> > both 'userkey' and 'tweak' even for random key mode.
> > >
> > > In fact, which might be better IMHO, we can simply disallow or ignore
> > 'userkey' and 'tweak' part for random key mode, since if we allow user to specify
> > both entropies, and if user passes value with all 1, we are effectively making the
> > key and tweak to be all 1, which is not random anymore.
> > >
> > > Instead, kernel can generate random for both entropies, or we can simply uses
> > 0, ignoring user input.
> > 
> > Kai,
> > I think my typo above, threw you off. We have the same understanding of the
> > key fields.
> > 
> > Is this the structure you are suggesting?
> > 
> > 	Options
> > 
> > 	key_type=	"user" or "CPU"
> > 
> > 	key=		If key_type == user
> > 				key= is the data key
> > 			If key_type == CPU
> > 				key= is not required
> > 				if key= is present
> > 					it is entropy to be mixed with
> > 					CPU generated data key
> > 
> > 	tweak=		If key_type == user
> > 				tweak= is the tweak key
> > 			If key_type == CPU
> > 				tweak= is not required
> > 				if tweak= is present
> > 					it is entropy to be mixed with
> > 					CPU generated tweak key
> 
> Exactly.
> 
> Although I am not sure whether we should support other 2 modes: Clear key  and  no encryption;

A hardware key does get CLEAR'ed when the userspace key is revoked.
I don't think we identified any other user directed need to clear a key.

The no encryption option is currently considered not a requirement.
That means, although you see it in the Intel HW Spec, we don't have
use case that is driving us to implement it.

For other's info - no encryption would be an option where the key
tells the hardware not to do any encryption at all on this piece of memory.
All of memory not encrypted with these MKTME keys, is by default, encrypted
with the system level TME, Total Memory Encryption algorithm. (OK - not
really *all*, there is also a BIOS settable exclusion zone for TME)

> 
> Thanks,
> -Kai
> > 
> > 
> > Alison
> > >
> > > Thanks,
> > > -Kai
> > 
> > ........snip...........
> 

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

T24gTW9uLCAyMDE4LTA5LTEwIGF0IDE3OjQ1IC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBPbiBNb24sIFNlcCAxMCwgMjAxOCBhdCAwNTozMzozM1BNIC0wNzAwLCBIdWFuZywgS2Fp
IHdyb3RlOg0KPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+IEZyb206IG93
bmVyLWxpbnV4LW1tQGt2YWNrLm9yZyBbbWFpbHRvOm93bmVyLWxpbnV4LW1tQGt2YWNrLm9yZ10N
Cj4gPiA+IE9uDQo+ID4gPiBCZWhhbGYgT2YgQWxpc29uIFNjaG9maWVsZA0KPiA+ID4gU2VudDog
VHVlc2RheSwgU2VwdGVtYmVyIDExLCAyMDE4IDEyOjEzIFBNDQo+ID4gPiBUbzogSHVhbmcsIEth
aSA8a2FpLmh1YW5nQGludGVsLmNvbT4NCj4gPiA+IENjOiBkaG93ZWxsc0ByZWRoYXQuY29tOyB0
Z2x4QGxpbnV0cm9uaXguZGU7IE5ha2FqaW1hLCBKdW4NCj4gPiA+IDxqdW4ubmFrYWppbWFAaW50
ZWwuY29tPjsgU2h1dGVtb3YsIEtpcmlsbCA8a2lyaWxsLnNodXRlbW92QGludGVsDQo+ID4gPiAu
Y29tPjsNCj4gPiA+IEhhbnNlbiwgRGF2ZSA8ZGF2ZS5oYW5zZW5AaW50ZWwuY29tPjsgU2Fra2lu
ZW4sIEphcmtrbw0KPiA+ID4gPGphcmtrby5zYWtraW5lbkBpbnRlbC5jb20+OyBqbW9ycmlzQG5h
bWVpLm9yZzsga2V5cmluZ3NAdmdlci5rZXINCj4gPiA+IG5lbC5vcmc7DQo+ID4gPiBsaW51eC1z
ZWN1cml0eS1tb2R1bGVAdmdlci5rZXJuZWwub3JnOyBtaW5nb0ByZWRoYXQuY29tOyBocGFAenl0
bw0KPiA+ID4gci5jb207DQo+ID4gPiB4ODZAa2VybmVsLm9yZzsgbGludXgtbW1Aa3ZhY2sub3Jn
DQo+ID4gPiBTdWJqZWN0OiBSZTogW1JGQyAwMS8xMl0gZG9jcy94ODY6IERvY3VtZW50IHRoZSBN
dWx0aS1LZXkgVG90YWwNCj4gPiA+IE1lbW9yeQ0KPiA+ID4gRW5jcnlwdGlvbiBBUEkNCj4gPiA+
IA0KPiA+ID4gT24gU3VuLCBTZXAgMDksIDIwMTggYXQgMDY6Mjg6MjhQTSAtMDcwMCwgSHVhbmcs
IEthaSB3cm90ZToNCj4gPiA+ID4gDQo+ID4gPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0t
LS0NCj4gPiA+ID4gPiBGcm9tOiBvd25lci1saW51eC1tbUBrdmFjay5vcmcgW21haWx0bzpvd25l
ci1saW51eC1tbUBrdmFjay5vDQo+ID4gPiA+ID4gcmddIE9uDQo+ID4gPiA+ID4gQmVoYWxmIE9m
IEFsaXNvbiBTY2hvZmllbGQNCj4gPiA+ID4gPiBTZW50OiBTYXR1cmRheSwgU2VwdGVtYmVyIDgs
IDIwMTggMTA6MzQgQU0NCj4gPiA+ID4gPiBUbzogZGhvd2VsbHNAcmVkaGF0LmNvbTsgdGdseEBs
aW51dHJvbml4LmRlDQo+ID4gPiA+ID4gQ2M6IEh1YW5nLCBLYWkgPGthaS5odWFuZ0BpbnRlbC5j
b20+OyBOYWthamltYSwgSnVuDQo+ID4gPiA+ID4gPGp1bi5uYWthamltYUBpbnRlbC5jb20+OyBT
aHV0ZW1vdiwgS2lyaWxsDQo+ID4gPiA+ID4gPGtpcmlsbC5zaHV0ZW1vdkBpbnRlbC5jb20+OyBI
YW5zZW4sIERhdmUgPGRhdmUuaGFuc2VuQGludGVsLg0KPiA+ID4gPiA+IGNvbT47DQo+ID4gPiA+
ID4gU2Fra2luZW4sIEphcmtrbyA8amFya2tvLnNha2tpbmVuQGludGVsLmNvbT47IGptb3JyaXNA
bmFtZWkubw0KPiA+ID4gPiA+IHJnOw0KPiA+ID4gPiA+IGtleXJpbmdzQHZnZXIua2VybmVsLm9y
ZzsgbGludXgtc2VjdXJpdHktbW9kdWxlQHZnZXIua2VybmVsLm8NCj4gPiA+ID4gPiByZzsNCj4g
PiA+ID4gPiBtaW5nb0ByZWRoYXQuY29tOyBocGFAenl0b3IuY29tOyB4ODZAa2VybmVsLm9yZzsg
bGludXgtDQo+ID4gPiANCj4gPiA+IG1tQGt2YWNrLm9yZw0KPiA+ID4gPiA+IFN1YmplY3Q6IFtS
RkMgMDEvMTJdIGRvY3MveDg2OiBEb2N1bWVudCB0aGUgTXVsdGktS2V5IFRvdGFsDQo+ID4gPiA+
ID4gTWVtb3J5DQo+ID4gPiA+ID4gRW5jcnlwdGlvbiBBUEkNCj4gPiA+ID4gPiANCj4gPiA+ID4g
PiBEb2N1bWVudCB0aGUgQVBJJ3MgdXNlZCBmb3IgTUtUTUUgb24gSW50ZWwgcGxhdGZvcm1zLg0K
PiA+ID4gPiA+IE1LVE1FOiBNdWx0aS1LRVkgVG90YWwgTWVtb3J5IEVuY3J5cHRpb24NCj4gPiA+
ID4gPiANCj4gPiA+ID4gPiBTaWduZWQtb2ZmLWJ5OiBBbGlzb24gU2Nob2ZpZWxkIDxhbGlzb24u
c2Nob2ZpZWxkQGludGVsLmNvbT4NCj4gPiA+ID4gPiAtLS0NCj4gPiA+ID4gPiAgRG9jdW1lbnRh
dGlvbi94ODYvbWt0bWUta2V5cy50eHQgfCAxNTMNCj4gPiA+ID4gPiArKysrKysrKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysNCj4gPiA+ID4gPiAgMSBmaWxlIGNoYW5nZWQsIDE1MyBp
bnNlcnRpb25zKCspDQo+ID4gPiA+ID4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBEb2N1bWVudGF0aW9u
L3g4Ni9ta3RtZS1rZXlzLnR4dA0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IGRpZmYgLS1naXQgYS9E
b2N1bWVudGF0aW9uL3g4Ni9ta3RtZS1rZXlzLnR4dA0KPiA+ID4gPiA+IGIvRG9jdW1lbnRhdGlv
bi94ODYvbWt0bWUtIGtleXMudHh0IG5ldyBmaWxlIG1vZGUgMTAwNjQ0DQo+ID4gPiA+ID4gaW5k
ZXgNCj4gPiA+ID4gPiAwMDAwMDAwMDAwMDAuLjJkZWE3YWNkMmExNw0KPiA+ID4gPiA+IC0tLSAv
ZGV2L251bGwNCj4gPiA+ID4gPiArKysgYi9Eb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS1rZXlzLnR4
dA0KPiA+ID4gPiA+IEBAIC0wLDAgKzEsMTUzIEBADQo+ID4gPiA+ID4gK01LVE1FIChNdWx0aS1L
ZXkgVG90YWwgTWVtb3J5IEVuY3J5cHRpb24pIGlzIGEgdGVjaG5vbG9neQ0KPiA+ID4gPiA+IHRo
YXQNCj4gPiA+ID4gPiArYWxsb3dzIG1lbW9yeSBlbmNyeXB0aW9uIG9uIEludGVsIHBsYXRmb3Jt
cy4gV2hlcmVhcyBUTUUNCj4gPiA+ID4gPiAoVG90YWwNCj4gPiA+ID4gPiArTWVtb3J5DQo+ID4g
PiA+ID4gK0VuY3J5cHRpb24pIGFsbG93cyBlbmNyeXB0aW9uIG9mIHRoZSBlbnRpcmUgc3lzdGVt
IG1lbW9yeQ0KPiA+ID4gPiA+IHVzaW5nIGENCj4gPiA+ID4gPiArc2luZ2xlIGtleSwgTUtUTUUg
YWxsb3dzIG11bHRpcGxlIGVuY3J5cHRpb24gZG9tYWlucywgZWFjaA0KPiA+ID4gPiA+IGhhdmlu
Zw0KPiA+ID4gPiA+ICt0aGVpciBvd24ga2V5LiBUaGUgbWFpbiB1c2UgY2FzZSBmb3IgdGhlIGZl
YXR1cmUgaXMgdmlydHVhbA0KPiA+ID4gPiA+IG1hY2hpbmUNCj4gPiA+ID4gPiAraXNvbGF0aW9u
LiBUaGUgQVBJJ3MgaW50cm9kdWNlZCBoZXJlIGFyZSBpbnRlbmRlZCB0byBvZmZlcg0KPiA+ID4g
PiA+ICtmbGV4aWJpbGl0eSB0byB3b3JrIGluIGENCj4gPiA+ID4gPiB3aWRlIHJhbmdlIG9mIHVz
ZXMuDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICtUaGUgZXh0ZXJuYWxseSBhdmFpbGFibGUgSW50
ZWwgQXJjaGl0ZWN0dXJlIFNwZWM6DQo+ID4gPiA+ID4gK2h0dHBzOi8vc29mdHdhcmUuaW50ZWwu
Y29tL3NpdGVzL2RlZmF1bHQvZmlsZXMvbWFuYWdlZC9hNS8xNg0KPiA+ID4gPiA+IC9NdWx0aS0N
Cj4gPiA+ID4gPiArS2V5LQ0KPiA+ID4gPiA+ICtUb3RhbC1NZW1vcnktRW5jcnlwdGlvbi1TcGVj
LnBkZg0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiArPT09PT09PT09PT09PT09PT09PT09PT09PT09
PSAgQVBJIE92ZXJ2aWV3DQo+ID4gPiA+ID4gKz09PT09PT09PT09PT09PT09PT09PT09PT09PT0N
Cj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gK1RoZXJlIGFyZSAyIE1LVE1FIHNwZWNpZmljIEFQSSdz
IHRoYXQgZW5hYmxlIHVzZXJzcGFjZSB0bw0KPiA+ID4gPiA+IGNyZWF0ZQ0KPiA+ID4gPiA+ICth
bmQgdXNlIHRoZSBtZW1vcnkgZW5jcnlwdGlvbiBrZXlzOg0KPiA+ID4gPiA+ICsNCj4gPiA+ID4g
PiArMSkgS2VybmVsIEtleSBTZXJ2aWNlOiBNS1RNRSBUeXBlDQo+ID4gPiA+ID4gKw0KPiA+ID4g
PiA+ICsgICBNS1RNRSBpcyBhIG5ldyBrZXkgdHlwZSBhZGRlZCB0byB0aGUgZXhpc3RpbmcgS2Vy
bmVsIEtleQ0KPiA+ID4gPiA+IFNlcnZpY2VzDQo+ID4gPiA+ID4gKyAgIHRvIHN1cHBvcnQgdGhl
IG1lbW9yeSBlbmNyeXB0aW9uIGtleXMuIFRoZSBNS1RNRSBzZXJ2aWNlDQo+ID4gPiA+ID4gbWFu
YWdlcw0KPiA+ID4gPiA+ICsgICB0aGUgYWRkaXRpb24gYW5kIHJlbW92YWwgb2YgTUtUTUUga2V5
cy4gSXQgbWFwcyB1c2Vyc3BhY2UNCj4gPiA+ID4gPiBrZXlzDQo+ID4gPiA+ID4gKyAgIHRvIGhh
cmR3YXJlIGtleWlkcyBhbmQgcHJvZ3JhbXMgdGhlIGhhcmR3YXJlIHdpdGggdXNlcg0KPiA+ID4g
PiA+IHJlcXVlc3RlZA0KPiA+ID4gPiA+ICsgICBlbmNyeXB0aW9uIHBhcmFtZXRlcnMuDQo+ID4g
PiA+ID4gKw0KPiA+ID4gPiA+ICsgICBvIEFuIHVuZGVyc3RhbmRpbmcgb2YgdGhlIEtlcm5lbCBL
ZXkgU2VydmljZSBpcyByZXF1aXJlZA0KPiA+ID4gPiA+IGluIG9yZGVyDQo+ID4gPiA+ID4gKyAg
ICAgdG8gdXNlIHRoZSBNS1RNRSBrZXkgdHlwZSBhcyBpdCBpcyBhIHN1YnNldCBvZiB0aGF0DQo+
ID4gPiA+ID4gc2VydmljZS4NCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgIG8gTUtUTUUga2V5
cyBhcmUgYSBsaW1pdGVkIHJlc291cmNlLiBUaGVyZSBpcyBhIHNpbmdsZQ0KPiA+ID4gPiA+IHBv
b2wgb2YNCj4gPiA+ID4gPiArICAgICBNS1RNRSBrZXlzIGZvciBhIHN5c3RlbSBhbmQgdGhhdCBw
b29sIGNhbiBiZSBmcm9tIDMgdG8NCj4gPiA+ID4gPiA2MyBrZXlzLg0KPiA+ID4gPiANCj4gPiA+
ID4gV2h5IDMgdG8gNjMga2V5cz8gQXJjaGl0ZWN0dXJhbGx5IHdlIGFyZSBhYmxlIHRvIHN1cHBv
cnQgdXAgdG8NCj4gPiA+ID4gMTUtYml0IGtleUlELA0KPiA+ID4gDQo+ID4gPiBhbHRob3VnaCBp
biB0aGUgZmlyc3QgZ2VuZXJhdGlvbiBzZXJ2ZXIgd2Ugb25seSBzdXBwb3J0IDYtYml0DQo+ID4g
PiBrZXlJRCwgd2hpY2ggaXMgNjMNCj4gPiA+IGtleS9rZXlJRHMgKGV4Y2x1ZGluZyBrZXlJRCAw
LCB3aGljaCBpcyBUTUUncyBrZXlJRCkuDQo+ID4gPiANCj4gPiA+IE15IHVuZGVyc3RhbmRpbmcg
aXMgdGhhdCBsb3cgbGV2ZWwgU0tVJ3MgY291bGQgaGF2ZSBhcyBmZXcgYXMgMw0KPiA+ID4gYml0
cyBhdmFpbGFibGUgdG8NCj4gPiA+IGhvbGQgdGhlIGtleWlkLCBhbmQgdGhhdCB0aGUgbWF4IGlz
IDYgYml0cywgaGVuY2UgNjQuDQo+ID4gPiBJIHByb2JhYmx5IGRvbid0IG5lZWQgdG8gYmUgc3Rh
dGluZyB0aGF0IGxldmVsIG9mIGRldGFpbCBoZXJlLA0KPiA+ID4gYnV0IHJhdGhlciBqdXN0DQo+
ID4gPiBpdGVyYXRlIHRoZSBpbXBvcnRhbnQgcG9pbnQgdGhhdCB0aGUgcmVzb3VyY2UgaXMgbGlt
aXRlZCENCj4gPiA+IA0KPiA+ID4gPiANCj4gPiA+ID4gPiArICAgICBXaXRoIHRoYXQgaW4gbWlu
ZCwgdXNlcnNwYWNlIG1heSB0YWtlIGFkdmFudGFnZSBvZiB0aGUNCj4gPiA+ID4gPiBrZXJuZWwN
Cj4gPiA+ID4gPiArICAgICBrZXkgc2VydmljZXMgc2hhcmluZyBhbmQgcGVybWlzc2lvbnMgbW9k
ZWwgZm9yDQo+ID4gPiA+ID4gdXNlcnNwYWNlIGtleXMuDQo+ID4gPiA+ID4gKyAgICAgT25lIGtl
eSBjYW4gYmUgc2hhcmVkIGFzIGxvbmcgYXMgZWFjaCB1c2VyIGhhcyB0aGUNCj4gPiA+ID4gPiBw
ZXJtaXNzaW9uDQo+ID4gPiA+ID4gKyAgICAgb2YgIktFWV9ORUVEX1ZJRVciIHRvIHVzZSBpdC4N
Cj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgIG8gTUtUTUUga2V5IHR5cGUgdXNlcyBjYXBhYmls
aXRpZXMgdG8gcmVzdHJpY3QgdGhlDQo+ID4gPiA+ID4gYWxsb2NhdGlvbg0KPiA+ID4gPiA+ICsg
ICAgIG9mIGtleXMuIEl0IG9ubHkgcmVxdWlyZXMgQ0FQX1NZU19SRVNPVVJDRSwgYnV0IHdpbGwN
Cj4gPiA+ID4gPiBhY2NlcHQNCj4gPiA+ID4gPiArICAgICB0aGUgYnJvYWRlciBjYXBhYmlsaXR5
IG9mIENBUF9TWVNfQURNSU4uICBTZWUNCj4gPiA+ID4gPiBjYXBhYmlsaXRpZXMoNykuDQo+ID4g
PiA+ID4gKw0KPiA+ID4gPiA+ICsgICBvIFRoZSBNS1RNRSBrZXkgc2VydmljZSBibG9ja3Mga2Vy
bmVsIGtleSBzZXJ2aWNlDQo+ID4gPiA+ID4gY29tbWFuZHMgdGhhdA0KPiA+ID4gPiA+ICsgICAg
IGNvdWxkIGxlYWQgdG8gcmVwcm9ncmFtbWluZyBvZiBpbiB1c2Uga2V5cywgb3IgbG9zcyBvZg0K
PiA+ID4gPiA+IGtleXMgZnJvbQ0KPiA+ID4gPiA+ICsgICAgIHRoZSBwb29sLiBUaGlzIG1lYW5z
IE1LVE1FIGRvZXMgbm90IGFsbG93IGEga2V5IHRvIGJlDQo+ID4gPiA+ID4gaW52YWxpZGF0ZWQs
DQo+ID4gPiA+ID4gKyAgICAgdW5saW5rZWQsIG9yIHRpbWVkIG91dC4gVGhlc2Ugb3BlcmF0aW9u
cyBhcmUgYmxvY2tlZCBieQ0KPiA+ID4gPiA+IE1LVE1FIGFzDQo+ID4gPiA+ID4gKyAgICAgaXQg
Y3JlYXRlcyBhbGwga2V5cyB3aXRoIHRoZSBpbnRlcm5hbCBmbGFnDQo+ID4gPiA+ID4gS0VZX0ZM
QUdfS0VFUC4NCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgIG8gTUtUTUUgZG9lcyBub3Qgc3Vw
cG9ydCB0aGUga2V5Y3RsIG9wdGlvbiBvZiBVUERBVEUuDQo+ID4gPiA+ID4gVXNlcnNwYWNlDQo+
ID4gPiA+ID4gKyAgICAgbWF5IGNoYW5nZSB0aGUgcHJvZ3JhbW1pbmcgb2YgYSBrZXkgYnkgcmV2
b2tpbmcgaXQgYW5kDQo+ID4gPiA+ID4gYWRkaW5nDQo+ID4gPiA+ID4gKyAgICAgYSBuZXcga2V5
IHdpdGggdGhlIHVwZGF0ZWQgZW5jcnlwdGlvbiBvcHRpb25zIChvciB2aWNlLQ0KPiA+ID4gPiA+
IHZlcnNhKS4NCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKzIpIFN5c3RlbSBDYWxsOiBlbmNyeXB0
X21wcm90ZWN0KCkNCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgIE1LVE1FIGVuY3J5cHRpb24g
aXMgcmVxdWVzdGVkIGJ5IGNhbGxpbmcNCj4gPiA+ID4gPiBlbmNyeXB0X21wcm90ZWN0KCkuIFRo
ZQ0KPiA+ID4gPiA+ICsgICBjYWxsZXIgcGFzc2VzIHRoZSBzZXJpYWwgbnVtYmVyIHRvIGEgcHJl
dmlvdXNseSBhbGxvY2F0ZWQNCj4gPiA+ID4gPiBhbmQNCj4gPiA+ID4gPiArICAgcHJvZ3JhbW1l
ZCBlbmNyeXB0aW9uIGtleS4gVGhhdCBoYW5kbGUgd2FzIGNyZWF0ZWQgd2l0aA0KPiA+ID4gPiA+
IHRoZSBNS1RNRQ0KPiA+ID4gPiA+ICsgICBLZXkgU2VydmljZS4NCj4gPiA+ID4gPiArDQo+ID4g
PiA+ID4gKyAgIG8gVGhlIGNhbGxlciBtdXN0IGhhdmUgS0VZX05FRURfVklFVyBwZXJtaXNzaW9u
IG9uIHRoZQ0KPiA+ID4gPiA+IGtleQ0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiArICAgbyBUaGUg
cmFuZ2Ugb2YgbWVtb3J5IHRoYXQgaXMgdG8gYmUgcHJvdGVjdGVkIG11c3QgYmUNCj4gPiA+ID4g
PiBtYXBwZWQgYXMNCj4gPiA+ID4gPiArICAgICBBTk9OWU1PVVMuIElmIGl0IGlzIG5vdCwgdGhl
IGVudGlyZSBlbmNyeXB0X21wcm90ZWN0KCkNCj4gPiA+ID4gPiByZXF1ZXN0DQo+ID4gPiA+ID4g
KyAgICAgZmFpbHMgd2l0aCBFSU5WQUwuDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICsgICBvIEFz
IGFuIGV4dGVuc2lvbiB0byB0aGUgZXhpc3RpbmcgbXByb3RlY3QoKSBzeXN0ZW0gY2FsbCwNCj4g
PiA+ID4gPiArICAgICBlbmNyeXB0X21wcm90ZWN0KCkgc3VwcG9ydHMgdGhlIGxlZ2FjeSBtcHJv
dGVjdA0KPiA+ID4gPiA+IGJlaGF2aW9yIHBsdXMNCj4gPiA+ID4gPiArICAgICB0aGUgZW5hYmxp
bmcgb2YgbWVtb3J5IGVuY3J5cHRpb24uIFRoYXQgbWVhbnMgdGhhdCBpbg0KPiA+ID4gPiA+IGFk
ZGl0aW9uDQo+ID4gPiA+ID4gKyAgICAgdG8gZW5jcnlwdGluZyB0aGUgbWVtb3J5LCB0aGUgcHJv
dGVjdGlvbiBmbGFncyB3aWxsIGJlDQo+ID4gPiA+ID4gdXBkYXRlZA0KPiA+ID4gPiA+ICsgICAg
IGFzIHJlcXVlc3RlZCBpbiB0aGUgY2FsbC4NCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgIG8g
QWRkaXRpb25hbCBtcHJvdGVjdCgpIGNhbGxzIHRvIG1lbW9yeSBhbHJlYWR5IHByb3RlY3RlZA0K
PiA+ID4gPiA+IHdpdGgNCj4gPiA+ID4gPiArICAgICBNS1RNRSB3aWxsIG5vdCBhbHRlciB0aGUg
TUtUTUUgc3RhdHVzLg0KPiA+ID4gPiANCj4gPiA+ID4gSSB0aGluayBpdCdzIGJldHRlciB0byBz
ZXBhcmF0ZSBlbmNyeXB0X21wcm90ZWN0KCkgaW50byBhbm90aGVyDQo+ID4gPiA+IGRvYyBzbyBi
b3RoDQo+ID4gPiANCj4gPiA+IHBhcnRzIGNhbiBiZSByZXZpZXdlZCBlYXNpZXIuDQo+ID4gPiAN
Cj4gPiA+IEkgY2FuIGRvIHRoYXQuDQo+ID4gPiBBbHNvLCBJIGRvIGtub3cgSSBuZWVkIG1hbiBw
YWdlIGZvciB0aGF0IHRvby4NCj4gPiA+ID4gDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICs9PT09
PT09PT09PT09PT09PT09PT09ICBVc2FnZTogTUtUTUUgS2V5IFNlcnZpY2UNCj4gPiA+ID4gPiAr
PT09PT09PT09PT09PT09PT09PT09PQ0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiArTUtUTUUgaXMg
ZW5hYmxlZCBvbiBzdXBwb3J0ZWQgSW50ZWwgcGxhdGZvcm1zIGJ5IHNlbGVjdGluZw0KPiA+ID4g
PiA+ICtDT05GSUdfWDg2X0lOVEVMX01LVE1FIHdoaWNoIHNlbGVjdHMgQ09ORklHX01LVE1FX0tF
WVMuDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICtBbGxvY2F0aW5nIE1LVE1FIEtleXMgdmlhIGNv
bW1hbmQgbGluZSBvciBzeXN0ZW0gY2FsbDoNCj4gPiA+ID4gPiArICAgIGtleWN0bCBhZGQgbWt0
bWUgbmFtZSAiW29wdGlvbnNdIiByaW5nDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICsgICAga2V5
X3NlcmlhbF90IGFkZF9rZXkoY29uc3QgY2hhciAqdHlwZSwgY29uc3QgY2hhcg0KPiA+ID4gPiA+
ICpkZXNjcmlwdGlvbiwNCj4gPiA+ID4gPiArICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnN0
IHZvaWQgKnBheWxvYWQsIHNpemVfdCBwbGVuLA0KPiA+ID4gPiA+ICsgICAgICAgICAgICAgICAg
ICAgICAgICAga2V5X3NlcmlhbF90IGtleXJpbmcpOw0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiAr
UmV2b2tpbmcgTUtUTUUgS2V5cyB2aWEgY29tbWFuZCBsaW5lIG9yIHN5c3RlbSBjYWxsOjoNCj4g
PiA+ID4gPiArICAga2V5Y3RsIHJldm9rZSA8a2V5Pg0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiAr
ICAgbG9uZyBrZXljdGwoS0VZQ1RMX1JFVk9LRSwga2V5X3NlcmlhbF90IGtleSk7DQo+ID4gPiA+
ID4gKw0KPiA+ID4gPiA+ICtPcHRpb25zIEZpZWxkIERlZmluaXRpb246DQo+ID4gPiA+ID4gKyAg
ICB1c2Vya2V5PSAgICAgIEFTQ0lJIEhFWCB2YWx1ZSBlbmNyeXB0aW9uIGtleS4gRGVmYXVsdHMN
Cj4gPiA+ID4gPiB0byBhIENQVQ0KPiA+ID4gPiA+ICsJCSAgZ2VuZXJhdGVkIGtleSBpZiBhIHVz
ZXJrZXkgaXMgbm90IGRlZmluZWQNCj4gPiA+ID4gPiBoZXJlLg0KPiA+ID4gPiA+ICsNCj4gPiA+
ID4gPiArICAgIGFsZ29yaXRobT0gICAgRW5jcnlwdGlvbiBhbGdvcml0aG0gbmFtZSBhcyBhIHN0
cmluZy4NCj4gPiA+ID4gPiArCQkgIFZhbGlkIGFsZ29yaXRobTogImFlcy14dHMtMTI4Ig0KPiA+
ID4gPiA+ICsNCj4gPiA+ID4gPiArICAgIHR3ZWFrPSAgICAgICAgQVNDSUkgSEVYIHZhbHVlIHR3
ZWFrIGtleS4gVHdlYWsga2V5IHdpbGwNCj4gPiA+ID4gPiBiZSBhZGRlZCB0byB0aGUNCj4gPiA+
ID4gPiArICAgICAgICAgICAgICAgICAgdXNlcmtleS4uLiAgKG5lZWQgdG8gYmUgY2xlYXIgaGVy
ZSB0aGF0DQo+ID4gPiA+ID4gdGhpcyBpcyBiZWluZyBzZW50DQo+ID4gPiA+ID4gKyAgICAgICAg
ICAgICAgICAgIHRvIHRoZSBoYXJkd2FyZSAtIGtlcm5lbCBub3QgbWVzc2luZyB3IGl0KQ0KPiA+
ID4gPiA+ICsNCj4gPiA+ID4gPiArICAgIGVudHJvcHk9ICAgICAgYXNjaWkgaGV4IHZhbHVlIGVu
dHJvcHkuDQo+ID4gPiA+ID4gKyAgICAgICAgICAgICAgICAgIFRoaXMgZW50cm9weSB3aWxsIGJl
IHVzZWQgdG8gZ2VuZXJhdGVkIHRoZQ0KPiA+ID4gPiA+IENQVSBrZXkgYW5kDQo+ID4gPiA+ID4g
KwkJICB0aGUgdHdlYWsga2V5IHdoZW4gQ1BVIGdlbmVyYXRlZCBrZXkgaXMNCj4gPiA+ID4gPiBy
ZXF1ZXN0ZWQuDQo+ID4gPiA+ID4gKw0KPiA+ID4gPiA+ICtBbGdvcml0aG0gRGVwZW5kZW5jaWVz
Og0KPiA+ID4gPiA+ICsgICAgQUVTLVhUUyAxMjggaXMgdGhlIG9ubHkgc3VwcG9ydGVkIGFsZ29y
aXRobS4NCj4gPiA+ID4gPiArICAgIFRoZXJlIGFyZSBvbmx5IDIgd2F5cyB0aGF0IEFFUy1YVFMg
MTI4IG1heSBiZSB1c2VkOg0KPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiArICAgIDEpIFVzZXIgc3Bl
Y2lmaWVkIGVuY3J5cHRpb24ga2V5DQo+ID4gPiA+ID4gKwktIFRoZSB1c2VyIHNwZWNpZmllZCBl
bmNyeXB0aW9uIGtleSBtdXN0IGJlIGV4YWN0bHkNCj4gPiA+ID4gPiArCSAgMTYgQVNDSUkgSGV4
IGJ5dGVzICgxMjggYml0cykuDQo+ID4gPiA+ID4gKwktIEEgdHdlYWsga2V5IG11c3QgYmUgc3Bl
Y2lmaWVkIGFuZCBpdCBtdXN0IGJlDQo+ID4gPiA+ID4gZXhhY3RseQ0KPiA+ID4gPiA+ICsJICAx
NiBBU0NJSSBIZXggYnl0ZXMgKDEyOCBiaXRzKS4NCj4gPiA+ID4gPiArCS0gTm8gZW50cm9weSBm
aWVsZCBpcyBhY2NlcHRlZC4NCj4gPiA+ID4gPiArDQo+ID4gPiA+ID4gKyAgICAyKSBDUFUgZ2Vu
ZXJhdGVkIGVuY3J5cHRpb24ga2V5DQo+ID4gPiA+ID4gKwktIFdoZW4gbm8gdXNlciBzcGVjaWZp
ZWQgZW5jcnlwdGlvbiBrZXkgaXMgcHJvdmlkZWQsDQo+ID4gPiA+ID4gdGhlDQo+ID4gPiA+ID4g
KwkgIGRlZmF1bHQgZW5jcnlwdGlvbiBrZXkgd2lsbCBiZSBDUFUgZ2VuZXJhdGVkLg0KPiA+ID4g
PiA+ICsJLSBVc2VyIG11c3Qgc3BlY2lmeSAxNiBBU0NJSSBIZXggYnl0ZXMgb2YgZW50cm9weS4N
Cj4gPiA+ID4gPiBUaGlzDQo+ID4gPiA+ID4gKwkgIGVudHJvcHkgd2lsbCBiZSB1c2VkIGJ5IHRo
ZSBDUFUgdG8gZ2VuZXJhdGUgYm90aA0KPiA+ID4gPiA+IHRoZQ0KPiA+ID4gPiA+ICsJICBlbmNy
eXB0aW9uIGtleSBhbmQgdGhlIHR3ZWFrIGtleS4NCj4gPiA+ID4gPiArCS0gTm8gZW50cm9weSBm
aWVsZCBpcyBhY2NlcHRlZC4NCj4gPiA+IA0KPiA+ID4gICAgICAgICAgICAgIF5eXl5eXl4gc2hv
dWxkIGJlIHR3ZWFrDQo+ID4gPiANCj4gPiA+ID4gDQo+ID4gPiA+IFRoaXMgaXMgbm90IHRydWUu
IFRoZSBzcGVjIHNheXMgaW4gQ1BVIGdlbmVyYXRlZCByYW5kb20gbW9kZSwNCj4gPiA+ID4gYm90
aCAna2V5JyBhbmQNCj4gPiA+IA0KPiA+ID4gJ3R3ZWFrJyBwYXJ0IGFyZSB1c2VkIHRvIGdlbmVy
YXRlIHRoZSBmaW5hbCBrZXkgYW5kIHR3ZWFrDQo+ID4gPiByZXNwZWN0aXZlbHkuDQo+ID4gPiA+
IA0KPiA+ID4gPiBBY3R1YWxseSwgc2ltcGxlICdYT1InIGlzIHVzZWQgdG8gZ2VuZXJhdGUgdGhl
IGZpbmFsIGtleToNCj4gPiA+ID4gDQo+ID4gPiA+IGNhc2UgS0VZSURfU0VUX0tFWV9SQU5ET006
DQo+ID4gPiA+IAkuLi4uLi4NCj4gPiA+ID4gCSgqIE1peCB1c2VyIHN1cHBsaWVkIGVudHJvcHkg
dG8gdGhlIGRhdGEga2V5IGFuZCB0d2Vhaw0KPiA+ID4gPiBrZXkgKikNCj4gPiA+ID4gCVRNUF9S
TkRfREFUQV9LRVkgPSBUTVBfUk5EX0tFWSBYT1INCj4gPiA+ID4gCQlUTVBfS0VZX1BST0dSQU1f
U1RSVUNULktFWV9GSUVMRF8xLkJZVEVTWzE1OjBdOw0KPiA+ID4gPiAJVE1QX1JORF9UV0VBS19L
RVkgPSBUTVBfUk5EX1RXRUFLX0tFWSBYT1INCj4gPiA+ID4gCQlUTVBfS0VZX1BST0dSQU1fU1RS
VUNULktFWV9GSUVMRF8yLkJZVEVTWzE1OjBdOw0KPiA+ID4gPiANCj4gPiA+ID4gU28gSSB0aGlu
ayB3ZSBjYW4gZWl0aGVyIGp1c3QgcmVtb3ZlICdlbnRyb3B5JyBwYXJhbWV0ZXIsIHNpbmNlDQo+
ID4gPiA+IHdlIGNhbiB1c2UNCj4gPiA+IA0KPiA+ID4gYm90aCAndXNlcmtleScgYW5kICd0d2Vh
aycgZXZlbiBmb3IgcmFuZG9tIGtleSBtb2RlLg0KPiA+ID4gPiANCj4gPiA+ID4gSW4gZmFjdCwg
d2hpY2ggbWlnaHQgYmUgYmV0dGVyIElNSE8sIHdlIGNhbiBzaW1wbHkgZGlzYWxsb3cgb3INCj4g
PiA+ID4gaWdub3JlDQo+ID4gPiANCj4gPiA+ICd1c2Vya2V5JyBhbmQgJ3R3ZWFrJyBwYXJ0IGZv
ciByYW5kb20ga2V5IG1vZGUsIHNpbmNlIGlmIHdlIGFsbG93DQo+ID4gPiB1c2VyIHRvIHNwZWNp
ZnkNCj4gPiA+IGJvdGggZW50cm9waWVzLCBhbmQgaWYgdXNlciBwYXNzZXMgdmFsdWUgd2l0aCBh
bGwgMSwgd2UgYXJlDQo+ID4gPiBlZmZlY3RpdmVseSBtYWtpbmcgdGhlDQo+ID4gPiBrZXkgYW5k
IHR3ZWFrIHRvIGJlIGFsbCAxLCB3aGljaCBpcyBub3QgcmFuZG9tIGFueW1vcmUuDQo+ID4gPiA+
IA0KPiA+ID4gPiBJbnN0ZWFkLCBrZXJuZWwgY2FuIGdlbmVyYXRlIHJhbmRvbSBmb3IgYm90aCBl
bnRyb3BpZXMsIG9yIHdlDQo+ID4gPiA+IGNhbiBzaW1wbHkgdXNlcw0KPiA+ID4gDQo+ID4gPiAw
LCBpZ25vcmluZyB1c2VyIGlucHV0Lg0KPiA+ID4gDQo+ID4gPiBLYWksDQo+ID4gPiBJIHRoaW5r
IG15IHR5cG8gYWJvdmUsIHRocmV3IHlvdSBvZmYuIFdlIGhhdmUgdGhlIHNhbWUNCj4gPiA+IHVu
ZGVyc3RhbmRpbmcgb2YgdGhlDQo+ID4gPiBrZXkgZmllbGRzLg0KPiA+ID4gDQo+ID4gPiBJcyB0
aGlzIHRoZSBzdHJ1Y3R1cmUgeW91IGFyZSBzdWdnZXN0aW5nPw0KPiA+ID4gDQo+ID4gPiAJT3B0
aW9ucw0KPiA+ID4gDQo+ID4gPiAJa2V5X3R5cGU9CSJ1c2VyIiBvciAiQ1BVIg0KPiA+ID4gDQo+
ID4gPiAJa2V5PQkJSWYga2V5X3R5cGUgPT0gdXNlcg0KPiA+ID4gCQkJCWtleT0gaXMgdGhlIGRh
dGEga2V5DQo+ID4gPiAJCQlJZiBrZXlfdHlwZSA9PSBDUFUNCj4gPiA+IAkJCQlrZXk9IGlzIG5v
dCByZXF1aXJlZA0KPiA+ID4gCQkJCWlmIGtleT0gaXMgcHJlc2VudA0KPiA+ID4gCQkJCQlpdCBp
cyBlbnRyb3B5IHRvIGJlIG1peGVkIHdpdGgNCj4gPiA+IAkJCQkJQ1BVIGdlbmVyYXRlZCBkYXRh
IGtleQ0KPiA+ID4gDQo+ID4gPiAJdHdlYWs9CQlJZiBrZXlfdHlwZSA9PSB1c2VyDQo+ID4gPiAJ
CQkJdHdlYWs9IGlzIHRoZSB0d2VhayBrZXkNCj4gPiA+IAkJCUlmIGtleV90eXBlID09IENQVQ0K
PiA+ID4gCQkJCXR3ZWFrPSBpcyBub3QgcmVxdWlyZWQNCj4gPiA+IAkJCQlpZiB0d2Vhaz0gaXMg
cHJlc2VudA0KPiA+ID4gCQkJCQlpdCBpcyBlbnRyb3B5IHRvIGJlIG1peGVkIHdpdGgNCj4gPiA+
IAkJCQkJQ1BVIGdlbmVyYXRlZCB0d2VhayBrZXkNCj4gPiANCj4gPiBFeGFjdGx5Lg0KPiA+IA0K
PiA+IEFsdGhvdWdoIEkgYW0gbm90IHN1cmUgd2hldGhlciB3ZSBzaG91bGQgc3VwcG9ydCBvdGhl
ciAyIG1vZGVzOg0KPiA+IENsZWFyIGtleSAgYW5kICBubyBlbmNyeXB0aW9uOw0KPiANCj4gQSBo
YXJkd2FyZSBrZXkgZG9lcyBnZXQgQ0xFQVInZWQgd2hlbiB0aGUgdXNlcnNwYWNlIGtleSBpcyBy
ZXZva2VkLg0KPiBJIGRvbid0IHRoaW5rIHdlIGlkZW50aWZpZWQgYW55IG90aGVyIHVzZXIgZGly
ZWN0ZWQgbmVlZCB0byBjbGVhciBhDQo+IGtleS4NCj4gDQo+IFRoZSBubyBlbmNyeXB0aW9uIG9w
dGlvbiBpcyBjdXJyZW50bHkgY29uc2lkZXJlZCBub3QgYSByZXF1aXJlbWVudC4NCj4gVGhhdCBt
ZWFucywgYWx0aG91Z2ggeW91IHNlZSBpdCBpbiB0aGUgSW50ZWwgSFcgU3BlYywgd2UgZG9uJ3Qg
aGF2ZQ0KPiB1c2UgY2FzZSB0aGF0IGlzIGRyaXZpbmcgdXMgdG8gaW1wbGVtZW50IGl0Lg0KPiAN
Cj4gRm9yIG90aGVyJ3MgaW5mbyAtIG5vIGVuY3J5cHRpb24gd291bGQgYmUgYW4gb3B0aW9uIHdo
ZXJlIHRoZSBrZXkNCj4gdGVsbHMgdGhlIGhhcmR3YXJlIG5vdCB0byBkbyBhbnkgZW5jcnlwdGlv
biBhdCBhbGwgb24gdGhpcyBwaWVjZSBvZg0KPiBtZW1vcnkuDQo+IEFsbCBvZiBtZW1vcnkgbm90
IGVuY3J5cHRlZCB3aXRoIHRoZXNlIE1LVE1FIGtleXMsIGlzIGJ5IGRlZmF1bHQsDQo+IGVuY3J5
cHRlZA0KPiB3aXRoIHRoZSBzeXN0ZW0gbGV2ZWwgVE1FLCBUb3RhbCBNZW1vcnkgRW5jcnlwdGlv
biBhbGdvcml0aG0uIChPSyAtDQo+IG5vdA0KPiByZWFsbHkgKmFsbCosIHRoZXJlIGlzIGFsc28g
YSBCSU9TIHNldHRhYmxlIGV4Y2x1c2lvbiB6b25lIGZvciBUTUUpDQoNCkFncmVlZC4gTGV0J3Mg
Zm9jdXMgb24gdXNlciBtb2RlIGFuZCBDUFUgbW9kZSBmb3Igbm93Lg0KDQpUaGFua3MsDQotS2Fp
DQo+IA0KPiA+IA0KPiA+IFRoYW5rcywNCj4gPiAtS2FpDQo+ID4gPiANCj4gPiA+IA0KPiA+ID4g
QWxpc29uDQo+ID4gPiA+IA0KPiA+ID4gPiBUaGFua3MsDQo+ID4gPiA+IC1LYWkNCj4gPiA+IA0K
PiA+ID4gLi4uLi4uLi5zbmlwLi4uLi4uLi4uLi4

[RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

On Mon, 2018-09-10 at 17:45 -0700, Alison Schofield wrote:
> On Mon, Sep 10, 2018 at 05:33:33PM -0700, Huang, Kai wrote:
> > > -----Original Message-----
> > > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.org]
> > > On
> > > Behalf Of Alison Schofield
> > > Sent: Tuesday, September 11, 2018 12:13 PM
> > > To: Huang, Kai <kai.huang@intel.com>
> > > Cc: dhowells at redhat.com; tglx at linutronix.de; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel
> > > .com>;
> > > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.ker
> > > nel.org;
> > > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zyto
> > > r.com;
> > > x86 at kernel.org; linux-mm at kvack.org
> > > Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total
> > > Memory
> > > Encryption API
> > > 
> > > On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> > > > 
> > > > > -----Original Message-----
> > > > > From: owner-linux-mm at kvack.org [mailto:owner-linux-mm at kvack.o
> > > > > rg] On
> > > > > Behalf Of Alison Schofield
> > > > > Sent: Saturday, September 8, 2018 10:34 AM
> > > > > To: dhowells at redhat.com; tglx at linutronix.de
> > > > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.
> > > > > com>;
> > > > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris at namei.o
> > > > > rg;
> > > > > keyrings at vger.kernel.org; linux-security-module at vger.kernel.o
> > > > > rg;
> > > > > mingo at redhat.com; hpa at zytor.com; x86 at kernel.org; linux-
> > > 
> > > mm at kvack.org
> > > > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total
> > > > > Memory
> > > > > Encryption API
> > > > > 
> > > > > Document the API's used for MKTME on Intel platforms.
> > > > > MKTME: Multi-KEY Total Memory Encryption
> > > > > 
> > > > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > > > ---
> > > > >  Documentation/x86/mktme-keys.txt | 153
> > > > > +++++++++++++++++++++++++++++++++++++++
> > > > >  1 file changed, 153 insertions(+)
> > > > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > > > > 
> > > > > diff --git a/Documentation/x86/mktme-keys.txt
> > > > > b/Documentation/x86/mktme- keys.txt new file mode 100644
> > > > > index
> > > > > 000000000000..2dea7acd2a17
> > > > > --- /dev/null
> > > > > +++ b/Documentation/x86/mktme-keys.txt
> > > > > @@ -0,0 +1,153 @@
> > > > > +MKTME (Multi-Key Total Memory Encryption) is a technology
> > > > > that
> > > > > +allows memory encryption on Intel platforms. Whereas TME
> > > > > (Total
> > > > > +Memory
> > > > > +Encryption) allows encryption of the entire system memory
> > > > > using a
> > > > > +single key, MKTME allows multiple encryption domains, each
> > > > > having
> > > > > +their own key. The main use case for the feature is virtual
> > > > > machine
> > > > > +isolation. The API's introduced here are intended to offer
> > > > > +flexibility to work in a
> > > > > wide range of uses.
> > > > > +
> > > > > +The externally available Intel Architecture Spec:
> > > > > +https://software.intel.com/sites/default/files/managed/a5/16
> > > > > /Multi-
> > > > > +Key-
> > > > > +Total-Memory-Encryption-Spec.pdf
> > > > > +
> > > > > +============================  API Overview
> > > > > +============================
> > > > > +
> > > > > +There are 2 MKTME specific API's that enable userspace to
> > > > > create
> > > > > +and use the memory encryption keys:
> > > > > +
> > > > > +1) Kernel Key Service: MKTME Type
> > > > > +
> > > > > +   MKTME is a new key type added to the existing Kernel Key
> > > > > Services
> > > > > +   to support the memory encryption keys. The MKTME service
> > > > > manages
> > > > > +   the addition and removal of MKTME keys. It maps userspace
> > > > > keys
> > > > > +   to hardware keyids and programs the hardware with user
> > > > > requested
> > > > > +   encryption parameters.
> > > > > +
> > > > > +   o An understanding of the Kernel Key Service is required
> > > > > in order
> > > > > +     to use the MKTME key type as it is a subset of that
> > > > > service.
> > > > > +
> > > > > +   o MKTME keys are a limited resource. There is a single
> > > > > pool of
> > > > > +     MKTME keys for a system and that pool can be from 3 to
> > > > > 63 keys.
> > > > 
> > > > Why 3 to 63 keys? Architecturally we are able to support up to
> > > > 15-bit keyID,
> > > 
> > > although in the first generation server we only support 6-bit
> > > keyID, which is 63
> > > key/keyIDs (excluding keyID 0, which is TME's keyID).
> > > 
> > > My understanding is that low level SKU's could have as few as 3
> > > bits available to
> > > hold the keyid, and that the max is 6 bits, hence 64.
> > > I probably don't need to be stating that level of detail here,
> > > but rather just
> > > iterate the important point that the resource is limited!
> > > 
> > > > 
> > > > > +     With that in mind, userspace may take advantage of the
> > > > > kernel
> > > > > +     key services sharing and permissions model for
> > > > > userspace keys.
> > > > > +     One key can be shared as long as each user has the
> > > > > permission
> > > > > +     of "KEY_NEED_VIEW" to use it.
> > > > > +
> > > > > +   o MKTME key type uses capabilities to restrict the
> > > > > allocation
> > > > > +     of keys. It only requires CAP_SYS_RESOURCE, but will
> > > > > accept
> > > > > +     the broader capability of CAP_SYS_ADMIN.  See
> > > > > capabilities(7).
> > > > > +
> > > > > +   o The MKTME key service blocks kernel key service
> > > > > commands that
> > > > > +     could lead to reprogramming of in use keys, or loss of
> > > > > keys from
> > > > > +     the pool. This means MKTME does not allow a key to be
> > > > > invalidated,
> > > > > +     unlinked, or timed out. These operations are blocked by
> > > > > MKTME as
> > > > > +     it creates all keys with the internal flag
> > > > > KEY_FLAG_KEEP.
> > > > > +
> > > > > +   o MKTME does not support the keyctl option of UPDATE.
> > > > > Userspace
> > > > > +     may change the programming of a key by revoking it and
> > > > > adding
> > > > > +     a new key with the updated encryption options (or vice-
> > > > > versa).
> > > > > +
> > > > > +2) System Call: encrypt_mprotect()
> > > > > +
> > > > > +   MKTME encryption is requested by calling
> > > > > encrypt_mprotect(). The
> > > > > +   caller passes the serial number to a previously allocated
> > > > > and
> > > > > +   programmed encryption key. That handle was created with
> > > > > the MKTME
> > > > > +   Key Service.
> > > > > +
> > > > > +   o The caller must have KEY_NEED_VIEW permission on the
> > > > > key
> > > > > +
> > > > > +   o The range of memory that is to be protected must be
> > > > > mapped as
> > > > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect()
> > > > > request
> > > > > +     fails with EINVAL.
> > > > > +
> > > > > +   o As an extension to the existing mprotect() system call,
> > > > > +     encrypt_mprotect() supports the legacy mprotect
> > > > > behavior plus
> > > > > +     the enabling of memory encryption. That means that in
> > > > > addition
> > > > > +     to encrypting the memory, the protection flags will be
> > > > > updated
> > > > > +     as requested in the call.
> > > > > +
> > > > > +   o Additional mprotect() calls to memory already protected
> > > > > with
> > > > > +     MKTME will not alter the MKTME status.
> > > > 
> > > > I think it's better to separate encrypt_mprotect() into another
> > > > doc so both
> > > 
> > > parts can be reviewed easier.
> > > 
> > > I can do that.
> > > Also, I do know I need man page for that too.
> > > > 
> > > > > +
> > > > > +======================  Usage: MKTME Key Service
> > > > > +======================
> > > > > +
> > > > > +MKTME is enabled on supported Intel platforms by selecting
> > > > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > > > +
> > > > > +Allocating MKTME Keys via command line or system call:
> > > > > +    keyctl add mktme name "[options]" ring
> > > > > +
> > > > > +    key_serial_t add_key(const char *type, const char
> > > > > *description,
> > > > > +                         const void *payload, size_t plen,
> > > > > +                         key_serial_t keyring);
> > > > > +
> > > > > +Revoking MKTME Keys via command line or system call::
> > > > > +   keyctl revoke <key>
> > > > > +
> > > > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > > > +
> > > > > +Options Field Definition:
> > > > > +    userkey=      ASCII HEX value encryption key. Defaults
> > > > > to a CPU
> > > > > +		  generated key if a userkey is not defined
> > > > > here.
> > > > > +
> > > > > +    algorithm=    Encryption algorithm name as a string.
> > > > > +		  Valid algorithm: "aes-xts-128"
> > > > > +
> > > > > +    tweak=        ASCII HEX value tweak key. Tweak key will
> > > > > be added to the
> > > > > +                  userkey...  (need to be clear here that
> > > > > this is being sent
> > > > > +                  to the hardware - kernel not messing w it)
> > > > > +
> > > > > +    entropy=      ascii hex value entropy.
> > > > > +                  This entropy will be used to generated the
> > > > > CPU key and
> > > > > +		  the tweak key when CPU generated key is
> > > > > requested.
> > > > > +
> > > > > +Algorithm Dependencies:
> > > > > +    AES-XTS 128 is the only supported algorithm.
> > > > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > > > +
> > > > > +    1) User specified encryption key
> > > > > +	- The user specified encryption key must be exactly
> > > > > +	  16 ASCII Hex bytes (128 bits).
> > > > > +	- A tweak key must be specified and it must be
> > > > > exactly
> > > > > +	  16 ASCII Hex bytes (128 bits).
> > > > > +	- No entropy field is accepted.
> > > > > +
> > > > > +    2) CPU generated encryption key
> > > > > +	- When no user specified encryption key is provided,
> > > > > the
> > > > > +	  default encryption key will be CPU generated.
> > > > > +	- User must specify 16 ASCII Hex bytes of entropy.
> > > > > This
> > > > > +	  entropy will be used by the CPU to generate both
> > > > > the
> > > > > +	  encryption key and the tweak key.
> > > > > +	- No entropy field is accepted.
> > > 
> > >              ^^^^^^^ should be tweak
> > > 
> > > > 
> > > > This is not true. The spec says in CPU generated random mode,
> > > > both 'key' and
> > > 
> > > 'tweak' part are used to generate the final key and tweak
> > > respectively.
> > > > 
> > > > Actually, simple 'XOR' is used to generate the final key:
> > > > 
> > > > case KEYID_SET_KEY_RANDOM:
> > > > 	......
> > > > 	(* Mix user supplied entropy to the data key and tweak
> > > > key *)
> > > > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > > > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> > > > 
> > > > So I think we can either just remove 'entropy' parameter, since
> > > > we can use
> > > 
> > > both 'userkey' and 'tweak' even for random key mode.
> > > > 
> > > > In fact, which might be better IMHO, we can simply disallow or
> > > > ignore
> > > 
> > > 'userkey' and 'tweak' part for random key mode, since if we allow
> > > user to specify
> > > both entropies, and if user passes value with all 1, we are
> > > effectively making the
> > > key and tweak to be all 1, which is not random anymore.
> > > > 
> > > > Instead, kernel can generate random for both entropies, or we
> > > > can simply uses
> > > 
> > > 0, ignoring user input.
> > > 
> > > Kai,
> > > I think my typo above, threw you off. We have the same
> > > understanding of the
> > > key fields.
> > > 
> > > Is this the structure you are suggesting?
> > > 
> > > 	Options
> > > 
> > > 	key_type=	"user" or "CPU"
> > > 
> > > 	key=		If key_type == user
> > > 				key= is the data key
> > > 			If key_type == CPU
> > > 				key= is not required
> > > 				if key= is present
> > > 					it is entropy to be mixed with
> > > 					CPU generated data key
> > > 
> > > 	tweak=		If key_type == user
> > > 				tweak= is the tweak key
> > > 			If key_type == CPU
> > > 				tweak= is not required
> > > 				if tweak= is present
> > > 					it is entropy to be mixed with
> > > 					CPU generated tweak key
> > 
> > Exactly.
> > 
> > Although I am not sure whether we should support other 2 modes:
> > Clear key  and  no encryption;
> 
> A hardware key does get CLEAR'ed when the userspace key is revoked.
> I don't think we identified any other user directed need to clear a
> key.
> 
> The no encryption option is currently considered not a requirement.
> That means, although you see it in the Intel HW Spec, we don't have
> use case that is driving us to implement it.
> 
> For other's info - no encryption would be an option where the key
> tells the hardware not to do any encryption at all on this piece of
> memory.
> All of memory not encrypted with these MKTME keys, is by default,
> encrypted
> with the system level TME, Total Memory Encryption algorithm. (OK -
> not
> really *all*, there is also a BIOS settable exclusion zone for TME)

Agreed. Let's focus on user mode and CPU mode for now.

Thanks,
-Kai
> 
> > 
> > Thanks,
> > -Kai
> > > 
> > > 
> > > Alison
> > > > 
> > > > Thanks,
> > > > -Kai
> > > 
> > > ........snip...........

Re: [RFC 01/12] docs/x86: Document the Multi-Key Total Memory Encryption API

[Huang, Kai]

On Mon, 2018-09-10 at 17:45 -0700, Alison Schofield wrote:
> On Mon, Sep 10, 2018 at 05:33:33PM -0700, Huang, Kai wrote:
> > > -----Original Message-----
> > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org]
> > > On
> > > Behalf Of Alison Schofield
> > > Sent: Tuesday, September 11, 2018 12:13 PM
> > > To: Huang, Kai <kai.huang@intel.com>
> > > Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel
> > > .com>;
> > > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.ker
> > > nel.org;
> > > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zyto
> > > r.com;
> > > x86@kernel.org; linux-mm@kvack.org
> > > Subject: Re: [RFC 01/12] docs/x86: Document the Multi-Key Total
> > > Memory
> > > Encryption API
> > > 
> > > On Sun, Sep 09, 2018 at 06:28:28PM -0700, Huang, Kai wrote:
> > > > 
> > > > > -----Original Message-----
> > > > > From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.o
> > > > > rg] On
> > > > > Behalf Of Alison Schofield
> > > > > Sent: Saturday, September 8, 2018 10:34 AM
> > > > > To: dhowells@redhat.com; tglx@linutronix.de
> > > > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > > > <jun.nakajima@intel.com>; Shutemov, Kirill
> > > > > <kirill.shutemov@intel.com>; Hansen, Dave <dave.hansen@intel.
> > > > > com>;
> > > > > Sakkinen, Jarkko <jarkko.sakkinen@intel.com>; jmorris@namei.o
> > > > > rg;
> > > > > keyrings@vger.kernel.org; linux-security-module@vger.kernel.o
> > > > > rg;
> > > > > mingo@redhat.com; hpa@zytor.com; x86@kernel.org; linux-
> > > 
> > > mm@kvack.org
> > > > > Subject: [RFC 01/12] docs/x86: Document the Multi-Key Total
> > > > > Memory
> > > > > Encryption API
> > > > > 
> > > > > Document the API's used for MKTME on Intel platforms.
> > > > > MKTME: Multi-KEY Total Memory Encryption
> > > > > 
> > > > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > > > > ---
> > > > >  Documentation/x86/mktme-keys.txt | 153
> > > > > +++++++++++++++++++++++++++++++++++++++
> > > > >  1 file changed, 153 insertions(+)
> > > > >  create mode 100644 Documentation/x86/mktme-keys.txt
> > > > > 
> > > > > diff --git a/Documentation/x86/mktme-keys.txt
> > > > > b/Documentation/x86/mktme- keys.txt new file mode 100644
> > > > > index
> > > > > 000000000000..2dea7acd2a17
> > > > > --- /dev/null
> > > > > +++ b/Documentation/x86/mktme-keys.txt
> > > > > @@ -0,0 +1,153 @@
> > > > > +MKTME (Multi-Key Total Memory Encryption) is a technology
> > > > > that
> > > > > +allows memory encryption on Intel platforms. Whereas TME
> > > > > (Total
> > > > > +Memory
> > > > > +Encryption) allows encryption of the entire system memory
> > > > > using a
> > > > > +single key, MKTME allows multiple encryption domains, each
> > > > > having
> > > > > +their own key. The main use case for the feature is virtual
> > > > > machine
> > > > > +isolation. The API's introduced here are intended to offer
> > > > > +flexibility to work in a
> > > > > wide range of uses.
> > > > > +
> > > > > +The externally available Intel Architecture Spec:
> > > > > +https://software.intel.com/sites/default/files/managed/a5/16
> > > > > /Multi-
> > > > > +Key-
> > > > > +Total-Memory-Encryption-Spec.pdf
> > > > > +
> > > > > +============================  API Overview
> > > > > +============================
> > > > > +
> > > > > +There are 2 MKTME specific API's that enable userspace to
> > > > > create
> > > > > +and use the memory encryption keys:
> > > > > +
> > > > > +1) Kernel Key Service: MKTME Type
> > > > > +
> > > > > +   MKTME is a new key type added to the existing Kernel Key
> > > > > Services
> > > > > +   to support the memory encryption keys. The MKTME service
> > > > > manages
> > > > > +   the addition and removal of MKTME keys. It maps userspace
> > > > > keys
> > > > > +   to hardware keyids and programs the hardware with user
> > > > > requested
> > > > > +   encryption parameters.
> > > > > +
> > > > > +   o An understanding of the Kernel Key Service is required
> > > > > in order
> > > > > +     to use the MKTME key type as it is a subset of that
> > > > > service.
> > > > > +
> > > > > +   o MKTME keys are a limited resource. There is a single
> > > > > pool of
> > > > > +     MKTME keys for a system and that pool can be from 3 to
> > > > > 63 keys.
> > > > 
> > > > Why 3 to 63 keys? Architecturally we are able to support up to
> > > > 15-bit keyID,
> > > 
> > > although in the first generation server we only support 6-bit
> > > keyID, which is 63
> > > key/keyIDs (excluding keyID 0, which is TME's keyID).
> > > 
> > > My understanding is that low level SKU's could have as few as 3
> > > bits available to
> > > hold the keyid, and that the max is 6 bits, hence 64.
> > > I probably don't need to be stating that level of detail here,
> > > but rather just
> > > iterate the important point that the resource is limited!
> > > 
> > > > 
> > > > > +     With that in mind, userspace may take advantage of the
> > > > > kernel
> > > > > +     key services sharing and permissions model for
> > > > > userspace keys.
> > > > > +     One key can be shared as long as each user has the
> > > > > permission
> > > > > +     of "KEY_NEED_VIEW" to use it.
> > > > > +
> > > > > +   o MKTME key type uses capabilities to restrict the
> > > > > allocation
> > > > > +     of keys. It only requires CAP_SYS_RESOURCE, but will
> > > > > accept
> > > > > +     the broader capability of CAP_SYS_ADMIN.  See
> > > > > capabilities(7).
> > > > > +
> > > > > +   o The MKTME key service blocks kernel key service
> > > > > commands that
> > > > > +     could lead to reprogramming of in use keys, or loss of
> > > > > keys from
> > > > > +     the pool. This means MKTME does not allow a key to be
> > > > > invalidated,
> > > > > +     unlinked, or timed out. These operations are blocked by
> > > > > MKTME as
> > > > > +     it creates all keys with the internal flag
> > > > > KEY_FLAG_KEEP.
> > > > > +
> > > > > +   o MKTME does not support the keyctl option of UPDATE.
> > > > > Userspace
> > > > > +     may change the programming of a key by revoking it and
> > > > > adding
> > > > > +     a new key with the updated encryption options (or vice-
> > > > > versa).
> > > > > +
> > > > > +2) System Call: encrypt_mprotect()
> > > > > +
> > > > > +   MKTME encryption is requested by calling
> > > > > encrypt_mprotect(). The
> > > > > +   caller passes the serial number to a previously allocated
> > > > > and
> > > > > +   programmed encryption key. That handle was created with
> > > > > the MKTME
> > > > > +   Key Service.
> > > > > +
> > > > > +   o The caller must have KEY_NEED_VIEW permission on the
> > > > > key
> > > > > +
> > > > > +   o The range of memory that is to be protected must be
> > > > > mapped as
> > > > > +     ANONYMOUS. If it is not, the entire encrypt_mprotect()
> > > > > request
> > > > > +     fails with EINVAL.
> > > > > +
> > > > > +   o As an extension to the existing mprotect() system call,
> > > > > +     encrypt_mprotect() supports the legacy mprotect
> > > > > behavior plus
> > > > > +     the enabling of memory encryption. That means that in
> > > > > addition
> > > > > +     to encrypting the memory, the protection flags will be
> > > > > updated
> > > > > +     as requested in the call.
> > > > > +
> > > > > +   o Additional mprotect() calls to memory already protected
> > > > > with
> > > > > +     MKTME will not alter the MKTME status.
> > > > 
> > > > I think it's better to separate encrypt_mprotect() into another
> > > > doc so both
> > > 
> > > parts can be reviewed easier.
> > > 
> > > I can do that.
> > > Also, I do know I need man page for that too.
> > > > 
> > > > > +
> > > > > +======================  Usage: MKTME Key Service
> > > > > +======================
> > > > > +
> > > > > +MKTME is enabled on supported Intel platforms by selecting
> > > > > +CONFIG_X86_INTEL_MKTME which selects CONFIG_MKTME_KEYS.
> > > > > +
> > > > > +Allocating MKTME Keys via command line or system call:
> > > > > +    keyctl add mktme name "[options]" ring
> > > > > +
> > > > > +    key_serial_t add_key(const char *type, const char
> > > > > *description,
> > > > > +                         const void *payload, size_t plen,
> > > > > +                         key_serial_t keyring);
> > > > > +
> > > > > +Revoking MKTME Keys via command line or system call::
> > > > > +   keyctl revoke <key>
> > > > > +
> > > > > +   long keyctl(KEYCTL_REVOKE, key_serial_t key);
> > > > > +
> > > > > +Options Field Definition:
> > > > > +    userkey=      ASCII HEX value encryption key. Defaults
> > > > > to a CPU
> > > > > +		  generated key if a userkey is not defined
> > > > > here.
> > > > > +
> > > > > +    algorithm=    Encryption algorithm name as a string.
> > > > > +		  Valid algorithm: "aes-xts-128"
> > > > > +
> > > > > +    tweak=        ASCII HEX value tweak key. Tweak key will
> > > > > be added to the
> > > > > +                  userkey...  (need to be clear here that
> > > > > this is being sent
> > > > > +                  to the hardware - kernel not messing w it)
> > > > > +
> > > > > +    entropy=      ascii hex value entropy.
> > > > > +                  This entropy will be used to generated the
> > > > > CPU key and
> > > > > +		  the tweak key when CPU generated key is
> > > > > requested.
> > > > > +
> > > > > +Algorithm Dependencies:
> > > > > +    AES-XTS 128 is the only supported algorithm.
> > > > > +    There are only 2 ways that AES-XTS 128 may be used:
> > > > > +
> > > > > +    1) User specified encryption key
> > > > > +	- The user specified encryption key must be exactly
> > > > > +	  16 ASCII Hex bytes (128 bits).
> > > > > +	- A tweak key must be specified and it must be
> > > > > exactly
> > > > > +	  16 ASCII Hex bytes (128 bits).
> > > > > +	- No entropy field is accepted.
> > > > > +
> > > > > +    2) CPU generated encryption key
> > > > > +	- When no user specified encryption key is provided,
> > > > > the
> > > > > +	  default encryption key will be CPU generated.
> > > > > +	- User must specify 16 ASCII Hex bytes of entropy.
> > > > > This
> > > > > +	  entropy will be used by the CPU to generate both
> > > > > the
> > > > > +	  encryption key and the tweak key.
> > > > > +	- No entropy field is accepted.
> > > 
> > >              ^^^^^^^ should be tweak
> > > 
> > > > 
> > > > This is not true. The spec says in CPU generated random mode,
> > > > both 'key' and
> > > 
> > > 'tweak' part are used to generate the final key and tweak
> > > respectively.
> > > > 
> > > > Actually, simple 'XOR' is used to generate the final key:
> > > > 
> > > > case KEYID_SET_KEY_RANDOM:
> > > > 	......
> > > > 	(* Mix user supplied entropy to the data key and tweak
> > > > key *)
> > > > 	TMP_RND_DATA_KEY = TMP_RND_KEY XOR
> > > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_1.BYTES[15:0];
> > > > 	TMP_RND_TWEAK_KEY = TMP_RND_TWEAK_KEY XOR
> > > > 		TMP_KEY_PROGRAM_STRUCT.KEY_FIELD_2.BYTES[15:0];
> > > > 
> > > > So I think we can either just remove 'entropy' parameter, since
> > > > we can use
> > > 
> > > both 'userkey' and 'tweak' even for random key mode.
> > > > 
> > > > In fact, which might be better IMHO, we can simply disallow or
> > > > ignore
> > > 
> > > 'userkey' and 'tweak' part for random key mode, since if we allow
> > > user to specify
> > > both entropies, and if user passes value with all 1, we are
> > > effectively making the
> > > key and tweak to be all 1, which is not random anymore.
> > > > 
> > > > Instead, kernel can generate random for both entropies, or we
> > > > can simply uses
> > > 
> > > 0, ignoring user input.
> > > 
> > > Kai,
> > > I think my typo above, threw you off. We have the same
> > > understanding of the
> > > key fields.
> > > 
> > > Is this the structure you are suggesting?
> > > 
> > > 	Options
> > > 
> > > 	key_type=	"user" or "CPU"
> > > 
> > > 	key=		If key_type == user
> > > 				key= is the data key
> > > 			If key_type == CPU
> > > 				key= is not required
> > > 				if key= is present
> > > 					it is entropy to be mixed with
> > > 					CPU generated data key
> > > 
> > > 	tweak=		If key_type == user
> > > 				tweak= is the tweak key
> > > 			If key_type == CPU
> > > 				tweak= is not required
> > > 				if tweak= is present
> > > 					it is entropy to be mixed with
> > > 					CPU generated tweak key
> > 
> > Exactly.
> > 
> > Although I am not sure whether we should support other 2 modes:
> > Clear key  and  no encryption;
> 
> A hardware key does get CLEAR'ed when the userspace key is revoked.
> I don't think we identified any other user directed need to clear a
> key.
> 
> The no encryption option is currently considered not a requirement.
> That means, although you see it in the Intel HW Spec, we don't have
> use case that is driving us to implement it.
> 
> For other's info - no encryption would be an option where the key
> tells the hardware not to do any encryption at all on this piece of
> memory.
> All of memory not encrypted with these MKTME keys, is by default,
> encrypted
> with the system level TME, Total Memory Encryption algorithm. (OK -
> not
> really *all*, there is also a BIOS settable exclusion zone for TME)

Agreed. Let's focus on user mode and CPU mode for now.

Thanks,
-Kai
> 
> > 
> > Thanks,
> > -Kai
> > > 
> > > 
> > > Alison
> > > > 
> > > > Thanks,
> > > > -Kai
> > > 
> > > ........snip...........

Re: [RFC 06/12] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:02:43PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> > Implement memory encryption with a new system call that is an
> > extension of the legacy mprotect() system call.
> > 
> > In encrypt_mprotect the caller must pass a handle to a previously
> > allocated and programmed encryption key. Validate the key and store
> > the keyid bits in the vm_page_prot for each VMA in the protection
> > range.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  fs/exec.c           |  4 ++--
> >  include/linux/key.h |  2 ++
> >  include/linux/mm.h  |  3 ++-
> >  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> > ---
> >  4 files changed, 67 insertions(+), 9 deletions(-)
> > 
> > diff --git a/fs/exec.c b/fs/exec.c
> > index a1a246062561..b681a413db9c 100644
> > --- a/fs/exec.c
> > +++ b/fs/exec.c
> > @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
> >  	vm_flags |= mm->def_flags;
> >  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
> >  
> > -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > -			vm_flags);
> > +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > vm_flags,
> > +			     -1);
> 
> Why you pass a magic number here when you went the trouble having
> a named constant?

The named constant isn't available over here in fs/exec.c.
The special case is documented further down ... (scroll down)
That caller has no knowledge of proctection or encryption keys.

> 
> >  	if (ret)
> >  		goto out_unlock;
> >  	BUG_ON(prev != vma);
> > diff --git a/include/linux/key.h b/include/linux/key.h
> > index e58ee10f6e58..fb8a7d5f6149 100644
> > --- a/include/linux/key.h
> > +++ b/include/linux/key.h
> > @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> > *key)
> >  
> >  extern void key_set_timeout(struct key *, unsigned);
> >  
> > +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> > +				 key_perm_t perm);
> >  /*
> >   * The permissions required on a key that we're looking up.
> >   */
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index ac85c0805761..0f9422c7841e 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> > vm_area_struct *vma, unsigned long
> >  			      int dirty_accountable, int prot_numa);
> >  extern int mprotect_fixup(struct vm_area_struct *vma,
> >  			  struct vm_area_struct **pprev, unsigned long start,
> > -			  unsigned long end, unsigned long newflags);
> > +			  unsigned long end, unsigned long newflags,
> > +			  int newkeyid);
> >  
> >  /*
> >   * doesn't attempt to fault and will return short.
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 56e64ef7931e..6c2e1106525c 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -28,14 +28,17 @@
> >  #include <linux/ksm.h>
> >  #include <linux/uaccess.h>
> >  #include <linux/mm_inline.h>
> > +#include <linux/key.h>
> >  #include <asm/pgtable.h>
> >  #include <asm/cacheflush.h>
> >  #include <asm/mmu_context.h>
> >  #include <asm/tlbflush.h>
> > +#include <asm/mktme.h>
> >  
> >  #include "internal.h"
> >  
> >  #define NO_PKEY  -1
> > +#define NO_KEYID -1
> 
> Should have only single named constant IMHO. This ambiguity
> is worse than some reasonable constant name for both cases.
> Maybe NO_KEYID would be adequate?

Yes, this could be NO_KEY for both. I was worried about readability,
but most of the usages compare it to a well-named variable, as in 
(pkey = NO_KEY) or (keyid = NO_KEY) so it seems to work. 
Will do!

> 
> >  
> >  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
> >  		unsigned long addr, unsigned long end, pgprot_t newprot,
> > @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> > *vma, unsigned long start,
> >  
> >  int
> >  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> > -	unsigned long start, unsigned long end, unsigned long newflags)
> > +	       unsigned long start, unsigned long end, unsigned long
> > newflags,
> > +	       int newkeyid)
> >  {
> >  	struct mm_struct *mm = vma->vm_mm;
> >  	unsigned long oldflags = vma->vm_flags;
> > @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	int error;
> >  	int dirty_accountable = 0;
> >  
> > +	/*
> > +	 * Flags match and Keyids match or we have NO_KEYID.
> > +	 * This _fixup is usually called from do_mprotect_ext() except
> > +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> > +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> > +	 */

Above is that special case explanation.

> > +	if (newflags = oldflags &&
> > +	    (newkeyid = vma_keyid(vma) || newkeyid = NO_KEYID)) {
> > +		*pprev = vma;
> > +		return 0;
> > +	}
> > +	/* Flags match and Keyid changes */
> >  	if (newflags = oldflags) {
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  		*pprev = vma;
> >  		return 0;
> >  	}
> > +	/* Flags and Keyids both change, continue. */
> >  
> >  	/*
> >  	 * If we make a private mapping writable we increase our commit;
> > @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	}
> >  
> >  success:
> > +	if (newkeyid != NO_KEYID)
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  	/*
> >  	 * vm_flags and vm_page_prot are protected by the mmap_sem
> >  	 * held in write mode.
> > @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  }
> >  
> >  /*
> > - * When pkey=NO_PKEY we get legacy mprotect behavior here.
> > + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> > + * for protection keys and memory encryption keys. These extensions are
> > + * mutually exclusive and the behavior is:
> > + *	(pkey=NO_PKEY && keyid=NO_KEYID) => legacy mprotect
> > + *	(pkey is valid)  => legacy mprotect plus protection key extensions
> > + *	(keyid is valid) => legacy mprotect plus encryption key extensions
> >   */
> 
> The header does not follow
> 
> https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

Jarkko,

I see the "how to" on kernel doc formatting, but I'm not getting the
when? When should it be done? I can imagine it for new code, like
the mktme_keys.c where new code is being added and the kernel docs
could give a complete narrative of the module.

Here, in mprotect.c, there are no kernel doc comments. I realize
sometime we just need to start somewhere ;), but I find it odd to
pluck out one function header to be extracted for kernel docs.

Alison

> 
> >  static int do_mprotect_ext(unsigned long start, size_t len,
> > -		unsigned long prot, int pkey)
> > +			   unsigned long prot, int pkey, int keyid)
> >  {
> >  	unsigned long nstart, end, tmp, reqprot;
> >  	struct vm_area_struct *vma, *prev;
> > @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  		tmp = vma->vm_end;
> >  		if (tmp > end)
> >  			tmp = end;
> > -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> > +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> > +				       keyid);
> >  		if (error)
> >  			goto out;
> >  		nstart = tmp;
> > @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot)
> >  {
> > -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> > +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
> >  }
> >  
> >  #ifdef CONFIG_ARCH_HAS_PKEYS
> > @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> > len,
> >  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot, int, pkey)
> >  {
> > -	return do_mprotect_ext(start, len, prot, pkey);
> > +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
> >  }
> >  
> >  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> > @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
> >  }
> >  
> >  #endif /* CONFIG_ARCH_HAS_PKEYS */
> > +
> > +#ifdef CONFIG_X86_INTEL_MKTME
> > +
> > +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> > +		unsigned long, prot, key_serial_t, serial)
> > +{
> > +	key_ref_t key_ref;
> > +	int ret, keyid;
> > +
> > +	/* TODO MKTME key service must be initialized */
> > +
> > +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> > +	if (IS_ERR(key_ref))
> > +		return PTR_ERR(key_ref);
> > +
> > +	mktme_map_lock();
> > +	keyid = mktme_map_keyid_from_serial(serial);
> > +	if (!keyid) {
> > +		mktme_map_unlock();
> > +		key_ref_put(key_ref);
> > +		return -EINVAL;
> > +	}
> > +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> > +	mktme_map_unlock();
> > +	key_ref_put(key_ref);
> > +	return ret;
> > +}
> > +
> > +#endif /* CONFIG_X86_INTEL_MKTME */
> 
> /Jarkko

[RFC 06/12] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:02:43PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> > Implement memory encryption with a new system call that is an
> > extension of the legacy mprotect() system call.
> > 
> > In encrypt_mprotect the caller must pass a handle to a previously
> > allocated and programmed encryption key. Validate the key and store
> > the keyid bits in the vm_page_prot for each VMA in the protection
> > range.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  fs/exec.c           |  4 ++--
> >  include/linux/key.h |  2 ++
> >  include/linux/mm.h  |  3 ++-
> >  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> > ---
> >  4 files changed, 67 insertions(+), 9 deletions(-)
> > 
> > diff --git a/fs/exec.c b/fs/exec.c
> > index a1a246062561..b681a413db9c 100644
> > --- a/fs/exec.c
> > +++ b/fs/exec.c
> > @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
> >  	vm_flags |= mm->def_flags;
> >  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
> >  
> > -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > -			vm_flags);
> > +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > vm_flags,
> > +			     -1);
> 
> Why you pass a magic number here when you went the trouble having
> a named constant?

The named constant isn't available over here in fs/exec.c.
The special case is documented further down ... (scroll down)
That caller has no knowledge of proctection or encryption keys.

> 
> >  	if (ret)
> >  		goto out_unlock;
> >  	BUG_ON(prev != vma);
> > diff --git a/include/linux/key.h b/include/linux/key.h
> > index e58ee10f6e58..fb8a7d5f6149 100644
> > --- a/include/linux/key.h
> > +++ b/include/linux/key.h
> > @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> > *key)
> >  
> >  extern void key_set_timeout(struct key *, unsigned);
> >  
> > +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> > +				 key_perm_t perm);
> >  /*
> >   * The permissions required on a key that we're looking up.
> >   */
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index ac85c0805761..0f9422c7841e 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> > vm_area_struct *vma, unsigned long
> >  			      int dirty_accountable, int prot_numa);
> >  extern int mprotect_fixup(struct vm_area_struct *vma,
> >  			  struct vm_area_struct **pprev, unsigned long start,
> > -			  unsigned long end, unsigned long newflags);
> > +			  unsigned long end, unsigned long newflags,
> > +			  int newkeyid);
> >  
> >  /*
> >   * doesn't attempt to fault and will return short.
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 56e64ef7931e..6c2e1106525c 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -28,14 +28,17 @@
> >  #include <linux/ksm.h>
> >  #include <linux/uaccess.h>
> >  #include <linux/mm_inline.h>
> > +#include <linux/key.h>
> >  #include <asm/pgtable.h>
> >  #include <asm/cacheflush.h>
> >  #include <asm/mmu_context.h>
> >  #include <asm/tlbflush.h>
> > +#include <asm/mktme.h>
> >  
> >  #include "internal.h"
> >  
> >  #define NO_PKEY  -1
> > +#define NO_KEYID -1
> 
> Should have only single named constant IMHO. This ambiguity
> is worse than some reasonable constant name for both cases.
> Maybe NO_KEYID would be adequate?

Yes, this could be NO_KEY for both. I was worried about readability,
but most of the usages compare it to a well-named variable, as in 
(pkey == NO_KEY) or (keyid == NO_KEY) so it seems to work. 
Will do!

> 
> >  
> >  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
> >  		unsigned long addr, unsigned long end, pgprot_t newprot,
> > @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> > *vma, unsigned long start,
> >  
> >  int
> >  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> > -	unsigned long start, unsigned long end, unsigned long newflags)
> > +	       unsigned long start, unsigned long end, unsigned long
> > newflags,
> > +	       int newkeyid)
> >  {
> >  	struct mm_struct *mm = vma->vm_mm;
> >  	unsigned long oldflags = vma->vm_flags;
> > @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	int error;
> >  	int dirty_accountable = 0;
> >  
> > +	/*
> > +	 * Flags match and Keyids match or we have NO_KEYID.
> > +	 * This _fixup is usually called from do_mprotect_ext() except
> > +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> > +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> > +	 */

Above is that special case explanation.

> > +	if (newflags == oldflags &&
> > +	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEYID)) {
> > +		*pprev = vma;
> > +		return 0;
> > +	}
> > +	/* Flags match and Keyid changes */
> >  	if (newflags == oldflags) {
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  		*pprev = vma;
> >  		return 0;
> >  	}
> > +	/* Flags and Keyids both change, continue. */
> >  
> >  	/*
> >  	 * If we make a private mapping writable we increase our commit;
> > @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	}
> >  
> >  success:
> > +	if (newkeyid != NO_KEYID)
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  	/*
> >  	 * vm_flags and vm_page_prot are protected by the mmap_sem
> >  	 * held in write mode.
> > @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  }
> >  
> >  /*
> > - * When pkey==NO_PKEY we get legacy mprotect behavior here.
> > + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> > + * for protection keys and memory encryption keys. These extensions are
> > + * mutually exclusive and the behavior is:
> > + *	(pkey==NO_PKEY && keyid==NO_KEYID) ==> legacy mprotect
> > + *	(pkey is valid)  ==> legacy mprotect plus protection key extensions
> > + *	(keyid is valid) ==> legacy mprotect plus encryption key extensions
> >   */
> 
> The header does not follow
> 
> https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

Jarkko,

I see the "how to" on kernel doc formatting, but I'm not getting the
when? When should it be done? I can imagine it for new code, like
the mktme_keys.c where new code is being added and the kernel docs
could give a complete narrative of the module.

Here, in mprotect.c, there are no kernel doc comments. I realize
sometime we just need to start somewhere ;), but I find it odd to
pluck out one function header to be extracted for kernel docs.

Alison

> 
> >  static int do_mprotect_ext(unsigned long start, size_t len,
> > -		unsigned long prot, int pkey)
> > +			   unsigned long prot, int pkey, int keyid)
> >  {
> >  	unsigned long nstart, end, tmp, reqprot;
> >  	struct vm_area_struct *vma, *prev;
> > @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  		tmp = vma->vm_end;
> >  		if (tmp > end)
> >  			tmp = end;
> > -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> > +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> > +				       keyid);
> >  		if (error)
> >  			goto out;
> >  		nstart = tmp;
> > @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot)
> >  {
> > -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> > +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
> >  }
> >  
> >  #ifdef CONFIG_ARCH_HAS_PKEYS
> > @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> > len,
> >  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot, int, pkey)
> >  {
> > -	return do_mprotect_ext(start, len, prot, pkey);
> > +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
> >  }
> >  
> >  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> > @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
> >  }
> >  
> >  #endif /* CONFIG_ARCH_HAS_PKEYS */
> > +
> > +#ifdef CONFIG_X86_INTEL_MKTME
> > +
> > +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> > +		unsigned long, prot, key_serial_t, serial)
> > +{
> > +	key_ref_t key_ref;
> > +	int ret, keyid;
> > +
> > +	/* TODO MKTME key service must be initialized */
> > +
> > +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> > +	if (IS_ERR(key_ref))
> > +		return PTR_ERR(key_ref);
> > +
> > +	mktme_map_lock();
> > +	keyid = mktme_map_keyid_from_serial(serial);
> > +	if (!keyid) {
> > +		mktme_map_unlock();
> > +		key_ref_put(key_ref);
> > +		return -EINVAL;
> > +	}
> > +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> > +	mktme_map_unlock();
> > +	key_ref_put(key_ref);
> > +	return ret;
> > +}
> > +
> > +#endif /* CONFIG_X86_INTEL_MKTME */
> 
> /Jarkko

Re: [RFC 06/12] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:02:43PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:36 -0700, Alison Schofield wrote:
> > Implement memory encryption with a new system call that is an
> > extension of the legacy mprotect() system call.
> > 
> > In encrypt_mprotect the caller must pass a handle to a previously
> > allocated and programmed encryption key. Validate the key and store
> > the keyid bits in the vm_page_prot for each VMA in the protection
> > range.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  fs/exec.c           |  4 ++--
> >  include/linux/key.h |  2 ++
> >  include/linux/mm.h  |  3 ++-
> >  mm/mprotect.c       | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--
> > ---
> >  4 files changed, 67 insertions(+), 9 deletions(-)
> > 
> > diff --git a/fs/exec.c b/fs/exec.c
> > index a1a246062561..b681a413db9c 100644
> > --- a/fs/exec.c
> > +++ b/fs/exec.c
> > @@ -754,8 +754,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
> >  	vm_flags |= mm->def_flags;
> >  	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
> >  
> > -	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > -			vm_flags);
> > +	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
> > vm_flags,
> > +			     -1);
> 
> Why you pass a magic number here when you went the trouble having
> a named constant?

The named constant isn't available over here in fs/exec.c.
The special case is documented further down ... (scroll down)
That caller has no knowledge of proctection or encryption keys.

> 
> >  	if (ret)
> >  		goto out_unlock;
> >  	BUG_ON(prev != vma);
> > diff --git a/include/linux/key.h b/include/linux/key.h
> > index e58ee10f6e58..fb8a7d5f6149 100644
> > --- a/include/linux/key.h
> > +++ b/include/linux/key.h
> > @@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key
> > *key)
> >  
> >  extern void key_set_timeout(struct key *, unsigned);
> >  
> > +extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
> > +				 key_perm_t perm);
> >  /*
> >   * The permissions required on a key that we're looking up.
> >   */
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index ac85c0805761..0f9422c7841e 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -1579,7 +1579,8 @@ extern unsigned long change_protection(struct
> > vm_area_struct *vma, unsigned long
> >  			      int dirty_accountable, int prot_numa);
> >  extern int mprotect_fixup(struct vm_area_struct *vma,
> >  			  struct vm_area_struct **pprev, unsigned long start,
> > -			  unsigned long end, unsigned long newflags);
> > +			  unsigned long end, unsigned long newflags,
> > +			  int newkeyid);
> >  
> >  /*
> >   * doesn't attempt to fault and will return short.
> > diff --git a/mm/mprotect.c b/mm/mprotect.c
> > index 56e64ef7931e..6c2e1106525c 100644
> > --- a/mm/mprotect.c
> > +++ b/mm/mprotect.c
> > @@ -28,14 +28,17 @@
> >  #include <linux/ksm.h>
> >  #include <linux/uaccess.h>
> >  #include <linux/mm_inline.h>
> > +#include <linux/key.h>
> >  #include <asm/pgtable.h>
> >  #include <asm/cacheflush.h>
> >  #include <asm/mmu_context.h>
> >  #include <asm/tlbflush.h>
> > +#include <asm/mktme.h>
> >  
> >  #include "internal.h"
> >  
> >  #define NO_PKEY  -1
> > +#define NO_KEYID -1
> 
> Should have only single named constant IMHO. This ambiguity
> is worse than some reasonable constant name for both cases.
> Maybe NO_KEYID would be adequate?

Yes, this could be NO_KEY for both. I was worried about readability,
but most of the usages compare it to a well-named variable, as in 
(pkey == NO_KEY) or (keyid == NO_KEY) so it seems to work. 
Will do!

> 
> >  
> >  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
> >  		unsigned long addr, unsigned long end, pgprot_t newprot,
> > @@ -310,7 +313,8 @@ unsigned long change_protection(struct vm_area_struct
> > *vma, unsigned long start,
> >  
> >  int
> >  mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
> > -	unsigned long start, unsigned long end, unsigned long newflags)
> > +	       unsigned long start, unsigned long end, unsigned long
> > newflags,
> > +	       int newkeyid)
> >  {
> >  	struct mm_struct *mm = vma->vm_mm;
> >  	unsigned long oldflags = vma->vm_flags;
> > @@ -320,10 +324,24 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	int error;
> >  	int dirty_accountable = 0;
> >  
> > +	/*
> > +	 * Flags match and Keyids match or we have NO_KEYID.
> > +	 * This _fixup is usually called from do_mprotect_ext() except
> > +	 * for one special case: caller fs/exec.c/setup_arg_pages()
> > +	 * In that case, newkeyid is passed as -1 (NO_KEYID).
> > +	 */

Above is that special case explanation.

> > +	if (newflags == oldflags &&
> > +	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEYID)) {
> > +		*pprev = vma;
> > +		return 0;
> > +	}
> > +	/* Flags match and Keyid changes */
> >  	if (newflags == oldflags) {
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  		*pprev = vma;
> >  		return 0;
> >  	}
> > +	/* Flags and Keyids both change, continue. */
> >  
> >  	/*
> >  	 * If we make a private mapping writable we increase our commit;
> > @@ -373,6 +391,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  	}
> >  
> >  success:
> > +	if (newkeyid != NO_KEYID)
> > +		mprotect_set_encrypt(vma, newkeyid);
> >  	/*
> >  	 * vm_flags and vm_page_prot are protected by the mmap_sem
> >  	 * held in write mode.
> > @@ -404,10 +424,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> > vm_area_struct **pprev,
> >  }
> >  
> >  /*
> > - * When pkey==NO_PKEY we get legacy mprotect behavior here.
> > + * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
> > + * for protection keys and memory encryption keys. These extensions are
> > + * mutually exclusive and the behavior is:
> > + *	(pkey==NO_PKEY && keyid==NO_KEYID) ==> legacy mprotect
> > + *	(pkey is valid)  ==> legacy mprotect plus protection key extensions
> > + *	(keyid is valid) ==> legacy mprotect plus encryption key extensions
> >   */
> 
> The header does not follow
> 
> https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

Jarkko,

I see the "how to" on kernel doc formatting, but I'm not getting the
when? When should it be done? I can imagine it for new code, like
the mktme_keys.c where new code is being added and the kernel docs
could give a complete narrative of the module.

Here, in mprotect.c, there are no kernel doc comments. I realize
sometime we just need to start somewhere ;), but I find it odd to
pluck out one function header to be extracted for kernel docs.

Alison

> 
> >  static int do_mprotect_ext(unsigned long start, size_t len,
> > -		unsigned long prot, int pkey)
> > +			   unsigned long prot, int pkey, int keyid)
> >  {
> >  	unsigned long nstart, end, tmp, reqprot;
> >  	struct vm_area_struct *vma, *prev;
> > @@ -505,7 +530,8 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  		tmp = vma->vm_end;
> >  		if (tmp > end)
> >  			tmp = end;
> > -		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
> > +		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
> > +				       keyid);
> >  		if (error)
> >  			goto out;
> >  		nstart = tmp;
> > @@ -530,7 +556,7 @@ static int do_mprotect_ext(unsigned long start, size_t
> > len,
> >  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot)
> >  {
> > -	return do_mprotect_ext(start, len, prot, NO_PKEY);
> > +	return do_mprotect_ext(start, len, prot, NO_PKEY, NO_KEYID);
> >  }
> >  
> >  #ifdef CONFIG_ARCH_HAS_PKEYS
> > @@ -538,7 +564,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> > len,
> >  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
> >  		unsigned long, prot, int, pkey)
> >  {
> > -	return do_mprotect_ext(start, len, prot, pkey);
> > +	return do_mprotect_ext(start, len, prot, pkey, NO_KEYID);
> >  }
> >  
> >  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
> > @@ -587,3 +613,32 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
> >  }
> >  
> >  #endif /* CONFIG_ARCH_HAS_PKEYS */
> > +
> > +#ifdef CONFIG_X86_INTEL_MKTME
> > +
> > +SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
> > +		unsigned long, prot, key_serial_t, serial)
> > +{
> > +	key_ref_t key_ref;
> > +	int ret, keyid;
> > +
> > +	/* TODO MKTME key service must be initialized */
> > +
> > +	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
> > +	if (IS_ERR(key_ref))
> > +		return PTR_ERR(key_ref);
> > +
> > +	mktme_map_lock();
> > +	keyid = mktme_map_keyid_from_serial(serial);
> > +	if (!keyid) {
> > +		mktme_map_unlock();
> > +		key_ref_put(key_ref);
> > +		return -EINVAL;
> > +	}
> > +	ret = do_mprotect_ext(start, len, prot, NO_PKEY, keyid);
> > +	mktme_map_unlock();
> > +	key_ref_put(key_ref);
> > +	return ret;
> > +}
> > +
> > +#endif /* CONFIG_X86_INTEL_MKTME */
> 
> /Jarkko

Re: [RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:20:45PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> > Keep track of the VMA's oustanding for each memory encryption keyid.
> > The count is used by the MKTME (Multi-Key Total Memory Encryption)
> > Key Service to determine when it is safe to reprogram a hardware
> > encryption key.
> 
> Maybe a stupid question but why they are tracked and what do you 
> mean by tracking?
> 
> /Jarkko

Perhaps 'Keep a count of' instead of 'Keep track of' will be clearer.

Counting VMA's using each keyid prevents in use keys from being cleared
and reused. The counting is done here, and the mtkme key service checks
these counts to decide if it is OK to allow a userspace key to be revoked.
A successful userspace key revoke will clear the hardware keyid slot and
leave the key available to be reprogrammed.

> 
> > Approach here is to do gets and puts on the encryption reference
> > wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> > A couple of these locations will not be hit until cgroup support is
> > added. One of these locations should never hit, so use a VM_WARN_ON.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/mm/mktme.c |  2 ++
> >  kernel/fork.c       |  2 ++
> >  mm/mmap.c           | 12 ++++++++++++
> >  mm/nommu.c          |  4 ++++
> >  4 files changed, 20 insertions(+)
> > 

.... snip ....

[RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:20:45PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> > Keep track of the VMA's oustanding for each memory encryption keyid.
> > The count is used by the MKTME (Multi-Key Total Memory Encryption)
> > Key Service to determine when it is safe to reprogram a hardware
> > encryption key.
> 
> Maybe a stupid question but why they are tracked and what do you 
> mean by tracking?
> 
> /Jarkko

Perhaps 'Keep a count of' instead of 'Keep track of' will be clearer.

Counting VMA's using each keyid prevents in use keys from being cleared
and reused. The counting is done here, and the mtkme key service checks
these counts to decide if it is OK to allow a userspace key to be revoked.
A successful userspace key revoke will clear the hardware keyid slot and
leave the key available to be reprogrammed.

> 
> > Approach here is to do gets and puts on the encryption reference
> > wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> > A couple of these locations will not be hit until cgroup support is
> > added. One of these locations should never hit, so use a VM_WARN_ON.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/mm/mktme.c |  2 ++
> >  kernel/fork.c       |  2 ++
> >  mm/mmap.c           | 12 ++++++++++++
> >  mm/nommu.c          |  4 ++++
> >  4 files changed, 20 insertions(+)
> > 

.... snip ....

Re: [RFC 08/12] mm: Track VMA's in use for each memory encryption keyid

[Alison Schofield]

On Mon, Sep 10, 2018 at 09:20:45PM +0300, Jarkko Sakkinen wrote:
> On Fri, 2018-09-07 at 15:37 -0700, Alison Schofield wrote:
> > Keep track of the VMA's oustanding for each memory encryption keyid.
> > The count is used by the MKTME (Multi-Key Total Memory Encryption)
> > Key Service to determine when it is safe to reprogram a hardware
> > encryption key.
> 
> Maybe a stupid question but why they are tracked and what do you 
> mean by tracking?
> 
> /Jarkko

Perhaps 'Keep a count of' instead of 'Keep track of' will be clearer.

Counting VMA's using each keyid prevents in use keys from being cleared
and reused. The counting is done here, and the mtkme key service checks
these counts to decide if it is OK to allow a userspace key to be revoked.
A successful userspace key revoke will clear the hardware keyid slot and
leave the key available to be reprogrammed.

> 
> > Approach here is to do gets and puts on the encryption reference
> > wherever kmem_cache_alloc/free's of vma_area_cachep's are executed.
> > A couple of these locations will not be hit until cgroup support is
> > added. One of these locations should never hit, so use a VM_WARN_ON.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > ---
> >  arch/x86/mm/mktme.c |  2 ++
> >  kernel/fork.c       |  2 ++
> >  mm/mmap.c           | 12 ++++++++++++
> >  mm/nommu.c          |  4 ++++
> >  4 files changed, 20 insertions(+)
> > 

.... snip ....

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > as a system wide single pool of keys. The hardware, however, manages
> > the keys on a per physical package basis. Each physical package
> > maintains a key table that all CPU's in that package share.
> > 
> > In order to maintain the consistent, system wide view that the kernel
> > requires, program all physical packages during a key program request.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> 
> Just kind of checking that are you talking about multiple cores in
> a single package or really multiple packages?

System wide pool.
System has multiple packages.  
Packages have multiple CPU's.

The hardware KEY TABLE is per package. I need that per package KEY TABLE
to be the same in every package across the system. So, I pick one 'lead'
CPU in each package to program that packages KEY TABLE.

(BTW - I'm going to look into Kai's suggestion to move the system wide view
of this key programming into the key service. Not sure if that's a go.)

> 
> /Jarkko

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > as a system wide single pool of keys. The hardware, however, manages
> > the keys on a per physical package basis. Each physical package
> > maintains a key table that all CPU's in that package share.
> > 
> > In order to maintain the consistent, system wide view that the kernel
> > requires, program all physical packages during a key program request.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> 
> Just kind of checking that are you talking about multiple cores in
> a single package or really multiple packages?

System wide pool.
System has multiple packages.  
Packages have multiple CPU's.

The hardware KEY TABLE is per package. I need that per package KEY TABLE
to be the same in every package across the system. So, I pick one 'lead'
CPU in each package to program that packages KEY TABLE.

(BTW - I'm going to look into Kai's suggestion to move the system wide view
of this key programming into the key service. Not sure if that's a go.)

> 
> /Jarkko

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Alison Schofield]

On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > as a system wide single pool of keys. The hardware, however, manages
> > the keys on a per physical package basis. Each physical package
> > maintains a key table that all CPU's in that package share.
> > 
> > In order to maintain the consistent, system wide view that the kernel
> > requires, program all physical packages during a key program request.
> > 
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> 
> Just kind of checking that are you talking about multiple cores in
> a single package or really multiple packages?

System wide pool.
System has multiple packages.  
Packages have multiple CPU's.

The hardware KEY TABLE is per package. I need that per package KEY TABLE
to be the same in every package across the system. So, I pick one 'lead'
CPU in each package to program that packages KEY TABLE.

(BTW - I'm going to look into Kai's suggestion to move the system wide view
of this key programming into the key service. Not sure if that's a go.)

> 
> /Jarkko

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

T24gTW9uLCAyMDE4LTA5LTEwIGF0IDEyOjEwIC0wNzAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBPbiBTdW4sIFNlcCAwOSwgMjAxOCBhdCAwNjoxMDoxOVBNIC0wNzAwLCBIdWFuZywgS2Fp
IHdyb3RlOg0KPiA+IA0KPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiA+IEZy
b206IGtleXJpbmdzLW93bmVyQHZnZXIua2VybmVsLm9yZyBbbWFpbHRvOmtleXJpbmdzLQ0KPiA+
ID4gb3duZXJAdmdlci5rZXJuZWwub3JnXSBPbiBCZWhhbGYgT2YgQWxpc29uIFNjaG9maWVsZA0K
PiA+ID4gU2VudDogU2F0dXJkYXksIFNlcHRlbWJlciA4LCAyMDE4IDEwOjIzIEFNDQo+ID4gPiBU
bzogZGhvd2VsbHNAcmVkaGF0LmNvbTsgdGdseEBsaW51dHJvbml4LmRlDQo+ID4gPiBDYzogSHVh
bmcsIEthaSA8a2FpLmh1YW5nQGludGVsLmNvbT47IE5ha2FqaW1hLCBKdW4NCj4gPiA+IDxqdW4u
bmFrYWppbWFAaW50ZWwuY29tPjsgU2h1dGVtb3YsIEtpcmlsbCA8a2lyaWxsLnNodXRlbW92QGlu
dGVsDQo+ID4gPiAuY29tPjsNCj4gPiA+IEhhbnNlbiwgRGF2ZSA8ZGF2ZS5oYW5zZW5AaW50ZWwu
Y29tPjsgU2Fra2luZW4sIEphcmtrbw0KPiA+ID4gPGphcmtrby5zYWtraW5lbkBpbnRlbC5jb20+
OyBqbW9ycmlzQG5hbWVpLm9yZzsga2V5cmluZ3NAdmdlci5rZXINCj4gPiA+IG5lbC5vcmc7DQo+
ID4gPiBsaW51eC1zZWN1cml0eS1tb2R1bGVAdmdlci5rZXJuZWwub3JnOyBtaW5nb0ByZWRoYXQu
Y29tOyBocGFAenl0bw0KPiA+ID4gci5jb207DQo+ID4gPiB4ODZAa2VybmVsLm9yZzsgbGludXgt
bW1Aa3ZhY2sub3JnDQo+ID4gPiBTdWJqZWN0OiBbUkZDIDAwLzEyXSBNdWx0aS1LZXkgVG90YWwg
TWVtb3J5IEVuY3J5cHRpb24gQVBJDQo+ID4gPiAoTUtUTUUpDQo+ID4gPiANCj4gPiA+IFNlZWtp
bmcgY29tbWVudHMgb24gdGhlIEFQSXMgc3VwcG9ydGluZyBNS1RNRSBvbiBmdXR1cmUgSW50ZWwN
Cj4gPiA+IHBsYXRmb3Jtcy4NCj4gPiA+IA0KPiA+ID4gTUtUTUUgKE11bHRpLUtleSBUb3RhbCBN
ZW1vcnkgRW5jcnlwdGlvbikgaXMgYSB0ZWNobm9sb2d5DQo+ID4gPiBzdXBwb3J0aW5nDQo+ID4g
PiBtZW1vcnkgZW5jcnlwdGlvbiBvbiB1cGNvbWluZyBJbnRlbCBwbGF0Zm9ybXMuIFdoZXJlYXMg
VE1FIGFsbG93cw0KPiA+ID4gZW5jcnlwdGlvbiBvZiB0aGUgZW50aXJlIHN5c3RlbSBtZW1vcnkg
dXNpbmcgYSBzaW5nbGUga2V5LCBNS1RNRQ0KPiA+ID4gYWxsb3dzDQo+ID4gPiBtdWxpdHBsZSBl
bmNyeXB0aW9uIGRvbWFpbnMsIGVhY2ggaGF2aW5nIHRoZWlyIG93biBrZXkuIFdoaWxlIHRoZQ0K
PiA+ID4gbWFpbiB1c2UNCj4gPiA+IGNhc2UgZm9yIHRoZSBmZWF0dXJlIGlzIHZpcnR1YWwgbWFj
aGluZSBpc29sYXRpb24sIHRoZSBBUEkgbmVlZHMNCj4gPiA+IHRoZSBmbGV4aWJpbGl0eSB0bw0K
PiA+ID4gd29yayBmb3IgYSB3aWRlIHJhbmdlIG9mIHVzZSBjYXNlcy4NCj4gPiA+IA0KPiA+ID4g
VGhpcyBSRkMgcHJlc2VudHMgdGhlIDIgQVBJIGFkZGl0aW9ucyB0aGF0IGVuYWJsZSB1c2Vyc3Bh
Y2UgdG86DQo+ID4gPiAgMSkgQ3JlYXRlIEVuY3J5cHRpb24gS2V5czogS2VybmVsIEtleSBTZXJ2
aWNlIHR5cGUgIm1rdG1lIg0KPiA+ID4gIDIpIFVzZSB0aGUgRW5jcnlwdGlvbiBLZXlzOiBzeXN0
ZW0gY2FsbCBlbmNyeXB0X21wcm90ZWN0KCkNCj4gPiA+IA0KPiA+ID4gSW4gb3JkZXIgdG8gc2hh
cmUgYmV0d2VlbjogdGhlIEtlcm5lbCBLZXkgU2VydmljZSwgdGhlIG5ldyBzeXN0ZW0NCj4gPiA+
IGNhbGwsIGFuZCB0aGUNCj4gPiA+IGV4aXN0aW5nIG1tIGNvZGUsIGhlbHBlciBmdW5jdGlvbnMg
d2VyZSBjcmVhdGVkIGluIGFyY2gveDg2L21rdG1lDQo+ID4gDQo+ID4gSU1ITywgd2UgY2FuIHNl
cGFyYXRlIHRoaXMgc2VyaWVzIGludG8gMiBwYXJ0cywgYXMgeW91IGRpZCBhYm92ZSwNCj4gPiBh
bmQgc2VuZCBvdXQgdGhlbSBzZXBhcmF0ZWx5LiBUaGUgcmVhc29uIGlzLCBpbiBnZW5lcmFsIEkg
dGhpbmsNCj4gPiBhZGRpbmcgbmV3IE1LVE1FIHR5cGUgdG8ga2V5IHJldGVudGlvbiBzZXJ2aWNl
cyBpcyBub3QgdGhhdCByZWxhdGVkDQo+ID4gdG8gbWVtb3J5IG1hbmFnZW1lbnQgY29kZSwgbmFt
ZWx5IHRoZSBlbmNyeXB0X21wcm90ZWN0KCkgQVBJIHBhcnQuDQo+ID4gDQo+ID4gU28gaWYgd2Ug
c3BsaXQgdGhlIHR3byBwYXJ0cyBhbmQgc2VuZCB0aGVtIG91dCBzZXBhcmF0ZWx5LCB0aGUNCj4g
PiBmaXJzdCBwYXJ0IGNhbiBiZSByZXZpZXdlZCBieSBrZXlyaW5nIGFuZCBzZWN1cml0eSBndXlz
LCB3aXRob3V0DQo+ID4gaW52b2x2aW5nIG1tIGd1eXMsIGFuZCB0aGUgZW5jcnlwdF9tcHJvdGVj
dCgpIHBhcnQgY2FuIGJlIG1vcmUNCj4gPiByZXZpZXdlZCBtb3JlIGJ5IG1tIGd1eXMuIA0KPiA+
IA0KPiANCj4gS2FpLA0KPiANCj4gVGhhdCB3YXMgdGhlIGRpcmVjdGlvbiBJIGhhZCBpbiBtaW5k
IGF0IHRoZSBvbnNldDogdGhlIE1LVE1FIGtleQ0KPiBzZXJ2aWNlDQo+IHdvdWxkIGJlIG9uZSBw
YXRjaChzZXQpIGFuZCB0aGUgTUtUTUUgZW5jcnlwdF9tcHJvdGVjdCgpIHN5c3RlbSBjYWxsDQo+
IHdvdWxkDQo+IGJlIGRlbGl2ZXJlZCBpbiBhbm90aGVyIHBhdGNoKHNldCkuDQo+IA0KPiBUaGF0
IHNlcGFyYXRpb24gZmFsbHMgYXBhcnQgd2hlbiB0aGUgc2hhcmVkIHN0cnVjdHVyZXMgYW5kIGZ1
bmN0aW9ucw0KPiBhcmUNCj4gaW50cm9kdWNlZC4gVGhhdCAnbWt0bWVfbWFwJyAobWFwcyB1c2Vy
c3BhY2Uga2V5cyB0byBoYXJkd2FyZSBrZXlpZA0KPiBzbG90cyksDQo+IGFuZCB0aGUgJ2VuY3J5
cHRfY291bnQnIGFycmF5IChjb3VudHMgdm1hJ3Mgb3V0c3RhbmRpbmcgZm9yIGVhY2gga2V5KQ0K
PiBuZWVkDQo+IHRvIGJlIHNoYXJlZCBieSBib3RoIHBpZWNlcy4gVGhlc2UgbWt0bWUgc3BlY2lh
bCBzaGFyZWQgc3RydWN0dXJlcw0KPiBhbmQgdGhlDQo+IGZ1bmN0aW9ucyB0aGF0IG9wZXJhdGUg
b24gdGhlbSBhcmUgYWxsIGRlZmluZWQgaW4NCj4gYXJjaC94ODYvbW0vbWt0bWUuaCwuYy4NCj4g
RnJvbSB0aGVyZSB0aGV5IGNhbiBiZSBzaGFyZWQgd2l0aCB0aGUgc2VjdXJpdHkva2V5cy9ta3Rt
ZV9rZXlzLmMNCj4gDQo+IE9uY2UgSSBtYWRlIHRoYXQgc2VwYXJhdGlvbiwgSSBzdHVjayB3aXRo
IGl0LiBUaG9zZSBzdHJ1Y3R1cmVzLCBhbmQNCj4gYW55DQo+IGZ1bmN0aW9ucyB0aGF0IG1hbmlw
dWxhdGUgdGhvc2Ugc3RydWN0dXJlcyBsaXZlIGluDQo+IGFyY2gveDg2L21tL21rdG1lLmgsYw0K
PiANCj4gWW91IG5vdGVkIHRoYXQgc29tZSBvZiB0aGUgZnVuY3Rpb25zIHRoYXQgb3BlcmF0ZSBv
biB0aGUNCj4gZW5jcnlwdF9jb3VudA0KPiBtaWdodCBub3QgbmVlZCB0byBiZSBvdmVyIGluIGFy
Y2gveDg2L21tL21rdG1lLmMgYmVjYXVzZSB0aGV5IGFyZSBub3QNCj4gdXNlZA0KPiBpbiB0aGUg
bW0gY29kZS4gVGhhdCBpcyB0cnVlLiAgDQoNClllcyBJTU8gdGhpcyBpcyBiZXR0ZXIuDQoNCj4g
QnV0LCB0aGVuIEknZCBiZSBzcGxpdHRpbmcgdXAgdGhlIGRlZmluaXRpb24NCj4gb2YgdGhlIHN0
cnVjdCBhbmQgdGhlIGZ1bmNzIHRoYXQgb3BlcmF0ZSBvbiBpdC4gU28sIEkgc3R1Y2sgd2l0aA0K
PiBrZWVwaW5nIGl0DQo+IGFsbCB0b2dldGhlciBpbiB0aGUgYXJjaCBzcGVjaWZpYyBta3RtZSBm
aWxlcy4NCg0KRGVmaW5pdGlvbiBjYW4gYWxzbyBiZSBpbiBpbmNsdWRlL2tleXMvbWt0bWUtdHlw
ZS5oLCByaWdodD8NCg0KPiANCj4gSGF2aW5nIHNhaWQgYWxsIHRoZSBhYm92ZSwgSSBkbyB3ZWxj
b21lIG90aGVyIGlkZWFzIG9uIGhvdyB0byBiZXR0ZXINCj4gb3JnYW5pemUNCj4gdGhlIGNvZGUu
IA0KPiANCj4gQmFjayB0byB5b3VyIHJlcXVlc3QtIHRvIHNwbGl0IGl0IGludG8gc21hbGxlciBw
YXRjaHNldHMgbWlnaHQgbG9vaw0KPiBzb21ldGhpbmcNCj4gbGlrZToNCj4gMSkgdGhlIE1LVE1F
IEFQSSBoZWxwZXJzDQoNCkV4YWN0bHkgd2hhdCBoZWxwZXJzPw0KDQo+IDIpIHRoZSBNS1RNRSBL
ZXkgU2VydmljZQ0KPiAzKSB0aGUgTUtUTUUgc3lzY2FsbCBlbmNyeXB0X21wcm90ZWN0KCkNCj4g
DQo+IEknbSBub3QgY2xlYXIgdGhhdCB3b3VsZCBtYWtlIGFueW9uZXMgcmV2aWV3IGxpZmUgZWFz
aWVyLCB0aGFuDQo+IHBpY2tpbmcNCj4gdGhlIHNhbWUgcGllY2VzIG91dCBvZiB0aGUgZ3JlYXRl
ciBwYXRjaHNldC4NCg0KV2VsbCwgdGhlIHJlYXNvbiBJIHN1Z2dlc3RlZCB3ZSBzaG91bGQgc3Bs
aXQga2V5IHJldGVudGlvbiBzZXJ2aWNlcw0KZnJvbSBvdGhlciBzdGFmZiBhbmQgZ2V0IGl0IHJl
dmlld2VkIGZpcnN0IGlzIEkgdGhpbmsgdGhlDQpmdW5jdGlvbmFsaXRpZXMgb2YgTUtUTUUga2V5
IHR5cGUgaGFzIG5vdGhpbmcgdG8gZG8gdy8gTU0gY29kZS4gSU1ITw0Kd2hhdCBNS1RNRSBrZXkg
dHlwZSBzaG91bGQgcHJvdmlkZSBpbmNsdWRlczoNCg0KMSkgU3VwcG9ydCBhZGRfa2V5LCBrZXlj
dGwgKHJldm9rZSwgZXRjKSBvYnZpb3VzbHkuIFRIaXMgYWxzbyBpbXBsaWVzDQprZXlJRCBhbGxv
Y2F0aW9uLCBzbyB5b3Ugd2lsbCBuZWVkIHNvbWUgaW5mbyBmcm9tIE1NIG9yIHg4NiBjb2RlIHRv
IGdldA0KIGluZm8gc3VjaCBhcyBudW1iZXIgb2Yga2V5SURzLCBldGMuIEluIGZhY3QgSSB0aGlu
ayBudW1iZXIgb2Yga2V5SURzDQppcyAgdGhlIG9ubHkgaW5mbyB5b3UgbmVlZCAoMCBrZXlJRHMg
bWVhbnMgTUtUTUUgaXMgZGlzYWJsZWQpLg0KDQoyKSBZb3UgbmVlZCB0byBwcm92aWRlIHNvbWUg
QVBJcyB0byBtdW5pcHVsYXRlIGtleSByZWZlcmVuY2UgY291bnQuDQpUaG9zZSBmdW5jdGlvbnMg
Y2FuIGJlIGRlY2xhcmVkIGluIG1rdG1lLXR5cGUuaCwgYW5kIGltcGxlbWVudGVkIGluDQpta3Rt
ZV9rZXlzLmMuIEkgdGhpbmsgdGhpcyBpcyBtb3JlIHJlYXNvbmFibGUsIGFzIGxvZ2ljYWxseSBy
ZWZlcmVuY2UNCmNvdW50IGlzIHBhcnQgb2YgTUtUTUUga2V5IHNlcnZpY2UuDQoNCkkgZG9uJ3Qg
dGhpbmsgeW91IG5lZWQgdG8gZG8gYW55dGhpbmcgZWxzZSBpbiBNS1RNRSBrZXkgdHlwZT8NCg0K
SSB0aGluayB5b3UgY2FuIGV2ZW4gc3RhcnQgdXBzdHJlYW1pbmcgTUtUTUUga2V5IHR5cGUgbm93
IHcvbyBLaXJpbGwncw0KY29yZS1NS1RNRSBzdXBwb3J0LCBzaW5jZSAxKSBhZGRpbmcgTUtUTUUg
a2V5IHR5cGUgZG9lc24ndCBkbyBhbnkgaGFybQ0Kdy9vIGtpcmlsbCdzIGNvcmUtTUtUTUUgY29k
ZTsgMikgbG9naWNhbGx5IE1LVE1FIGtleSB0eXBlIGhhcyBubyB0cnVlDQpkZXBlbmRlbmN5IG9u
IGNvcmUtTUtUTUUgY29kZS4NCg0KU28gSU1ITywgYSBiZXR0ZXIgc3RydWN0dXJlIHNob3VsZCBi
ZToNCg0KMSkgQ3VycmVudCB1cHN0cmVhbSBjb2RlIChtYXliZSB3aXRoIHNvbWUgZWFybHkgZGV0
Y3Rpb24gY29kZSBjaGFuZ2UsDQppZSwgdG8gZGlzYWJsZSBNS1RNRSBmb3Igc29tZSBTVyByZWFz
b24sIGJ1dCBJIGFtIG5vdCBzdXJlIHdoZXRoZXIgdGhpcw0KaXMgdHJ1ZWx5IG5lZWRlZCwgc2lu
Y2Ugd2hlbiBNS1RNRSBrZXkgdHlwZSBnZXRzIGluaXRpYWxpemVkLCB0aGF0IHBhcnQNCnNob3Vs
ZCBiZWVuIGRvbmUgYWxyZWFkeSkuDQoyKSBNS1RNRSBrZXkgdHlwZQ0KMykgS2lyaWxsJ3MgY29y
ZS1NS1RNRQ0KNCkgbmV3IHN5c2NhbGwNCg0KQW5kIEkgdGhpbmsgQ09ORklHX01LVE1FX0tFWVMg
ZG9lc24ndCByZWFsbHkgbmVlZCB0byBkZXBlbmQgb24NCkNPTkZJR19JTlRFTF9NS1RNRSBhdCB0
aGlzIHBvaW50LCBhbHRob3VnaCB3ZSBjYW4gYWRkIHN1Y2ggaWYgbmVlZGVkDQp3aGVuIGtpcmls
bCdzIGNvcmUtTUtUTUUgY29kZSBnZXRzIHVwc3RyZWFtZWQuDQoNClRoYW5rcywNCi1LYWkNCj4g
DQo+IFN1Z2dlc3Rpb25zIHdlbGNvbWUsDQo+IEFsaXNvbg0KPiANCj4gPiBBbmQgc2luY2UgZW5j
cnlwdF9tcHJvdGVjdCgpIGlzIGEgbmV3IHN5c2NhbGwsIHlvdSBtYXkgbmVlZCB0byBhZGQNCj4g
PiBtb3JlIGxpc3RzIGZvciB0aGUgcmV2aWV3LCBpZSwgbGludXgtYXBpLCBhbmQgbWF5YmUgbGlu
dXgta2VybmVsIGFzDQo+ID4gd2VsbC4NCj4gDQo+IEdvdCBpdC4gV2lsbCBpbmNsdWRlIHRoZXNl
cyBpbiB2MS4NCj4gDQo+ID4gDQo+ID4gVGhhbmtzLA0KPiA+IC1LYWkNCj4gPiANCj4gPiA+IA0K
PiA+ID4gVGhpcyBwYXRjaHNldCBpcyBidWlsdCB1cG9uIEtpcmlsbCBTaHV0ZW1vdidzIHBhdGNo
c2V0IGZvciB0aGUNCj4gPiA+IGNvcmUgTUtUTUUNCj4gPiA+IHN1cHBvcnQuIFlvdSBjYW4gZmlu
ZCB0aGF0IGhlcmU6DQo+ID4gPiBnaXQ6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tl
cm5lbC9naXQva2FzL2xpbnV4LmdpdA0KPiA+ID4gbWt0bWUvd2lwDQo+ID4gPiANCj4gPiA+IA0K
PiA+ID4gQWxpc29uIFNjaG9maWVsZCAoMTIpOg0KPiA+ID4gICBkb2NzL3g4NjogRG9jdW1lbnQg
dGhlIE11bHRpLUtleSBUb3RhbCBNZW1vcnkgRW5jcnlwdGlvbiBBUEkNCj4gPiA+ICAgbW06IEdl
bmVyYWxpemUgdGhlIG1wcm90ZWN0IGltcGxlbWVudGF0aW9uIHRvIHN1cHBvcnQNCj4gPiA+IGV4
dGVuc2lvbnMNCj4gPiA+ICAgc3lzY2FsbC94ODY6IFdpcmUgdXAgYSBuZXcgc3lzdGVtIGNhbGwg
Zm9yIG1lbW9yeSBlbmNyeXB0aW9uDQo+ID4gPiBrZXlzDQo+ID4gPiAgIHg4Ni9tbTogQWRkIGhl
bHBlciBmdW5jdGlvbnMgdG8gbWFuYWdlIG1lbW9yeSBlbmNyeXB0aW9uIGtleXMNCj4gPiA+ICAg
eDg2L21tOiBBZGQgYSBoZWxwZXIgZnVuY3Rpb24gdG8gc2V0IGtleWlkIGJpdHMgaW4gZW5jcnlw
dGVkDQo+ID4gPiBWTUEncw0KPiA+ID4gICBtbTogQWRkIHRoZSBlbmNyeXB0X21wcm90ZWN0KCkg
c3lzdGVtIGNhbGwNCj4gPiA+ICAgeDg2L21tOiBBZGQgaGVscGVyIGZ1bmN0aW9ucyB0byB0cmFj
ayBlbmNyeXB0ZWQgVk1BJ3MNCj4gPiA+ICAgbW06IFRyYWNrIFZNQSdzIGluIHVzZSBmb3IgZWFj
aCBtZW1vcnkgZW5jcnlwdGlvbiBrZXlpZA0KPiA+ID4gICBtbTogUmVzdHJpY3QgbWVtb3J5IGVu
Y3J5cHRpb24gdG8gYW5vbnltb3VzIFZNQSdzDQo+ID4gPiAgIHg4Ni9wY29uZmlnOiBQcm9ncmFt
IG1lbW9yeSBlbmNyeXB0aW9uIGtleXMgb24gYSBzeXN0ZW0td2lkZQ0KPiA+ID4gYmFzaXMNCj4g
PiA+ICAga2V5cy9ta3RtZTogQWRkIGEgbmV3IGtleSBzZXJ2aWNlIHR5cGUgZm9yIG1lbW9yeSBl
bmNyeXB0aW9uDQo+ID4gPiBrZXlzDQo+ID4gPiAgIGtleXMvbWt0bWU6IERvIG5vdCByZXZva2Ug
aW4gdXNlIG1lbW9yeSBlbmNyeXB0aW9uIGtleXMNCj4gPiA+IA0KPiA+ID4gIERvY3VtZW50YXRp
b24veDg2L21rdG1lLWtleXMudHh0ICAgICAgIHwgMTUzICsrKysrKysrKysrKysrKysNCj4gPiA+
ICBhcmNoL3g4Ni9LY29uZmlnICAgICAgICAgICAgICAgICAgICAgICB8ICAgMSArDQo+ID4gPiAg
YXJjaC94ODYvZW50cnkvc3lzY2FsbHMvc3lzY2FsbF8zMi50YmwgfCAgIDEgKw0KPiA+ID4gIGFy
Y2gveDg2L2VudHJ5L3N5c2NhbGxzL3N5c2NhbGxfNjQudGJsIHwgICAxICsNCj4gPiA+ICBhcmNo
L3g4Ni9pbmNsdWRlL2FzbS9pbnRlbF9wY29uZmlnLmggICB8ICA0MiArKysrLQ0KPiA+ID4gIGFy
Y2gveDg2L2luY2x1ZGUvYXNtL21rdG1lLmggICAgICAgICAgIHwgIDIxICsrKw0KPiA+ID4gIGFy
Y2gveDg2L21tL21rdG1lLmMgICAgICAgICAgICAgICAgICAgIHwgMTQxICsrKysrKysrKysrKysr
DQo+ID4gPiAgZnMvZXhlYy5jICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgIDQgKy0N
Cj4gPiA+ICBpbmNsdWRlL2tleXMvbWt0bWUtdHlwZS5oICAgICAgICAgICAgICB8ICAyOCArKysN
Cj4gPiA+ICBpbmNsdWRlL2xpbnV4L2tleS5oICAgICAgICAgICAgICAgICAgICB8ICAgMiArDQo+
ID4gPiAgaW5jbHVkZS9saW51eC9tbS5oICAgICAgICAgICAgICAgICAgICAgfCAgIDkgKy0NCj4g
PiA+ICBpbmNsdWRlL2xpbnV4L3N5c2NhbGxzLmggICAgICAgICAgICAgICB8ICAgMiArDQo+ID4g
PiAgaW5jbHVkZS91YXBpL2FzbS1nZW5lcmljL3VuaXN0ZC5oICAgICAgfCAgIDQgKy0NCj4gPiA+
ICBrZXJuZWwvZm9yay5jICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgMiArDQo+ID4gPiAg
a2VybmVsL3N5c19uaS5jICAgICAgICAgICAgICAgICAgICAgICAgfCAgIDIgKw0KPiA+ID4gIG1t
L21tYXAuYyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIDEyICsrDQo+ID4gPiAgbW0v
bXByb3RlY3QuYyAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgOTMgKysrKysrKysrLQ0KPiA+
ID4gIG1tL25vbW11LmMgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICA0ICsNCj4gPiA+
ICBzZWN1cml0eS9rZXlzL0tjb25maWcgICAgICAgICAgICAgICAgICB8ICAxMSArKw0KPiA+ID4g
IHNlY3VyaXR5L2tleXMvTWFrZWZpbGUgICAgICAgICAgICAgICAgIHwgICAxICsNCj4gPiA+ICBz
ZWN1cml0eS9rZXlzL2ludGVybmFsLmggICAgICAgICAgICAgICB8ICAgNiArDQo+ID4gPiAgc2Vj
dXJpdHkva2V5cy9rZXljdGwuYyAgICAgICAgICAgICAgICAgfCAgIDcgKw0KPiA+ID4gIHNlY3Vy
aXR5L2tleXMvbWt0bWVfa2V5cy5jICAgICAgICAgICAgIHwgMzI1DQo+ID4gPiArKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysNCj4gPiA+ICAyMyBmaWxlcyBjaGFuZ2VkLCA4NTUgaW5z
ZXJ0aW9ucygrKSwgMTcgZGVsZXRpb25zKC0pICBjcmVhdGUNCj4gPiA+IG1vZGUgMTAwNjQ0DQo+
ID4gPiBEb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS1rZXlzLnR4dCAgY3JlYXRlIG1vZGUgMTAwNjQ0
DQo+ID4gPiBpbmNsdWRlL2tleXMvbWt0bWUtDQo+ID4gPiB0eXBlLmggIGNyZWF0ZSBtb2RlIDEw
MDY0NCBzZWN1cml0eS9rZXlzL21rdG1lX2tleXMuYw0KPiA+ID4gDQo+ID4gPiAtLQ0KPiA+ID4g
Mi4xNC4x

[RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Mon, 2018-09-10 at 12:10 -0700, Alison Schofield wrote:
> On Sun, Sep 09, 2018 at 06:10:19PM -0700, Huang, Kai wrote:
> > 
> > > -----Original Message-----
> > > From: keyrings-owner at vger.kernel.org [mailto:keyrings-
> > > owner at vger.kernel.org] On Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:23 AM
> > > To: dhowells at redhat.com; tglx at linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel
> > > .com>;
> > > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > > <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.ker
> > > nel.org;
> > > linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zyto
> > > r.com;
> > > x86 at kernel.org; linux-mm at kvack.org
> > > Subject: [RFC 00/12] Multi-Key Total Memory Encryption API
> > > (MKTME)
> > > 
> > > Seeking comments on the APIs supporting MKTME on future Intel
> > > platforms.
> > > 
> > > MKTME (Multi-Key Total Memory Encryption) is a technology
> > > supporting
> > > memory encryption on upcoming Intel platforms. Whereas TME allows
> > > encryption of the entire system memory using a single key, MKTME
> > > allows
> > > mulitple encryption domains, each having their own key. While the
> > > main use
> > > case for the feature is virtual machine isolation, the API needs
> > > the flexibility to
> > > work for a wide range of use cases.
> > > 
> > > This RFC presents the 2 API additions that enable userspace to:
> > >  1) Create Encryption Keys: Kernel Key Service type "mktme"
> > >  2) Use the Encryption Keys: system call encrypt_mprotect()
> > > 
> > > In order to share between: the Kernel Key Service, the new system
> > > call, and the
> > > existing mm code, helper functions were created in arch/x86/mktme
> > 
> > IMHO, we can separate this series into 2 parts, as you did above,
> > and send out them separately. The reason is, in general I think
> > adding new MKTME type to key retention services is not that related
> > to memory management code, namely the encrypt_mprotect() API part.
> > 
> > So if we split the two parts and send them out separately, the
> > first part can be reviewed by keyring and security guys, without
> > involving mm guys, and the encrypt_mprotect() part can be more
> > reviewed more by mm guys. 
> > 
> 
> Kai,
> 
> That was the direction I had in mind at the onset: the MKTME key
> service
> would be one patch(set) and the MKTME encrypt_mprotect() system call
> would
> be delivered in another patch(set).
> 
> That separation falls apart when the shared structures and functions
> are
> introduced. That 'mktme_map' (maps userspace keys to hardware keyid
> slots),
> and the 'encrypt_count' array (counts vma's outstanding for each key)
> need
> to be shared by both pieces. These mktme special shared structures
> and the
> functions that operate on them are all defined in
> arch/x86/mm/mktme.h,.c.
> From there they can be shared with the security/keys/mktme_keys.c
> 
> Once I made that separation, I stuck with it. Those structures, and
> any
> functions that manipulate those structures live in
> arch/x86/mm/mktme.h,c
> 
> You noted that some of the functions that operate on the
> encrypt_count
> might not need to be over in arch/x86/mm/mktme.c because they are not
> used
> in the mm code. That is true.  

Yes IMO this is better.

> But, then I'd be splitting up the definition
> of the struct and the funcs that operate on it. So, I stuck with
> keeping it
> all together in the arch specific mktme files.

Definition can also be in include/keys/mktme-type.h, right?

> 
> Having said all the above, I do welcome other ideas on how to better
> organize
> the code. 
> 
> Back to your request- to split it into smaller patchsets might look
> something
> like:
> 1) the MKTME API helpers

Exactly what helpers?

> 2) the MKTME Key Service
> 3) the MKTME syscall encrypt_mprotect()
> 
> I'm not clear that would make anyones review life easier, than
> picking
> the same pieces out of the greater patchset.

Well, the reason I suggested we should split key retention services
from other staff and get it reviewed first is I think the
functionalities of MKTME key type has nothing to do w/ MM code. IMHO
what MKTME key type should provide includes:

1) Support add_key, keyctl (revoke, etc) obviously. THis also implies
keyID allocation, so you will need some info from MM or x86 code to get
 info such as number of keyIDs, etc. In fact I think number of keyIDs
is  the only info you need (0 keyIDs means MKTME is disabled).

2) You need to provide some APIs to munipulate key reference count.
Those functions can be declared in mktme-type.h, and implemented in
mktme_keys.c. I think this is more reasonable, as logically reference
count is part of MKTME key service.

I don't think you need to do anything else in MKTME key type?

I think you can even start upstreaming MKTME key type now w/o Kirill's
core-MKTME support, since 1) adding MKTME key type doesn't do any harm
w/o kirill's core-MKTME code; 2) logically MKTME key type has no true
dependency on core-MKTME code.

So IMHO, a better structure should be:

1) Current upstream code (maybe with some early detction code change,
ie, to disable MKTME for some SW reason, but I am not sure whether this
is truely needed, since when MKTME key type gets initialized, that part
should been done already).
2) MKTME key type
3) Kirill's core-MKTME
4) new syscall

And I think CONFIG_MKTME_KEYS doesn't really need to depend on
CONFIG_INTEL_MKTME at this point, although we can add such if needed
when kirill's core-MKTME code gets upstreamed.

Thanks,
-Kai
> 
> Suggestions welcome,
> Alison
> 
> > And since encrypt_mprotect() is a new syscall, you may need to add
> > more lists for the review, ie, linux-api, and maybe linux-kernel as
> > well.
> 
> Got it. Will include theses in v1.
> 
> > 
> > Thanks,
> > -Kai
> > 
> > > 
> > > This patchset is built upon Kirill Shutemov's patchset for the
> > > core MKTME
> > > support. You can find that here:
> > > git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git
> > > mktme/wip
> > > 
> > > 
> > > Alison Schofield (12):
> > >   docs/x86: Document the Multi-Key Total Memory Encryption API
> > >   mm: Generalize the mprotect implementation to support
> > > extensions
> > >   syscall/x86: Wire up a new system call for memory encryption
> > > keys
> > >   x86/mm: Add helper functions to manage memory encryption keys
> > >   x86/mm: Add a helper function to set keyid bits in encrypted
> > > VMA's
> > >   mm: Add the encrypt_mprotect() system call
> > >   x86/mm: Add helper functions to track encrypted VMA's
> > >   mm: Track VMA's in use for each memory encryption keyid
> > >   mm: Restrict memory encryption to anonymous VMA's
> > >   x86/pconfig: Program memory encryption keys on a system-wide
> > > basis
> > >   keys/mktme: Add a new key service type for memory encryption
> > > keys
> > >   keys/mktme: Do not revoke in use memory encryption keys
> > > 
> > >  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
> > >  arch/x86/Kconfig                       |   1 +
> > >  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
> > >  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
> > >  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
> > >  arch/x86/include/asm/mktme.h           |  21 +++
> > >  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
> > >  fs/exec.c                              |   4 +-
> > >  include/keys/mktme-type.h              |  28 +++
> > >  include/linux/key.h                    |   2 +
> > >  include/linux/mm.h                     |   9 +-
> > >  include/linux/syscalls.h               |   2 +
> > >  include/uapi/asm-generic/unistd.h      |   4 +-
> > >  kernel/fork.c                          |   2 +
> > >  kernel/sys_ni.c                        |   2 +
> > >  mm/mmap.c                              |  12 ++
> > >  mm/mprotect.c                          |  93 +++++++++-
> > >  mm/nommu.c                             |   4 +
> > >  security/keys/Kconfig                  |  11 ++
> > >  security/keys/Makefile                 |   1 +
> > >  security/keys/internal.h               |   6 +
> > >  security/keys/keyctl.c                 |   7 +
> > >  security/keys/mktme_keys.c             | 325
> > > +++++++++++++++++++++++++++++++++
> > >  23 files changed, 855 insertions(+), 17 deletions(-)  create
> > > mode 100644
> > > Documentation/x86/mktme-keys.txt  create mode 100644
> > > include/keys/mktme-
> > > type.h  create mode 100644 security/keys/mktme_keys.c
> > > 
> > > --
> > > 2.14.1

Re: [RFC 00/12] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Mon, 2018-09-10 at 12:10 -0700, Alison Schofield wrote:
> On Sun, Sep 09, 2018 at 06:10:19PM -0700, Huang, Kai wrote:
> > 
> > > -----Original Message-----
> > > From: keyrings-owner@vger.kernel.org [mailto:keyrings-
> > > owner@vger.kernel.org] On Behalf Of Alison Schofield
> > > Sent: Saturday, September 8, 2018 10:23 AM
> > > To: dhowells@redhat.com; tglx@linutronix.de
> > > Cc: Huang, Kai <kai.huang@intel.com>; Nakajima, Jun
> > > <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel
> > > .com>;
> > > Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> > > <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.ker
> > > nel.org;
> > > linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zyto
> > > r.com;
> > > x86@kernel.org; linux-mm@kvack.org
> > > Subject: [RFC 00/12] Multi-Key Total Memory Encryption API
> > > (MKTME)
> > > 
> > > Seeking comments on the APIs supporting MKTME on future Intel
> > > platforms.
> > > 
> > > MKTME (Multi-Key Total Memory Encryption) is a technology
> > > supporting
> > > memory encryption on upcoming Intel platforms. Whereas TME allows
> > > encryption of the entire system memory using a single key, MKTME
> > > allows
> > > mulitple encryption domains, each having their own key. While the
> > > main use
> > > case for the feature is virtual machine isolation, the API needs
> > > the flexibility to
> > > work for a wide range of use cases.
> > > 
> > > This RFC presents the 2 API additions that enable userspace to:
> > >  1) Create Encryption Keys: Kernel Key Service type "mktme"
> > >  2) Use the Encryption Keys: system call encrypt_mprotect()
> > > 
> > > In order to share between: the Kernel Key Service, the new system
> > > call, and the
> > > existing mm code, helper functions were created in arch/x86/mktme
> > 
> > IMHO, we can separate this series into 2 parts, as you did above,
> > and send out them separately. The reason is, in general I think
> > adding new MKTME type to key retention services is not that related
> > to memory management code, namely the encrypt_mprotect() API part.
> > 
> > So if we split the two parts and send them out separately, the
> > first part can be reviewed by keyring and security guys, without
> > involving mm guys, and the encrypt_mprotect() part can be more
> > reviewed more by mm guys. 
> > 
> 
> Kai,
> 
> That was the direction I had in mind at the onset: the MKTME key
> service
> would be one patch(set) and the MKTME encrypt_mprotect() system call
> would
> be delivered in another patch(set).
> 
> That separation falls apart when the shared structures and functions
> are
> introduced. That 'mktme_map' (maps userspace keys to hardware keyid
> slots),
> and the 'encrypt_count' array (counts vma's outstanding for each key)
> need
> to be shared by both pieces. These mktme special shared structures
> and the
> functions that operate on them are all defined in
> arch/x86/mm/mktme.h,.c.
> From there they can be shared with the security/keys/mktme_keys.c
> 
> Once I made that separation, I stuck with it. Those structures, and
> any
> functions that manipulate those structures live in
> arch/x86/mm/mktme.h,c
> 
> You noted that some of the functions that operate on the
> encrypt_count
> might not need to be over in arch/x86/mm/mktme.c because they are not
> used
> in the mm code. That is true.  

Yes IMO this is better.

> But, then I'd be splitting up the definition
> of the struct and the funcs that operate on it. So, I stuck with
> keeping it
> all together in the arch specific mktme files.

Definition can also be in include/keys/mktme-type.h, right?

> 
> Having said all the above, I do welcome other ideas on how to better
> organize
> the code. 
> 
> Back to your request- to split it into smaller patchsets might look
> something
> like:
> 1) the MKTME API helpers

Exactly what helpers?

> 2) the MKTME Key Service
> 3) the MKTME syscall encrypt_mprotect()
> 
> I'm not clear that would make anyones review life easier, than
> picking
> the same pieces out of the greater patchset.

Well, the reason I suggested we should split key retention services
from other staff and get it reviewed first is I think the
functionalities of MKTME key type has nothing to do w/ MM code. IMHO
what MKTME key type should provide includes:

1) Support add_key, keyctl (revoke, etc) obviously. THis also implies
keyID allocation, so you will need some info from MM or x86 code to get
 info such as number of keyIDs, etc. In fact I think number of keyIDs
is  the only info you need (0 keyIDs means MKTME is disabled).

2) You need to provide some APIs to munipulate key reference count.
Those functions can be declared in mktme-type.h, and implemented in
mktme_keys.c. I think this is more reasonable, as logically reference
count is part of MKTME key service.

I don't think you need to do anything else in MKTME key type?

I think you can even start upstreaming MKTME key type now w/o Kirill's
core-MKTME support, since 1) adding MKTME key type doesn't do any harm
w/o kirill's core-MKTME code; 2) logically MKTME key type has no true
dependency on core-MKTME code.

So IMHO, a better structure should be:

1) Current upstream code (maybe with some early detction code change,
ie, to disable MKTME for some SW reason, but I am not sure whether this
is truely needed, since when MKTME key type gets initialized, that part
should been done already).
2) MKTME key type
3) Kirill's core-MKTME
4) new syscall

And I think CONFIG_MKTME_KEYS doesn't really need to depend on
CONFIG_INTEL_MKTME at this point, although we can add such if needed
when kirill's core-MKTME code gets upstreamed.

Thanks,
-Kai
> 
> Suggestions welcome,
> Alison
> 
> > And since encrypt_mprotect() is a new syscall, you may need to add
> > more lists for the review, ie, linux-api, and maybe linux-kernel as
> > well.
> 
> Got it. Will include theses in v1.
> 
> > 
> > Thanks,
> > -Kai
> > 
> > > 
> > > This patchset is built upon Kirill Shutemov's patchset for the
> > > core MKTME
> > > support. You can find that here:
> > > git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git
> > > mktme/wip
> > > 
> > > 
> > > Alison Schofield (12):
> > >   docs/x86: Document the Multi-Key Total Memory Encryption API
> > >   mm: Generalize the mprotect implementation to support
> > > extensions
> > >   syscall/x86: Wire up a new system call for memory encryption
> > > keys
> > >   x86/mm: Add helper functions to manage memory encryption keys
> > >   x86/mm: Add a helper function to set keyid bits in encrypted
> > > VMA's
> > >   mm: Add the encrypt_mprotect() system call
> > >   x86/mm: Add helper functions to track encrypted VMA's
> > >   mm: Track VMA's in use for each memory encryption keyid
> > >   mm: Restrict memory encryption to anonymous VMA's
> > >   x86/pconfig: Program memory encryption keys on a system-wide
> > > basis
> > >   keys/mktme: Add a new key service type for memory encryption
> > > keys
> > >   keys/mktme: Do not revoke in use memory encryption keys
> > > 
> > >  Documentation/x86/mktme-keys.txt       | 153 ++++++++++++++++
> > >  arch/x86/Kconfig                       |   1 +
> > >  arch/x86/entry/syscalls/syscall_32.tbl |   1 +
> > >  arch/x86/entry/syscalls/syscall_64.tbl |   1 +
> > >  arch/x86/include/asm/intel_pconfig.h   |  42 ++++-
> > >  arch/x86/include/asm/mktme.h           |  21 +++
> > >  arch/x86/mm/mktme.c                    | 141 ++++++++++++++
> > >  fs/exec.c                              |   4 +-
> > >  include/keys/mktme-type.h              |  28 +++
> > >  include/linux/key.h                    |   2 +
> > >  include/linux/mm.h                     |   9 +-
> > >  include/linux/syscalls.h               |   2 +
> > >  include/uapi/asm-generic/unistd.h      |   4 +-
> > >  kernel/fork.c                          |   2 +
> > >  kernel/sys_ni.c                        |   2 +
> > >  mm/mmap.c                              |  12 ++
> > >  mm/mprotect.c                          |  93 +++++++++-
> > >  mm/nommu.c                             |   4 +
> > >  security/keys/Kconfig                  |  11 ++
> > >  security/keys/Makefile                 |   1 +
> > >  security/keys/internal.h               |   6 +
> > >  security/keys/keyctl.c                 |   7 +
> > >  security/keys/mktme_keys.c             | 325
> > > +++++++++++++++++++++++++++++++++
> > >  23 files changed, 855 insertions(+), 17 deletions(-)  create
> > > mode 100644
> > > Documentation/x86/mktme-keys.txt  create mode 100644
> > > include/keys/mktme-
> > > type.h  create mode 100644 security/keys/mktme_keys.c
> > > 
> > > --
> > > 2.14.1

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 07:46:57PM -0700, Alison Schofield wrote:
> On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> > On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > > as a system wide single pool of keys. The hardware, however, manages
> > > the keys on a per physical package basis. Each physical package
> > > maintains a key table that all CPU's in that package share.
> > > 
> > > In order to maintain the consistent, system wide view that the kernel
> > > requires, program all physical packages during a key program request.
> > > 
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > 
> > Just kind of checking that are you talking about multiple cores in
> > a single package or really multiple packages?
> 
> System wide pool.
> System has multiple packages.  
> Packages have multiple CPU's.
> 
> The hardware KEY TABLE is per package. I need that per package KEY TABLE
> to be the same in every package across the system. So, I pick one 'lead'
> CPU in each package to program that packages KEY TABLE.
> 
> (BTW - I'm going to look into Kai's suggestion to move the system wide view
> of this key programming into the key service. Not sure if that's a go.)

Thanks. I think could be perhaps a fair addition to the documentation?

/Jarkko

[RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 07:46:57PM -0700, Alison Schofield wrote:
> On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> > On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > > as a system wide single pool of keys. The hardware, however, manages
> > > the keys on a per physical package basis. Each physical package
> > > maintains a key table that all CPU's in that package share.
> > > 
> > > In order to maintain the consistent, system wide view that the kernel
> > > requires, program all physical packages during a key program request.
> > > 
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > 
> > Just kind of checking that are you talking about multiple cores in
> > a single package or really multiple packages?
> 
> System wide pool.
> System has multiple packages.  
> Packages have multiple CPU's.
> 
> The hardware KEY TABLE is per package. I need that per package KEY TABLE
> to be the same in every package across the system. So, I pick one 'lead'
> CPU in each package to program that packages KEY TABLE.
> 
> (BTW - I'm going to look into Kai's suggestion to move the system wide view
> of this key programming into the key service. Not sure if that's a go.)

Thanks. I think could be perhaps a fair addition to the documentation?

/Jarkko

Re: [RFC 10/12] x86/pconfig: Program memory encryption keys on a system-wide basis

[Jarkko Sakkinen]

On Mon, Sep 10, 2018 at 07:46:57PM -0700, Alison Schofield wrote:
> On Mon, Sep 10, 2018 at 11:24:20AM -0700, Sakkinen, Jarkko wrote:
> > On Fri, 2018-09-07 at 15:38 -0700, Alison Schofield wrote:
> > > The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
> > > as a system wide single pool of keys. The hardware, however, manages
> > > the keys on a per physical package basis. Each physical package
> > > maintains a key table that all CPU's in that package share.
> > > 
> > > In order to maintain the consistent, system wide view that the kernel
> > > requires, program all physical packages during a key program request.
> > > 
> > > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> > 
> > Just kind of checking that are you talking about multiple cores in
> > a single package or really multiple packages?
> 
> System wide pool.
> System has multiple packages.  
> Packages have multiple CPU's.
> 
> The hardware KEY TABLE is per package. I need that per package KEY TABLE
> to be the same in every package across the system. So, I pick one 'lead'
> CPU in each package to program that packages KEY TABLE.
> 
> (BTW - I'm going to look into Kai's suggestion to move the system wide view
> of this key programming into the key service. Not sure if that's a go.)

Thanks. I think could be perhaps a fair addition to the documentation?

/Jarkko

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> +{
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;

Everything prior to here looks like it should be in the ->preparse() routine.
I really should get round to making that mandatory.

> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}

David

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> +{
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;

Everything prior to here looks like it should be in the ->preparse() routine.
I really should get round to making that mandatory.

> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}

David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +/* Key Service Command: Creates a software key and programs hardware */
> +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> +{
> +	struct mktme_key_program *kprog = NULL;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret = 0;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog) {
> +		kzfree(options);
> +		return -ENOMEM;
> +	}
> +	ret = mktme_get_options(options, kprog);
> +	if (ret < 0)
> +		goto out;

Everything prior to here looks like it should be in the ->preparse() routine.
I really should get round to making that mandatory.

> +
> +	mktme_map_lock();
> +	ret = mktme_program_key(key->serial, kprog);
> +	mktme_map_unlock();
> +out:
> +	kzfree(options);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}

David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Tue, Sep 11, 2018 at 11:03:15PM +0100, David Howells wrote:
> Alison Schofield <alison.schofield@intel.com> wrote:
> 
> > +/* Key Service Command: Creates a software key and programs hardware */
> > +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> > +{
> > +	struct mktme_key_program *kprog = NULL;
> > +	size_t datalen = prep->datalen;
> > +	char *options;
> > +	int ret = 0;
> > +
> > +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> > +		return -EACCES;
> > +
> > +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> > +		return -EINVAL;
> > +
> > +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> > +	if (!options)
> > +		return -ENOMEM;
> > +
> > +	options[datalen] = '\0';
> > +
> > +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> > +	if (!kprog) {
> > +		kzfree(options);
> > +		return -ENOMEM;
> > +	}
> > +	ret = mktme_get_options(options, kprog);
> > +	if (ret < 0)
> > +		goto out;
> 
> Everything prior to here looks like it should be in the ->preparse() routine.
> I really should get round to making that mandatory.

Hi Dave,

If a preparse routine handles all the above, then if any of the
above failures occur, the key service has less backing out to do.
Is that the point?

How do I make the connection between the preparse and the instantiate? 
Do I just put what I need to remember about this key request in the
payload.data during preparse, so I can examine it again during
instantiate?

Thanks,
Alison

> 
> > +
> > +	mktme_map_lock();
> > +	ret = mktme_program_key(key->serial, kprog);
> > +	mktme_map_unlock();
> > +out:
> > +	kzfree(options);
> > +	kmem_cache_free(mktme_prog_cache, kprog);
> > +	return ret;
> > +}
> 
> David

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Tue, Sep 11, 2018 at 11:03:15PM +0100, David Howells wrote:
> Alison Schofield <alison.schofield@intel.com> wrote:
> 
> > +/* Key Service Command: Creates a software key and programs hardware */
> > +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> > +{
> > +	struct mktme_key_program *kprog = NULL;
> > +	size_t datalen = prep->datalen;
> > +	char *options;
> > +	int ret = 0;
> > +
> > +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> > +		return -EACCES;
> > +
> > +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> > +		return -EINVAL;
> > +
> > +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> > +	if (!options)
> > +		return -ENOMEM;
> > +
> > +	options[datalen] = '\0';
> > +
> > +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> > +	if (!kprog) {
> > +		kzfree(options);
> > +		return -ENOMEM;
> > +	}
> > +	ret = mktme_get_options(options, kprog);
> > +	if (ret < 0)
> > +		goto out;
> 
> Everything prior to here looks like it should be in the ->preparse() routine.
> I really should get round to making that mandatory.

Hi Dave,

If a preparse routine handles all the above, then if any of the
above failures occur, the key service has less backing out to do.
Is that the point?

How do I make the connection between the preparse and the instantiate? 
Do I just put what I need to remember about this key request in the
payload.data during preparse, so I can examine it again during
instantiate?

Thanks,
Alison

> 
> > +
> > +	mktme_map_lock();
> > +	ret = mktme_program_key(key->serial, kprog);
> > +	mktme_map_unlock();
> > +out:
> > +	kzfree(options);
> > +	kmem_cache_free(mktme_prog_cache, kprog);
> > +	return ret;
> > +}
> 
> David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Tue, Sep 11, 2018 at 11:03:15PM +0100, David Howells wrote:
> Alison Schofield <alison.schofield@intel.com> wrote:
> 
> > +/* Key Service Command: Creates a software key and programs hardware */
> > +int mktme_instantiate(struct key *key, struct key_preparsed_payload *prep)
> > +{
> > +	struct mktme_key_program *kprog = NULL;
> > +	size_t datalen = prep->datalen;
> > +	char *options;
> > +	int ret = 0;
> > +
> > +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> > +		return -EACCES;
> > +
> > +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> > +		return -EINVAL;
> > +
> > +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> > +	if (!options)
> > +		return -ENOMEM;
> > +
> > +	options[datalen] = '\0';
> > +
> > +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> > +	if (!kprog) {
> > +		kzfree(options);
> > +		return -ENOMEM;
> > +	}
> > +	ret = mktme_get_options(options, kprog);
> > +	if (ret < 0)
> > +		goto out;
> 
> Everything prior to here looks like it should be in the ->preparse() routine.
> I really should get round to making that mandatory.

Hi Dave,

If a preparse routine handles all the above, then if any of the
above failures occur, the key service has less backing out to do.
Is that the point?

How do I make the connection between the preparse and the instantiate? 
Do I just put what I need to remember about this key request in the
payload.data during preparse, so I can examine it again during
instantiate?

Thanks,
Alison

> 
> > +
> > +	mktme_map_lock();
> > +	ret = mktme_program_key(key->serial, kprog);
> > +	mktme_map_unlock();
> > +out:
> > +	kzfree(options);
> > +	kmem_cache_free(mktme_prog_cache, kprog);
> > +	return ret;
> > +}
> 
> David

Re: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +void mktme_map_set_keyid(int keyid, unsigned int serial)
> +{
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}

It appears that 'serial' should be key_serial_t.

Note that you *aren't* allowed to cache key serial numbers inside the kernel.
You must cache the struct key * instead and hold a reference to the key.  This
will prevent the key from being destroyed whilst it is in use.

David

[RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +void mktme_map_set_keyid(int keyid, unsigned int serial)
> +{
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}

It appears that 'serial' should be key_serial_t.

Note that you *aren't* allowed to cache key serial numbers inside the kernel.
You must cache the struct key * instead and hold a reference to the key.  This
will prevent the key from being destroyed whilst it is in use.

David

Re: [RFC 04/12] x86/mm: Add helper functions to manage memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +void mktme_map_set_keyid(int keyid, unsigned int serial)
> +{
> +	mktme_map->serial[keyid] = serial;
> +	mktme_map->mapped_keyids++;
> +}

It appears that 'serial' should be key_serial_t.

Note that you *aren't* allowed to cache key serial numbers inside the kernel.
You must cache the struct key * instead and hold a reference to the key.  This
will prevent the key from being destroyed whilst it is in use.

David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> If a preparse routine handles all the above, then if any of the
> above failures occur, the key service has less backing out to do.
> Is that the point?

Yes.  Ideally, ->instantiate() would never fail.

> How do I make the connection between the preparse and the instantiate? 
> Do I just put what I need to remember about this key request in the
> payload.data during preparse, so I can examine it again during
> instantiate?

Have a look at user_preparse().  It attaches the contribution to the supplied
key_preparsed_payload struct, which is then passed to ->instantiate() and
->update() as appropriate.  generic_key_instantiate() is used by the user key
type.

David

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> If a preparse routine handles all the above, then if any of the
> above failures occur, the key service has less backing out to do.
> Is that the point?

Yes.  Ideally, ->instantiate() would never fail.

> How do I make the connection between the preparse and the instantiate? 
> Do I just put what I need to remember about this key request in the
> payload.data during preparse, so I can examine it again during
> instantiate?

Have a look at user_preparse().  It attaches the contribution to the supplied
key_preparsed_payload struct, which is then passed to ->instantiate() and
->update() as appropriate.  generic_key_instantiate() is used by the user key
type.

David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> If a preparse routine handles all the above, then if any of the
> above failures occur, the key service has less backing out to do.
> Is that the point?

Yes.  Ideally, ->instantiate() would never fail.

> How do I make the connection between the preparse and the instantiate? 
> Do I just put what I need to remember about this key request in the
> payload.data during preparse, so I can examine it again during
> instantiate?

Have a look at user_preparse().  It attaches the contribution to the supplied
key_preparsed_payload struct, which is then passed to ->instantiate() and
->update() as appropriate.  generic_key_instantiate() is used by the user key
type.

David

Re: [RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +
> +	if (strcmp(key->type->name, "mktme") = 0)
> +		mktme_revoke_key(key);
> +

*Please* don't do that.

The core code shouldn't be making references to specific key types in this
way.  The only reason this is necessary for encrypted and trusted keys is
because they misused the ->update() hook and it took a while for this to be
noticed.

> The KEY_FLAG_KEEP bit offers good control. The mktme service uses
> that flag to prevent userspace keys from going away without proper
> synchronization with the mktme service type.

This is not the control you are looking for.  The point of KEY_FLAG_KEEP is to
allow the system to pin a key.  It's not meant to be a flag for the key type
to play with.

You say this:

	One example is that userspace keys should not be revoked while the
	hardware keyid slot is still in use.

but why not?  Revoking it causes accesses to return -EKEYREVOKED; it doesn't
stop the kernel from using the key.

Also, note that you don't *have* to provide a ->revoke() operation

If you really want to suppress revocation, then I would suggest adding another
type operation, say ->may_revoke(), that says whether you're allowed to do
that.

David

[RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +
> +	if (strcmp(key->type->name, "mktme") == 0)
> +		mktme_revoke_key(key);
> +

*Please* don't do that.

The core code shouldn't be making references to specific key types in this
way.  The only reason this is necessary for encrypted and trusted keys is
because they misused the ->update() hook and it took a while for this to be
noticed.

> The KEY_FLAG_KEEP bit offers good control. The mktme service uses
> that flag to prevent userspace keys from going away without proper
> synchronization with the mktme service type.

This is not the control you are looking for.  The point of KEY_FLAG_KEEP is to
allow the system to pin a key.  It's not meant to be a flag for the key type
to play with.

You say this:

	One example is that userspace keys should not be revoked while the
	hardware keyid slot is still in use.

but why not?  Revoking it causes accesses to return -EKEYREVOKED; it doesn't
stop the kernel from using the key.

Also, note that you don't *have* to provide a ->revoke() operation

If you really want to suppress revocation, then I would suggest adding another
type operation, say ->may_revoke(), that says whether you're allowed to do
that.

David

Re: [RFC 12/12] keys/mktme: Do not revoke in use memory encryption keys

[David Howells]

Alison Schofield <alison.schofield@intel.com> wrote:

> +
> +	if (strcmp(key->type->name, "mktme") == 0)
> +		mktme_revoke_key(key);
> +

*Please* don't do that.

The core code shouldn't be making references to specific key types in this
way.  The only reason this is necessary for encrypted and trusted keys is
because they misused the ->update() hook and it took a while for this to be
noticed.

> The KEY_FLAG_KEEP bit offers good control. The mktme service uses
> that flag to prevent userspace keys from going away without proper
> synchronization with the mktme service type.

This is not the control you are looking for.  The point of KEY_FLAG_KEEP is to
allow the system to pin a key.  It's not meant to be a flag for the key type
to play with.

You say this:

	One example is that userspace keys should not be revoked while the
	hardware keyid slot is still in use.

but why not?  Revoking it causes accesses to return -EKEYREVOKED; it doesn't
stop the kernel from using the key.

Also, note that you don't *have* to provide a ->revoke() operation

If you really want to suppress revocation, then I would suggest adding another
type operation, say ->may_revoke(), that says whether you're allowed to do
that.

David

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > + */
> > +static int mktme_build_cpumask(void)
> > +{
> > +	int online_cpu, mktme_cpu;
> > +	int online_pkgid, mktme_pkgid = -1;
> > +
> > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > +		return -ENOMEM;
> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid > > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid = online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Kai, 

I tried to simplify it and came up with code that looked like this:

	int lead_cpu, node;
	for_each_online_node(node) {
		lead_cpu = cpumask_first(cpumask_of_node(node));
		if (lead_cpu < nr_cpu_ids)
			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
	}
When I test it on an SNC (Sub Numa Cluster) system it gives me too many
CPU's. I get a CPU per Node (just like i asked for;) instead of per Socket.
It has 2 sockets and 4 NUMA nodes. 

I kind of remember this when I originally coded it, hence the bottoms up
approach using topology_physical_package_id()

Any ideas?

Alison

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > + */
> > +static int mktme_build_cpumask(void)
> > +{
> > +	int online_cpu, mktme_cpu;
> > +	int online_pkgid, mktme_pkgid = -1;
> > +
> > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > +		return -ENOMEM;
> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid =
> > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid == online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Kai, 

I tried to simplify it and came up with code that looked like this:

	int lead_cpu, node;
	for_each_online_node(node) {
		lead_cpu = cpumask_first(cpumask_of_node(node));
		if (lead_cpu < nr_cpu_ids)
			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
	}
When I test it on an SNC (Sub Numa Cluster) system it gives me too many
CPU's. I get a CPU per Node (just like i asked for;) instead of per Socket.
It has 2 sockets and 4 NUMA nodes. 

I kind of remember this when I originally coded it, hence the bottoms up
approach using topology_physical_package_id()

Any ideas?

Alison

Re: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Alison Schofield]

On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > + */
> > +static int mktme_build_cpumask(void)
> > +{
> > +	int online_cpu, mktme_cpu;
> > +	int online_pkgid, mktme_pkgid = -1;
> > +
> > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > +		return -ENOMEM;
> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > +			mktme_pkgid =
> > topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid == online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > +	}
> 
> Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the logic?

Kai, 

I tried to simplify it and came up with code that looked like this:

	int lead_cpu, node;
	for_each_online_node(node) {
		lead_cpu = cpumask_first(cpumask_of_node(node));
		if (lead_cpu < nr_cpu_ids)
			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
	}
When I test it on an SNC (Sub Numa Cluster) system it gives me too many
CPU's. I get a CPU per Node (just like i asked for;) instead of per Socket.
It has 2 sockets and 4 NUMA nodes. 

I kind of remember this when I originally coded it, hence the bottoms up
approach using topology_physical_package_id()

Any ideas?

Alison

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: Schofield, Alison
> Sent: Saturday, September 15, 2018 12:07 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > + */
> > > +static int mktme_build_cpumask(void) {
> > > +	int online_cpu, mktme_cpu;
> > > +	int online_pkgid, mktme_pkgid = -1;
> > > +
> > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > +		return -ENOMEM;
> > > +
> > > +	for_each_online_cpu(online_cpu) {
> > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > +
> > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > +			mktme_pkgid > > > topology_physical_package_id(mktme_cpu);
> > > +			if (mktme_pkgid = online_pkgid)
> > > +				break;
> > > +		}
> > > +		if (mktme_pkgid != online_pkgid)
> > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > +	}
> >
> > Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the
> logic?
> 
> Kai,
> 
> I tried to simplify it and came up with code that looked like this:
> 
> 	int lead_cpu, node;
> 	for_each_online_node(node) {
> 		lead_cpu = cpumask_first(cpumask_of_node(node));
> 		if (lead_cpu < nr_cpu_ids)
> 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> 	}
> When I test it on an SNC (Sub Numa Cluster) system it gives me too many CPU's.
> I get a CPU per Node (just like i asked for;) instead of per Socket.
> It has 2 sockets and 4 NUMA nodes.
> 
> I kind of remember this when I originally coded it, hence the bottoms up
> approach using topology_physical_package_id()
> 
> Any ideas?

Hmm.. I forgot the SNC case, sorry :(

So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure about this.

If PCONFIG is per-package, I don't have better idea than your original one. :)

Thanks,
-Kai
> 
> Alison
> 

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: Schofield, Alison
> Sent: Saturday, September 15, 2018 12:07 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells at redhat.com; tglx at linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris at namei.org; keyrings at vger.kernel.org;
> linux-security-module at vger.kernel.org; mingo at redhat.com; hpa at zytor.com;
> x86 at kernel.org; linux-mm at kvack.org
> Subject: Re: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > + */
> > > +static int mktme_build_cpumask(void) {
> > > +	int online_cpu, mktme_cpu;
> > > +	int online_pkgid, mktme_pkgid = -1;
> > > +
> > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > +		return -ENOMEM;
> > > +
> > > +	for_each_online_cpu(online_cpu) {
> > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > +
> > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > +			mktme_pkgid =
> > > topology_physical_package_id(mktme_cpu);
> > > +			if (mktme_pkgid == online_pkgid)
> > > +				break;
> > > +		}
> > > +		if (mktme_pkgid != online_pkgid)
> > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > +	}
> >
> > Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the
> logic?
> 
> Kai,
> 
> I tried to simplify it and came up with code that looked like this:
> 
> 	int lead_cpu, node;
> 	for_each_online_node(node) {
> 		lead_cpu = cpumask_first(cpumask_of_node(node));
> 		if (lead_cpu < nr_cpu_ids)
> 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> 	}
> When I test it on an SNC (Sub Numa Cluster) system it gives me too many CPU's.
> I get a CPU per Node (just like i asked for;) instead of per Socket.
> It has 2 sockets and 4 NUMA nodes.
> 
> I kind of remember this when I originally coded it, hence the bottoms up
> approach using topology_physical_package_id()
> 
> Any ideas?

Hmm.. I forgot the SNC case, sorry :(

So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure about this.

If PCONFIG is per-package, I don't have better idea than your original one. :)

Thanks,
-Kai
> 
> Alison
> 

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> -----Original Message-----
> From: Schofield, Alison
> Sent: Saturday, September 15, 2018 12:07 PM
> To: Huang, Kai <kai.huang@intel.com>
> Cc: dhowells@redhat.com; tglx@linutronix.de; Nakajima, Jun
> <jun.nakajima@intel.com>; Shutemov, Kirill <kirill.shutemov@intel.com>;
> Hansen, Dave <dave.hansen@intel.com>; Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com>; jmorris@namei.org; keyrings@vger.kernel.org;
> linux-security-module@vger.kernel.org; mingo@redhat.com; hpa@zytor.com;
> x86@kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC 11/12] keys/mktme: Add a new key service type for memory
> encryption keys
> 
> On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > + */
> > > +static int mktme_build_cpumask(void) {
> > > +	int online_cpu, mktme_cpu;
> > > +	int online_pkgid, mktme_pkgid = -1;
> > > +
> > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > +		return -ENOMEM;
> > > +
> > > +	for_each_online_cpu(online_cpu) {
> > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > +
> > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > +			mktme_pkgid =
> > > topology_physical_package_id(mktme_cpu);
> > > +			if (mktme_pkgid == online_pkgid)
> > > +				break;
> > > +		}
> > > +		if (mktme_pkgid != online_pkgid)
> > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > +	}
> >
> > Could we use 'for_each_online_node', 'cpumask_first/next', etc to simplify the
> logic?
> 
> Kai,
> 
> I tried to simplify it and came up with code that looked like this:
> 
> 	int lead_cpu, node;
> 	for_each_online_node(node) {
> 		lead_cpu = cpumask_first(cpumask_of_node(node));
> 		if (lead_cpu < nr_cpu_ids)
> 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> 	}
> When I test it on an SNC (Sub Numa Cluster) system it gives me too many CPU's.
> I get a CPU per Node (just like i asked for;) instead of per Socket.
> It has 2 sockets and 4 NUMA nodes.
> 
> I kind of remember this when I originally coded it, hence the bottoms up
> approach using topology_physical_package_id()
> 
> Any ideas?

Hmm.. I forgot the SNC case, sorry :(

So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure about this.

If PCONFIG is per-package, I don't have better idea than your original one. :)

Thanks,
-Kai
> 
> Alison
> 

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> > On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > > + */
> > > > +static int mktme_build_cpumask(void) {
> > > > +	int online_cpu, mktme_cpu;
> > > > +	int online_pkgid, mktme_pkgid = -1;
> > > > +
> > > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	for_each_online_cpu(online_cpu) {
> > > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > > +
> > > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > > +			mktme_pkgid > > > > topology_physical_package_id(mktme_cpu);
> > > > +			if (mktme_pkgid = online_pkgid)
> > > > +				break;
> > > > +		}
> > > > +		if (mktme_pkgid != online_pkgid)
> > > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > > +	}
> > >
> > > Could we use 'for_each_online_node', 'cpumask_first/next', etc to
> > > simplify the
> > logic?
> >
> > Kai,
> >
> > I tried to simplify it and came up with code that looked like this:
> >
> > 	int lead_cpu, node;
> > 	for_each_online_node(node) {
> > 		lead_cpu = cpumask_first(cpumask_of_node(node));
> > 		if (lead_cpu < nr_cpu_ids)
> > 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> > 	}
> > When I test it on an SNC (Sub Numa Cluster) system it gives me too many
> CPU's.
> > I get a CPU per Node (just like i asked for;) instead of per Socket.
> > It has 2 sockets and 4 NUMA nodes.
> >
> > I kind of remember this when I originally coded it, hence the bottoms
> > up approach using topology_physical_package_id()
> >
> > Any ideas?
> 
> Hmm.. I forgot the SNC case, sorry :(
> 
> So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure
> about this.

I have confirmed internally that PCONFIG is per-package even in SNC.

Thanks,
-Kai
> 
> If PCONFIG is per-package, I don't have better idea than your original one. :)
> 
> Thanks,
> -Kai
> >
> > Alison
> >

[RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> > On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > > + */
> > > > +static int mktme_build_cpumask(void) {
> > > > +	int online_cpu, mktme_cpu;
> > > > +	int online_pkgid, mktme_pkgid = -1;
> > > > +
> > > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	for_each_online_cpu(online_cpu) {
> > > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > > +
> > > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > > +			mktme_pkgid =
> > > > topology_physical_package_id(mktme_cpu);
> > > > +			if (mktme_pkgid == online_pkgid)
> > > > +				break;
> > > > +		}
> > > > +		if (mktme_pkgid != online_pkgid)
> > > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > > +	}
> > >
> > > Could we use 'for_each_online_node', 'cpumask_first/next', etc to
> > > simplify the
> > logic?
> >
> > Kai,
> >
> > I tried to simplify it and came up with code that looked like this:
> >
> > 	int lead_cpu, node;
> > 	for_each_online_node(node) {
> > 		lead_cpu = cpumask_first(cpumask_of_node(node));
> > 		if (lead_cpu < nr_cpu_ids)
> > 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> > 	}
> > When I test it on an SNC (Sub Numa Cluster) system it gives me too many
> CPU's.
> > I get a CPU per Node (just like i asked for;) instead of per Socket.
> > It has 2 sockets and 4 NUMA nodes.
> >
> > I kind of remember this when I originally coded it, hence the bottoms
> > up approach using topology_physical_package_id()
> >
> > Any ideas?
> 
> Hmm.. I forgot the SNC case, sorry :(
> 
> So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure
> about this.

I have confirmed internally that PCONFIG is per-package even in SNC.

Thanks,
-Kai
> 
> If PCONFIG is per-package, I don't have better idea than your original one. :)
> 
> Thanks,
> -Kai
> >
> > Alison
> >

RE: [RFC 11/12] keys/mktme: Add a new key service type for memory encryption keys

[Huang, Kai]

> > On Sun, Sep 09, 2018 at 08:29:29PM -0700, Huang, Kai wrote:
> > > > + */
> > > > +static int mktme_build_cpumask(void) {
> > > > +	int online_cpu, mktme_cpu;
> > > > +	int online_pkgid, mktme_pkgid = -1;
> > > > +
> > > > +	if (!zalloc_cpumask_var(&mktme_cpumask, GFP_KERNEL))
> > > > +		return -ENOMEM;
> > > > +
> > > > +	for_each_online_cpu(online_cpu) {
> > > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > > +
> > > > +		for_each_cpu(mktme_cpu, mktme_cpumask) {
> > > > +			mktme_pkgid =
> > > > topology_physical_package_id(mktme_cpu);
> > > > +			if (mktme_pkgid == online_pkgid)
> > > > +				break;
> > > > +		}
> > > > +		if (mktme_pkgid != online_pkgid)
> > > > +			cpumask_set_cpu(online_cpu, mktme_cpumask);
> > > > +	}
> > >
> > > Could we use 'for_each_online_node', 'cpumask_first/next', etc to
> > > simplify the
> > logic?
> >
> > Kai,
> >
> > I tried to simplify it and came up with code that looked like this:
> >
> > 	int lead_cpu, node;
> > 	for_each_online_node(node) {
> > 		lead_cpu = cpumask_first(cpumask_of_node(node));
> > 		if (lead_cpu < nr_cpu_ids)
> > 			cpumask_set_cpu(lead_cpu, mktme_cpumask_NEW);
> > 	}
> > When I test it on an SNC (Sub Numa Cluster) system it gives me too many
> CPU's.
> > I get a CPU per Node (just like i asked for;) instead of per Socket.
> > It has 2 sockets and 4 NUMA nodes.
> >
> > I kind of remember this when I originally coded it, hence the bottoms
> > up approach using topology_physical_package_id()
> >
> > Any ideas?
> 
> Hmm.. I forgot the SNC case, sorry :(
> 
> So in case of SNC, is PCONFIG per-package, or per-node? I am not quite sure
> about this.

I have confirmed internally that PCONFIG is per-package even in SNC.

Thanks,
-Kai
> 
> If PCONFIG is per-package, I don't have better idea than your original one. :)
> 
> Thanks,
> -Kai
> >
> > Alison
> >